[occi-wg] confusion about status of link / headers

Alexis Richardson alexis.richardson at gmail.com
Tue Oct 20 11:39:07 CDT 2009 

Sam,

On Tue, Oct 20, 2009 at 5:31 PM, Sam Johnston <samj at samj.net> wrote:
> Alexis,
>
> Sweeping, unjustified statements are intensely frustrating, not to
> mention unhelpful in the absence of a coherent alternative. If people
> choose not to disclose conflicts I happen to be aware of then you bet
> I'll disclose them for them - nothing personal about it.

I don't know what 'conflicts' you are referring to, or how these
relate to the matter at hand.  If by 'sweeping, unjustified' you refer
to the word 'unprecedented' then I shall justify it by pointing out
that - at least as I currently understand it - you are proposing use
of headers to a *greater* extent than has been done before and to
cover a *greater* set of use cases.


> Cutting to the chase, both Microsoft and Amazon are using headers for
> metadata to the same extent as what has been proposed.

Really - to the "same" extent?  That seems contrary to what has been
asserted on this thread previously.


> What premature optimisation? *Not* using XML? Do we really need to
> drag that carcass out again? Surely Adrian's performance tests put the
> last nail in that particular coffin.

Premature optimisation = making performance a critical design concern
when we want an API.

I have noted Adrian's points as have others.


> The question really is about whether we want to be an interface/
> protocol like HTTP or some mongrel hybrid of that along with a set of
> XML, JSON and/or text formats for everything we ever intend to model
> (knowing that work has already been done by others).

Work that has been done by others and shown to work, is a good thing.

alexis




> Sent from my iPhone
>
> On 20/10/2009, at 18:10, Alexis Richardson
> <alexis.richardson at gmail.com> wrote:
>
>> Sam
>>
>> Cut the personal stuff please.
>>
>> Re 'unproven', I think the issues people are having are:
>>
>> * The extent, scope and purpose of your application of headers to
>> metadata is unprecedented - as I understand it.  Nobody is saying it
>> has not been done in other and restricted cases.
>>
>> * Premature optimisation.
>>
>> alexis
>>
>>
>>
>> On Tue, Oct 20, 2009 at 4:44 PM, Sam Johnston <samj at samj.net> wrote:
>>> Gary,
>>>
>>> Please be specific with your objections so as I can have some hope of
>>> answering them... "several other emails" does not suffice as to the
>>> best of my knowledge there are no outstanding issues that I have not
>>> comprehensively and convincingly dealt with.
>>>
>>> Your repeated assertions than HTTP headers are... what is it now...
>>> "unproven"... are demonstrably unfounded (refer to the various cloud
>>> APIs referred to by myself and Adrian, including both MS Azure and
>>> Amazon S3, as well as the history of the Internet in general) and
>>> given the frequency, bordering on FUD.
>>>
>>> Regarding performance, OCCI should strive to be as efficient as
>>> possible as it primarily targets public cloud providers who operate
>>> at
>>> arbitrarily large "cloud" scale. The existing specification is
>>> *extremely* efficient and alternatives would at least want to be in
>>> the same neighbourhood. Efficiency may be an inconvenient factor for
>>> whatever your alternative is, but it's critical for many of our use
>>> cases.
>>>
>>> Finally, you reject OVF without providing any alternative beyond
>>> building a replacement from scratch ourselves, after our second and
>>> final deadline has just sailed by no less. Have you considered the
>>> implementation cost of this? And in the same breath you accuse my
>>> proposal of being "unproven"? Give me a break...
>>>
>>> Correct me if I'm wrong but the way I see it is that you want to
>>> revert the last 2-3 months of my work because you have an investment
>>> in an old version of the spec. With each day I spend on OCCI costing
>>> me well over €1,000 excuse me for being pissed about someone with
>>> such a blatantly obvious conflict attacking my work and refusing to
>>> provide a coherent argument, particularly with no "camera ready"
>>> alternative anywhere in sight.
>>>
>>> Sent from my iPhone
>>>
>>> On 20/10/2009, at 16:51, Gary Mazz <garymazzaferro at gmail.com> wrote:
>>>
>>>> I've understood Sam's proposal from the beginning. I have had issue
>>>> with from the beginning.
>>>>
>>>> Although I find it very interesting and valuable in terms of
>>>> economy, I have issue with it as the OCCI's primary interface
>>>> implementation for a few reasons which have been outlined in several
>>>> other emails.
>>>>
>>>> It appears that Sam has created a new technology, a lower cost RPC
>>>> mechanism, which is unproven for both small and wide scale
>>>> deployment. For interoperability, IMO,  occi should focus on proven
>>>> mainstream technologies as the primary interfaces. I am not
>>>> suggesting, shelving or not supporting Sam's proposal, based on
>>>> market acceptance, it may prove to be one of many occi interface
>>>> implementations.
>>>>
>>>> As for cost saving in interface technologies, I do not buy into the
>>>> idea that the occi traffic will be so overwhelming that there is a
>>>> requirement to maximally optimize http requests and responses. IMO,
>>>> in comparison to images and consumer data occi traffic will be
>>>> insignificant. I think we are worrying about spilled milk while the
>>>> house is burning.
>>>>
>>>> There has been some discussion on supporting technologies including
>>>> OVF. Whether occi supports OVF has not been fully discussed,
>>>> examined or any consensus reached. I believe adopting technologies
>>>> like OVF will only lead to mapping issues between occi attributes
>>>> and the external formats creating a long term management headache.
>>>>
>>>> cheers,
>>>> gary
>>>>
>>>>
>>>> Alexis Richardson wrote:
>>>>> OK
>>>>>
>>>>> Is Sam's statement something that we can achieve consensus around?
>>>>>
>>>>> a
>>>>>
>>>>>
>>>>> On Tue, Oct 20, 2009 at 12:47 PM, Sam Johnston <samj at samj.net>
>>>>> wrote:
>>>>>
>>>>>> Alexis,
>>>>>>
>>>>>> Seems our posts crossed in the mail. I hope the last one clears
>>>>>> things up
>>>>>> but the fact that you're using OVF and XML/JSON interchangeably
>>>>>> suggests
>>>>>> that you [are possibly one of many who] don't grok the proposal.
>>>>>> Worse, it
>>>>>> seems you've got it back to front in thinking of the "core" as
>>>>>> metadata,
>>>>>> which is not at all what I had intended.
>>>>>>
>>>>>> The focus is on the resources and their various representations.
>>>>>> The
>>>>>> metadata (that is, data about data) essentially provides
>>>>>> whatever is
>>>>>> necessary to "glue" these resources together. It is basically
>>>>>> using HTTP as
>>>>>> the meta-model rather than an envelope format like Atom/SOAP or,
>>>>>> worse, a
>>>>>> meta-model that we have to create ourselves from scratch.
>>>>>>
>>>>>> So it's more like:
>>>>>>
>>>>>> 1) OCCI is compatible with ALL existing formats, eg OVF
>>>>>>
>>>>>> 2) Existing formats are used are used for ALL representations
>>>>>>
>>>>>> 3) ALL additional metadata should be in the headers
>>>>>>
>>>>>> 4) ALL of the core stuff should be in the body (that is, the
>>>>>> metadata in the
>>>>>> headers is just there to support the data and wire it up in useful
>>>>>> ways)
>>>>>>
>>>>>> I think the confusion stems from your assuming that there is still
>>>>>> XML/JSON/Atom/etc. representations. There aren't. HTTP headers
>>>>>> provide a
>>>>>> single, performant, interoperable metadata channel that has
>>>>>> already been
>>>>>> both standardised and extensively implemented.
>>>>>>
>>>>>> If you've followed me this far then you should have realised that
>>>>>> every HTTP
>>>>>> user agent (e.g. Perl's LWP, Python's urllib2, C's libcurl, etc.)
>>>>>> is already
>>>>>> a basic OCCI client out-of-the-box. The result is that 5-line
>>>>>> scripts like
>>>>>> demo.py are functional with no dependence on client libraries
>>>>>> (that would
>>>>>> absolutely be necessary if you were to start trying to define your
>>>>>> own
>>>>>> formats).
>>>>>>
>>>>>> Sam
>>>>>>
>>>>>> PS I'm sorry my "bunk" comment came across as referring to you
>>>>>> personally -
>>>>>> it was directed at the argument.
>>>>>>
>>>>>> On Tue, Oct 20, 2009 at 12:44 PM, Alexis Richardson
>>>>>> <alexis.richardson at gmail.com> wrote:
>>>>>>
>>>>>>> I think what Sam is proposing is the following:
>>>>>>>
>>>>>>> 1) If OCCI can handle 'any' data representation, eg OVF, XML,
>>>>>>> JSON,
>>>>>>> then it needs some core / common model.  Otherwise: there can
>>>>>>> be no
>>>>>>> defined behaviour / interop
>>>>>>>
>>>>>>> 2) This core / common model is ALL metadata
>>>>>>>
>>>>>>> 3) ALL this metadata should be in the headers
>>>>>>>
>>>>>>> 4) ALL the non core stuff eg OVF payload, XML representation,
>>>>>>> should be in
>>>>>>> body
>>>>>>>
>>>>>>> Sam - is that right?
>>>>>>>
>>>>>>> a
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 20, 2009 at 11:39 AM, Sam Johnston <samj at samj.net>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Tue, Oct 20, 2009 at 12:28 PM, Edmonds, AndrewX
>>>>>>>> <andrewx.edmonds at intel.com> wrote:
>>>>>>>>
>>>>>>>>> To that point I'd ask - If we're talking about meta-data that
>>>>>>>>> infers
>>>>>>>>> there
>>>>>>>>> is some data to accompany it. Where are the examples of this
>>>>>>>>> data? This
>>>>>>>>> would help in forming the full picture.
>>>>>>>>>
>>>>>>>> Here, from the core spec (using OVF as an example, but could be
>>>>>>>> anything).
>>>>>>>> Request is green, response metadata is yellow & response data/
>>>>>>>> payload is
>>>>>>>> orange:
>>>>>>>>
>>>>>>>>
>>>>>>>>> GET /us-east/webapp/vm01 HTTP/1.1
>>>>>>>>> User-Agent: occi-client/1.0 (linux) libcurl/7.19.4 OCCI/1.0
>>>>>>>>>         Host: cloud.example.com
>>>>>>>>> Accept: */*
>>>>>>>>>
>>>>>>>> < HTTP/1.1 200 OK
>>>>>>>> < Date: Sat, 10 Oct 2009 12:56:51 GMT
>>>>>>>>
>>>>>>>> < Content-Type: application/ovf
>>>>>>>> < Link: </us-east/webapp/vm01;start>;
>>>>>>>>
>>>>>>>> <       rel="http://purl.org/occi/action/start";
>>>>>>>> <       title="Start"
>>>>>>>>
>>>>>>>> < Link: </us-east/webapp/build.pdf>;
>>>>>>>> <       rel="related";
>>>>>>>>
>>>>>>>> <       title="Documentation";
>>>>>>>> <       type="application/pdf"
>>>>>>>>
>>>>>>>> < Category: compute;
>>>>>>>> <       label="Compute Resource";
>>>>>>>>
>>>>>>>> <       scheme="http://purl.org/occi/kind/"
>>>>>>>> < Server: occi-server/1.0 (linux) OCCI/1.0
>>>>>>>>
>>>>>>>> < Connection: close
>>>>>>>> <
>>>>>>>> < <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>>
>>>>>>>> < <Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-
>>>>>>>> instance"
>>>>>>>> <           xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1"
>>>>>>>>
>>>>>>>> <           xmlns="http://schemas.dmtf.org/ovf/envelope/1"
>>>>>>>> <           xml:lang="en-US">
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: occi-wg-bounces at ogf.org [mailto:occi-wg-
>>>>>>>>> bounces at ogf.org] On
>>>>>>>>> Behalf
>>>>>>>>> Of Alexis Richardson
>>>>>>>>> Sent: 20 October 2009 09:49
>>>>>>>>> To: Sam Johnston
>>>>>>>>> Cc: Tim Bray; occi-wg at ogf.org
>>>>>>>>> Subject: Re: [occi-wg] confusion about status of link / headers
>>>>>>>>>
>>>>>>>>> Sam,
>>>>>>>>>
>>>>>>>>> Please don't throw around words like "bunk" willy nilly.
>>>>>>>>>
>>>>>>>>> I have just had a look at
>>>>>>>>> http://www.rackspacecloud.com/cloud_hosting_products/servers/
>>>>>>>>> api
>>>>>>>>>
>>>>>>>>> I can see some limited use of headers for metadata in a few
>>>>>>>>> places.
>>>>>>>>> That's never been controversial.  But the understanding that I
>>>>>>>>> have of
>>>>>>>>> your proposal is to use headers wholesale for all metadata.
>>>>>>>>> Rackspace
>>>>>>>>> don't appear to do that, unless I have misunderstood their
>>>>>>>>> document.
>>>>>>>>> I also may have misunderstood your proposal.  If nobody
>>>>>>>>> understands
>>>>>>>>> your proposal except you, then it will not be possible to gain
>>>>>>>>> consensus around it.
>>>>>>>>>
>>>>>>>>> Are you saying "OCCI metadata should be more like Rackspace
>>>>>>>>> metadata"?
>>>>>>>>>
>>>>>>>>> alexis
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Oct 20, 2009 at 9:07 AM, Sam Johnston <samj at samj.net>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On Tue, Oct 20, 2009 at 9:55 AM, Alexis Richardson
>>>>>>>>>> <alexis.richardson at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> What about compute clouds?
>>>>>>>>>>>
>>>>>>>>>> Rackspace Cloud API for a start:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> HTTP/1.1 204 No Content
>>>>>>>>>>> Date: Mon, 12 Nov 2007 15:32:21 GMT
>>>>>>>>>>> Server: Apache
>>>>>>>>>>> X-Server-Management-Url:
>>>>>>>>>>> https://servers.api.rackspacecloud.com/v1.0/35428
>>>>>>>>>>> X-Storage-Url: https://storage.clouddrive.com/v1/CloudFS_9c83b-5ed4
>>>>>>>>>>> X-CDN-Management-Url:
>>>>>>>>>>> https://cdn.clouddrive.com/v1/CloudFS_9c83b-5ed4
>>>>>>>>>>> X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
>>>>>>>>>>> Content-Length: 0
>>>>>>>>>>> Content-Type: text/plain; charset=UTF-8
>>>>>>>>>>>
>>>>>>>>>> Microsoft Azure uses headers for "attributes" as well, in the
>>>>>>>>>> same
>>>>>>>>>> way
>>>>>>>>>> as I
>>>>>>>>>> had originally proposed:
>>>>>>>>>>
>>>>>>>>>> x-ms-meta-name: value     Returns a metadata value for the
>>>>>>>>>> container.
>>>>>>>>>>
>>>>>>>>>> However prefer that the header names are static/opaque rather
>>>>>>>>>> than
>>>>>>>>>> using
>>>>>>>>>> a
>>>>>>>>>> template and parsing them:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Attribute: name=value
>>>>>>>>>>>
>>>>>>>>>> This to me is cleaner and more self-explanatory, plus it is
>>>>>>>>>> easily
>>>>>>>>>> collapsed
>>>>>>>>>> ala:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Attribute: name1=value1, name2=value2, ... namen=valuen
>>>>>>>>>>>
>>>>>>>>>> Anyway suffice to say that claims this is somehow
>>>>>>>>>> "experimental" are
>>>>>>>>>> bunk.
>>>>>>>>>>
>>>>>>>>>> Sam
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Tue, Oct 20, 2009 at 6:19 AM, Adrian Cole <adrian at jclouds.org
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Here are options for metadata used in some of the major
>>>>>>>>>>>> storage
>>>>>>>>>>>> clouds
>>>>>>>>>>>> FWIW:
>>>>>>>>>>>>
>>>>>>>>>>>> S3, Rackspace, EMC Atmos, Azure - Headers
>>>>>>>>>>>> Nirvanix - query params in, xml entity out
>>>>>>>>>>>> Mezeo - entity
>>>>>>>>>>>>
>>>>>>>>>>>> Of the ones using headers, S3, Rackspace and Azure use
>>>>>>>>>>>> prefix with
>>>>>>>>>>>> values stored as-us.  Atmos joins all metadata together into
>>>>>>>>>>>> one
>>>>>>>>>>>> header, making parsing trivial (split /,/), but necessary.
>>>>>>>>>>>>
>>>>>>>>>>>> The most expensive option of the above is entity, where each
>>>>>>>>>>>> metadata
>>>>>>>>>>>> value is a separate GET.  However, entities allow binary
>>>>>>>>>>>> metadata
>>>>>>>>>>>> and
>>>>>>>>>>>> zero restrictions on it, which may be useful.
>>>>>>>>>>>>
>>>>>>>>>>>> In jclouds, we time parsing of response values.  A simple
>>>>>>>>>>>> XML doc
>>>>>>>>>>>> with
>>>>>>>>>>>> only several elements written in SAX takes a few ms to
>>>>>>>>>>>> parse.  My
>>>>>>>>>>>> log
>>>>>>>>>>>> files are not precise enough to find the overhead in parsing
>>>>>>>>>>>> headers:
>>>>>>>>>>>> they always start and finish within the same millisecond.
>>>>>>>>>>>>
>>>>>>>>>>>> I hope this background helps, and also helps explain why I'm
>>>>>>>>>>>> vocal
>>>>>>>>>>>> on
>>>>>>>>>>>> such topics such as headers vs entities :)
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> -Adrian
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 19, 2009 at 4:21 PM, Sam Johnston
>>>>>>>>>>>> <samj at samj.net>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Oct 20, 2009 at 12:57 AM, gary mazzaferro
>>>>>>>>>>>>> <garymazzaferro at gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The http header and key/value pairs need to parsed also,
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> no
>>>>>>>>>>>>>> free
>>>>>>>>>>>>>> ride here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Every HTTP library I have ever used parses HTTP headers and
>>>>>>>>>>>>> puts
>>>>>>>>>>>>> them
>>>>>>>>>>>>> in a
>>>>>>>>>>>>> nice hash for you ready to consume. If we go for "Name:
>>>>>>>>>>>>> Value"
>>>>>>>>>>>>> then
>>>>>>>>>>>>> that's
>>>>>>>>>>>>> all there is to it. If we go for "Attribute: name=value" as
>>>>>>>>>>>>> is
>>>>>>>>>>>>> currently
>>>>>>>>>>>>> proposed (which is arguably cleaner, follows cookies'
>>>>>>>>>>>>> "prior art"
>>>>>>>>>>>>> and
>>>>>>>>>>>>> avoids
>>>>>>>>>>>>> Amazon's prefix hack) then you just have to split on '='.
>>>>>>>>>>>>>
>>>>>>>>>>>>> To illustrate how clean this is by example:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> #!/usr/bin/python
>>>>>>>>>>>>>> import urllib2
>>>>>>>>>>>>>> response = urllib2.urlopen('http://cloud.example.com/
>>>>>>>>>>>>>> myvm')
>>>>>>>>>>>>>> representation = response.read()
>>>>>>>>>>>>>> metadata = response.info()
>>>>>>>>>>>>>> print metadata['occi-compute-cores']
>>>>>>>>>>>>>>
>>>>>>>>>>>>> As soon as you start talking about payloads you have to
>>>>>>>>>>>>> fire up a
>>>>>>>>>>>>> parser
>>>>>>>>>>>>> (JSON/XML/Atom/etc.) or write your own (previous text
>>>>>>>>>>>>> rendering)
>>>>>>>>>>>>> which
>>>>>>>>>>>>> is
>>>>>>>>>>>>> significantly more work to do at both design and run times.
>>>>>>>>>>>>> Not
>>>>>>>>>>>>> to
>>>>>>>>>>>>> mention
>>>>>>>>>>>>> more work for us to do now and more scope for
>>>>>>>>>>>>> interoperability
>>>>>>>>>>>>> problems.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sam
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> occi-wg mailing list
>>>>>>>>>>>>> occi-wg at ogf.org
>>>>>>>>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> occi-wg mailing list
>>>>>>>>>>>> occi-wg at ogf.org
>>>>>>>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> occi-wg mailing list
>>>>>>>>> occi-wg at ogf.org
>>>>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg
>>>>>>>>> -------------------------------------------------------------
>>>>>>>>> Intel Ireland Limited (Branch)
>>>>>>>>> Collinstown Industrial Park, Leixlip, County Kildare, Ireland
>>>>>>>>> Registered Number: E902934
>>>>>>>>>
>>>>>>>>> This e-mail and any attachments may contain confidential
>>>>>>>>> material for
>>>>>>>>> the sole use of the intended recipient(s). Any review or
>>>>>>>>> distribution
>>>>>>>>> by others is strictly prohibited. If you are not the intended
>>>>>>>>> recipient, please contact the sender and delete all copies.
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> occi-wg mailing list
>>>>> occi-wg at ogf.org
>>>>> http://www.ogf.org/mailman/listinfo/occi-wg
>>>>>
>>>>>
>>>>
>>>
>

More information about the occi-wg mailing list