[occi-wg] confusion about status of link / headers

Alexis Richardson alexis.richardson at gmail.com
Tue Oct 20 11:10:42 CDT 2009 

Sam

Cut the personal stuff please.

Re 'unproven', I think the issues people are having are:

* The extent, scope and purpose of your application of headers to
metadata is unprecedented - as I understand it.  Nobody is saying it
has not been done in other and restricted cases.

* Premature optimisation.

alexis



On Tue, Oct 20, 2009 at 4:44 PM, Sam Johnston <samj at samj.net> wrote:
> Gary,
>
> Please be specific with your objections so as I can have some hope of
> answering them... "several other emails" does not suffice as to the
> best of my knowledge there are no outstanding issues that I have not
> comprehensively and convincingly dealt with.
>
> Your repeated assertions than HTTP headers are... what is it now...
> "unproven"... are demonstrably unfounded (refer to the various cloud
> APIs referred to by myself and Adrian, including both MS Azure and
> Amazon S3, as well as the history of the Internet in general) and
> given the frequency, bordering on FUD.
>
> Regarding performance, OCCI should strive to be as efficient as
> possible as it primarily targets public cloud providers who operate at
> arbitrarily large "cloud" scale. The existing specification is
> *extremely* efficient and alternatives would at least want to be in
> the same neighbourhood. Efficiency may be an inconvenient factor for
> whatever your alternative is, but it's critical for many of our use
> cases.
>
> Finally, you reject OVF without providing any alternative beyond
> building a replacement from scratch ourselves, after our second and
> final deadline has just sailed by no less. Have you considered the
> implementation cost of this? And in the same breath you accuse my
> proposal of being "unproven"? Give me a break...
>
> Correct me if I'm wrong but the way I see it is that you want to
> revert the last 2-3 months of my work because you have an investment
> in an old version of the spec. With each day I spend on OCCI costing
> me well over €1,000 excuse me for being pissed about someone with
> such a blatantly obvious conflict attacking my work and refusing to
> provide a coherent argument, particularly with no "camera ready"
> alternative anywhere in sight.
>
> Sent from my iPhone
>
> On 20/10/2009, at 16:51, Gary Mazz <garymazzaferro at gmail.com> wrote:
>
>> I've understood Sam's proposal from the beginning. I have had issue
>> with from the beginning.
>>
>> Although I find it very interesting and valuable in terms of
>> economy, I have issue with it as the OCCI's primary interface
>> implementation for a few reasons which have been outlined in several
>> other emails.
>>
>> It appears that Sam has created a new technology, a lower cost RPC
>> mechanism, which is unproven for both small and wide scale
>> deployment. For interoperability, IMO,  occi should focus on proven
>> mainstream technologies as the primary interfaces. I am not
>> suggesting, shelving or not supporting Sam's proposal, based on
>> market acceptance, it may prove to be one of many occi interface
>> implementations.
>>
>> As for cost saving in interface technologies, I do not buy into the
>> idea that the occi traffic will be so overwhelming that there is a
>> requirement to maximally optimize http requests and responses. IMO,
>> in comparison to images and consumer data occi traffic will be
>> insignificant. I think we are worrying about spilled milk while the
>> house is burning.
>>
>> There has been some discussion on supporting technologies including
>> OVF. Whether occi supports OVF has not been fully discussed,
>> examined or any consensus reached. I believe adopting technologies
>> like OVF will only lead to mapping issues between occi attributes
>> and the external formats creating a long term management headache.
>>
>> cheers,
>> gary
>>
>>
>> Alexis Richardson wrote:
>>> OK
>>>
>>> Is Sam's statement something that we can achieve consensus around?
>>>
>>> a
>>>
>>>
>>> On Tue, Oct 20, 2009 at 12:47 PM, Sam Johnston <samj at samj.net> wrote:
>>>
>>>> Alexis,
>>>>
>>>> Seems our posts crossed in the mail. I hope the last one clears
>>>> things up
>>>> but the fact that you're using OVF and XML/JSON interchangeably
>>>> suggests
>>>> that you [are possibly one of many who] don't grok the proposal.
>>>> Worse, it
>>>> seems you've got it back to front in thinking of the "core" as
>>>> metadata,
>>>> which is not at all what I had intended.
>>>>
>>>> The focus is on the resources and their various representations. The
>>>> metadata (that is, data about data) essentially provides whatever is
>>>> necessary to "glue" these resources together. It is basically
>>>> using HTTP as
>>>> the meta-model rather than an envelope format like Atom/SOAP or,
>>>> worse, a
>>>> meta-model that we have to create ourselves from scratch.
>>>>
>>>> So it's more like:
>>>>
>>>> 1) OCCI is compatible with ALL existing formats, eg OVF
>>>>
>>>> 2) Existing formats are used are used for ALL representations
>>>>
>>>> 3) ALL additional metadata should be in the headers
>>>>
>>>> 4) ALL of the core stuff should be in the body (that is, the
>>>> metadata in the
>>>> headers is just there to support the data and wire it up in useful
>>>> ways)
>>>>
>>>> I think the confusion stems from your assuming that there is still
>>>> XML/JSON/Atom/etc. representations. There aren't. HTTP headers
>>>> provide a
>>>> single, performant, interoperable metadata channel that has
>>>> already been
>>>> both standardised and extensively implemented.
>>>>
>>>> If you've followed me this far then you should have realised that
>>>> every HTTP
>>>> user agent (e.g. Perl's LWP, Python's urllib2, C's libcurl, etc.)
>>>> is already
>>>> a basic OCCI client out-of-the-box. The result is that 5-line
>>>> scripts like
>>>> demo.py are functional with no dependence on client libraries
>>>> (that would
>>>> absolutely be necessary if you were to start trying to define your
>>>> own
>>>> formats).
>>>>
>>>> Sam
>>>>
>>>> PS I'm sorry my "bunk" comment came across as referring to you
>>>> personally -
>>>> it was directed at the argument.
>>>>
>>>> On Tue, Oct 20, 2009 at 12:44 PM, Alexis Richardson
>>>> <alexis.richardson at gmail.com> wrote:
>>>>
>>>>> I think what Sam is proposing is the following:
>>>>>
>>>>> 1) If OCCI can handle 'any' data representation, eg OVF, XML, JSON,
>>>>> then it needs some core / common model.  Otherwise: there can be no
>>>>> defined behaviour / interop
>>>>>
>>>>> 2) This core / common model is ALL metadata
>>>>>
>>>>> 3) ALL this metadata should be in the headers
>>>>>
>>>>> 4) ALL the non core stuff eg OVF payload, XML representation,
>>>>> should be in
>>>>> body
>>>>>
>>>>> Sam - is that right?
>>>>>
>>>>> a
>>>>>
>>>>>
>>>>> On Tue, Oct 20, 2009 at 11:39 AM, Sam Johnston <samj at samj.net>
>>>>> wrote:
>>>>>
>>>>>> On Tue, Oct 20, 2009 at 12:28 PM, Edmonds, AndrewX
>>>>>> <andrewx.edmonds at intel.com> wrote:
>>>>>>
>>>>>>> To that point I'd ask - If we're talking about meta-data that
>>>>>>> infers
>>>>>>> there
>>>>>>> is some data to accompany it. Where are the examples of this
>>>>>>> data? This
>>>>>>> would help in forming the full picture.
>>>>>>>
>>>>>> Here, from the core spec (using OVF as an example, but could be
>>>>>> anything).
>>>>>> Request is green, response metadata is yellow & response data/
>>>>>> payload is
>>>>>> orange:
>>>>>>
>>>>>>
>>>>>>> GET /us-east/webapp/vm01 HTTP/1.1
>>>>>>> User-Agent: occi-client/1.0 (linux) libcurl/7.19.4 OCCI/1.0
>>>>>>>          Host: cloud.example.com
>>>>>>> Accept: */*
>>>>>>>
>>>>>> < HTTP/1.1 200 OK
>>>>>> < Date: Sat, 10 Oct 2009 12:56:51 GMT
>>>>>>
>>>>>> < Content-Type: application/ovf
>>>>>> < Link: </us-east/webapp/vm01;start>;
>>>>>>
>>>>>> <       rel="http://purl.org/occi/action/start";
>>>>>> <       title="Start"
>>>>>>
>>>>>> < Link: </us-east/webapp/build.pdf>;
>>>>>> <       rel="related";
>>>>>>
>>>>>> <       title="Documentation";
>>>>>> <       type="application/pdf"
>>>>>>
>>>>>> < Category: compute;
>>>>>> <       label="Compute Resource";
>>>>>>
>>>>>> <       scheme="http://purl.org/occi/kind/"
>>>>>> < Server: occi-server/1.0 (linux) OCCI/1.0
>>>>>>
>>>>>> < Connection: close
>>>>>> <
>>>>>> < <?xml version="1.0" encoding="UTF-8"?>
>>>>>>
>>>>>> < <Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>>> <           xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1"
>>>>>>
>>>>>> <           xmlns="http://schemas.dmtf.org/ovf/envelope/1"
>>>>>> <           xml:lang="en-US">
>>>>>>
>>>>>> ...
>>>>>>
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: occi-wg-bounces at ogf.org [mailto:occi-wg-bounces at ogf.org] On
>>>>>>> Behalf
>>>>>>> Of Alexis Richardson
>>>>>>> Sent: 20 October 2009 09:49
>>>>>>> To: Sam Johnston
>>>>>>> Cc: Tim Bray; occi-wg at ogf.org
>>>>>>> Subject: Re: [occi-wg] confusion about status of link / headers
>>>>>>>
>>>>>>> Sam,
>>>>>>>
>>>>>>> Please don't throw around words like "bunk" willy nilly.
>>>>>>>
>>>>>>> I have just had a look at
>>>>>>> http://www.rackspacecloud.com/cloud_hosting_products/servers/api
>>>>>>>
>>>>>>> I can see some limited use of headers for metadata in a few
>>>>>>> places.
>>>>>>> That's never been controversial.  But the understanding that I
>>>>>>> have of
>>>>>>> your proposal is to use headers wholesale for all metadata.
>>>>>>> Rackspace
>>>>>>> don't appear to do that, unless I have misunderstood their
>>>>>>> document.
>>>>>>> I also may have misunderstood your proposal.  If nobody
>>>>>>> understands
>>>>>>> your proposal except you, then it will not be possible to gain
>>>>>>> consensus around it.
>>>>>>>
>>>>>>> Are you saying "OCCI metadata should be more like Rackspace
>>>>>>> metadata"?
>>>>>>>
>>>>>>> alexis
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 20, 2009 at 9:07 AM, Sam Johnston <samj at samj.net>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Tue, Oct 20, 2009 at 9:55 AM, Alexis Richardson
>>>>>>>> <alexis.richardson at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> What about compute clouds?
>>>>>>>>>
>>>>>>>> Rackspace Cloud API for a start:
>>>>>>>>
>>>>>>>>
>>>>>>>>> HTTP/1.1 204 No Content
>>>>>>>>> Date: Mon, 12 Nov 2007 15:32:21 GMT
>>>>>>>>> Server: Apache
>>>>>>>>> X-Server-Management-Url:
>>>>>>>>> https://servers.api.rackspacecloud.com/v1.0/35428
>>>>>>>>> X-Storage-Url: https://storage.clouddrive.com/v1/CloudFS_9c83b-5ed4
>>>>>>>>> X-CDN-Management-Url:
>>>>>>>>> https://cdn.clouddrive.com/v1/CloudFS_9c83b-5ed4
>>>>>>>>> X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
>>>>>>>>> Content-Length: 0
>>>>>>>>> Content-Type: text/plain; charset=UTF-8
>>>>>>>>>
>>>>>>>> Microsoft Azure uses headers for "attributes" as well, in the
>>>>>>>> same
>>>>>>>> way
>>>>>>>> as I
>>>>>>>> had originally proposed:
>>>>>>>>
>>>>>>>> x-ms-meta-name: value     Returns a metadata value for the
>>>>>>>> container.
>>>>>>>>
>>>>>>>> However prefer that the header names are static/opaque rather
>>>>>>>> than
>>>>>>>> using
>>>>>>>> a
>>>>>>>> template and parsing them:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Attribute: name=value
>>>>>>>>>
>>>>>>>> This to me is cleaner and more self-explanatory, plus it is
>>>>>>>> easily
>>>>>>>> collapsed
>>>>>>>> ala:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Attribute: name1=value1, name2=value2, ... namen=valuen
>>>>>>>>>
>>>>>>>> Anyway suffice to say that claims this is somehow
>>>>>>>> "experimental" are
>>>>>>>> bunk.
>>>>>>>>
>>>>>>>> Sam
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Tue, Oct 20, 2009 at 6:19 AM, Adrian Cole <adrian at jclouds.org
>>>>>>>>> >
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Here are options for metadata used in some of the major
>>>>>>>>>> storage
>>>>>>>>>> clouds
>>>>>>>>>> FWIW:
>>>>>>>>>>
>>>>>>>>>> S3, Rackspace, EMC Atmos, Azure - Headers
>>>>>>>>>> Nirvanix - query params in, xml entity out
>>>>>>>>>> Mezeo - entity
>>>>>>>>>>
>>>>>>>>>> Of the ones using headers, S3, Rackspace and Azure use
>>>>>>>>>> prefix with
>>>>>>>>>> values stored as-us.  Atmos joins all metadata together into
>>>>>>>>>> one
>>>>>>>>>> header, making parsing trivial (split /,/), but necessary.
>>>>>>>>>>
>>>>>>>>>> The most expensive option of the above is entity, where each
>>>>>>>>>> metadata
>>>>>>>>>> value is a separate GET.  However, entities allow binary
>>>>>>>>>> metadata
>>>>>>>>>> and
>>>>>>>>>> zero restrictions on it, which may be useful.
>>>>>>>>>>
>>>>>>>>>> In jclouds, we time parsing of response values.  A simple
>>>>>>>>>> XML doc
>>>>>>>>>> with
>>>>>>>>>> only several elements written in SAX takes a few ms to
>>>>>>>>>> parse.  My
>>>>>>>>>> log
>>>>>>>>>> files are not precise enough to find the overhead in parsing
>>>>>>>>>> headers:
>>>>>>>>>> they always start and finish within the same millisecond.
>>>>>>>>>>
>>>>>>>>>> I hope this background helps, and also helps explain why I'm
>>>>>>>>>> vocal
>>>>>>>>>> on
>>>>>>>>>> such topics such as headers vs entities :)
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> -Adrian
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 19, 2009 at 4:21 PM, Sam Johnston <samj at samj.net>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Tue, Oct 20, 2009 at 12:57 AM, gary mazzaferro
>>>>>>>>>>> <garymazzaferro at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> The http header and key/value pairs need to parsed also,
>>>>>>>>>>>> there
>>>>>>>>>>>> is
>>>>>>>>>>>> no
>>>>>>>>>>>> free
>>>>>>>>>>>> ride here.
>>>>>>>>>>>>
>>>>>>>>>>> Every HTTP library I have ever used parses HTTP headers and
>>>>>>>>>>> puts
>>>>>>>>>>> them
>>>>>>>>>>> in a
>>>>>>>>>>> nice hash for you ready to consume. If we go for "Name:
>>>>>>>>>>> Value"
>>>>>>>>>>> then
>>>>>>>>>>> that's
>>>>>>>>>>> all there is to it. If we go for "Attribute: name=value" as
>>>>>>>>>>> is
>>>>>>>>>>> currently
>>>>>>>>>>> proposed (which is arguably cleaner, follows cookies'
>>>>>>>>>>> "prior art"
>>>>>>>>>>> and
>>>>>>>>>>> avoids
>>>>>>>>>>> Amazon's prefix hack) then you just have to split on '='.
>>>>>>>>>>>
>>>>>>>>>>> To illustrate how clean this is by example:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> #!/usr/bin/python
>>>>>>>>>>>> import urllib2
>>>>>>>>>>>> response = urllib2.urlopen('http://cloud.example.com/myvm')
>>>>>>>>>>>> representation = response.read()
>>>>>>>>>>>> metadata = response.info()
>>>>>>>>>>>> print metadata['occi-compute-cores']
>>>>>>>>>>>>
>>>>>>>>>>> As soon as you start talking about payloads you have to
>>>>>>>>>>> fire up a
>>>>>>>>>>> parser
>>>>>>>>>>> (JSON/XML/Atom/etc.) or write your own (previous text
>>>>>>>>>>> rendering)
>>>>>>>>>>> which
>>>>>>>>>>> is
>>>>>>>>>>> significantly more work to do at both design and run times.
>>>>>>>>>>> Not
>>>>>>>>>>> to
>>>>>>>>>>> mention
>>>>>>>>>>> more work for us to do now and more scope for
>>>>>>>>>>> interoperability
>>>>>>>>>>> problems.
>>>>>>>>>>>
>>>>>>>>>>> Sam
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> occi-wg mailing list
>>>>>>>>>>> occi-wg at ogf.org
>>>>>>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> occi-wg mailing list
>>>>>>>>>> occi-wg at ogf.org
>>>>>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> occi-wg mailing list
>>>>>>> occi-wg at ogf.org
>>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg
>>>>>>> -------------------------------------------------------------
>>>>>>> Intel Ireland Limited (Branch)
>>>>>>> Collinstown Industrial Park, Leixlip, County Kildare, Ireland
>>>>>>> Registered Number: E902934
>>>>>>>
>>>>>>> This e-mail and any attachments may contain confidential
>>>>>>> material for
>>>>>>> the sole use of the intended recipient(s). Any review or
>>>>>>> distribution
>>>>>>> by others is strictly prohibited. If you are not the intended
>>>>>>> recipient, please contact the sender and delete all copies.
>>>>>>>
>>>>>>
>>>>
>>> _______________________________________________
>>> occi-wg mailing list
>>> occi-wg at ogf.org
>>> http://www.ogf.org/mailman/listinfo/occi-wg
>>>
>>>
>>
>

More information about the occi-wg mailing list