Re: [occi-wg] confusion about status of link / headers
Alexis, Sweeping, unjustified statements are intensely frustrating, not to mention unhelpful in the absence of a coherent alternative. If people choose not to disclose conflicts I happen to be aware of then you bet I'll disclose them for them - nothing personal about it. Cutting to the chase, both Microsoft and Amazon are using headers for metadata to the same extent as what has been proposed. What premature optimisation? *Not* using XML? Do we really need to drag that carcass out again? Surely Adrian's performance tests put the last nail in that particular coffin. The question really is about whether we want to be an interface/ protocol like HTTP or some mongrel hybrid of that along with a set of XML, JSON and/or text formats for everything we ever intend to model (knowing that work has already been done by others). Sent from my iPhone On 20/10/2009, at 18:10, Alexis Richardson <alexis.richardson@gmail.com> wrote:
Sam
Cut the personal stuff please.
Re 'unproven', I think the issues people are having are:
* The extent, scope and purpose of your application of headers to metadata is unprecedented - as I understand it. Nobody is saying it has not been done in other and restricted cases.
* Premature optimisation.
alexis
On Tue, Oct 20, 2009 at 4:44 PM, Sam Johnston <samj@samj.net> wrote:
Gary,
Please be specific with your objections so as I can have some hope of answering them... "several other emails" does not suffice as to the best of my knowledge there are no outstanding issues that I have not comprehensively and convincingly dealt with.
Your repeated assertions than HTTP headers are... what is it now... "unproven"... are demonstrably unfounded (refer to the various cloud APIs referred to by myself and Adrian, including both MS Azure and Amazon S3, as well as the history of the Internet in general) and given the frequency, bordering on FUD.
Regarding performance, OCCI should strive to be as efficient as possible as it primarily targets public cloud providers who operate at arbitrarily large "cloud" scale. The existing specification is *extremely* efficient and alternatives would at least want to be in the same neighbourhood. Efficiency may be an inconvenient factor for whatever your alternative is, but it's critical for many of our use cases.
Finally, you reject OVF without providing any alternative beyond building a replacement from scratch ourselves, after our second and final deadline has just sailed by no less. Have you considered the implementation cost of this? And in the same breath you accuse my proposal of being "unproven"? Give me a break...
Correct me if I'm wrong but the way I see it is that you want to revert the last 2-3 months of my work because you have an investment in an old version of the spec. With each day I spend on OCCI costing me well over €1,000 excuse me for being pissed about someone with such a blatantly obvious conflict attacking my work and refusing to provide a coherent argument, particularly with no "camera ready" alternative anywhere in sight.
Sent from my iPhone
On 20/10/2009, at 16:51, Gary Mazz <garymazzaferro@gmail.com> wrote:
I've understood Sam's proposal from the beginning. I have had issue with from the beginning.
Although I find it very interesting and valuable in terms of economy, I have issue with it as the OCCI's primary interface implementation for a few reasons which have been outlined in several other emails.
It appears that Sam has created a new technology, a lower cost RPC mechanism, which is unproven for both small and wide scale deployment. For interoperability, IMO, occi should focus on proven mainstream technologies as the primary interfaces. I am not suggesting, shelving or not supporting Sam's proposal, based on market acceptance, it may prove to be one of many occi interface implementations.
As for cost saving in interface technologies, I do not buy into the idea that the occi traffic will be so overwhelming that there is a requirement to maximally optimize http requests and responses. IMO, in comparison to images and consumer data occi traffic will be insignificant. I think we are worrying about spilled milk while the house is burning.
There has been some discussion on supporting technologies including OVF. Whether occi supports OVF has not been fully discussed, examined or any consensus reached. I believe adopting technologies like OVF will only lead to mapping issues between occi attributes and the external formats creating a long term management headache.
cheers, gary
Alexis Richardson wrote:
OK
Is Sam's statement something that we can achieve consensus around?
a
On Tue, Oct 20, 2009 at 12:47 PM, Sam Johnston <samj@samj.net> wrote:
Alexis,
Seems our posts crossed in the mail. I hope the last one clears things up but the fact that you're using OVF and XML/JSON interchangeably suggests that you [are possibly one of many who] don't grok the proposal. Worse, it seems you've got it back to front in thinking of the "core" as metadata, which is not at all what I had intended.
The focus is on the resources and their various representations. The metadata (that is, data about data) essentially provides whatever is necessary to "glue" these resources together. It is basically using HTTP as the meta-model rather than an envelope format like Atom/SOAP or, worse, a meta-model that we have to create ourselves from scratch.
So it's more like:
1) OCCI is compatible with ALL existing formats, eg OVF
2) Existing formats are used are used for ALL representations
3) ALL additional metadata should be in the headers
4) ALL of the core stuff should be in the body (that is, the metadata in the headers is just there to support the data and wire it up in useful ways)
I think the confusion stems from your assuming that there is still XML/JSON/Atom/etc. representations. There aren't. HTTP headers provide a single, performant, interoperable metadata channel that has already been both standardised and extensively implemented.
If you've followed me this far then you should have realised that every HTTP user agent (e.g. Perl's LWP, Python's urllib2, C's libcurl, etc.) is already a basic OCCI client out-of-the-box. The result is that 5-line scripts like demo.py are functional with no dependence on client libraries (that would absolutely be necessary if you were to start trying to define your own formats).
Sam
PS I'm sorry my "bunk" comment came across as referring to you personally - it was directed at the argument.
On Tue, Oct 20, 2009 at 12:44 PM, Alexis Richardson <alexis.richardson@gmail.com> wrote:
I think what Sam is proposing is the following:
1) If OCCI can handle 'any' data representation, eg OVF, XML, JSON, then it needs some core / common model. Otherwise: there can be no defined behaviour / interop
2) This core / common model is ALL metadata
3) ALL this metadata should be in the headers
4) ALL the non core stuff eg OVF payload, XML representation, should be in body
Sam - is that right?
a
On Tue, Oct 20, 2009 at 11:39 AM, Sam Johnston <samj@samj.net> wrote:
> On Tue, Oct 20, 2009 at 12:28 PM, Edmonds, AndrewX > <andrewx.edmonds@intel.com> wrote: > >> To that point I'd ask - If we're talking about meta-data that >> infers >> there >> is some data to accompany it. Where are the examples of this >> data? This >> would help in forming the full picture. >> > Here, from the core spec (using OVF as an example, but could be > anything). > Request is green, response metadata is yellow & response data/ > payload is > orange: > > >> GET /us-east/webapp/vm01 HTTP/1.1 >> User-Agent: occi-client/1.0 (linux) libcurl/7.19.4 OCCI/1.0 >> Host: cloud.example.com >> Accept: */* >> > < HTTP/1.1 200 OK > < Date: Sat, 10 Oct 2009 12:56:51 GMT > > < Content-Type: application/ovf > < Link: </us-east/webapp/vm01;start>; > > < rel="http://purl.org/occi/action/start"; > < title="Start" > > < Link: </us-east/webapp/build.pdf>; > < rel="related"; > > < title="Documentation"; > < type="application/pdf" > > < Category: compute; > < label="Compute Resource"; > > < scheme="http://purl.org/occi/kind/" > < Server: occi-server/1.0 (linux) OCCI/1.0 > > < Connection: close > < > < <?xml version="1.0" encoding="UTF-8"?> > > < <Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema- > instance" > < xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" > > < xmlns="http://schemas.dmtf.org/ovf/envelope/1" > < xml:lang="en-US"> > > ... > > > >> -----Original Message----- >> From: occi-wg-bounces@ogf.org [mailto:occi-wg- >> bounces@ogf.org] On >> Behalf >> Of Alexis Richardson >> Sent: 20 October 2009 09:49 >> To: Sam Johnston >> Cc: Tim Bray; occi-wg@ogf.org >> Subject: Re: [occi-wg] confusion about status of link / headers >> >> Sam, >> >> Please don't throw around words like "bunk" willy nilly. >> >> I have just had a look at >> http://www.rackspacecloud.com/cloud_hosting_products/servers/ >> api >> >> I can see some limited use of headers for metadata in a few >> places. >> That's never been controversial. But the understanding that I >> have of >> your proposal is to use headers wholesale for all metadata. >> Rackspace >> don't appear to do that, unless I have misunderstood their >> document. >> I also may have misunderstood your proposal. If nobody >> understands >> your proposal except you, then it will not be possible to gain >> consensus around it. >> >> Are you saying "OCCI metadata should be more like Rackspace >> metadata"? >> >> alexis >> >> >> On Tue, Oct 20, 2009 at 9:07 AM, Sam Johnston <samj@samj.net> >> wrote: >> >>> On Tue, Oct 20, 2009 at 9:55 AM, Alexis Richardson >>> <alexis.richardson@gmail.com> wrote: >>> >>>> What about compute clouds? >>>> >>> Rackspace Cloud API for a start: >>> >>> >>>> HTTP/1.1 204 No Content >>>> Date: Mon, 12 Nov 2007 15:32:21 GMT >>>> Server: Apache >>>> X-Server-Management-Url: >>>> https://servers.api.rackspacecloud.com/v1.0/35428 >>>> X-Storage-Url: https://storage.clouddrive.com/v1/CloudFS_9c83b-5ed4 >>>> X-CDN-Management-Url: >>>> https://cdn.clouddrive.com/v1/CloudFS_9c83b-5ed4 >>>> X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb >>>> Content-Length: 0 >>>> Content-Type: text/plain; charset=UTF-8 >>>> >>> Microsoft Azure uses headers for "attributes" as well, in the >>> same >>> way >>> as I >>> had originally proposed: >>> >>> x-ms-meta-name: value Returns a metadata value for the >>> container. >>> >>> However prefer that the header names are static/opaque rather >>> than >>> using >>> a >>> template and parsing them: >>> >>> >>>> Attribute: name=value >>>> >>> This to me is cleaner and more self-explanatory, plus it is >>> easily >>> collapsed >>> ala: >>> >>> >>>> Attribute: name1=value1, name2=value2, ... namen=valuen >>>> >>> Anyway suffice to say that claims this is somehow >>> "experimental" are >>> bunk. >>> >>> Sam >>> >>> >>>> On Tue, Oct 20, 2009 at 6:19 AM, Adrian Cole <adrian@jclouds.org >>>>> >>>> wrote: >>>> >>>>> Here are options for metadata used in some of the major >>>>> storage >>>>> clouds >>>>> FWIW: >>>>> >>>>> S3, Rackspace, EMC Atmos, Azure - Headers >>>>> Nirvanix - query params in, xml entity out >>>>> Mezeo - entity >>>>> >>>>> Of the ones using headers, S3, Rackspace and Azure use >>>>> prefix with >>>>> values stored as-us. Atmos joins all metadata together into >>>>> one >>>>> header, making parsing trivial (split /,/), but necessary. >>>>> >>>>> The most expensive option of the above is entity, where each >>>>> metadata >>>>> value is a separate GET. However, entities allow binary >>>>> metadata >>>>> and >>>>> zero restrictions on it, which may be useful. >>>>> >>>>> In jclouds, we time parsing of response values. A simple >>>>> XML doc >>>>> with >>>>> only several elements written in SAX takes a few ms to >>>>> parse. My >>>>> log >>>>> files are not precise enough to find the overhead in parsing >>>>> headers: >>>>> they always start and finish within the same millisecond. >>>>> >>>>> I hope this background helps, and also helps explain why I'm >>>>> vocal >>>>> on >>>>> such topics such as headers vs entities :) >>>>> >>>>> Cheers, >>>>> -Adrian >>>>> >>>>> >>>>> On Mon, Oct 19, 2009 at 4:21 PM, Sam Johnston >>>>> <samj@samj.net> >>>>> wrote: >>>>> >>>>>> On Tue, Oct 20, 2009 at 12:57 AM, gary mazzaferro >>>>>> <garymazzaferro@gmail.com> >>>>>> wrote: >>>>>> >>>>>> >>>>>>> The http header and key/value pairs need to parsed also, >>>>>>> there >>>>>>> is >>>>>>> no >>>>>>> free >>>>>>> ride here. >>>>>>> >>>>>> Every HTTP library I have ever used parses HTTP headers and >>>>>> puts >>>>>> them >>>>>> in a >>>>>> nice hash for you ready to consume. If we go for "Name: >>>>>> Value" >>>>>> then >>>>>> that's >>>>>> all there is to it. If we go for "Attribute: name=value" as >>>>>> is >>>>>> currently >>>>>> proposed (which is arguably cleaner, follows cookies' >>>>>> "prior art" >>>>>> and >>>>>> avoids >>>>>> Amazon's prefix hack) then you just have to split on '='. >>>>>> >>>>>> To illustrate how clean this is by example: >>>>>> >>>>>> >>>>>>> #!/usr/bin/python >>>>>>> import urllib2 >>>>>>> response = urllib2.urlopen('http://cloud.example.com/ >>>>>>> myvm') >>>>>>> representation = response.read() >>>>>>> metadata = response.info() >>>>>>> print metadata['occi-compute-cores'] >>>>>>> >>>>>> As soon as you start talking about payloads you have to >>>>>> fire up a >>>>>> parser >>>>>> (JSON/XML/Atom/etc.) or write your own (previous text >>>>>> rendering) >>>>>> which >>>>>> is >>>>>> significantly more work to do at both design and run times. >>>>>> Not >>>>>> to >>>>>> mention >>>>>> more work for us to do now and more scope for >>>>>> interoperability >>>>>> problems. >>>>>> >>>>>> Sam >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> occi-wg mailing list >>>>>> occi-wg@ogf.org >>>>>> http://www.ogf.org/mailman/listinfo/occi-wg >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> occi-wg mailing list >>>>> occi-wg@ogf.org >>>>> http://www.ogf.org/mailman/listinfo/occi-wg >>>>> >>>>> >>> >> _______________________________________________ >> occi-wg mailing list >> occi-wg@ogf.org >> http://www.ogf.org/mailman/listinfo/occi-wg >> ------------------------------------------------------------- >> Intel Ireland Limited (Branch) >> Collinstown Industrial Park, Leixlip, County Kildare, Ireland >> Registered Number: E902934 >> >> This e-mail and any attachments may contain confidential >> material for >> the sole use of the intended recipient(s). Any review or >> distribution >> by others is strictly prohibited. If you are not the intended >> recipient, please contact the sender and delete all copies. >> >
_______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
Sam, On Tue, Oct 20, 2009 at 5:31 PM, Sam Johnston <samj@samj.net> wrote:
Alexis,
Sweeping, unjustified statements are intensely frustrating, not to mention unhelpful in the absence of a coherent alternative. If people choose not to disclose conflicts I happen to be aware of then you bet I'll disclose them for them - nothing personal about it.
I don't know what 'conflicts' you are referring to, or how these relate to the matter at hand. If by 'sweeping, unjustified' you refer to the word 'unprecedented' then I shall justify it by pointing out that - at least as I currently understand it - you are proposing use of headers to a *greater* extent than has been done before and to cover a *greater* set of use cases.
Cutting to the chase, both Microsoft and Amazon are using headers for metadata to the same extent as what has been proposed.
Really - to the "same" extent? That seems contrary to what has been asserted on this thread previously.
What premature optimisation? *Not* using XML? Do we really need to drag that carcass out again? Surely Adrian's performance tests put the last nail in that particular coffin.
Premature optimisation = making performance a critical design concern when we want an API. I have noted Adrian's points as have others.
The question really is about whether we want to be an interface/ protocol like HTTP or some mongrel hybrid of that along with a set of XML, JSON and/or text formats for everything we ever intend to model (knowing that work has already been done by others).
Work that has been done by others and shown to work, is a good thing. alexis
Sent from my iPhone
On 20/10/2009, at 18:10, Alexis Richardson <alexis.richardson@gmail.com> wrote:
Sam
Cut the personal stuff please.
Re 'unproven', I think the issues people are having are:
* The extent, scope and purpose of your application of headers to metadata is unprecedented - as I understand it. Nobody is saying it has not been done in other and restricted cases.
* Premature optimisation.
alexis
On Tue, Oct 20, 2009 at 4:44 PM, Sam Johnston <samj@samj.net> wrote:
Gary,
Please be specific with your objections so as I can have some hope of answering them... "several other emails" does not suffice as to the best of my knowledge there are no outstanding issues that I have not comprehensively and convincingly dealt with.
Your repeated assertions than HTTP headers are... what is it now... "unproven"... are demonstrably unfounded (refer to the various cloud APIs referred to by myself and Adrian, including both MS Azure and Amazon S3, as well as the history of the Internet in general) and given the frequency, bordering on FUD.
Regarding performance, OCCI should strive to be as efficient as possible as it primarily targets public cloud providers who operate at arbitrarily large "cloud" scale. The existing specification is *extremely* efficient and alternatives would at least want to be in the same neighbourhood. Efficiency may be an inconvenient factor for whatever your alternative is, but it's critical for many of our use cases.
Finally, you reject OVF without providing any alternative beyond building a replacement from scratch ourselves, after our second and final deadline has just sailed by no less. Have you considered the implementation cost of this? And in the same breath you accuse my proposal of being "unproven"? Give me a break...
Correct me if I'm wrong but the way I see it is that you want to revert the last 2-3 months of my work because you have an investment in an old version of the spec. With each day I spend on OCCI costing me well over €1,000 excuse me for being pissed about someone with such a blatantly obvious conflict attacking my work and refusing to provide a coherent argument, particularly with no "camera ready" alternative anywhere in sight.
Sent from my iPhone
On 20/10/2009, at 16:51, Gary Mazz <garymazzaferro@gmail.com> wrote:
I've understood Sam's proposal from the beginning. I have had issue with from the beginning.
Although I find it very interesting and valuable in terms of economy, I have issue with it as the OCCI's primary interface implementation for a few reasons which have been outlined in several other emails.
It appears that Sam has created a new technology, a lower cost RPC mechanism, which is unproven for both small and wide scale deployment. For interoperability, IMO, occi should focus on proven mainstream technologies as the primary interfaces. I am not suggesting, shelving or not supporting Sam's proposal, based on market acceptance, it may prove to be one of many occi interface implementations.
As for cost saving in interface technologies, I do not buy into the idea that the occi traffic will be so overwhelming that there is a requirement to maximally optimize http requests and responses. IMO, in comparison to images and consumer data occi traffic will be insignificant. I think we are worrying about spilled milk while the house is burning.
There has been some discussion on supporting technologies including OVF. Whether occi supports OVF has not been fully discussed, examined or any consensus reached. I believe adopting technologies like OVF will only lead to mapping issues between occi attributes and the external formats creating a long term management headache.
cheers, gary
Alexis Richardson wrote:
OK
Is Sam's statement something that we can achieve consensus around?
a
On Tue, Oct 20, 2009 at 12:47 PM, Sam Johnston <samj@samj.net> wrote:
Alexis,
Seems our posts crossed in the mail. I hope the last one clears things up but the fact that you're using OVF and XML/JSON interchangeably suggests that you [are possibly one of many who] don't grok the proposal. Worse, it seems you've got it back to front in thinking of the "core" as metadata, which is not at all what I had intended.
The focus is on the resources and their various representations. The metadata (that is, data about data) essentially provides whatever is necessary to "glue" these resources together. It is basically using HTTP as the meta-model rather than an envelope format like Atom/SOAP or, worse, a meta-model that we have to create ourselves from scratch.
So it's more like:
1) OCCI is compatible with ALL existing formats, eg OVF
2) Existing formats are used are used for ALL representations
3) ALL additional metadata should be in the headers
4) ALL of the core stuff should be in the body (that is, the metadata in the headers is just there to support the data and wire it up in useful ways)
I think the confusion stems from your assuming that there is still XML/JSON/Atom/etc. representations. There aren't. HTTP headers provide a single, performant, interoperable metadata channel that has already been both standardised and extensively implemented.
If you've followed me this far then you should have realised that every HTTP user agent (e.g. Perl's LWP, Python's urllib2, C's libcurl, etc.) is already a basic OCCI client out-of-the-box. The result is that 5-line scripts like demo.py are functional with no dependence on client libraries (that would absolutely be necessary if you were to start trying to define your own formats).
Sam
PS I'm sorry my "bunk" comment came across as referring to you personally - it was directed at the argument.
On Tue, Oct 20, 2009 at 12:44 PM, Alexis Richardson <alexis.richardson@gmail.com> wrote:
> I think what Sam is proposing is the following: > > 1) If OCCI can handle 'any' data representation, eg OVF, XML, > JSON, > then it needs some core / common model. Otherwise: there can > be no > defined behaviour / interop > > 2) This core / common model is ALL metadata > > 3) ALL this metadata should be in the headers > > 4) ALL the non core stuff eg OVF payload, XML representation, > should be in > body > > Sam - is that right? > > a > > > On Tue, Oct 20, 2009 at 11:39 AM, Sam Johnston <samj@samj.net> > wrote: > >> On Tue, Oct 20, 2009 at 12:28 PM, Edmonds, AndrewX >> <andrewx.edmonds@intel.com> wrote: >> >>> To that point I'd ask - If we're talking about meta-data that >>> infers >>> there >>> is some data to accompany it. Where are the examples of this >>> data? This >>> would help in forming the full picture. >>> >> Here, from the core spec (using OVF as an example, but could be >> anything). >> Request is green, response metadata is yellow & response data/ >> payload is >> orange: >> >> >>> GET /us-east/webapp/vm01 HTTP/1.1 >>> User-Agent: occi-client/1.0 (linux) libcurl/7.19.4 OCCI/1.0 >>> Host: cloud.example.com >>> Accept: */* >>> >> < HTTP/1.1 200 OK >> < Date: Sat, 10 Oct 2009 12:56:51 GMT >> >> < Content-Type: application/ovf >> < Link: </us-east/webapp/vm01;start>; >> >> < rel="http://purl.org/occi/action/start"; >> < title="Start" >> >> < Link: </us-east/webapp/build.pdf>; >> < rel="related"; >> >> < title="Documentation"; >> < type="application/pdf" >> >> < Category: compute; >> < label="Compute Resource"; >> >> < scheme="http://purl.org/occi/kind/" >> < Server: occi-server/1.0 (linux) OCCI/1.0 >> >> < Connection: close >> < >> < <?xml version="1.0" encoding="UTF-8"?> >> >> < <Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema- >> instance" >> < xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" >> >> < xmlns="http://schemas.dmtf.org/ovf/envelope/1" >> < xml:lang="en-US"> >> >> ... >> >> >> >>> -----Original Message----- >>> From: occi-wg-bounces@ogf.org [mailto:occi-wg- >>> bounces@ogf.org] On >>> Behalf >>> Of Alexis Richardson >>> Sent: 20 October 2009 09:49 >>> To: Sam Johnston >>> Cc: Tim Bray; occi-wg@ogf.org >>> Subject: Re: [occi-wg] confusion about status of link / headers >>> >>> Sam, >>> >>> Please don't throw around words like "bunk" willy nilly. >>> >>> I have just had a look at >>> http://www.rackspacecloud.com/cloud_hosting_products/servers/ >>> api >>> >>> I can see some limited use of headers for metadata in a few >>> places. >>> That's never been controversial. But the understanding that I >>> have of >>> your proposal is to use headers wholesale for all metadata. >>> Rackspace >>> don't appear to do that, unless I have misunderstood their >>> document. >>> I also may have misunderstood your proposal. If nobody >>> understands >>> your proposal except you, then it will not be possible to gain >>> consensus around it. >>> >>> Are you saying "OCCI metadata should be more like Rackspace >>> metadata"? >>> >>> alexis >>> >>> >>> On Tue, Oct 20, 2009 at 9:07 AM, Sam Johnston <samj@samj.net> >>> wrote: >>> >>>> On Tue, Oct 20, 2009 at 9:55 AM, Alexis Richardson >>>> <alexis.richardson@gmail.com> wrote: >>>> >>>>> What about compute clouds? >>>>> >>>> Rackspace Cloud API for a start: >>>> >>>> >>>>> HTTP/1.1 204 No Content >>>>> Date: Mon, 12 Nov 2007 15:32:21 GMT >>>>> Server: Apache >>>>> X-Server-Management-Url: >>>>> https://servers.api.rackspacecloud.com/v1.0/35428 >>>>> X-Storage-Url: https://storage.clouddrive.com/v1/CloudFS_9c83b-5ed4 >>>>> X-CDN-Management-Url: >>>>> https://cdn.clouddrive.com/v1/CloudFS_9c83b-5ed4 >>>>> X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb >>>>> Content-Length: 0 >>>>> Content-Type: text/plain; charset=UTF-8 >>>>> >>>> Microsoft Azure uses headers for "attributes" as well, in the >>>> same >>>> way >>>> as I >>>> had originally proposed: >>>> >>>> x-ms-meta-name: value Returns a metadata value for the >>>> container. >>>> >>>> However prefer that the header names are static/opaque rather >>>> than >>>> using >>>> a >>>> template and parsing them: >>>> >>>> >>>>> Attribute: name=value >>>>> >>>> This to me is cleaner and more self-explanatory, plus it is >>>> easily >>>> collapsed >>>> ala: >>>> >>>> >>>>> Attribute: name1=value1, name2=value2, ... namen=valuen >>>>> >>>> Anyway suffice to say that claims this is somehow >>>> "experimental" are >>>> bunk. >>>> >>>> Sam >>>> >>>> >>>>> On Tue, Oct 20, 2009 at 6:19 AM, Adrian Cole <adrian@jclouds.org >>>>>> >>>>> wrote: >>>>> >>>>>> Here are options for metadata used in some of the major >>>>>> storage >>>>>> clouds >>>>>> FWIW: >>>>>> >>>>>> S3, Rackspace, EMC Atmos, Azure - Headers >>>>>> Nirvanix - query params in, xml entity out >>>>>> Mezeo - entity >>>>>> >>>>>> Of the ones using headers, S3, Rackspace and Azure use >>>>>> prefix with >>>>>> values stored as-us. Atmos joins all metadata together into >>>>>> one >>>>>> header, making parsing trivial (split /,/), but necessary. >>>>>> >>>>>> The most expensive option of the above is entity, where each >>>>>> metadata >>>>>> value is a separate GET. However, entities allow binary >>>>>> metadata >>>>>> and >>>>>> zero restrictions on it, which may be useful. >>>>>> >>>>>> In jclouds, we time parsing of response values. A simple >>>>>> XML doc >>>>>> with >>>>>> only several elements written in SAX takes a few ms to >>>>>> parse. My >>>>>> log >>>>>> files are not precise enough to find the overhead in parsing >>>>>> headers: >>>>>> they always start and finish within the same millisecond. >>>>>> >>>>>> I hope this background helps, and also helps explain why I'm >>>>>> vocal >>>>>> on >>>>>> such topics such as headers vs entities :) >>>>>> >>>>>> Cheers, >>>>>> -Adrian >>>>>> >>>>>> >>>>>> On Mon, Oct 19, 2009 at 4:21 PM, Sam Johnston >>>>>> <samj@samj.net> >>>>>> wrote: >>>>>> >>>>>>> On Tue, Oct 20, 2009 at 12:57 AM, gary mazzaferro >>>>>>> <garymazzaferro@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>>> The http header and key/value pairs need to parsed also, >>>>>>>> there >>>>>>>> is >>>>>>>> no >>>>>>>> free >>>>>>>> ride here. >>>>>>>> >>>>>>> Every HTTP library I have ever used parses HTTP headers and >>>>>>> puts >>>>>>> them >>>>>>> in a >>>>>>> nice hash for you ready to consume. If we go for "Name: >>>>>>> Value" >>>>>>> then >>>>>>> that's >>>>>>> all there is to it. If we go for "Attribute: name=value" as >>>>>>> is >>>>>>> currently >>>>>>> proposed (which is arguably cleaner, follows cookies' >>>>>>> "prior art" >>>>>>> and >>>>>>> avoids >>>>>>> Amazon's prefix hack) then you just have to split on '='. >>>>>>> >>>>>>> To illustrate how clean this is by example: >>>>>>> >>>>>>> >>>>>>>> #!/usr/bin/python >>>>>>>> import urllib2 >>>>>>>> response = urllib2.urlopen('http://cloud.example.com/ >>>>>>>> myvm') >>>>>>>> representation = response.read() >>>>>>>> metadata = response.info() >>>>>>>> print metadata['occi-compute-cores'] >>>>>>>> >>>>>>> As soon as you start talking about payloads you have to >>>>>>> fire up a >>>>>>> parser >>>>>>> (JSON/XML/Atom/etc.) or write your own (previous text >>>>>>> rendering) >>>>>>> which >>>>>>> is >>>>>>> significantly more work to do at both design and run times. >>>>>>> Not >>>>>>> to >>>>>>> mention >>>>>>> more work for us to do now and more scope for >>>>>>> interoperability >>>>>>> problems. >>>>>>> >>>>>>> Sam >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> occi-wg mailing list >>>>>>> occi-wg@ogf.org >>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> occi-wg mailing list >>>>>> occi-wg@ogf.org >>>>>> http://www.ogf.org/mailman/listinfo/occi-wg >>>>>> >>>>>> >>>> >>> _______________________________________________ >>> occi-wg mailing list >>> occi-wg@ogf.org >>> http://www.ogf.org/mailman/listinfo/occi-wg >>> ------------------------------------------------------------- >>> Intel Ireland Limited (Branch) >>> Collinstown Industrial Park, Leixlip, County Kildare, Ireland >>> Registered Number: E902934 >>> >>> This e-mail and any attachments may contain confidential >>> material for >>> the sole use of the intended recipient(s). Any review or >>> distribution >>> by others is strictly prohibited. If you are not the intended >>> recipient, please contact the sender and delete all copies. >>> >>
_______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
As we aren't introducing new information, I think we are at a point where a vote can take place. This issue isn't infinitely complex, and reasonable to model in choices. OCCI desires to be a community process, so let's see where we fall on this. Do you agree? -Adrian On Tue, Oct 20, 2009 at 9:39 AM, Alexis Richardson <alexis.richardson@gmail.com> wrote:
Sam,
On Tue, Oct 20, 2009 at 5:31 PM, Sam Johnston <samj@samj.net> wrote:
Alexis,
Sweeping, unjustified statements are intensely frustrating, not to mention unhelpful in the absence of a coherent alternative. If people choose not to disclose conflicts I happen to be aware of then you bet I'll disclose them for them - nothing personal about it.
I don't know what 'conflicts' you are referring to, or how these relate to the matter at hand. If by 'sweeping, unjustified' you refer to the word 'unprecedented' then I shall justify it by pointing out that - at least as I currently understand it - you are proposing use of headers to a *greater* extent than has been done before and to cover a *greater* set of use cases.
Cutting to the chase, both Microsoft and Amazon are using headers for metadata to the same extent as what has been proposed.
Really - to the "same" extent? That seems contrary to what has been asserted on this thread previously.
What premature optimisation? *Not* using XML? Do we really need to drag that carcass out again? Surely Adrian's performance tests put the last nail in that particular coffin.
Premature optimisation = making performance a critical design concern when we want an API.
I have noted Adrian's points as have others.
The question really is about whether we want to be an interface/ protocol like HTTP or some mongrel hybrid of that along with a set of XML, JSON and/or text formats for everything we ever intend to model (knowing that work has already been done by others).
Work that has been done by others and shown to work, is a good thing.
alexis
Sent from my iPhone
On 20/10/2009, at 18:10, Alexis Richardson <alexis.richardson@gmail.com> wrote:
Sam
Cut the personal stuff please.
Re 'unproven', I think the issues people are having are:
* The extent, scope and purpose of your application of headers to metadata is unprecedented - as I understand it. Nobody is saying it has not been done in other and restricted cases.
* Premature optimisation.
alexis
On Tue, Oct 20, 2009 at 4:44 PM, Sam Johnston <samj@samj.net> wrote:
Gary,
Please be specific with your objections so as I can have some hope of answering them... "several other emails" does not suffice as to the best of my knowledge there are no outstanding issues that I have not comprehensively and convincingly dealt with.
Your repeated assertions than HTTP headers are... what is it now... "unproven"... are demonstrably unfounded (refer to the various cloud APIs referred to by myself and Adrian, including both MS Azure and Amazon S3, as well as the history of the Internet in general) and given the frequency, bordering on FUD.
Regarding performance, OCCI should strive to be as efficient as possible as it primarily targets public cloud providers who operate at arbitrarily large "cloud" scale. The existing specification is *extremely* efficient and alternatives would at least want to be in the same neighbourhood. Efficiency may be an inconvenient factor for whatever your alternative is, but it's critical for many of our use cases.
Finally, you reject OVF without providing any alternative beyond building a replacement from scratch ourselves, after our second and final deadline has just sailed by no less. Have you considered the implementation cost of this? And in the same breath you accuse my proposal of being "unproven"? Give me a break...
Correct me if I'm wrong but the way I see it is that you want to revert the last 2-3 months of my work because you have an investment in an old version of the spec. With each day I spend on OCCI costing me well over €1,000 excuse me for being pissed about someone with such a blatantly obvious conflict attacking my work and refusing to provide a coherent argument, particularly with no "camera ready" alternative anywhere in sight.
Sent from my iPhone
On 20/10/2009, at 16:51, Gary Mazz <garymazzaferro@gmail.com> wrote:
I've understood Sam's proposal from the beginning. I have had issue with from the beginning.
Although I find it very interesting and valuable in terms of economy, I have issue with it as the OCCI's primary interface implementation for a few reasons which have been outlined in several other emails.
It appears that Sam has created a new technology, a lower cost RPC mechanism, which is unproven for both small and wide scale deployment. For interoperability, IMO, occi should focus on proven mainstream technologies as the primary interfaces. I am not suggesting, shelving or not supporting Sam's proposal, based on market acceptance, it may prove to be one of many occi interface implementations.
As for cost saving in interface technologies, I do not buy into the idea that the occi traffic will be so overwhelming that there is a requirement to maximally optimize http requests and responses. IMO, in comparison to images and consumer data occi traffic will be insignificant. I think we are worrying about spilled milk while the house is burning.
There has been some discussion on supporting technologies including OVF. Whether occi supports OVF has not been fully discussed, examined or any consensus reached. I believe adopting technologies like OVF will only lead to mapping issues between occi attributes and the external formats creating a long term management headache.
cheers, gary
Alexis Richardson wrote:
OK
Is Sam's statement something that we can achieve consensus around?
a
On Tue, Oct 20, 2009 at 12:47 PM, Sam Johnston <samj@samj.net> wrote:
> Alexis, > > Seems our posts crossed in the mail. I hope the last one clears > things up > but the fact that you're using OVF and XML/JSON interchangeably > suggests > that you [are possibly one of many who] don't grok the proposal. > Worse, it > seems you've got it back to front in thinking of the "core" as > metadata, > which is not at all what I had intended. > > The focus is on the resources and their various representations. > The > metadata (that is, data about data) essentially provides > whatever is > necessary to "glue" these resources together. It is basically > using HTTP as > the meta-model rather than an envelope format like Atom/SOAP or, > worse, a > meta-model that we have to create ourselves from scratch. > > So it's more like: > > 1) OCCI is compatible with ALL existing formats, eg OVF > > 2) Existing formats are used are used for ALL representations > > 3) ALL additional metadata should be in the headers > > 4) ALL of the core stuff should be in the body (that is, the > metadata in the > headers is just there to support the data and wire it up in useful > ways) > > I think the confusion stems from your assuming that there is still > XML/JSON/Atom/etc. representations. There aren't. HTTP headers > provide a > single, performant, interoperable metadata channel that has > already been > both standardised and extensively implemented. > > If you've followed me this far then you should have realised that > every HTTP > user agent (e.g. Perl's LWP, Python's urllib2, C's libcurl, etc.) > is already > a basic OCCI client out-of-the-box. The result is that 5-line > scripts like > demo.py are functional with no dependence on client libraries > (that would > absolutely be necessary if you were to start trying to define your > own > formats). > > Sam > > PS I'm sorry my "bunk" comment came across as referring to you > personally - > it was directed at the argument. > > On Tue, Oct 20, 2009 at 12:44 PM, Alexis Richardson > <alexis.richardson@gmail.com> wrote: > >> I think what Sam is proposing is the following: >> >> 1) If OCCI can handle 'any' data representation, eg OVF, XML, >> JSON, >> then it needs some core / common model. Otherwise: there can >> be no >> defined behaviour / interop >> >> 2) This core / common model is ALL metadata >> >> 3) ALL this metadata should be in the headers >> >> 4) ALL the non core stuff eg OVF payload, XML representation, >> should be in >> body >> >> Sam - is that right? >> >> a >> >> >> On Tue, Oct 20, 2009 at 11:39 AM, Sam Johnston <samj@samj.net> >> wrote: >> >>> On Tue, Oct 20, 2009 at 12:28 PM, Edmonds, AndrewX >>> <andrewx.edmonds@intel.com> wrote: >>> >>>> To that point I'd ask - If we're talking about meta-data that >>>> infers >>>> there >>>> is some data to accompany it. Where are the examples of this >>>> data? This >>>> would help in forming the full picture. >>>> >>> Here, from the core spec (using OVF as an example, but could be >>> anything). >>> Request is green, response metadata is yellow & response data/ >>> payload is >>> orange: >>> >>> >>>> GET /us-east/webapp/vm01 HTTP/1.1 >>>> User-Agent: occi-client/1.0 (linux) libcurl/7.19.4 OCCI/1.0 >>>> Host: cloud.example.com >>>> Accept: */* >>>> >>> < HTTP/1.1 200 OK >>> < Date: Sat, 10 Oct 2009 12:56:51 GMT >>> >>> < Content-Type: application/ovf >>> < Link: </us-east/webapp/vm01;start>; >>> >>> < rel="http://purl.org/occi/action/start"; >>> < title="Start" >>> >>> < Link: </us-east/webapp/build.pdf>; >>> < rel="related"; >>> >>> < title="Documentation"; >>> < type="application/pdf" >>> >>> < Category: compute; >>> < label="Compute Resource"; >>> >>> < scheme="http://purl.org/occi/kind/" >>> < Server: occi-server/1.0 (linux) OCCI/1.0 >>> >>> < Connection: close >>> < >>> < <?xml version="1.0" encoding="UTF-8"?> >>> >>> < <Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema- >>> instance" >>> < xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" >>> >>> < xmlns="http://schemas.dmtf.org/ovf/envelope/1" >>> < xml:lang="en-US"> >>> >>> ... >>> >>> >>> >>>> -----Original Message----- >>>> From: occi-wg-bounces@ogf.org [mailto:occi-wg- >>>> bounces@ogf.org] On >>>> Behalf >>>> Of Alexis Richardson >>>> Sent: 20 October 2009 09:49 >>>> To: Sam Johnston >>>> Cc: Tim Bray; occi-wg@ogf.org >>>> Subject: Re: [occi-wg] confusion about status of link / headers >>>> >>>> Sam, >>>> >>>> Please don't throw around words like "bunk" willy nilly. >>>> >>>> I have just had a look at >>>> http://www.rackspacecloud.com/cloud_hosting_products/servers/ >>>> api >>>> >>>> I can see some limited use of headers for metadata in a few >>>> places. >>>> That's never been controversial. But the understanding that I >>>> have of >>>> your proposal is to use headers wholesale for all metadata. >>>> Rackspace >>>> don't appear to do that, unless I have misunderstood their >>>> document. >>>> I also may have misunderstood your proposal. If nobody >>>> understands >>>> your proposal except you, then it will not be possible to gain >>>> consensus around it. >>>> >>>> Are you saying "OCCI metadata should be more like Rackspace >>>> metadata"? >>>> >>>> alexis >>>> >>>> >>>> On Tue, Oct 20, 2009 at 9:07 AM, Sam Johnston <samj@samj.net> >>>> wrote: >>>> >>>>> On Tue, Oct 20, 2009 at 9:55 AM, Alexis Richardson >>>>> <alexis.richardson@gmail.com> wrote: >>>>> >>>>>> What about compute clouds? >>>>>> >>>>> Rackspace Cloud API for a start: >>>>> >>>>> >>>>>> HTTP/1.1 204 No Content >>>>>> Date: Mon, 12 Nov 2007 15:32:21 GMT >>>>>> Server: Apache >>>>>> X-Server-Management-Url: >>>>>> https://servers.api.rackspacecloud.com/v1.0/35428 >>>>>> X-Storage-Url: https://storage.clouddrive.com/v1/CloudFS_9c83b-5ed4 >>>>>> X-CDN-Management-Url: >>>>>> https://cdn.clouddrive.com/v1/CloudFS_9c83b-5ed4 >>>>>> X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb >>>>>> Content-Length: 0 >>>>>> Content-Type: text/plain; charset=UTF-8 >>>>>> >>>>> Microsoft Azure uses headers for "attributes" as well, in the >>>>> same >>>>> way >>>>> as I >>>>> had originally proposed: >>>>> >>>>> x-ms-meta-name: value Returns a metadata value for the >>>>> container. >>>>> >>>>> However prefer that the header names are static/opaque rather >>>>> than >>>>> using >>>>> a >>>>> template and parsing them: >>>>> >>>>> >>>>>> Attribute: name=value >>>>>> >>>>> This to me is cleaner and more self-explanatory, plus it is >>>>> easily >>>>> collapsed >>>>> ala: >>>>> >>>>> >>>>>> Attribute: name1=value1, name2=value2, ... namen=valuen >>>>>> >>>>> Anyway suffice to say that claims this is somehow >>>>> "experimental" are >>>>> bunk. >>>>> >>>>> Sam >>>>> >>>>> >>>>>> On Tue, Oct 20, 2009 at 6:19 AM, Adrian Cole <adrian@jclouds.org >>>>>>> >>>>>> wrote: >>>>>> >>>>>>> Here are options for metadata used in some of the major >>>>>>> storage >>>>>>> clouds >>>>>>> FWIW: >>>>>>> >>>>>>> S3, Rackspace, EMC Atmos, Azure - Headers >>>>>>> Nirvanix - query params in, xml entity out >>>>>>> Mezeo - entity >>>>>>> >>>>>>> Of the ones using headers, S3, Rackspace and Azure use >>>>>>> prefix with >>>>>>> values stored as-us. Atmos joins all metadata together into >>>>>>> one >>>>>>> header, making parsing trivial (split /,/), but necessary. >>>>>>> >>>>>>> The most expensive option of the above is entity, where each >>>>>>> metadata >>>>>>> value is a separate GET. However, entities allow binary >>>>>>> metadata >>>>>>> and >>>>>>> zero restrictions on it, which may be useful. >>>>>>> >>>>>>> In jclouds, we time parsing of response values. A simple >>>>>>> XML doc >>>>>>> with >>>>>>> only several elements written in SAX takes a few ms to >>>>>>> parse. My >>>>>>> log >>>>>>> files are not precise enough to find the overhead in parsing >>>>>>> headers: >>>>>>> they always start and finish within the same millisecond. >>>>>>> >>>>>>> I hope this background helps, and also helps explain why I'm >>>>>>> vocal >>>>>>> on >>>>>>> such topics such as headers vs entities :) >>>>>>> >>>>>>> Cheers, >>>>>>> -Adrian >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 19, 2009 at 4:21 PM, Sam Johnston >>>>>>> <samj@samj.net> >>>>>>> wrote: >>>>>>> >>>>>>>> On Tue, Oct 20, 2009 at 12:57 AM, gary mazzaferro >>>>>>>> <garymazzaferro@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>>> The http header and key/value pairs need to parsed also, >>>>>>>>> there >>>>>>>>> is >>>>>>>>> no >>>>>>>>> free >>>>>>>>> ride here. >>>>>>>>> >>>>>>>> Every HTTP library I have ever used parses HTTP headers and >>>>>>>> puts >>>>>>>> them >>>>>>>> in a >>>>>>>> nice hash for you ready to consume. If we go for "Name: >>>>>>>> Value" >>>>>>>> then >>>>>>>> that's >>>>>>>> all there is to it. If we go for "Attribute: name=value" as >>>>>>>> is >>>>>>>> currently >>>>>>>> proposed (which is arguably cleaner, follows cookies' >>>>>>>> "prior art" >>>>>>>> and >>>>>>>> avoids >>>>>>>> Amazon's prefix hack) then you just have to split on '='. >>>>>>>> >>>>>>>> To illustrate how clean this is by example: >>>>>>>> >>>>>>>> >>>>>>>>> #!/usr/bin/python >>>>>>>>> import urllib2 >>>>>>>>> response = urllib2.urlopen('http://cloud.example.com/ >>>>>>>>> myvm') >>>>>>>>> representation = response.read() >>>>>>>>> metadata = response.info() >>>>>>>>> print metadata['occi-compute-cores'] >>>>>>>>> >>>>>>>> As soon as you start talking about payloads you have to >>>>>>>> fire up a >>>>>>>> parser >>>>>>>> (JSON/XML/Atom/etc.) or write your own (previous text >>>>>>>> rendering) >>>>>>>> which >>>>>>>> is >>>>>>>> significantly more work to do at both design and run times. >>>>>>>> Not >>>>>>>> to >>>>>>>> mention >>>>>>>> more work for us to do now and more scope for >>>>>>>> interoperability >>>>>>>> problems. >>>>>>>> >>>>>>>> Sam >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> occi-wg mailing list >>>>>>>> occi-wg@ogf.org >>>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> occi-wg mailing list >>>>>>> occi-wg@ogf.org >>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg >>>>>>> >>>>>>> >>>>> >>>> _______________________________________________ >>>> occi-wg mailing list >>>> occi-wg@ogf.org >>>> http://www.ogf.org/mailman/listinfo/occi-wg >>>> ------------------------------------------------------------- >>>> Intel Ireland Limited (Branch) >>>> Collinstown Industrial Park, Leixlip, County Kildare, Ireland >>>> Registered Number: E902934 >>>> >>>> This e-mail and any attachments may contain confidential >>>> material for >>>> the sole use of the intended recipient(s). Any review or >>>> distribution >>>> by others is strictly prohibited. If you are not the intended >>>> recipient, please contact the sender and delete all copies. >>>> >>> > _______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
_______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
I don't think we are ready to vote by any means. Sam's asked for a time out - it's after 7pm in Zurich now. So let's continue tomorrow. On Tue, Oct 20, 2009 at 5:44 PM, Adrian Cole <adrian@jclouds.org> wrote:
As we aren't introducing new information, I think we are at a point where a vote can take place. This issue isn't infinitely complex, and reasonable to model in choices. OCCI desires to be a community process, so let's see where we fall on this.
Do you agree? -Adrian
On Tue, Oct 20, 2009 at 9:39 AM, Alexis Richardson <alexis.richardson@gmail.com> wrote:
Sam,
On Tue, Oct 20, 2009 at 5:31 PM, Sam Johnston <samj@samj.net> wrote:
Alexis,
Sweeping, unjustified statements are intensely frustrating, not to mention unhelpful in the absence of a coherent alternative. If people choose not to disclose conflicts I happen to be aware of then you bet I'll disclose them for them - nothing personal about it.
I don't know what 'conflicts' you are referring to, or how these relate to the matter at hand. If by 'sweeping, unjustified' you refer to the word 'unprecedented' then I shall justify it by pointing out that - at least as I currently understand it - you are proposing use of headers to a *greater* extent than has been done before and to cover a *greater* set of use cases.
Cutting to the chase, both Microsoft and Amazon are using headers for metadata to the same extent as what has been proposed.
Really - to the "same" extent? That seems contrary to what has been asserted on this thread previously.
What premature optimisation? *Not* using XML? Do we really need to drag that carcass out again? Surely Adrian's performance tests put the last nail in that particular coffin.
Premature optimisation = making performance a critical design concern when we want an API.
I have noted Adrian's points as have others.
The question really is about whether we want to be an interface/ protocol like HTTP or some mongrel hybrid of that along with a set of XML, JSON and/or text formats for everything we ever intend to model (knowing that work has already been done by others).
Work that has been done by others and shown to work, is a good thing.
alexis
Sent from my iPhone
On 20/10/2009, at 18:10, Alexis Richardson <alexis.richardson@gmail.com> wrote:
Sam
Cut the personal stuff please.
Re 'unproven', I think the issues people are having are:
* The extent, scope and purpose of your application of headers to metadata is unprecedented - as I understand it. Nobody is saying it has not been done in other and restricted cases.
* Premature optimisation.
alexis
On Tue, Oct 20, 2009 at 4:44 PM, Sam Johnston <samj@samj.net> wrote:
Gary,
Please be specific with your objections so as I can have some hope of answering them... "several other emails" does not suffice as to the best of my knowledge there are no outstanding issues that I have not comprehensively and convincingly dealt with.
Your repeated assertions than HTTP headers are... what is it now... "unproven"... are demonstrably unfounded (refer to the various cloud APIs referred to by myself and Adrian, including both MS Azure and Amazon S3, as well as the history of the Internet in general) and given the frequency, bordering on FUD.
Regarding performance, OCCI should strive to be as efficient as possible as it primarily targets public cloud providers who operate at arbitrarily large "cloud" scale. The existing specification is *extremely* efficient and alternatives would at least want to be in the same neighbourhood. Efficiency may be an inconvenient factor for whatever your alternative is, but it's critical for many of our use cases.
Finally, you reject OVF without providing any alternative beyond building a replacement from scratch ourselves, after our second and final deadline has just sailed by no less. Have you considered the implementation cost of this? And in the same breath you accuse my proposal of being "unproven"? Give me a break...
Correct me if I'm wrong but the way I see it is that you want to revert the last 2-3 months of my work because you have an investment in an old version of the spec. With each day I spend on OCCI costing me well over €1,000 excuse me for being pissed about someone with such a blatantly obvious conflict attacking my work and refusing to provide a coherent argument, particularly with no "camera ready" alternative anywhere in sight.
Sent from my iPhone
On 20/10/2009, at 16:51, Gary Mazz <garymazzaferro@gmail.com> wrote:
I've understood Sam's proposal from the beginning. I have had issue with from the beginning.
Although I find it very interesting and valuable in terms of economy, I have issue with it as the OCCI's primary interface implementation for a few reasons which have been outlined in several other emails.
It appears that Sam has created a new technology, a lower cost RPC mechanism, which is unproven for both small and wide scale deployment. For interoperability, IMO, occi should focus on proven mainstream technologies as the primary interfaces. I am not suggesting, shelving or not supporting Sam's proposal, based on market acceptance, it may prove to be one of many occi interface implementations.
As for cost saving in interface technologies, I do not buy into the idea that the occi traffic will be so overwhelming that there is a requirement to maximally optimize http requests and responses. IMO, in comparison to images and consumer data occi traffic will be insignificant. I think we are worrying about spilled milk while the house is burning.
There has been some discussion on supporting technologies including OVF. Whether occi supports OVF has not been fully discussed, examined or any consensus reached. I believe adopting technologies like OVF will only lead to mapping issues between occi attributes and the external formats creating a long term management headache.
cheers, gary
Alexis Richardson wrote: > OK > > Is Sam's statement something that we can achieve consensus around? > > a > > > On Tue, Oct 20, 2009 at 12:47 PM, Sam Johnston <samj@samj.net> > wrote: > >> Alexis, >> >> Seems our posts crossed in the mail. I hope the last one clears >> things up >> but the fact that you're using OVF and XML/JSON interchangeably >> suggests >> that you [are possibly one of many who] don't grok the proposal. >> Worse, it >> seems you've got it back to front in thinking of the "core" as >> metadata, >> which is not at all what I had intended. >> >> The focus is on the resources and their various representations. >> The >> metadata (that is, data about data) essentially provides >> whatever is >> necessary to "glue" these resources together. It is basically >> using HTTP as >> the meta-model rather than an envelope format like Atom/SOAP or, >> worse, a >> meta-model that we have to create ourselves from scratch. >> >> So it's more like: >> >> 1) OCCI is compatible with ALL existing formats, eg OVF >> >> 2) Existing formats are used are used for ALL representations >> >> 3) ALL additional metadata should be in the headers >> >> 4) ALL of the core stuff should be in the body (that is, the >> metadata in the >> headers is just there to support the data and wire it up in useful >> ways) >> >> I think the confusion stems from your assuming that there is still >> XML/JSON/Atom/etc. representations. There aren't. HTTP headers >> provide a >> single, performant, interoperable metadata channel that has >> already been >> both standardised and extensively implemented. >> >> If you've followed me this far then you should have realised that >> every HTTP >> user agent (e.g. Perl's LWP, Python's urllib2, C's libcurl, etc.) >> is already >> a basic OCCI client out-of-the-box. The result is that 5-line >> scripts like >> demo.py are functional with no dependence on client libraries >> (that would >> absolutely be necessary if you were to start trying to define your >> own >> formats). >> >> Sam >> >> PS I'm sorry my "bunk" comment came across as referring to you >> personally - >> it was directed at the argument. >> >> On Tue, Oct 20, 2009 at 12:44 PM, Alexis Richardson >> <alexis.richardson@gmail.com> wrote: >> >>> I think what Sam is proposing is the following: >>> >>> 1) If OCCI can handle 'any' data representation, eg OVF, XML, >>> JSON, >>> then it needs some core / common model. Otherwise: there can >>> be no >>> defined behaviour / interop >>> >>> 2) This core / common model is ALL metadata >>> >>> 3) ALL this metadata should be in the headers >>> >>> 4) ALL the non core stuff eg OVF payload, XML representation, >>> should be in >>> body >>> >>> Sam - is that right? >>> >>> a >>> >>> >>> On Tue, Oct 20, 2009 at 11:39 AM, Sam Johnston <samj@samj.net> >>> wrote: >>> >>>> On Tue, Oct 20, 2009 at 12:28 PM, Edmonds, AndrewX >>>> <andrewx.edmonds@intel.com> wrote: >>>> >>>>> To that point I'd ask - If we're talking about meta-data that >>>>> infers >>>>> there >>>>> is some data to accompany it. Where are the examples of this >>>>> data? This >>>>> would help in forming the full picture. >>>>> >>>> Here, from the core spec (using OVF as an example, but could be >>>> anything). >>>> Request is green, response metadata is yellow & response data/ >>>> payload is >>>> orange: >>>> >>>> >>>>> GET /us-east/webapp/vm01 HTTP/1.1 >>>>> User-Agent: occi-client/1.0 (linux) libcurl/7.19.4 OCCI/1.0 >>>>> Host: cloud.example.com >>>>> Accept: */* >>>>> >>>> < HTTP/1.1 200 OK >>>> < Date: Sat, 10 Oct 2009 12:56:51 GMT >>>> >>>> < Content-Type: application/ovf >>>> < Link: </us-east/webapp/vm01;start>; >>>> >>>> < rel="http://purl.org/occi/action/start"; >>>> < title="Start" >>>> >>>> < Link: </us-east/webapp/build.pdf>; >>>> < rel="related"; >>>> >>>> < title="Documentation"; >>>> < type="application/pdf" >>>> >>>> < Category: compute; >>>> < label="Compute Resource"; >>>> >>>> < scheme="http://purl.org/occi/kind/" >>>> < Server: occi-server/1.0 (linux) OCCI/1.0 >>>> >>>> < Connection: close >>>> < >>>> < <?xml version="1.0" encoding="UTF-8"?> >>>> >>>> < <Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema- >>>> instance" >>>> < xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" >>>> >>>> < xmlns="http://schemas.dmtf.org/ovf/envelope/1" >>>> < xml:lang="en-US"> >>>> >>>> ... >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: occi-wg-bounces@ogf.org [mailto:occi-wg- >>>>> bounces@ogf.org] On >>>>> Behalf >>>>> Of Alexis Richardson >>>>> Sent: 20 October 2009 09:49 >>>>> To: Sam Johnston >>>>> Cc: Tim Bray; occi-wg@ogf.org >>>>> Subject: Re: [occi-wg] confusion about status of link / headers >>>>> >>>>> Sam, >>>>> >>>>> Please don't throw around words like "bunk" willy nilly. >>>>> >>>>> I have just had a look at >>>>> http://www.rackspacecloud.com/cloud_hosting_products/servers/ >>>>> api >>>>> >>>>> I can see some limited use of headers for metadata in a few >>>>> places. >>>>> That's never been controversial. But the understanding that I >>>>> have of >>>>> your proposal is to use headers wholesale for all metadata. >>>>> Rackspace >>>>> don't appear to do that, unless I have misunderstood their >>>>> document. >>>>> I also may have misunderstood your proposal. If nobody >>>>> understands >>>>> your proposal except you, then it will not be possible to gain >>>>> consensus around it. >>>>> >>>>> Are you saying "OCCI metadata should be more like Rackspace >>>>> metadata"? >>>>> >>>>> alexis >>>>> >>>>> >>>>> On Tue, Oct 20, 2009 at 9:07 AM, Sam Johnston <samj@samj.net> >>>>> wrote: >>>>> >>>>>> On Tue, Oct 20, 2009 at 9:55 AM, Alexis Richardson >>>>>> <alexis.richardson@gmail.com> wrote: >>>>>> >>>>>>> What about compute clouds? >>>>>>> >>>>>> Rackspace Cloud API for a start: >>>>>> >>>>>> >>>>>>> HTTP/1.1 204 No Content >>>>>>> Date: Mon, 12 Nov 2007 15:32:21 GMT >>>>>>> Server: Apache >>>>>>> X-Server-Management-Url: >>>>>>> https://servers.api.rackspacecloud.com/v1.0/35428 >>>>>>> X-Storage-Url: https://storage.clouddrive.com/v1/CloudFS_9c83b-5ed4 >>>>>>> X-CDN-Management-Url: >>>>>>> https://cdn.clouddrive.com/v1/CloudFS_9c83b-5ed4 >>>>>>> X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb >>>>>>> Content-Length: 0 >>>>>>> Content-Type: text/plain; charset=UTF-8 >>>>>>> >>>>>> Microsoft Azure uses headers for "attributes" as well, in the >>>>>> same >>>>>> way >>>>>> as I >>>>>> had originally proposed: >>>>>> >>>>>> x-ms-meta-name: value Returns a metadata value for the >>>>>> container. >>>>>> >>>>>> However prefer that the header names are static/opaque rather >>>>>> than >>>>>> using >>>>>> a >>>>>> template and parsing them: >>>>>> >>>>>> >>>>>>> Attribute: name=value >>>>>>> >>>>>> This to me is cleaner and more self-explanatory, plus it is >>>>>> easily >>>>>> collapsed >>>>>> ala: >>>>>> >>>>>> >>>>>>> Attribute: name1=value1, name2=value2, ... namen=valuen >>>>>>> >>>>>> Anyway suffice to say that claims this is somehow >>>>>> "experimental" are >>>>>> bunk. >>>>>> >>>>>> Sam >>>>>> >>>>>> >>>>>>> On Tue, Oct 20, 2009 at 6:19 AM, Adrian Cole <adrian@jclouds.org >>>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>>> Here are options for metadata used in some of the major >>>>>>>> storage >>>>>>>> clouds >>>>>>>> FWIW: >>>>>>>> >>>>>>>> S3, Rackspace, EMC Atmos, Azure - Headers >>>>>>>> Nirvanix - query params in, xml entity out >>>>>>>> Mezeo - entity >>>>>>>> >>>>>>>> Of the ones using headers, S3, Rackspace and Azure use >>>>>>>> prefix with >>>>>>>> values stored as-us. Atmos joins all metadata together into >>>>>>>> one >>>>>>>> header, making parsing trivial (split /,/), but necessary. >>>>>>>> >>>>>>>> The most expensive option of the above is entity, where each >>>>>>>> metadata >>>>>>>> value is a separate GET. However, entities allow binary >>>>>>>> metadata >>>>>>>> and >>>>>>>> zero restrictions on it, which may be useful. >>>>>>>> >>>>>>>> In jclouds, we time parsing of response values. A simple >>>>>>>> XML doc >>>>>>>> with >>>>>>>> only several elements written in SAX takes a few ms to >>>>>>>> parse. My >>>>>>>> log >>>>>>>> files are not precise enough to find the overhead in parsing >>>>>>>> headers: >>>>>>>> they always start and finish within the same millisecond. >>>>>>>> >>>>>>>> I hope this background helps, and also helps explain why I'm >>>>>>>> vocal >>>>>>>> on >>>>>>>> such topics such as headers vs entities :) >>>>>>>> >>>>>>>> Cheers, >>>>>>>> -Adrian >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Oct 19, 2009 at 4:21 PM, Sam Johnston >>>>>>>> <samj@samj.net> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Tue, Oct 20, 2009 at 12:57 AM, gary mazzaferro >>>>>>>>> <garymazzaferro@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> The http header and key/value pairs need to parsed also, >>>>>>>>>> there >>>>>>>>>> is >>>>>>>>>> no >>>>>>>>>> free >>>>>>>>>> ride here. >>>>>>>>>> >>>>>>>>> Every HTTP library I have ever used parses HTTP headers and >>>>>>>>> puts >>>>>>>>> them >>>>>>>>> in a >>>>>>>>> nice hash for you ready to consume. If we go for "Name: >>>>>>>>> Value" >>>>>>>>> then >>>>>>>>> that's >>>>>>>>> all there is to it. If we go for "Attribute: name=value" as >>>>>>>>> is >>>>>>>>> currently >>>>>>>>> proposed (which is arguably cleaner, follows cookies' >>>>>>>>> "prior art" >>>>>>>>> and >>>>>>>>> avoids >>>>>>>>> Amazon's prefix hack) then you just have to split on '='. >>>>>>>>> >>>>>>>>> To illustrate how clean this is by example: >>>>>>>>> >>>>>>>>> >>>>>>>>>> #!/usr/bin/python >>>>>>>>>> import urllib2 >>>>>>>>>> response = urllib2.urlopen('http://cloud.example.com/ >>>>>>>>>> myvm') >>>>>>>>>> representation = response.read() >>>>>>>>>> metadata = response.info() >>>>>>>>>> print metadata['occi-compute-cores'] >>>>>>>>>> >>>>>>>>> As soon as you start talking about payloads you have to >>>>>>>>> fire up a >>>>>>>>> parser >>>>>>>>> (JSON/XML/Atom/etc.) or write your own (previous text >>>>>>>>> rendering) >>>>>>>>> which >>>>>>>>> is >>>>>>>>> significantly more work to do at both design and run times. >>>>>>>>> Not >>>>>>>>> to >>>>>>>>> mention >>>>>>>>> more work for us to do now and more scope for >>>>>>>>> interoperability >>>>>>>>> problems. >>>>>>>>> >>>>>>>>> Sam >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> occi-wg mailing list >>>>>>>>> occi-wg@ogf.org >>>>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> occi-wg mailing list >>>>>>>> occi-wg@ogf.org >>>>>>>> http://www.ogf.org/mailman/listinfo/occi-wg >>>>>>>> >>>>>>>> >>>>>> >>>>> _______________________________________________ >>>>> occi-wg mailing list >>>>> occi-wg@ogf.org >>>>> http://www.ogf.org/mailman/listinfo/occi-wg >>>>> ------------------------------------------------------------- >>>>> Intel Ireland Limited (Branch) >>>>> Collinstown Industrial Park, Leixlip, County Kildare, Ireland >>>>> Registered Number: E902934 >>>>> >>>>> This e-mail and any attachments may contain confidential >>>>> material for >>>>> the sole use of the intended recipient(s). Any review or >>>>> distribution >>>>> by others is strictly prohibited. If you are not the intended >>>>> recipient, please contact the sender and delete all copies. >>>>> >>>> >> > _______________________________________________ > occi-wg mailing list > occi-wg@ogf.org > http://www.ogf.org/mailman/listinfo/occi-wg > >
_______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
participants (3)
-
Adrian Cole -
Alexis Richardson -
Sam Johnston