thought on interoperability vs integration
Hi all, Thanks for a thought-provoking week of emails on the OCCI-WG list. Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of the issues. One link that I found useful was this: http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we find the following statement: --- if Cloud technology is going to take off, there’ll have to be a competitive ecosystem; so that when you bet on a service provider, if the relationship doesn’t work out there’s a way to take your business to another provider with relatively little operational pain. Put another way: no lock-in. ... I got all excited about this back in January at that Cloud Interop session. Anant Jhingran, an IBM VIP, spoke up and said “Customers don’t want interoperability, they want integration.” ... “Bzzzzzzzzzt! Wrong!” I thought. But then I realized he was only half wrong; anyone going down this road needs integration and interoperability. --- What we have been discussing is "for customers". So it is about both of these things: * Interoperability ("interop") * Integration This made me realise that our OCCI discussions have correctly been about both issues. But, incorrectly, we have been commingling the two. For example a person will say "you need X for interop" and the reply will be "but you need Y" when in fact Y is for integration. And vice versa. This is a problem for us because it leads to confusion. But it's a symptom of a larger problem which is that interop and integration have opposite requirements. * Interop is about reducing the entropy in a general purpose system down to some level of behavioural invariance. This minimises the cost of meeting the defined behaviour correctly, and minimises the risk of different behaviours breaking interop. This is in turn about having a defined behaviour (in some format) and minimising its scope. * Integration is about minimising the frictions between (a) the general purpose system and (b) any specific purpose external system that may be coupled to it. It is also, often, about maximising the number of specific systems (or "features") that may be connected. Since we don't know ex ante what those are, this tends to maximise scope (eg "feature creep"). Too much specificity is the same as complexity. Because interop requires minimal scope, and integration pushes for maximal scope, they are in tension. They are BOTH about simplicity. Simplicity cannot be invoked on its own, as a reason for preferring interop over integration: * Interop is about simplicity of definition * Integration is about simplicity of end use BUT * Interop simplicity is ex ante * Integration simplicity is ex post We cannot predict all ex post issues, but we can try and make simple definitions ex ante. I argue below this means we have to have ONE definition. So let's look at interop first: It's really important that interop be based on defined behaviour. Or it cannot be verified. Lack of verifiability is almost always a symptom that something is opaque, complex and ambiguous, which will break interop in subtle ways that cannot be seen in advance (due to the opacity). This later leads to brittleness when interop and integration are attempted in practice; and that leads to expensive patching at all levels, ... which is one thing that WS-* exhibits. NOTE 1 on WS-* ---- IMO this was not accidental, and due to focussing on solving integration before solving interop. IIRC, the first WS-* was an Interop committee to fix minor vendor mismatches and versioning issues in prior WS protocols such as WSDL and SOAP. The formalisms of WSDL and SOAP were not precise enough to spot these issues upfront, so they were left until it was too late. Now let's look at the implications of having a definition of behaviour: You cannot define a system using multiple definitions. You have to have one definition, preferably in a formalism that admits of automatic conformance testing. In the case of data formats, this leads us to three possible choices: 1) We remove data formats from the interop profile. They are not part of interop. Data interop is excluded. Data payloads are opaque blobs. OR 2) We have one data format that can be defined unambiguously. OR 3) We have multiple formats and derive them from a single unambiguous definition using canonical, verifiable, automatable, mappings. This definition can either be in a new format which we invent (option 3a) or it can be in an existing format (option 3b). I am going to rule out option (3a) on grounds of time and due to the prevalence of existing candidates. Also, I think choice (1) is a complete cop-out -- I don't see how we can claim 'useful' interop without data interop. This leaves options (2) and (3b). BOTH of these options require one format. By occam's razor, option 2 is preferable on grounds of simplicity - option (3b) does not add anything except one mapping for every extra format that is based on the core definition. Such complexity MUST be deferred as long as possible, and MAY be pushed into integration use cases. The latter is preferable. Complexity MAY be added later if necessary and then, reluctantly. As a general observation if you have complexity (entropy) in a system, it is very hard to take out. It may permeate the system at all levels. Removal of the complexity is known as 'complete refactoring'. This is a bad outcome. Even worse is when refactoring cannot be done for commercial reasons, eg because the systems are in production. Then the complexity must be encapsulated and hidden. Attempts to wrap complex behaviours in simpler packaging usually fail. NOTE 2 on WS-* ---- This tried to do too much and ended up trying to wrap the complexity in simple packaging. This usually fails to work as we have seen. Another general observation: * Integration requires interop, ie. the use of a common general interop model; otherwise it is piecemeal and pointless. Example - the old 'integration brokers' that had N*N connections to maintain, and which got replaced by N connections to a common system. * But you can have interop without integration - it just means 'you have a smaller audience'. This is fine because you can always grow your audience by integrating more case with the interoperating core. It is easier to do that when the interoperating core is programmatically simple (as in low entropy, small code blocks, easy to test conformance to the definition). I would like to add some observations from the world of AMQP... AMQP-1) It is a protocol with one simple common data format - it gets that right. We leave it to integration products and services to support data formats at the edge (eg "ESBs"). OCCI should not be like an ESB - that is for products and services in the integration business. AMQP-2) That AMQP data format is not XML - see below for more thoughts on that. XML can be carried as an attached payload (just as it can be in the JSON case btw). AMQP-3) The 0-8 and 0-9 specs took 18 months of production use to show up the many very tiny interop bugs. We used those to create 0-9-1 which does have interop (we have tested this) and is only 40 pages long. This would not have been possible with a complex spec. It would not have been possible with multiple data formats. AMQP-4) The 0-10 spec was focussed on supporting a lot of use cases eg "integrate with multiple transport formats including SCTP and TCP and UDP" and adding lots of features at all levels (eg JMS, support for hardware, ..). That spec is really complicated and nearly 300 pages long. Some great new ideas are in it, but it's long and in my own opinion not supportive of two interoperating implementations. AMQP-5) All these painful lessons have taken the AMQP working group to a much happier place with AMQP 1.0 which tries to simplify everything by taking stuff out that is not needed for interop, plus refactoring (see above my comments on how removing entropy is hard) and clean-up. All of the above has taken time because we did not learn from WS-*. We did too much too fast and confused interop with integration. We are back on track now. Now to the issue of data formats. I have already argued that FOR INTEROP, there must be one definition. I argued that the best way to do this is via a suitable single format. We can support as many recommended ways as we like FOR INTEGRATION ... and they can be evolved over time. Here is my 2c on XML. XML-1) XML lets you do too much - because of namespaces and xsd it is in effect multiple formats. This is bad - we want a single, testable, constrained definition for data interop. XML-2) To enforce compliance with a simple XML definition, you need to have an extra definition of a well formed 'small' document. But creating a new definition of a data format in XML is equivalent to defining a new data format, the same as 'option 3a' above. But that option was ruled out above, on grounds of time constraint.. provided that a suitable alternative exists already (see JSON claims below). NOTE 3 on WS-* ---- IMHO a third reason why WS-* failed to be simple enough to get happy, quick and wide adoption, is that (XML-1) issue left too much data integration in the hands of customers, because vendors could not produce useful products when XML could be all things to all people. By not delivering on data integration, it became hard to deliver on the promise of integration at all. And recall that lowering integration costs was the selling point... So let's get INTEROP right and then do INTEGRATION. Interop requires: * Model - yes * Data format - one definition * Metadata - ? tbd As an aside - I think that GData is the nuts but it is also really an *integration techonology*. Now, here are some claims about JSON: JSON-1) Sun have demonstrated that it is plausible as a data model for a cloud API. That makes it plausible for us to ask: can it be used as the core definition for interop? JSON-2) It is lower entropy than XML. This makes it easy to test conformance. JSON-3) This means we do things the right way round -- simpler 'ex ante' choices make it EASIER for us to extend and enhance the protocol on a by need basis. For example carrying OVF payloads, or other integration points. Many will be done by users. So my recommendation is that the best way to avoid WS-* outcomes is A) Use one format for interop. Do interop first. Integration later. B) *At least for now* and *during the working draft* stage, to use JSON C) Other formats, for now, are "integration". But we do interop first. OK.... I have to run off. I wrote this down in one go and don't have time to review it. Apologies for any mistakes in the presentation. What do you all think? alexis
Alexis et al, a) Wiki Yep, time to have a Wiki or some form of threaded discussion, other than e-mail. Having said that, let me add to the entropy ;o) b) Interop vs Integration Agreed. Just as one more POV, interop tests the functionality while integration tests the ability to work with various systems. So interop first, with one format, is good. As you said, our first goal is to be functionally complete and have that canonical model, which when done, means, collectively we have understood the essential domain complexity (at least the current state of the art, as Tim pointed out a while ago) c) "Transport" Interesting pointer to AMQP. What about XMPP as the substrate ? (Of course, it could mean XML ;o)) d) CDATA The CDATA issue pointed by Sam is an interesting one. I was also thinking of the implications and had the question whether we plan to chuck OVF through our APIs? I think, we can take a position like html - i.e. use href mechanism as far as possible and base64 for the rest. e) OGF-25 Presentation Most probably we will not get to agreement on all the important points by OGF-25. We don't have to. At GGF-25, we present the state of our work - demark what we have agreed upon, and the alternatives we are working on. And until 1.0, things will be unstable anyway. Plus we can honestly represent the different camps and seek insights from the crowd. So I do not think GGF-25 is in critical path in anyway. f) Disagreeing with Sam I also agree that we should not en-masse disagree with Sam. But as Alexis' points out, the minimal scope vs. maximum coverage is the key. We can restrict or curtail the APIs to a minimum set - without any reduction in functionality; for example use hrefs instead of embedding. Constraints are wonderful thongs ! Cheers & have a nice day <k/> |-----Original Message----- |From: occi-wg-bounces@ogf.org [mailto:occi-wg-bounces@ogf.org] On Behalf |Of Alexis Richardson |Sent: Saturday, May 09, 2009 11:48 AM |To: occi-wg@ogf.org |Subject: [occi-wg] thought on interoperability vs. integration | |Hi all, | |Thanks for a thought-provoking week of emails on the OCCI-WG list. |Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of |the issues. | |One link that I found useful was this: |http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we |find the following statement: | |--- |if Cloud technology is going to take off, there'll have to be a |competitive ecosystem; so that when you bet on a service provider, if |the relationship doesn't work out there's a way to take your business |to another provider with relatively little operational pain. Put |another way: no lock-in. ... I got all excited about this back in |January at that Cloud Interop session. Anant Jhingran, an IBM VIP, |spoke up and said "Customers don't want interoperability, they want |integration." ... "Bzzzzzzzzzt! Wrong!" I thought. But then I |realized he was only half wrong; anyone going down this road needs |integration and interoperability. |--- | |What we have been discussing is "for customers". So it is about both |of these things: | |* Interoperability ("interop") |* Integration | |This made me realise that our OCCI discussions have correctly been |about both issues. But, incorrectly, we have been commingling the |two. For example a person will say "you need X for interop" and the |reply will be "but you need Y" when in fact Y is for integration. And |vice versa. | |This is a problem for us because it leads to confusion. But it's a |symptom of a larger problem which is that interop and integration have |opposite requirements. | |* Interop is about reducing the entropy in a general purpose system |down to some level of behavioural invariance. This minimises the cost |of meeting the defined behaviour correctly, and minimises the risk of |different behaviours breaking interop. This is in turn about having a |defined behaviour (in some format) and minimising its scope. | |* Integration is about minimising the frictions between (a) the |general purpose system and (b) any specific purpose external system |that may be coupled to it. It is also, often, about maximising the |number of specific systems (or "features") that may be connected. |Since we don't know ex ante what those are, this tends to maximise |scope (eg "feature creep"). Too much specificity is the same as |complexity. | |Because interop requires minimal scope, and integration pushes for |maximal scope, they are in tension. | |They are BOTH about simplicity. Simplicity cannot be invoked on its |own, as a reason for preferring interop over integration: | |* Interop is about simplicity of definition | |* Integration is about simplicity of end use | |BUT | |* Interop simplicity is ex ante | |* Integration simplicity is ex post | |We cannot predict all ex post issues, but we can try and make simple |definitions ex ante. I argue below this means we have to have ONE |definition. | |So let's look at interop first: | |It's really important that interop be based on defined behaviour. Or |it cannot be verified. Lack of verifiability is almost always a |symptom that something is opaque, complex and ambiguous, which will |break interop in subtle ways that cannot be seen in advance (due to |the opacity). This later leads to brittleness when interop and |integration are attempted in practice; and that leads to expensive |patching at all levels, ... which is one thing that WS-* exhibits. | |NOTE 1 on WS-* ---- IMO this was not accidental, and due to focussing |on solving integration before solving interop. IIRC, the first WS-* |was an Interop committee to fix minor vendor mismatches and versioning |issues in prior WS protocols such as WSDL and SOAP. The formalisms of |WSDL and SOAP were not precise enough to spot these issues upfront, so |they were left until it was too late. | |Now let's look at the implications of having a definition of behaviour: | |You cannot define a system using multiple definitions. You have to |have one definition, preferably in a formalism that admits of |automatic conformance testing. In the case of data formats, this |leads us to three possible choices: | |1) We remove data formats from the interop profile. They are not part |of interop. Data interop is excluded. Data payloads are opaque |blobs. | |OR |2) We have one data format that can be defined unambiguously. | |OR |3) We have multiple formats and derive them from a single unambiguous |definition using canonical, verifiable, automatable, mappings. This |definition can either be in a new format which we invent (option 3a) |or it can be in an existing format (option 3b). | |I am going to rule out option (3a) on grounds of time and due to the |prevalence of existing candidates. | |Also, I think choice (1) is a complete cop-out -- I don't see how we |can claim 'useful' interop without data interop. This leaves options |(2) and (3b). BOTH of these options require one format. | |By occam's razor, option 2 is preferable on grounds of simplicity - |option (3b) does not add anything except one mapping for every extra |format that is based on the core definition. Such complexity MUST be |deferred as long as possible, and MAY be pushed into integration use |cases. The latter is preferable. Complexity MAY be added later if |necessary and then, reluctantly. | |As a general observation if you have complexity (entropy) in a system, |it is very hard to take out. It may permeate the system at all |levels. Removal of the complexity is known as 'complete refactoring'. | This is a bad outcome. Even worse is when refactoring cannot be done |for commercial reasons, eg because the systems are in production. |Then the complexity must be encapsulated and hidden. Attempts to wrap |complex behaviours in simpler packaging usually fail. | |NOTE 2 on WS-* ---- This tried to do too much and ended up trying to |wrap the complexity in simple packaging. This usually fails to work |as we have seen. | |Another general observation: | |* Integration requires interop, ie. the use of a common general |interop model; otherwise it is piecemeal and pointless. Example - the |old 'integration brokers' that had N*N connections to maintain, and |which got replaced by N connections to a common system. | |* But you can have interop without integration - it just means 'you |have a smaller audience'. This is fine because you can always grow |your audience by integrating more case with the interoperating core. |It is easier to do that when the interoperating core is |programmatically simple (as in low entropy, small code blocks, easy to |test conformance to the definition). | |I would like to add some observations from the world of AMQP... | |AMQP-1) It is a protocol with one simple common data format - it gets |that right. We leave it to integration products and services to |support data formats at the edge (eg "ESBs"). OCCI should not be like |an ESB - that is for products and services in the integration |business. | |AMQP-2) That AMQP data format is not XML - see below for more thoughts |on that. XML can be carried as an attached payload (just as it can be |in the JSON case btw). | |AMQP-3) The 0-8 and 0-9 specs took 18 months of production use to show |up the many very tiny interop bugs. We used those to create 0-9-1 |which does have interop (we have tested this) and is only 40 pages |long. This would not have been possible with a complex spec. It |would not have been possible with multiple data formats. | |AMQP-4) The 0-10 spec was focussed on supporting a lot of use cases eg |"integrate with multiple transport formats including SCTP and TCP and |UDP" and adding lots of features at all levels (eg JMS, support for |hardware, ..). That spec is really complicated and nearly 300 pages |long. Some great new ideas are in it, but it's long and in my own |opinion not supportive of two interoperating implementations. | |AMQP-5) All these painful lessons have taken the AMQP working group to |a much happier place with AMQP 1.0 which tries to simplify everything |by taking stuff out that is not needed for interop, plus refactoring |(see above my comments on how removing entropy is hard) and clean-up. | |All of the above has taken time because we did not learn from WS-*. |We did too much too fast and confused interop with integration. We |are back on track now. | |Now to the issue of data formats. I have already argued that FOR |INTEROP, there must be one definition. I argued that the best way to |do this is via a suitable single format. We can support as many |recommended ways as we like FOR INTEGRATION ... and they can be |evolved over time. | |Here is my 2c on XML. | |XML-1) XML lets you do too much - because of namespaces and xsd it is |in effect multiple formats. This is bad - we want a single, testable, |constrained definition for data interop. | |XML-2) To enforce compliance with a simple XML definition, you need to |have an extra definition of a well formed 'small' document. But |creating a new definition of a data format in XML is equivalent to |defining a new data format, the same as 'option 3a' above. But that |option was ruled out above, on grounds of time constraint.. provided |that a suitable alternative exists already (see JSON claims below). | |NOTE 3 on WS-* ---- IMHO a third reason why WS-* failed to be simple |enough to get happy, quick and wide adoption, is that (XML-1) issue |left too much data integration in the hands of customers, because |vendors could not produce useful products when XML could be all things |to all people. By not delivering on data integration, it became hard |to deliver on the promise of integration at all. And recall that |lowering integration costs was the selling point... | |So let's get INTEROP right and then do INTEGRATION. | |Interop requires: | |* Model - yes |* Data format - one definition |* Metadata - ? tbd | |As an aside - I think that GData is the nuts but it is also really an |*integration techonology*. | |Now, here are some claims about JSON: | |JSON-1) Sun have demonstrated that it is plausible as a data model for |a cloud API. That makes it plausible for us to ask: can it be used as |the core definition for interop? | |JSON-2) It is lower entropy than XML. This makes it easy to test |conformance. | |JSON-3) This means we do things the right way round -- simpler 'ex |ante' choices make it EASIER for us to extend and enhance the protocol |on a by need basis. For example carrying OVF payloads, or other |integration points. Many will be done by users. | |So my recommendation is that the best way to avoid WS-* outcomes is | |A) Use one format for interop. Do interop first. Integration later. |B) *At least for now* and *during the working draft* stage, to use JSON |C) Other formats, for now, are "integration". But we do interop first. | |OK.... I have to run off. I wrote this down in one go and don't have |time to review it. Apologies for any mistakes in the presentation. | |What do you all think? | |alexis |_______________________________________________ |occi-wg mailing list |occi-wg@ogf.org |http://www.ogf.org/mailman/listinfo/occi-wg
Ok so since we're talking about trashing countless hours of my work and missing our deadlines for what appears to be (assuming perhaps a little too much good faith) nothing more than personal preference, and bearing in mind that I've been working on this since 4am this morning, on my birthday, which I've since had to cancel due to a nasty plane flu (thus the surly meter is already well and truly pegged), I'm of course going to have something to say - even if I lack the energy and inclination to give this the comprehensive retort that it deserves at this time. For a start I know nothing about AMQP so I have nothing to say about the AMQP assertions, but the XML and JSON assertions are patently false and bordering on FUD (XML having extensibility and validation features when JSON doesn't is a drawback? JSON has less "entropy" so it's easier to test conformance even in the absence of schemas? XML is to blame for WS-*? Sun Cloud API, currently a work in progress, has proven something? Seriously, WTF?). There is absolutely no question from anyone that XML is up to the task (Google have already proven it on a large scale) but significant concerns about whether JSON is - concerns that remain unanswered despite my repeated requests for answers and concerns that this proposal seeks to postpone or avoid entirely, thus making them someone else's problem. While most APIs currently support multiple formats (even GData offer a JSON option for convenience if not interop) I am yet to find a single successful JSON-only API so this all sounds a bit too experimental for my liking. Furthermore, given that things that are trivial with XML (transforms, embedding, signatures, encryption, etc.) are currently impossible with JSON, this is the mother of all externalisations (that is, leaving our users up the creek without a paddle when we are in the best position to solve the problem by making practical choices). It's also a disappointing attempt to foist personal preferences on others when we can trivially cater for the entire spectrum of needs using XML & XSLT (work which I have alread done and demonstrated I might add - the absence of any volunteers to reinvent the wheel given I'm not about to is also disconcerting). Other difficult questions like the serialisation and serious performance problems of using hrefs (1 request for XML vs 1+n for JSON) have also been conveniently ignored thus far and the best answer I've got about the absence of embedding functionality in JSON is that as XML isn't perfect we're better off with nothing?!? Most critically though, by telling SNIA, Google and pretty much everyone else in the cloud standard space that they can go fornicate themselves, and in doing so missing the [only] opportunity to standardise a well proven, risk-free, extensible protocol on which others can safely build, I think we may well be witnessing the inception of CC-* (that is, WS-*, JSON edition). More specifically, if we slam the door on the possibility of others building on our efforts then they will have little choice but to blaze their own trail, forcing clients to support a myriad standards (OCCI for trivial management tasks, OVF for compute resources, SNIA's stuff for storage and possibly networking, etc.). I for one have negative interest in supporting something that I believe could do more damage than good and knowing what happened previously, learning from our mistakes and practicing the precautionary principle<http://en.wikipedia.org/wiki/Precautionary_principle> would be well advised. I'm currently OCCI's number one fan but if this attempt to pwn the process is successful then those who witnessed the CCIF goat rodeo well know that I have no problems saying it how I see it. What I would suggest is that those who want to see JSON adopted get to work on cutting code to prove the concept as I have done already for XML - loose consensus and running code remember. Sam (who can't help but to observe that the key JSON junkies all have significant business investment in same) On Sat, May 9, 2009 at 10:25 PM, Krishna Sankar (ksankar) <ksankar@cisco.com
wrote:
Alexis et al, a) Wiki Yep, time to have a Wiki or some form of threaded discussion, other than e-mail. Having said that, let me add to the entropy ;o)
b) Interop vs Integration Agreed. Just as one more POV, interop tests the functionality while integration tests the ability to work with various systems. So interop first, with one format, is good. As you said, our first goal is to be functionally complete and have that canonical model, which when done, means, collectively we have understood the essential domain complexity (at least the current state of the art, as Tim pointed out a while ago)
c) "Transport" Interesting pointer to AMQP. What about XMPP as the substrate ? (Of course, it could mean XML ;o))
d) CDATA The CDATA issue pointed by Sam is an interesting one. I was also thinking of the implications and had the question whether we plan to chuck OVF through our APIs? I think, we can take a position like html - i.e. use href mechanism as far as possible and base64 for the rest.
e) OGF-25 Presentation Most probably we will not get to agreement on all the important points by OGF-25. We don't have to. At GGF-25, we present the state of our work - demark what we have agreed upon, and the alternatives we are working on. And until 1.0, things will be unstable anyway. Plus we can honestly represent the different camps and seek insights from the crowd. So I do not think GGF-25 is in critical path in anyway.
f) Disagreeing with Sam I also agree that we should not en-masse disagree with Sam. But as Alexis' points out, the minimal scope vs. maximum coverage is the key. We can restrict or curtail the APIs to a minimum set - without any reduction in functionality; for example use hrefs instead of embedding. Constraints are wonderful thongs !
Cheers & have a nice day <k/>
|-----Original Message----- |From: occi-wg-bounces@ogf.org [mailto:occi-wg-bounces@ogf.org] On Behalf |Of Alexis Richardson |Sent: Saturday, May 09, 2009 11:48 AM |To: occi-wg@ogf.org |Subject: [occi-wg] thought on interoperability vs. integration | |Hi all, | |Thanks for a thought-provoking week of emails on the OCCI-WG list. |Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of |the issues. | |One link that I found useful was this: |http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we |find the following statement: | |--- |if Cloud technology is going to take off, there'll have to be a |competitive ecosystem; so that when you bet on a service provider, if |the relationship doesn't work out there's a way to take your business |to another provider with relatively little operational pain. Put |another way: no lock-in. ... I got all excited about this back in |January at that Cloud Interop session. Anant Jhingran, an IBM VIP, |spoke up and said "Customers don't want interoperability, they want |integration." ... "Bzzzzzzzzzt! Wrong!" I thought. But then I |realized he was only half wrong; anyone going down this road needs |integration and interoperability. |--- | |What we have been discussing is "for customers". So it is about both |of these things: | |* Interoperability ("interop") |* Integration | |This made me realise that our OCCI discussions have correctly been |about both issues. But, incorrectly, we have been commingling the |two. For example a person will say "you need X for interop" and the |reply will be "but you need Y" when in fact Y is for integration. And |vice versa. | |This is a problem for us because it leads to confusion. But it's a |symptom of a larger problem which is that interop and integration have |opposite requirements. | |* Interop is about reducing the entropy in a general purpose system |down to some level of behavioural invariance. This minimises the cost |of meeting the defined behaviour correctly, and minimises the risk of |different behaviours breaking interop. This is in turn about having a |defined behaviour (in some format) and minimising its scope. | |* Integration is about minimising the frictions between (a) the |general purpose system and (b) any specific purpose external system |that may be coupled to it. It is also, often, about maximising the |number of specific systems (or "features") that may be connected. |Since we don't know ex ante what those are, this tends to maximise |scope (eg "feature creep"). Too much specificity is the same as |complexity. | |Because interop requires minimal scope, and integration pushes for |maximal scope, they are in tension. | |They are BOTH about simplicity. Simplicity cannot be invoked on its |own, as a reason for preferring interop over integration: | |* Interop is about simplicity of definition | |* Integration is about simplicity of end use | |BUT | |* Interop simplicity is ex ante | |* Integration simplicity is ex post | |We cannot predict all ex post issues, but we can try and make simple |definitions ex ante. I argue below this means we have to have ONE |definition. | |So let's look at interop first: | |It's really important that interop be based on defined behaviour. Or |it cannot be verified. Lack of verifiability is almost always a |symptom that something is opaque, complex and ambiguous, which will |break interop in subtle ways that cannot be seen in advance (due to |the opacity). This later leads to brittleness when interop and |integration are attempted in practice; and that leads to expensive |patching at all levels, ... which is one thing that WS-* exhibits. | |NOTE 1 on WS-* ---- IMO this was not accidental, and due to focussing |on solving integration before solving interop. IIRC, the first WS-* |was an Interop committee to fix minor vendor mismatches and versioning |issues in prior WS protocols such as WSDL and SOAP. The formalisms of |WSDL and SOAP were not precise enough to spot these issues upfront, so |they were left until it was too late. | |Now let's look at the implications of having a definition of behaviour: | |You cannot define a system using multiple definitions. You have to |have one definition, preferably in a formalism that admits of |automatic conformance testing. In the case of data formats, this |leads us to three possible choices: | |1) We remove data formats from the interop profile. They are not part |of interop. Data interop is excluded. Data payloads are opaque |blobs. | |OR |2) We have one data format that can be defined unambiguously. | |OR |3) We have multiple formats and derive them from a single unambiguous |definition using canonical, verifiable, automatable, mappings. This |definition can either be in a new format which we invent (option 3a) |or it can be in an existing format (option 3b). | |I am going to rule out option (3a) on grounds of time and due to the |prevalence of existing candidates. | |Also, I think choice (1) is a complete cop-out -- I don't see how we |can claim 'useful' interop without data interop. This leaves options |(2) and (3b). BOTH of these options require one format. | |By occam's razor, option 2 is preferable on grounds of simplicity - |option (3b) does not add anything except one mapping for every extra |format that is based on the core definition. Such complexity MUST be |deferred as long as possible, and MAY be pushed into integration use |cases. The latter is preferable. Complexity MAY be added later if |necessary and then, reluctantly. | |As a general observation if you have complexity (entropy) in a system, |it is very hard to take out. It may permeate the system at all |levels. Removal of the complexity is known as 'complete refactoring'. | This is a bad outcome. Even worse is when refactoring cannot be done |for commercial reasons, eg because the systems are in production. |Then the complexity must be encapsulated and hidden. Attempts to wrap |complex behaviours in simpler packaging usually fail. | |NOTE 2 on WS-* ---- This tried to do too much and ended up trying to |wrap the complexity in simple packaging. This usually fails to work |as we have seen. | |Another general observation: | |* Integration requires interop, ie. the use of a common general |interop model; otherwise it is piecemeal and pointless. Example - the |old 'integration brokers' that had N*N connections to maintain, and |which got replaced by N connections to a common system. | |* But you can have interop without integration - it just means 'you |have a smaller audience'. This is fine because you can always grow |your audience by integrating more case with the interoperating core. |It is easier to do that when the interoperating core is |programmatically simple (as in low entropy, small code blocks, easy to |test conformance to the definition). | |I would like to add some observations from the world of AMQP... | |AMQP-1) It is a protocol with one simple common data format - it gets |that right. We leave it to integration products and services to |support data formats at the edge (eg "ESBs"). OCCI should not be like |an ESB - that is for products and services in the integration |business. | |AMQP-2) That AMQP data format is not XML - see below for more thoughts |on that. XML can be carried as an attached payload (just as it can be |in the JSON case btw). | |AMQP-3) The 0-8 and 0-9 specs took 18 months of production use to show |up the many very tiny interop bugs. We used those to create 0-9-1 |which does have interop (we have tested this) and is only 40 pages |long. This would not have been possible with a complex spec. It |would not have been possible with multiple data formats. | |AMQP-4) The 0-10 spec was focussed on supporting a lot of use cases eg |"integrate with multiple transport formats including SCTP and TCP and |UDP" and adding lots of features at all levels (eg JMS, support for |hardware, ..). That spec is really complicated and nearly 300 pages |long. Some great new ideas are in it, but it's long and in my own |opinion not supportive of two interoperating implementations. | |AMQP-5) All these painful lessons have taken the AMQP working group to |a much happier place with AMQP 1.0 which tries to simplify everything |by taking stuff out that is not needed for interop, plus refactoring |(see above my comments on how removing entropy is hard) and clean-up. | |All of the above has taken time because we did not learn from WS-*. |We did too much too fast and confused interop with integration. We |are back on track now. | |Now to the issue of data formats. I have already argued that FOR |INTEROP, there must be one definition. I argued that the best way to |do this is via a suitable single format. We can support as many |recommended ways as we like FOR INTEGRATION ... and they can be |evolved over time. | |Here is my 2c on XML. | |XML-1) XML lets you do too much - because of namespaces and xsd it is |in effect multiple formats. This is bad - we want a single, testable, |constrained definition for data interop. | |XML-2) To enforce compliance with a simple XML definition, you need to |have an extra definition of a well formed 'small' document. But |creating a new definition of a data format in XML is equivalent to |defining a new data format, the same as 'option 3a' above. But that |option was ruled out above, on grounds of time constraint.. provided |that a suitable alternative exists already (see JSON claims below). | |NOTE 3 on WS-* ---- IMHO a third reason why WS-* failed to be simple |enough to get happy, quick and wide adoption, is that (XML-1) issue |left too much data integration in the hands of customers, because |vendors could not produce useful products when XML could be all things |to all people. By not delivering on data integration, it became hard |to deliver on the promise of integration at all. And recall that |lowering integration costs was the selling point... | |So let's get INTEROP right and then do INTEGRATION. | |Interop requires: | |* Model - yes |* Data format - one definition |* Metadata - ? tbd | |As an aside - I think that GData is the nuts but it is also really an |*integration techonology*. | |Now, here are some claims about JSON: | |JSON-1) Sun have demonstrated that it is plausible as a data model for |a cloud API. That makes it plausible for us to ask: can it be used as |the core definition for interop? | |JSON-2) It is lower entropy than XML. This makes it easy to test |conformance. | |JSON-3) This means we do things the right way round -- simpler 'ex |ante' choices make it EASIER for us to extend and enhance the protocol |on a by need basis. For example carrying OVF payloads, or other |integration points. Many will be done by users. | |So my recommendation is that the best way to avoid WS-* outcomes is | |A) Use one format for interop. Do interop first. Integration later. |B) *At least for now* and *during the working draft* stage, to use JSON |C) Other formats, for now, are "integration". But we do interop first. | |OK.... I have to run off. I wrote this down in one go and don't have |time to review it. Apologies for any mistakes in the presentation. | |What do you all think? | |alexis |_______________________________________________ |occi-wg mailing list |occi-wg@ogf.org |http://www.ogf.org/mailman/listinfo/occi-wg _______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
Sam, Briefly - I shall reply tomorrow: - Please enjoy the weekend ffs! - I admit my argument for JSON vs XML is weaker than the argument I made for one format, which I think is more robust, but please do criticise either argument as you see fit - Do note that my argument for JSON was to use it as the common format *for now* - We are not telling anyone to f off, we are trying to produce something good that people can integrate with; I am not trying to trash anything you or anyone else has done - we are however trying to reach some sort of consensus which we all agree we have not reached yet - I agree, and have stated very clearly, that JSON folks should lay out some work and in fact Richard et al. already volunteered to do so. It would be churlish to not give them that opportunity. alexis On Sat, May 9, 2009 at 11:03 PM, Sam Johnston <samj@samj.net> wrote:
Ok so since we're talking about trashing countless hours of my work and missing our deadlines for what appears to be (assuming perhaps a little too much good faith) nothing more than personal preference, and bearing in mind that I've been working on this since 4am this morning, on my birthday, which I've since had to cancel due to a nasty plane flu (thus the surly meter is already well and truly pegged), I'm of course going to have something to say - even if I lack the energy and inclination to give this the comprehensive retort that it deserves at this time. For a start I know nothing about AMQP so I have nothing to say about the AMQP assertions, but the XML and JSON assertions are patently false and bordering on FUD (XML having extensibility and validation features when JSON doesn't is a drawback? JSON has less "entropy" so it's easier to test conformance even in the absence of schemas? XML is to blame for WS-*? Sun Cloud API, currently a work in progress, has proven something? Seriously, WTF?). There is absolutely no question from anyone that XML is up to the task (Google have already proven it on a large scale) but significant concerns about whether JSON is - concerns that remain unanswered despite my repeated requests for answers and concerns that this proposal seeks to postpone or avoid entirely, thus making them someone else's problem. While most APIs currently support multiple formats (even GData offer a JSON option for convenience if not interop) I am yet to find a single successful JSON-only API so this all sounds a bit too experimental for my liking. Furthermore, given that things that are trivial with XML (transforms, embedding, signatures, encryption, etc.) are currently impossible with JSON, this is the mother of all externalisations (that is, leaving our users up the creek without a paddle when we are in the best position to solve the problem by making practical choices). It's also a disappointing attempt to foist personal preferences on others when we can trivially cater for the entire spectrum of needs using XML & XSLT (work which I have alread done and demonstrated I might add - the absence of any volunteers to reinvent the wheel given I'm not about to is also disconcerting). Other difficult questions like the serialisation and serious performance problems of using hrefs (1 request for XML vs 1+n for JSON) have also been conveniently ignored thus far and the best answer I've got about the absence of embedding functionality in JSON is that as XML isn't perfect we're better off with nothing?!? Most critically though, by telling SNIA, Google and pretty much everyone else in the cloud standard space that they can go fornicate themselves, and in doing so missing the [only] opportunity to standardise a well proven, risk-free, extensible protocol on which others can safely build, I think we may well be witnessing the inception of CC-* (that is, WS-*, JSON edition). More specifically, if we slam the door on the possibility of others building on our efforts then they will have little choice but to blaze their own trail, forcing clients to support a myriad standards (OCCI for trivial management tasks, OVF for compute resources, SNIA's stuff for storage and possibly networking, etc.). I for one have negative interest in supporting something that I believe could do more damage than good and knowing what happened previously, learning from our mistakes and practicing the precautionary principle would be well advised. I'm currently OCCI's number one fan but if this attempt to pwn the process is successful then those who witnessed the CCIF goat rodeo well know that I have no problems saying it how I see it. What I would suggest is that those who want to see JSON adopted get to work on cutting code to prove the concept as I have done already for XML - loose consensus and running code remember. Sam (who can't help but to observe that the key JSON junkies all have significant business investment in same) On Sat, May 9, 2009 at 10:25 PM, Krishna Sankar (ksankar) <ksankar@cisco.com> wrote:
Alexis et al, a) Wiki Yep, time to have a Wiki or some form of threaded discussion, other than e-mail. Having said that, let me add to the entropy ;o)
b) Interop vs Integration Agreed. Just as one more POV, interop tests the functionality while integration tests the ability to work with various systems. So interop first, with one format, is good. As you said, our first goal is to be functionally complete and have that canonical model, which when done, means, collectively we have understood the essential domain complexity (at least the current state of the art, as Tim pointed out a while ago)
c) "Transport" Interesting pointer to AMQP. What about XMPP as the substrate ? (Of course, it could mean XML ;o))
d) CDATA The CDATA issue pointed by Sam is an interesting one. I was also thinking of the implications and had the question whether we plan to chuck OVF through our APIs? I think, we can take a position like html - i.e. use href mechanism as far as possible and base64 for the rest.
e) OGF-25 Presentation Most probably we will not get to agreement on all the important points by OGF-25. We don't have to. At GGF-25, we present the state of our work - demark what we have agreed upon, and the alternatives we are working on. And until 1.0, things will be unstable anyway. Plus we can honestly represent the different camps and seek insights from the crowd. So I do not think GGF-25 is in critical path in anyway.
f) Disagreeing with Sam I also agree that we should not en-masse disagree with Sam. But as Alexis' points out, the minimal scope vs. maximum coverage is the key. We can restrict or curtail the APIs to a minimum set - without any reduction in functionality; for example use hrefs instead of embedding. Constraints are wonderful thongs !
Cheers & have a nice day <k/>
|-----Original Message----- |From: occi-wg-bounces@ogf.org [mailto:occi-wg-bounces@ogf.org] On Behalf |Of Alexis Richardson |Sent: Saturday, May 09, 2009 11:48 AM |To: occi-wg@ogf.org |Subject: [occi-wg] thought on interoperability vs. integration | |Hi all, | |Thanks for a thought-provoking week of emails on the OCCI-WG list. |Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of |the issues. | |One link that I found useful was this: |http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we |find the following statement: | |--- |if Cloud technology is going to take off, there'll have to be a |competitive ecosystem; so that when you bet on a service provider, if |the relationship doesn't work out there's a way to take your business |to another provider with relatively little operational pain. Put |another way: no lock-in. ... I got all excited about this back in |January at that Cloud Interop session. Anant Jhingran, an IBM VIP, |spoke up and said "Customers don't want interoperability, they want |integration." ... "Bzzzzzzzzzt! Wrong!" I thought. But then I |realized he was only half wrong; anyone going down this road needs |integration and interoperability. |--- | |What we have been discussing is "for customers". So it is about both |of these things: | |* Interoperability ("interop") |* Integration | |This made me realise that our OCCI discussions have correctly been |about both issues. But, incorrectly, we have been commingling the |two. For example a person will say "you need X for interop" and the |reply will be "but you need Y" when in fact Y is for integration. And |vice versa. | |This is a problem for us because it leads to confusion. But it's a |symptom of a larger problem which is that interop and integration have |opposite requirements. | |* Interop is about reducing the entropy in a general purpose system |down to some level of behavioural invariance. This minimises the cost |of meeting the defined behaviour correctly, and minimises the risk of |different behaviours breaking interop. This is in turn about having a |defined behaviour (in some format) and minimising its scope. | |* Integration is about minimising the frictions between (a) the |general purpose system and (b) any specific purpose external system |that may be coupled to it. It is also, often, about maximising the |number of specific systems (or "features") that may be connected. |Since we don't know ex ante what those are, this tends to maximise |scope (eg "feature creep"). Too much specificity is the same as |complexity. | |Because interop requires minimal scope, and integration pushes for |maximal scope, they are in tension. | |They are BOTH about simplicity. Simplicity cannot be invoked on its |own, as a reason for preferring interop over integration: | |* Interop is about simplicity of definition | |* Integration is about simplicity of end use | |BUT | |* Interop simplicity is ex ante | |* Integration simplicity is ex post | |We cannot predict all ex post issues, but we can try and make simple |definitions ex ante. I argue below this means we have to have ONE |definition. | |So let's look at interop first: | |It's really important that interop be based on defined behaviour. Or |it cannot be verified. Lack of verifiability is almost always a |symptom that something is opaque, complex and ambiguous, which will |break interop in subtle ways that cannot be seen in advance (due to |the opacity). This later leads to brittleness when interop and |integration are attempted in practice; and that leads to expensive |patching at all levels, ... which is one thing that WS-* exhibits. | |NOTE 1 on WS-* ---- IMO this was not accidental, and due to focussing |on solving integration before solving interop. IIRC, the first WS-* |was an Interop committee to fix minor vendor mismatches and versioning |issues in prior WS protocols such as WSDL and SOAP. The formalisms of |WSDL and SOAP were not precise enough to spot these issues upfront, so |they were left until it was too late. | |Now let's look at the implications of having a definition of behaviour: | |You cannot define a system using multiple definitions. You have to |have one definition, preferably in a formalism that admits of |automatic conformance testing. In the case of data formats, this |leads us to three possible choices: | |1) We remove data formats from the interop profile. They are not part |of interop. Data interop is excluded. Data payloads are opaque |blobs. | |OR |2) We have one data format that can be defined unambiguously. | |OR |3) We have multiple formats and derive them from a single unambiguous |definition using canonical, verifiable, automatable, mappings. This |definition can either be in a new format which we invent (option 3a) |or it can be in an existing format (option 3b). | |I am going to rule out option (3a) on grounds of time and due to the |prevalence of existing candidates. | |Also, I think choice (1) is a complete cop-out -- I don't see how we |can claim 'useful' interop without data interop. This leaves options |(2) and (3b). BOTH of these options require one format. | |By occam's razor, option 2 is preferable on grounds of simplicity - |option (3b) does not add anything except one mapping for every extra |format that is based on the core definition. Such complexity MUST be |deferred as long as possible, and MAY be pushed into integration use |cases. The latter is preferable. Complexity MAY be added later if |necessary and then, reluctantly. | |As a general observation if you have complexity (entropy) in a system, |it is very hard to take out. It may permeate the system at all |levels. Removal of the complexity is known as 'complete refactoring'. | This is a bad outcome. Even worse is when refactoring cannot be done |for commercial reasons, eg because the systems are in production. |Then the complexity must be encapsulated and hidden. Attempts to wrap |complex behaviours in simpler packaging usually fail. | |NOTE 2 on WS-* ---- This tried to do too much and ended up trying to |wrap the complexity in simple packaging. This usually fails to work |as we have seen. | |Another general observation: | |* Integration requires interop, ie. the use of a common general |interop model; otherwise it is piecemeal and pointless. Example - the |old 'integration brokers' that had N*N connections to maintain, and |which got replaced by N connections to a common system. | |* But you can have interop without integration - it just means 'you |have a smaller audience'. This is fine because you can always grow |your audience by integrating more case with the interoperating core. |It is easier to do that when the interoperating core is |programmatically simple (as in low entropy, small code blocks, easy to |test conformance to the definition). | |I would like to add some observations from the world of AMQP... | |AMQP-1) It is a protocol with one simple common data format - it gets |that right. We leave it to integration products and services to |support data formats at the edge (eg "ESBs"). OCCI should not be like |an ESB - that is for products and services in the integration |business. | |AMQP-2) That AMQP data format is not XML - see below for more thoughts |on that. XML can be carried as an attached payload (just as it can be |in the JSON case btw). | |AMQP-3) The 0-8 and 0-9 specs took 18 months of production use to show |up the many very tiny interop bugs. We used those to create 0-9-1 |which does have interop (we have tested this) and is only 40 pages |long. This would not have been possible with a complex spec. It |would not have been possible with multiple data formats. | |AMQP-4) The 0-10 spec was focussed on supporting a lot of use cases eg |"integrate with multiple transport formats including SCTP and TCP and |UDP" and adding lots of features at all levels (eg JMS, support for |hardware, ..). That spec is really complicated and nearly 300 pages |long. Some great new ideas are in it, but it's long and in my own |opinion not supportive of two interoperating implementations. | |AMQP-5) All these painful lessons have taken the AMQP working group to |a much happier place with AMQP 1.0 which tries to simplify everything |by taking stuff out that is not needed for interop, plus refactoring |(see above my comments on how removing entropy is hard) and clean-up. | |All of the above has taken time because we did not learn from WS-*. |We did too much too fast and confused interop with integration. We |are back on track now. | |Now to the issue of data formats. I have already argued that FOR |INTEROP, there must be one definition. I argued that the best way to |do this is via a suitable single format. We can support as many |recommended ways as we like FOR INTEGRATION ... and they can be |evolved over time. | |Here is my 2c on XML. | |XML-1) XML lets you do too much - because of namespaces and xsd it is |in effect multiple formats. This is bad - we want a single, testable, |constrained definition for data interop. | |XML-2) To enforce compliance with a simple XML definition, you need to |have an extra definition of a well formed 'small' document. But |creating a new definition of a data format in XML is equivalent to |defining a new data format, the same as 'option 3a' above. But that |option was ruled out above, on grounds of time constraint.. provided |that a suitable alternative exists already (see JSON claims below). | |NOTE 3 on WS-* ---- IMHO a third reason why WS-* failed to be simple |enough to get happy, quick and wide adoption, is that (XML-1) issue |left too much data integration in the hands of customers, because |vendors could not produce useful products when XML could be all things |to all people. By not delivering on data integration, it became hard |to deliver on the promise of integration at all. And recall that |lowering integration costs was the selling point... | |So let's get INTEROP right and then do INTEGRATION. | |Interop requires: | |* Model - yes |* Data format - one definition |* Metadata - ? tbd | |As an aside - I think that GData is the nuts but it is also really an |*integration techonology*. | |Now, here are some claims about JSON: | |JSON-1) Sun have demonstrated that it is plausible as a data model for |a cloud API. That makes it plausible for us to ask: can it be used as |the core definition for interop? | |JSON-2) It is lower entropy than XML. This makes it easy to test |conformance. | |JSON-3) This means we do things the right way round -- simpler 'ex |ante' choices make it EASIER for us to extend and enhance the protocol |on a by need basis. For example carrying OVF payloads, or other |integration points. Many will be done by users. | |So my recommendation is that the best way to avoid WS-* outcomes is | |A) Use one format for interop. Do interop first. Integration later. |B) *At least for now* and *during the working draft* stage, to use JSON |C) Other formats, for now, are "integration". But we do interop first. | |OK.... I have to run off. I wrote this down in one go and don't have |time to review it. Apologies for any mistakes in the presentation. | |What do you all think? | |alexis |_______________________________________________ |occi-wg mailing list |occi-wg@ogf.org |http://www.ogf.org/mailman/listinfo/occi-wg _______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
Surely it makes a good deal more sense to use a mature format we know works *for now* and migrate to something more elegant later, once we have everyone (Google, SNIA, etc.) on board/working together and once glaring gaps like schemas and transforms have been filled? As I said before, the current JSON output is the worst case (it's a generic XML->JSON transform) but an Atom->JSON transform would be much better. That way you get your JSON, Chris (and the sysadmins) get their text, my enterprise guys get their XML and we get a coherent web interface for free. Sam on iPhone On 5/10/09, Alexis Richardson <alexis.richardson@gmail.com> wrote:
Sam,
Briefly - I shall reply tomorrow:
- Please enjoy the weekend ffs! - I admit my argument for JSON vs XML is weaker than the argument I made for one format, which I think is more robust, but please do criticise either argument as you see fit - Do note that my argument for JSON was to use it as the common format *for now* - We are not telling anyone to f off, we are trying to produce something good that people can integrate with; I am not trying to trash anything you or anyone else has done - we are however trying to reach some sort of consensus which we all agree we have not reached yet - I agree, and have stated very clearly, that JSON folks should lay out some work and in fact Richard et al. already volunteered to do so. It would be churlish to not give them that opportunity.
alexis
On Sat, May 9, 2009 at 11:03 PM, Sam Johnston <samj@samj.net> wrote:
Ok so since we're talking about trashing countless hours of my work and missing our deadlines for what appears to be (assuming perhaps a little too much good faith) nothing more than personal preference, and bearing in mind that I've been working on this since 4am this morning, on my birthday, which I've since had to cancel due to a nasty plane flu (thus the surly meter is already well and truly pegged), I'm of course going to have something to say - even if I lack the energy and inclination to give this the comprehensive retort that it deserves at this time. For a start I know nothing about AMQP so I have nothing to say about the AMQP assertions, but the XML and JSON assertions are patently false and bordering on FUD (XML having extensibility and validation features when JSON doesn't is a drawback? JSON has less "entropy" so it's easier to test conformance even in the absence of schemas? XML is to blame for WS-*? Sun Cloud API, currently a work in progress, has proven something? Seriously, WTF?). There is absolutely no question from anyone that XML is up to the task (Google have already proven it on a large scale) but significant concerns about whether JSON is - concerns that remain unanswered despite my repeated requests for answers and concerns that this proposal seeks to postpone or avoid entirely, thus making them someone else's problem. While most APIs currently support multiple formats (even GData offer a JSON option for convenience if not interop) I am yet to find a single successful JSON-only API so this all sounds a bit too experimental for my liking. Furthermore, given that things that are trivial with XML (transforms, embedding, signatures, encryption, etc.) are currently impossible with JSON, this is the mother of all externalisations (that is, leaving our users up the creek without a paddle when we are in the best position to solve the problem by making practical choices). It's also a disappointing attempt to foist personal preferences on others when we can trivially cater for the entire spectrum of needs using XML & XSLT (work which I have alread done and demonstrated I might add - the absence of any volunteers to reinvent the wheel given I'm not about to is also disconcerting). Other difficult questions like the serialisation and serious performance problems of using hrefs (1 request for XML vs 1+n for JSON) have also been conveniently ignored thus far and the best answer I've got about the absence of embedding functionality in JSON is that as XML isn't perfect we're better off with nothing?!? Most critically though, by telling SNIA, Google and pretty much everyone else in the cloud standard space that they can go fornicate themselves, and in doing so missing the [only] opportunity to standardise a well proven, risk-free, extensible protocol on which others can safely build, I think we may well be witnessing the inception of CC-* (that is, WS-*, JSON edition). More specifically, if we slam the door on the possibility of others building on our efforts then they will have little choice but to blaze their own trail, forcing clients to support a myriad standards (OCCI for trivial management tasks, OVF for compute resources, SNIA's stuff for storage and possibly networking, etc.). I for one have negative interest in supporting something that I believe could do more damage than good and knowing what happened previously, learning from our mistakes and practicing the precautionary principle would be well advised. I'm currently OCCI's number one fan but if this attempt to pwn the process is successful then those who witnessed the CCIF goat rodeo well know that I have no problems saying it how I see it. What I would suggest is that those who want to see JSON adopted get to work on cutting code to prove the concept as I have done already for XML - loose consensus and running code remember. Sam (who can't help but to observe that the key JSON junkies all have significant business investment in same) On Sat, May 9, 2009 at 10:25 PM, Krishna Sankar (ksankar) <ksankar@cisco.com> wrote:
Alexis et al, a) Wiki Yep, time to have a Wiki or some form of threaded discussion, other than e-mail. Having said that, let me add to the entropy ;o)
b) Interop vs Integration Agreed. Just as one more POV, interop tests the functionality while integration tests the ability to work with various systems. So interop first, with one format, is good. As you said, our first goal is to be functionally complete and have that canonical model, which when done, means, collectively we have understood the essential domain complexity (at least the current state of the art, as Tim pointed out a while ago)
c) "Transport" Interesting pointer to AMQP. What about XMPP as the substrate ? (Of course, it could mean XML ;o))
d) CDATA The CDATA issue pointed by Sam is an interesting one. I was also thinking of the implications and had the question whether we plan to chuck OVF through our APIs? I think, we can take a position like html - i.e. use href mechanism as far as possible and base64 for the rest.
e) OGF-25 Presentation Most probably we will not get to agreement on all the important points by OGF-25. We don't have to. At GGF-25, we present the state of our work - demark what we have agreed upon, and the alternatives we are working on. And until 1.0, things will be unstable anyway. Plus we can honestly represent the different camps and seek insights from the crowd. So I do not think GGF-25 is in critical path in anyway.
f) Disagreeing with Sam I also agree that we should not en-masse disagree with Sam. But as Alexis' points out, the minimal scope vs. maximum coverage is the key. We can restrict or curtail the APIs to a minimum set - without any reduction in functionality; for example use hrefs instead of embedding. Constraints are wonderful thongs !
Cheers & have a nice day <k/>
|-----Original Message----- |From: occi-wg-bounces@ogf.org [mailto:occi-wg-bounces@ogf.org] On Behalf |Of Alexis Richardson |Sent: Saturday, May 09, 2009 11:48 AM |To: occi-wg@ogf.org |Subject: [occi-wg] thought on interoperability vs. integration | |Hi all, | |Thanks for a thought-provoking week of emails on the OCCI-WG list. |Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of |the issues. | |One link that I found useful was this: |http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we |find the following statement: | |--- |if Cloud technology is going to take off, there'll have to be a |competitive ecosystem; so that when you bet on a service provider, if |the relationship doesn't work out there's a way to take your business |to another provider with relatively little operational pain. Put |another way: no lock-in. ... I got all excited about this back in |January at that Cloud Interop session. Anant Jhingran, an IBM VIP, |spoke up and said "Customers don't want interoperability, they want |integration." ... "Bzzzzzzzzzt! Wrong!" I thought. But then I |realized he was only half wrong; anyone going down this road needs |integration and interoperability. |--- | |What we have been discussing is "for customers". So it is about both |of these things: | |* Interoperability ("interop") |* Integration | |This made me realise that our OCCI discussions have correctly been |about both issues. But, incorrectly, we have been commingling the |two. For example a person will say "you need X for interop" and the |reply will be "but you need Y" when in fact Y is for integration. And |vice versa. | |This is a problem for us because it leads to confusion. But it's a |symptom of a larger problem which is that interop and integration have |opposite requirements. | |* Interop is about reducing the entropy in a general purpose system |down to some level of behavioural invariance. This minimises the cost |of meeting the defined behaviour correctly, and minimises the risk of |different behaviours breaking interop. This is in turn about having a |defined behaviour (in some format) and minimising its scope. | |* Integration is about minimising the frictions between (a) the |general purpose system and (b) any specific purpose external system |that may be coupled to it. It is also, often, about maximising the |number of specific systems (or "features") that may be connected. |Since we don't know ex ante what those are, this tends to maximise |scope (eg "feature creep"). Too much specificity is the same as |complexity. | |Because interop requires minimal scope, and integration pushes for |maximal scope, they are in tension. | |They are BOTH about simplicity. Simplicity cannot be invoked on its |own, as a reason for preferring interop over integration: | |* Interop is about simplicity of definition | |* Integration is about simplicity of end use | |BUT | |* Interop simplicity is ex ante | |* Integration simplicity is ex post | |We cannot predict all ex post issues, but we can try and make simple |definitions ex ante. I argue below this means we have to have ONE |definition. | |So let's look at interop first: | |It's really important that interop be based on defined behaviour. Or |it cannot be verified. Lack of verifiability is almost always a |symptom that something is opaque, complex and ambiguous, which will |break interop in subtle ways that cannot be seen in advance (due to |the opacity). This later leads to brittleness when interop and |integration are attempted in practice; and that leads to expensive |patching at all levels, ... which is one thing that WS-* exhibits. | |NOTE 1 on WS-* ---- IMO this was not accidental, and due to focussing |on solving integration before solving interop. IIRC, the first WS-* |was an Interop committee to fix minor vendor mismatches and versioning |issues in prior WS protocols such as WSDL and SOAP. The formalisms of |WSDL and SOAP were not precise enough to spot these issues upfront, so |they were left until it was too late. | |Now let's look at the implications of having a definition of behaviour: | |You cannot define a system using multiple definitions. You have to |have one definition, preferably in a formalism that admits of |automatic conformance testing. In the case of data formats, this |leads us to three possible choices: | |1) We remove data formats from the interop profile. They are not part |of interop. Data interop is excluded. Data payloads are opaque |blobs. | |OR |2) We have one data format that can be defined unambiguously. | |OR |3) We have multiple formats and derive them from a single unambiguous |definition using canonical, verifiable, automatable, mappings. This |definition can either be in a new format which we invent (option 3a) |or it can be in an existing format (option 3b). | |I am going to rule out option (3a) on grounds of time and due to the |prevalence of existing candidates. | |Also, I think choice (1) is a complete cop-out -- I don't see how we |can claim 'useful' interop without data interop. This leaves options |(2) and (3b). BOTH of these options require one format. | |By occam's razor, option 2 is preferable on grounds of simplicity - |option (3b) does not add anything except one mapping for every extra |format that is based on the core definition. Such complexity MUST be |deferred as long as possible, and MAY be pushed into integration use |cases. The latter is preferable. Complexity MAY be added later if |necessary and then, reluctantly. | |As a general observation if you have complexity (entropy) in a system, |it is very hard to take out. It may permeate the system at all |levels. Removal of the complexity is known as 'complete refactoring'. | This is a bad outcome. Even worse is when refactoring cannot be done |for commercial reasons, eg because the systems are in production. |Then the complexity must be encapsulated and hidden. Attempts to wrap |complex behaviours in simpler packaging usually fail. | |NOTE 2 on WS-* ---- This tried to do too much and ended up trying to |wrap the complexity in simple packaging. This usually fails to work |as we have seen. | |Another general observation: | |* Integration requires interop, ie. the use of a common general |interop model; otherwise it is piecemeal and pointless. Example - the |old 'integration brokers' that had N*N connections to maintain, and |which got replaced by N connections to a common system. | |* But you can have interop without integration - it just means 'you |have a smaller audience'. This is fine because you can always grow |your audience by integrating more case with the interoperating core. |It is easier to do that when the interoperating core is |programmatically simple (as in low entropy, small code blocks, easy to |test conformance to the definition). | |I would like to add some observations from the world of AMQP... | |AMQP-1) It is a protocol with one simple common data format - it gets |that right. We leave it to integration products and services to |support data formats at the edge (eg "ESBs"). OCCI should not be like |an ESB - that is for products and services in the integration |business. | |AMQP-2) That AMQP data format is not XML - see below for more thoughts |on that. XML can be carried as an attached payload (just as it can be |in the JSON case btw). | |AMQP-3) The 0-8 and 0-9 specs took 18 months of production use to show |up the many very tiny interop bugs. We used those to create 0-9-1 |which does have interop (we have tested this) and is only 40 pages |long. This would not have been possible with a complex spec. It |would not have been possible with multiple data formats. | |AMQP-4) The 0-10 spec was focussed on supporting a lot of use cases eg |"integrate with multiple transport formats including SCTP and TCP and |UDP" and adding lots of features at all levels (eg JMS, support for |hardware, ..). That spec is really complicated and nearly 300 pages |long. Some great new ideas are in it, but it's long and in my own |opinion not supportive of two interoperating implementations. | |AMQP-5) All these painful lessons have taken the AMQP working group to |a much happier place with AMQP 1.0 which tries to simplify everything |by taking stuff out that is not needed for interop, plus refactoring |(see above my comments on how removing entropy is hard) and clean-up. | |All of the above has taken time because we did not learn from WS-*. |We did too much too fast and confused interop with integration. We |are back on track now. | |Now to the issue of data formats. I have already argued that FOR |INTEROP, there must be one definition. I argued that the best way to |do this is via a suitable single format. We can support as many |recommended ways as we like FOR INTEGRATION ... and they can be |evolved over time. | |Here is my 2c on XML. | |XML-1) XML lets you do too much - because of namespaces and xsd it is |in effect multiple formats. This is bad - we want a single, testable, |constrained definition for data interop. | |XML-2) To enforce compliance with a simple XML definition, you need to |have an extra definition of a well formed 'small' document. But |creating a new definition of a data format in XML is equivalent to |defining a new data format, the same as 'option 3a' above. But that |option was ruled out above, on grounds of time constraint.. provided |that a suitable alternative exists already (see JSON claims below). | |NOTE 3 on WS-* ---- IMHO a third reason why WS-* failed to be simple |enough to get happy, quick and wide adoption, is that (XML-1) issue |left too much data integration in the hands of customers, because |vendors could not produce useful products when XML could be all things |to all people. By not delivering on data integration, it became hard |to deliver on the promise of integration at all. And recall that |lowering integration costs was the selling point... | |So let's get INTEROP right and then do INTEGRATION. | |Interop requires: | |* Model - yes |* Data format - one definition |* Metadata - ? tbd | |As an aside - I think that GData is the nuts but it is also really an |*integration techonology*. | |Now, here are some claims about JSON: | |JSON-1) Sun have demonstrated that it is plausible as a data model for |a cloud API. That makes it plausible for us to ask: can it be used as |the core definition for interop? | |JSON-2) It is lower entropy than XML. This makes it easy to test |conformance. | |JSON-3) This means we do things the right way round -- simpler 'ex |ante' choices make it EASIER for us to extend and enhance the protocol |on a by need basis. For example carrying OVF payloads, or other |integration points. Many will be done by users. | |So my recommendation is that the best way to avoid WS-* outcomes is | |A) Use one format for interop. Do interop first. Integration later. |B) *At least for now* and *during the working draft* stage, to use JSON |C) Other formats, for now, are "integration". But we do interop first. | |OK.... I have to run off. I wrote this down in one go and don't have |time to review it. Apologies for any mistakes in the presentation. | |What do you all think? | |alexis |_______________________________________________ |occi-wg mailing list |occi-wg@ogf.org |http://www.ogf.org/mailman/listinfo/occi-wg _______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
BTW, Happy Birthday.... Some general observations on this thread.... I don't believe anyone is asking to disregard your work, at least I hope not. But what is happening here is a pitfall of starting with an light model and an API without clear definition of requirements based on use models. Lets touch on some of the issues, at least from my perspective, sorry for the terseness and directness, I'm very busy... 1) There are several public strategic use models emerging for clouds, occurring out of control of this group. Many are still evolving 2) Success of this effort, in part, is dependent on the ability to adopt and integrate other well know and deployed standards and technologies. 3) An api is a REPRESENTATION of actions and data with implicate or explicate organization into groups of interfaces. Actions, data and organization will defined by the use models, or at least provide enough that an api can be extrapolated from them. 4) Any particular api can be RENDERED based on the native need of the api consumer. We call them mappings. For example, you would not directly use a PHP or javascript api in C/C++ program without translating it into a form (a mapping) that the C/C++ compilers can accept and the executing code can reach. C/C++ doesn't care about CDATA 5) Many standards, specifications and technologies, deployed for decades, are defined and implemented in XML. 6) Entropy can occur though the lack of precision. ie: clearly defined bounds. Ambiguity equals entropy Summarizing: The api represents a model, the rendered format of the api will depend on the use model (including programming languages, development tools chains and execution environments). We, meaning the industry and the world, don't have well defined or detailed use model today. My Opinions Simple is better, but who cares if you can only interoperate with yourself. A JSON only solution may alienate most of the world. Although all the consultants will have a field day with mapping, until the mapping have to interoperate. The occi effort may and probably will support more than one api and rendering. The occi NEEDS to develop profile based use cases. The profiles can be used to set requirements and set priorities for mappings. Trading off definition in the front of a processes for expedience always results in exponential expenditures as you continue down the path of deployments. JSON lacks definition and increases ambiguity. I'm not sure what interop validation will look like, but the bounds of JSON data may require significantly more work and definition before validation can be achieved. BTW, I have a requirement for binary defined data... -gary Sam Johnston wrote:
Ok so since we're talking about trashing countless hours of my work and missing our deadlines for what appears to be (assuming perhaps a little too much good faith) nothing more than personal preference, and bearing in mind that I've been working on this since 4am this morning, on my birthday, which I've since had to cancel due to a nasty plane flu (thus the surly meter is already well and truly pegged), I'm of course going to have something to say - even if I lack the energy and inclination to give this the comprehensive retort that it deserves at this time.
For a start I know nothing about AMQP so I have nothing to say about the AMQP assertions, but the XML and JSON assertions are patently false and bordering on FUD (XML having extensibility and validation features when JSON doesn't is a drawback? JSON has less "entropy" so it's easier to test conformance even in the absence of schemas? XML is to blame for WS-*? Sun Cloud API, currently a work in progress, has proven something? Seriously, WTF?).
There is absolutely no question from anyone that XML is up to the task (Google have already proven it on a large scale) but significant concerns about whether JSON is - concerns that remain unanswered despite my repeated requests for answers and concerns that this proposal seeks to postpone or avoid entirely, thus making them someone else's problem. While most APIs currently support multiple formats (even GData offer a JSON option for convenience if not interop) I am yet to find a single successful JSON-only API so this all sounds a bit too experimental for my liking.
Furthermore, given that things that are trivial with XML (transforms, embedding, signatures, encryption, etc.) are currently impossible with JSON, this is the mother of all externalisations (that is, leaving our users up the creek without a paddle when we are in the best position to solve the problem by making practical choices). It's also a disappointing attempt to foist personal preferences on others when we can trivially cater for the entire spectrum of needs using XML & XSLT (work which I have alread done and demonstrated I might add - the absence of any volunteers to reinvent the wheel given I'm not about to is also disconcerting). Other difficult questions like the serialisation and serious performance problems of using hrefs (1 request for XML vs 1+n for JSON) have also been conveniently ignored thus far and the best answer I've got about the absence of embedding functionality in JSON is that as XML isn't perfect we're better off with nothing?!?
Most critically though, by telling SNIA, Google and pretty much everyone else in the cloud standard space that they can go fornicate themselves, and in doing so missing the [only] opportunity to standardise a well proven, risk-free, extensible protocol on which others can safely build, I think we may well be witnessing the inception of CC-* (that is, WS-*, JSON edition). More specifically, if we slam the door on the possibility of others building on our efforts then they will have little choice but to blaze their own trail, forcing clients to support a myriad standards (OCCI for trivial management tasks, OVF for compute resources, SNIA's stuff for storage and possibly networking, etc.).
I for one have negative interest in supporting something that I believe could do more damage than good and knowing what happened previously, learning from our mistakes and practicing the precautionary principle <http://en.wikipedia.org/wiki/Precautionary_principle> would be well advised. I'm currently OCCI's number one fan but if this attempt to pwn the process is successful then those who witnessed the CCIF goat rodeo well know that I have no problems saying it how I see it.
What I would suggest is that those who want to see JSON adopted get to work on cutting code to prove the concept as I have done already for XML - loose consensus and running code remember.
Sam (who can't help but to observe that the key JSON junkies all have significant business investment in same)
On Sat, May 9, 2009 at 10:25 PM, Krishna Sankar (ksankar) <ksankar@cisco.com <mailto:ksankar@cisco.com>> wrote:
Alexis et al, a) Wiki Yep, time to have a Wiki or some form of threaded discussion, other than e-mail. Having said that, let me add to the entropy ;o)
b) Interop vs Integration Agreed. Just as one more POV, interop tests the functionality while integration tests the ability to work with various systems. So interop first, with one format, is good. As you said, our first goal is to be functionally complete and have that canonical model, which when done, means, collectively we have understood the essential domain complexity (at least the current state of the art, as Tim pointed out a while ago)
c) "Transport" Interesting pointer to AMQP. What about XMPP as the substrate ? (Of course, it could mean XML ;o))
d) CDATA The CDATA issue pointed by Sam is an interesting one. I was also thinking of the implications and had the question whether we plan to chuck OVF through our APIs? I think, we can take a position like html - i.e. use href mechanism as far as possible and base64 for the rest.
e) OGF-25 Presentation Most probably we will not get to agreement on all the important points by OGF-25. We don't have to. At GGF-25, we present the state of our work - demark what we have agreed upon, and the alternatives we are working on. And until 1.0, things will be unstable anyway. Plus we can honestly represent the different camps and seek insights from the crowd. So I do not think GGF-25 is in critical path in anyway.
f) Disagreeing with Sam I also agree that we should not en-masse disagree with Sam. But as Alexis' points out, the minimal scope vs. maximum coverage is the key. We can restrict or curtail the APIs to a minimum set - without any reduction in functionality; for example use hrefs instead of embedding. Constraints are wonderful thongs !
Cheers & have a nice day <k/>
|-----Original Message----- |From: occi-wg-bounces@ogf.org <mailto:occi-wg-bounces@ogf.org> [mailto:occi-wg-bounces@ogf.org <mailto:occi-wg-bounces@ogf.org>] On Behalf |Of Alexis Richardson |Sent: Saturday, May 09, 2009 11:48 AM |To: occi-wg@ogf.org <mailto:occi-wg@ogf.org> |Subject: [occi-wg] thought on interoperability vs. integration | |Hi all, | |Thanks for a thought-provoking week of emails on the OCCI-WG list. |Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of |the issues. | |One link that I found useful was this: |http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we |find the following statement: | |--- |if Cloud technology is going to take off, there'll have to be a |competitive ecosystem; so that when you bet on a service provider, if |the relationship doesn't work out there's a way to take your business |to another provider with relatively little operational pain. Put |another way: no lock-in. ... I got all excited about this back in |January at that Cloud Interop session. Anant Jhingran, an IBM VIP, |spoke up and said "Customers don't want interoperability, they want |integration." ... "Bzzzzzzzzzt! Wrong!" I thought. But then I |realized he was only half wrong; anyone going down this road needs |integration and interoperability. |--- | |What we have been discussing is "for customers". So it is about both |of these things: | |* Interoperability ("interop") |* Integration | |This made me realise that our OCCI discussions have correctly been |about both issues. But, incorrectly, we have been commingling the |two. For example a person will say "you need X for interop" and the |reply will be "but you need Y" when in fact Y is for integration. And |vice versa. | |This is a problem for us because it leads to confusion. But it's a |symptom of a larger problem which is that interop and integration have |opposite requirements. | |* Interop is about reducing the entropy in a general purpose system |down to some level of behavioural invariance. This minimises the cost |of meeting the defined behaviour correctly, and minimises the risk of |different behaviours breaking interop. This is in turn about having a |defined behaviour (in some format) and minimising its scope. | |* Integration is about minimising the frictions between (a) the |general purpose system and (b) any specific purpose external system |that may be coupled to it. It is also, often, about maximising the |number of specific systems (or "features") that may be connected. |Since we don't know ex ante what those are, this tends to maximise |scope (eg "feature creep"). Too much specificity is the same as |complexity. | |Because interop requires minimal scope, and integration pushes for |maximal scope, they are in tension. | |They are BOTH about simplicity. Simplicity cannot be invoked on its |own, as a reason for preferring interop over integration: | |* Interop is about simplicity of definition | |* Integration is about simplicity of end use | |BUT | |* Interop simplicity is ex ante | |* Integration simplicity is ex post | |We cannot predict all ex post issues, but we can try and make simple |definitions ex ante. I argue below this means we have to have ONE |definition. | |So let's look at interop first: | |It's really important that interop be based on defined behaviour. Or |it cannot be verified. Lack of verifiability is almost always a |symptom that something is opaque, complex and ambiguous, which will |break interop in subtle ways that cannot be seen in advance (due to |the opacity). This later leads to brittleness when interop and |integration are attempted in practice; and that leads to expensive |patching at all levels, ... which is one thing that WS-* exhibits. | |NOTE 1 on WS-* ---- IMO this was not accidental, and due to focussing |on solving integration before solving interop. IIRC, the first WS-* |was an Interop committee to fix minor vendor mismatches and versioning |issues in prior WS protocols such as WSDL and SOAP. The formalisms of |WSDL and SOAP were not precise enough to spot these issues upfront, so |they were left until it was too late. | |Now let's look at the implications of having a definition of behaviour: | |You cannot define a system using multiple definitions. You have to |have one definition, preferably in a formalism that admits of |automatic conformance testing. In the case of data formats, this |leads us to three possible choices: | |1) We remove data formats from the interop profile. They are not part |of interop. Data interop is excluded. Data payloads are opaque |blobs. | |OR |2) We have one data format that can be defined unambiguously. | |OR |3) We have multiple formats and derive them from a single unambiguous |definition using canonical, verifiable, automatable, mappings. This |definition can either be in a new format which we invent (option 3a) |or it can be in an existing format (option 3b). | |I am going to rule out option (3a) on grounds of time and due to the |prevalence of existing candidates. | |Also, I think choice (1) is a complete cop-out -- I don't see how we |can claim 'useful' interop without data interop. This leaves options |(2) and (3b). BOTH of these options require one format. | |By occam's razor, option 2 is preferable on grounds of simplicity - |option (3b) does not add anything except one mapping for every extra |format that is based on the core definition. Such complexity MUST be |deferred as long as possible, and MAY be pushed into integration use |cases. The latter is preferable. Complexity MAY be added later if |necessary and then, reluctantly. | |As a general observation if you have complexity (entropy) in a system, |it is very hard to take out. It may permeate the system at all |levels. Removal of the complexity is known as 'complete refactoring'. | This is a bad outcome. Even worse is when refactoring cannot be done |for commercial reasons, eg because the systems are in production. |Then the complexity must be encapsulated and hidden. Attempts to wrap |complex behaviours in simpler packaging usually fail. | |NOTE 2 on WS-* ---- This tried to do too much and ended up trying to |wrap the complexity in simple packaging. This usually fails to work |as we have seen. | |Another general observation: | |* Integration requires interop, ie. the use of a common general |interop model; otherwise it is piecemeal and pointless. Example - the |old 'integration brokers' that had N*N connections to maintain, and |which got replaced by N connections to a common system. | |* But you can have interop without integration - it just means 'you |have a smaller audience'. This is fine because you can always grow |your audience by integrating more case with the interoperating core. |It is easier to do that when the interoperating core is |programmatically simple (as in low entropy, small code blocks, easy to |test conformance to the definition). | |I would like to add some observations from the world of AMQP... | |AMQP-1) It is a protocol with one simple common data format - it gets |that right. We leave it to integration products and services to |support data formats at the edge (eg "ESBs"). OCCI should not be like |an ESB - that is for products and services in the integration |business. | |AMQP-2) That AMQP data format is not XML - see below for more thoughts |on that. XML can be carried as an attached payload (just as it can be |in the JSON case btw). | |AMQP-3) The 0-8 and 0-9 specs took 18 months of production use to show |up the many very tiny interop bugs. We used those to create 0-9-1 |which does have interop (we have tested this) and is only 40 pages |long. This would not have been possible with a complex spec. It |would not have been possible with multiple data formats. | |AMQP-4) The 0-10 spec was focussed on supporting a lot of use cases eg |"integrate with multiple transport formats including SCTP and TCP and |UDP" and adding lots of features at all levels (eg JMS, support for |hardware, ..). That spec is really complicated and nearly 300 pages |long. Some great new ideas are in it, but it's long and in my own |opinion not supportive of two interoperating implementations. | |AMQP-5) All these painful lessons have taken the AMQP working group to |a much happier place with AMQP 1.0 which tries to simplify everything |by taking stuff out that is not needed for interop, plus refactoring |(see above my comments on how removing entropy is hard) and clean-up. | |All of the above has taken time because we did not learn from WS-*. |We did too much too fast and confused interop with integration. We |are back on track now. | |Now to the issue of data formats. I have already argued that FOR |INTEROP, there must be one definition. I argued that the best way to |do this is via a suitable single format. We can support as many |recommended ways as we like FOR INTEGRATION ... and they can be |evolved over time. | |Here is my 2c on XML. | |XML-1) XML lets you do too much - because of namespaces and xsd it is |in effect multiple formats. This is bad - we want a single, testable, |constrained definition for data interop. | |XML-2) To enforce compliance with a simple XML definition, you need to |have an extra definition of a well formed 'small' document. But |creating a new definition of a data format in XML is equivalent to |defining a new data format, the same as 'option 3a' above. But that |option was ruled out above, on grounds of time constraint.. provided |that a suitable alternative exists already (see JSON claims below). | |NOTE 3 on WS-* ---- IMHO a third reason why WS-* failed to be simple |enough to get happy, quick and wide adoption, is that (XML-1) issue |left too much data integration in the hands of customers, because |vendors could not produce useful products when XML could be all things |to all people. By not delivering on data integration, it became hard |to deliver on the promise of integration at all. And recall that |lowering integration costs was the selling point... | |So let's get INTEROP right and then do INTEGRATION. | |Interop requires: | |* Model - yes |* Data format - one definition |* Metadata - ? tbd | |As an aside - I think that GData is the nuts but it is also really an |*integration techonology*. | |Now, here are some claims about JSON: | |JSON-1) Sun have demonstrated that it is plausible as a data model for |a cloud API. That makes it plausible for us to ask: can it be used as |the core definition for interop? | |JSON-2) It is lower entropy than XML. This makes it easy to test |conformance. | |JSON-3) This means we do things the right way round -- simpler 'ex |ante' choices make it EASIER for us to extend and enhance the protocol |on a by need basis. For example carrying OVF payloads, or other |integration points. Many will be done by users. | |So my recommendation is that the best way to avoid WS-* outcomes is | |A) Use one format for interop. Do interop first. Integration later. |B) *At least for now* and *during the working draft* stage, to use JSON |C) Other formats, for now, are "integration". But we do interop first. | |OK.... I have to run off. I wrote this down in one go and don't have |time to review it. Apologies for any mistakes in the presentation. | |What do you all think? | |alexis |_______________________________________________ |occi-wg mailing list |occi-wg@ogf.org <mailto:occi-wg@ogf.org> |http://www.ogf.org/mailman/listinfo/occi-wg _______________________________________________ occi-wg mailing list occi-wg@ogf.org <mailto:occi-wg@ogf.org> http://www.ogf.org/mailman/listinfo/occi-wg
------------------------------------------------------------------------
_______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
Gary, Thanks for this note. My claim is that interoperability, if it describes data which IMO it has to, must use a formalism to do so, and this means choosing or making a format. I prefer choosing over making. While there can only be one format/formalism for interop, there can be many for integration. You say "A JSON only solution may alienate most of the world." The world will use OCCI directly or through an integration point. OCCI will not be "X only" at the integration points because it cannot be. The question of "how OCCI is adopted" is one of integration. The whole point is to not impose one cloud implementation or format on people at the cloud integration endpoint. I am against interop formats being user-extensible because this WILL break interop. I am in favour of integration ("rendering") formats being user extensible. (Sidenote: extending interop behaviour is foreseeable but should be done by agreement and only after OCCI 1.0 is defined.) More later ... got to run. alexis On Sun, May 10, 2009 at 1:07 AM, Gary Mazz <garymazzaferro@gmail.com> wrote:
BTW, Happy Birthday....
Some general observations on this thread....
I don't believe anyone is asking to disregard your work, at least I hope not. But what is happening here is a pitfall of starting with an light model and an API without clear definition of requirements based on use models.
Lets touch on some of the issues, at least from my perspective, sorry for the terseness and directness, I'm very busy...
1) There are several public strategic use models emerging for clouds, occurring out of control of this group. Many are still evolving
2) Success of this effort, in part, is dependent on the ability to adopt and integrate other well know and deployed standards and technologies.
3) An api is a REPRESENTATION of actions and data with implicate or explicate organization into groups of interfaces. Actions, data and organization will defined by the use models, or at least provide enough that an api can be extrapolated from them.
4) Any particular api can be RENDERED based on the native need of the api consumer. We call them mappings. For example, you would not directly use a PHP or javascript api in C/C++ program without translating it into a form (a mapping) that the C/C++ compilers can accept and the executing code can reach. C/C++ doesn't care about CDATA
5) Many standards, specifications and technologies, deployed for decades, are defined and implemented in XML.
6) Entropy can occur though the lack of precision. ie: clearly defined bounds. Ambiguity equals entropy
Summarizing: The api represents a model, the rendered format of the api will depend on the use model (including programming languages, development tools chains and execution environments). We, meaning the industry and the world, don't have well defined or detailed use model today.
My Opinions Simple is better, but who cares if you can only interoperate with yourself.
A JSON only solution may alienate most of the world. Although all the consultants will have a field day with mapping, until the mapping have to interoperate.
The occi effort may and probably will support more than one api and rendering.
The occi NEEDS to develop profile based use cases. The profiles can be used to set requirements and set priorities for mappings.
Trading off definition in the front of a processes for expedience always results in exponential expenditures as you continue down the path of deployments. JSON lacks definition and increases ambiguity. I'm not sure what interop validation will look like, but the bounds of JSON data may require significantly more work and definition before validation can be achieved.
BTW, I have a requirement for binary defined data...
-gary
Sam Johnston wrote:
Ok so since we're talking about trashing countless hours of my work and missing our deadlines for what appears to be (assuming perhaps a little too much good faith) nothing more than personal preference, and bearing in mind that I've been working on this since 4am this morning, on my birthday, which I've since had to cancel due to a nasty plane flu (thus the surly meter is already well and truly pegged), I'm of course going to have something to say - even if I lack the energy and inclination to give this the comprehensive retort that it deserves at this time.
For a start I know nothing about AMQP so I have nothing to say about the AMQP assertions, but the XML and JSON assertions are patently false and bordering on FUD (XML having extensibility and validation features when JSON doesn't is a drawback? JSON has less "entropy" so it's easier to test conformance even in the absence of schemas? XML is to blame for WS-*? Sun Cloud API, currently a work in progress, has proven something? Seriously, WTF?).
There is absolutely no question from anyone that XML is up to the task (Google have already proven it on a large scale) but significant concerns about whether JSON is - concerns that remain unanswered despite my repeated requests for answers and concerns that this proposal seeks to postpone or avoid entirely, thus making them someone else's problem. While most APIs currently support multiple formats (even GData offer a JSON option for convenience if not interop) I am yet to find a single successful JSON-only API so this all sounds a bit too experimental for my liking.
Furthermore, given that things that are trivial with XML (transforms, embedding, signatures, encryption, etc.) are currently impossible with JSON, this is the mother of all externalisations (that is, leaving our users up the creek without a paddle when we are in the best position to solve the problem by making practical choices). It's also a disappointing attempt to foist personal preferences on others when we can trivially cater for the entire spectrum of needs using XML & XSLT (work which I have alread done and demonstrated I might add - the absence of any volunteers to reinvent the wheel given I'm not about to is also disconcerting). Other difficult questions like the serialisation and serious performance problems of using hrefs (1 request for XML vs 1+n for JSON) have also been conveniently ignored thus far and the best answer I've got about the absence of embedding functionality in JSON is that as XML isn't perfect we're better off with nothing?!?
Most critically though, by telling SNIA, Google and pretty much everyone else in the cloud standard space that they can go fornicate themselves, and in doing so missing the [only] opportunity to standardise a well proven, risk-free, extensible protocol on which others can safely build, I think we may well be witnessing the inception of CC-* (that is, WS-*, JSON edition). More specifically, if we slam the door on the possibility of others building on our efforts then they will have little choice but to blaze their own trail, forcing clients to support a myriad standards (OCCI for trivial management tasks, OVF for compute resources, SNIA's stuff for storage and possibly networking, etc.).
I for one have negative interest in supporting something that I believe could do more damage than good and knowing what happened previously, learning from our mistakes and practicing the precautionary principle <http://en.wikipedia.org/wiki/Precautionary_principle> would be well advised. I'm currently OCCI's number one fan but if this attempt to pwn the process is successful then those who witnessed the CCIF goat rodeo well know that I have no problems saying it how I see it.
What I would suggest is that those who want to see JSON adopted get to work on cutting code to prove the concept as I have done already for XML - loose consensus and running code remember.
Sam (who can't help but to observe that the key JSON junkies all have significant business investment in same)
On Sat, May 9, 2009 at 10:25 PM, Krishna Sankar (ksankar) <ksankar@cisco.com <mailto:ksankar@cisco.com>> wrote:
Alexis et al, a) Wiki Yep, time to have a Wiki or some form of threaded discussion, other than e-mail. Having said that, let me add to the entropy ;o)
b) Interop vs Integration Agreed. Just as one more POV, interop tests the functionality while integration tests the ability to work with various systems. So interop first, with one format, is good. As you said, our first goal is to be functionally complete and have that canonical model, which when done, means, collectively we have understood the essential domain complexity (at least the current state of the art, as Tim pointed out a while ago)
c) "Transport" Interesting pointer to AMQP. What about XMPP as the substrate ? (Of course, it could mean XML ;o))
d) CDATA The CDATA issue pointed by Sam is an interesting one. I was also thinking of the implications and had the question whether we plan to chuck OVF through our APIs? I think, we can take a position like html - i.e. use href mechanism as far as possible and base64 for the rest.
e) OGF-25 Presentation Most probably we will not get to agreement on all the important points by OGF-25. We don't have to. At GGF-25, we present the state of our work - demark what we have agreed upon, and the alternatives we are working on. And until 1.0, things will be unstable anyway. Plus we can honestly represent the different camps and seek insights from the crowd. So I do not think GGF-25 is in critical path in anyway.
f) Disagreeing with Sam I also agree that we should not en-masse disagree with Sam. But as Alexis' points out, the minimal scope vs. maximum coverage is the key. We can restrict or curtail the APIs to a minimum set - without any reduction in functionality; for example use hrefs instead of embedding. Constraints are wonderful thongs !
Cheers & have a nice day <k/>
|-----Original Message----- |From: occi-wg-bounces@ogf.org <mailto:occi-wg-bounces@ogf.org> [mailto:occi-wg-bounces@ogf.org <mailto:occi-wg-bounces@ogf.org>] On Behalf |Of Alexis Richardson |Sent: Saturday, May 09, 2009 11:48 AM |To: occi-wg@ogf.org <mailto:occi-wg@ogf.org> |Subject: [occi-wg] thought on interoperability vs. integration | |Hi all, | |Thanks for a thought-provoking week of emails on the OCCI-WG list. |Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of |the issues. | |One link that I found useful was this: |http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we |find the following statement: | |--- |if Cloud technology is going to take off, there'll have to be a |competitive ecosystem; so that when you bet on a service provider, if |the relationship doesn't work out there's a way to take your business |to another provider with relatively little operational pain. Put |another way: no lock-in. ... I got all excited about this back in |January at that Cloud Interop session. Anant Jhingran, an IBM VIP, |spoke up and said "Customers don't want interoperability, they want |integration." ... "Bzzzzzzzzzt! Wrong!" I thought. But then I |realized he was only half wrong; anyone going down this road needs |integration and interoperability. |--- | |What we have been discussing is "for customers". So it is about both |of these things: | |* Interoperability ("interop") |* Integration | |This made me realise that our OCCI discussions have correctly been |about both issues. But, incorrectly, we have been commingling the |two. For example a person will say "you need X for interop" and the |reply will be "but you need Y" when in fact Y is for integration. And |vice versa. | |This is a problem for us because it leads to confusion. But it's a |symptom of a larger problem which is that interop and integration have |opposite requirements. | |* Interop is about reducing the entropy in a general purpose system |down to some level of behavioural invariance. This minimises the cost |of meeting the defined behaviour correctly, and minimises the risk of |different behaviours breaking interop. This is in turn about having a |defined behaviour (in some format) and minimising its scope. | |* Integration is about minimising the frictions between (a) the |general purpose system and (b) any specific purpose external system |that may be coupled to it. It is also, often, about maximising the |number of specific systems (or "features") that may be connected. |Since we don't know ex ante what those are, this tends to maximise |scope (eg "feature creep"). Too much specificity is the same as |complexity. | |Because interop requires minimal scope, and integration pushes for |maximal scope, they are in tension. | |They are BOTH about simplicity. Simplicity cannot be invoked on its |own, as a reason for preferring interop over integration: | |* Interop is about simplicity of definition | |* Integration is about simplicity of end use | |BUT | |* Interop simplicity is ex ante | |* Integration simplicity is ex post | |We cannot predict all ex post issues, but we can try and make simple |definitions ex ante. I argue below this means we have to have ONE |definition. | |So let's look at interop first: | |It's really important that interop be based on defined behaviour. Or |it cannot be verified. Lack of verifiability is almost always a |symptom that something is opaque, complex and ambiguous, which will |break interop in subtle ways that cannot be seen in advance (due to |the opacity). This later leads to brittleness when interop and |integration are attempted in practice; and that leads to expensive |patching at all levels, ... which is one thing that WS-* exhibits. | |NOTE 1 on WS-* ---- IMO this was not accidental, and due to focussing |on solving integration before solving interop. IIRC, the first WS-* |was an Interop committee to fix minor vendor mismatches and versioning |issues in prior WS protocols such as WSDL and SOAP. The formalisms of |WSDL and SOAP were not precise enough to spot these issues upfront, so |they were left until it was too late. | |Now let's look at the implications of having a definition of behaviour: | |You cannot define a system using multiple definitions. You have to |have one definition, preferably in a formalism that admits of |automatic conformance testing. In the case of data formats, this |leads us to three possible choices: | |1) We remove data formats from the interop profile. They are not part |of interop. Data interop is excluded. Data payloads are opaque |blobs. | |OR |2) We have one data format that can be defined unambiguously. | |OR |3) We have multiple formats and derive them from a single unambiguous |definition using canonical, verifiable, automatable, mappings. This |definition can either be in a new format which we invent (option 3a) |or it can be in an existing format (option 3b). | |I am going to rule out option (3a) on grounds of time and due to the |prevalence of existing candidates. | |Also, I think choice (1) is a complete cop-out -- I don't see how we |can claim 'useful' interop without data interop. This leaves options |(2) and (3b). BOTH of these options require one format. | |By occam's razor, option 2 is preferable on grounds of simplicity - |option (3b) does not add anything except one mapping for every extra |format that is based on the core definition. Such complexity MUST be |deferred as long as possible, and MAY be pushed into integration use |cases. The latter is preferable. Complexity MAY be added later if |necessary and then, reluctantly. | |As a general observation if you have complexity (entropy) in a system, |it is very hard to take out. It may permeate the system at all |levels. Removal of the complexity is known as 'complete refactoring'. | This is a bad outcome. Even worse is when refactoring cannot be done |for commercial reasons, eg because the systems are in production. |Then the complexity must be encapsulated and hidden. Attempts to wrap |complex behaviours in simpler packaging usually fail. | |NOTE 2 on WS-* ---- This tried to do too much and ended up trying to |wrap the complexity in simple packaging. This usually fails to work |as we have seen. | |Another general observation: | |* Integration requires interop, ie. the use of a common general |interop model; otherwise it is piecemeal and pointless. Example - the |old 'integration brokers' that had N*N connections to maintain, and |which got replaced by N connections to a common system. | |* But you can have interop without integration - it just means 'you |have a smaller audience'. This is fine because you can always grow |your audience by integrating more case with the interoperating core. |It is easier to do that when the interoperating core is |programmatically simple (as in low entropy, small code blocks, easy to |test conformance to the definition). | |I would like to add some observations from the world of AMQP... | |AMQP-1) It is a protocol with one simple common data format - it gets |that right. We leave it to integration products and services to |support data formats at the edge (eg "ESBs"). OCCI should not be like |an ESB - that is for products and services in the integration |business. | |AMQP-2) That AMQP data format is not XML - see below for more thoughts |on that. XML can be carried as an attached payload (just as it can be |in the JSON case btw). | |AMQP-3) The 0-8 and 0-9 specs took 18 months of production use to show |up the many very tiny interop bugs. We used those to create 0-9-1 |which does have interop (we have tested this) and is only 40 pages |long. This would not have been possible with a complex spec. It |would not have been possible with multiple data formats. | |AMQP-4) The 0-10 spec was focussed on supporting a lot of use cases eg |"integrate with multiple transport formats including SCTP and TCP and |UDP" and adding lots of features at all levels (eg JMS, support for |hardware, ..). That spec is really complicated and nearly 300 pages |long. Some great new ideas are in it, but it's long and in my own |opinion not supportive of two interoperating implementations. | |AMQP-5) All these painful lessons have taken the AMQP working group to |a much happier place with AMQP 1.0 which tries to simplify everything |by taking stuff out that is not needed for interop, plus refactoring |(see above my comments on how removing entropy is hard) and clean-up. | |All of the above has taken time because we did not learn from WS-*. |We did too much too fast and confused interop with integration. We |are back on track now. | |Now to the issue of data formats. I have already argued that FOR |INTEROP, there must be one definition. I argued that the best way to |do this is via a suitable single format. We can support as many |recommended ways as we like FOR INTEGRATION ... and they can be |evolved over time. | |Here is my 2c on XML. | |XML-1) XML lets you do too much - because of namespaces and xsd it is |in effect multiple formats. This is bad - we want a single, testable, |constrained definition for data interop. | |XML-2) To enforce compliance with a simple XML definition, you need to |have an extra definition of a well formed 'small' document. But |creating a new definition of a data format in XML is equivalent to |defining a new data format, the same as 'option 3a' above. But that |option was ruled out above, on grounds of time constraint.. provided |that a suitable alternative exists already (see JSON claims below). | |NOTE 3 on WS-* ---- IMHO a third reason why WS-* failed to be simple |enough to get happy, quick and wide adoption, is that (XML-1) issue |left too much data integration in the hands of customers, because |vendors could not produce useful products when XML could be all things |to all people. By not delivering on data integration, it became hard |to deliver on the promise of integration at all. And recall that |lowering integration costs was the selling point... | |So let's get INTEROP right and then do INTEGRATION. | |Interop requires: | |* Model - yes |* Data format - one definition |* Metadata - ? tbd | |As an aside - I think that GData is the nuts but it is also really an |*integration techonology*. | |Now, here are some claims about JSON: | |JSON-1) Sun have demonstrated that it is plausible as a data model for |a cloud API. That makes it plausible for us to ask: can it be used as |the core definition for interop? | |JSON-2) It is lower entropy than XML. This makes it easy to test |conformance. | |JSON-3) This means we do things the right way round -- simpler 'ex |ante' choices make it EASIER for us to extend and enhance the protocol |on a by need basis. For example carrying OVF payloads, or other |integration points. Many will be done by users. | |So my recommendation is that the best way to avoid WS-* outcomes is | |A) Use one format for interop. Do interop first. Integration later. |B) *At least for now* and *during the working draft* stage, to use JSON |C) Other formats, for now, are "integration". But we do interop first. | |OK.... I have to run off. I wrote this down in one go and don't have |time to review it. Apologies for any mistakes in the presentation. | |What do you all think? | |alexis |_______________________________________________ |occi-wg mailing list |occi-wg@ogf.org <mailto:occi-wg@ogf.org> |http://www.ogf.org/mailman/listinfo/occi-wg _______________________________________________ occi-wg mailing list occi-wg@ogf.org <mailto:occi-wg@ogf.org> http://www.ogf.org/mailman/listinfo/occi-wg
------------------------------------------------------------------------
_______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
_______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
Sam, Relax. As Winston would say it, we should disagree agreeably (and in that spirit, am going to ignore the last two paragraphs (and signature) of your e-mail). (And might be surprising to you, but I am pretty sure that JSON is not the cause of plane flu ;o) - it only happens to XML fans !) BTW, the e-mail chain doesn't deserve a comprehensive retort at all. a) Your work is not lost at all, the canonical model is required whether in JSON or XML or YAML or AOML. b) Let us leave WS-* and the rest aside (pun intended!) BTW, if you care to notice, you will not see WS-* or JSON vs. XML in my previous e-mail. c) Our essential question is "What is the best format to capture the canonical cloud apis ?" d) And in addition to Alexis's Interop and Integration we also need a third criteria namely "InterStandard". i.e. if SNIA and other cloud standards are using XML would we better off using XML ? If, as you say, signatures and encryption can be done easier, would XML be better ? e) BTW, if you think OCCI is supportable only if we use XML and if not OCCI is comparable to CCIF is pure bs(tm). OCCI is supportable because it solves a problem which we all care about. f) And about comparing to CCIF - Et tu Sam'e ? Cheers <k/> P.S : BTW, Happy Birthday - Live long and prosper (but of course, life would be easier if you support JSON ;o)) From: Sam Johnston [mailto:samj@samj.net] Sent: Saturday, May 09, 2009 3:04 PM To: Krishna Sankar (ksankar) Cc: Alexis Richardson; occi-wg@ogf.org Subject: Re: [occi-wg] thought on interoperability vs. integration Ok so since we're talking about trashing countless hours of my work and missing our deadlines for what appears to be (assuming perhaps a little too much good faith) nothing more than personal preference, and bearing in mind that I've been working on this since 4am this morning, on my birthday, which I've since had to cancel due to a nasty plane flu (thus the surly meter is already well and truly pegged), I'm of course going to have something to say - even if I lack the energy and inclination to give this the comprehensive retort that it deserves at this time. For a start I know nothing about AMQP so I have nothing to say about the AMQP assertions, but the XML and JSON assertions are patently false and bordering on FUD (XML having extensibility and validation features when JSON doesn't is a drawback? JSON has less "entropy" so it's easier to test conformance even in the absence of schemas? XML is to blame for WS-*? Sun Cloud API, currently a work in progress, has proven something? Seriously, WTF?). There is absolutely no question from anyone that XML is up to the task (Google have already proven it on a large scale) but significant concerns about whether JSON is - concerns that remain unanswered despite my repeated requests for answers and concerns that this proposal seeks to postpone or avoid entirely, thus making them someone else's problem. While most APIs currently support multiple formats (even GData offer a JSON option for convenience if not interop) I am yet to find a single successful JSON-only API so this all sounds a bit too experimental for my liking. Furthermore, given that things that are trivial with XML (transforms, embedding, signatures, encryption, etc.) are currently impossible with JSON, this is the mother of all externalisations (that is, leaving our users up the creek without a paddle when we are in the best position to solve the problem by making practical choices). It's also a disappointing attempt to foist personal preferences on others when we can trivially cater for the entire spectrum of needs using XML & XSLT (work which I have alread done and demonstrated I might add - the absence of any volunteers to reinvent the wheel given I'm not about to is also disconcerting). Other difficult questions like the serialisation and serious performance problems of using hrefs (1 request for XML vs 1+n for JSON) have also been conveniently ignored thus far and the best answer I've got about the absence of embedding functionality in JSON is that as XML isn't perfect we're better off with nothing?!? Most critically though, by telling SNIA, Google and pretty much everyone else in the cloud standard space that they can go fornicate themselves, and in doing so missing the [only] opportunity to standardise a well proven, risk-free, extensible protocol on which others can safely build, I think we may well be witnessing the inception of CC-* (that is, WS-*, JSON edition). More specifically, if we slam the door on the possibility of others building on our efforts then they will have little choice but to blaze their own trail, forcing clients to support a myriad standards (OCCI for trivial management tasks, OVF for compute resources, SNIA's stuff for storage and possibly networking, etc.). I for one have negative interest in supporting something that I believe could do more damage than good and knowing what happened previously, learning from our mistakes and practicing the precautionary principle <http://en.wikipedia.org/wiki/Precautionary_principle> would be well advised. I'm currently OCCI's number one fan but if this attempt to pwn the process is successful then those who witnessed the CCIF goat rodeo well know that I have no problems saying it how I see it. What I would suggest is that those who want to see JSON adopted get to work on cutting code to prove the concept as I have done already for XML - loose consensus and running code remember. Sam (who can't help but to observe that the key JSON junkies all have significant business investment in same) On Sat, May 9, 2009 at 10:25 PM, Krishna Sankar (ksankar) <ksankar@cisco.com> wrote: Alexis et al, a) Wiki Yep, time to have a Wiki or some form of threaded discussion, other than e-mail. Having said that, let me add to the entropy ;o) b) Interop vs Integration Agreed. Just as one more POV, interop tests the functionality while integration tests the ability to work with various systems. So interop first, with one format, is good. As you said, our first goal is to be functionally complete and have that canonical model, which when done, means, collectively we have understood the essential domain complexity (at least the current state of the art, as Tim pointed out a while ago) c) "Transport" Interesting pointer to AMQP. What about XMPP as the substrate ? (Of course, it could mean XML ;o)) d) CDATA The CDATA issue pointed by Sam is an interesting one. I was also thinking of the implications and had the question whether we plan to chuck OVF through our APIs? I think, we can take a position like html - i.e. use href mechanism as far as possible and base64 for the rest. e) OGF-25 Presentation Most probably we will not get to agreement on all the important points by OGF-25. We don't have to. At GGF-25, we present the state of our work - demark what we have agreed upon, and the alternatives we are working on. And until 1.0, things will be unstable anyway. Plus we can honestly represent the different camps and seek insights from the crowd. So I do not think GGF-25 is in critical path in anyway. f) Disagreeing with Sam I also agree that we should not en-masse disagree with Sam. But as Alexis' points out, the minimal scope vs. maximum coverage is the key. We can restrict or curtail the APIs to a minimum set - without any reduction in functionality; for example use hrefs instead of embedding. Constraints are wonderful thongs ! Cheers & have a nice day <k/> |-----Original Message----- |From: occi-wg-bounces@ogf.org [mailto:occi-wg-bounces@ogf.org] On Behalf |Of Alexis Richardson |Sent: Saturday, May 09, 2009 11:48 AM |To: occi-wg@ogf.org |Subject: [occi-wg] thought on interoperability vs. integration | |Hi all, | |Thanks for a thought-provoking week of emails on the OCCI-WG list. |Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of |the issues. | |One link that I found useful was this: |http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we |find the following statement: | |--- |if Cloud technology is going to take off, there'll have to be a |competitive ecosystem; so that when you bet on a service provider, if |the relationship doesn't work out there's a way to take your business |to another provider with relatively little operational pain. Put |another way: no lock-in. ... I got all excited about this back in |January at that Cloud Interop session. Anant Jhingran, an IBM VIP, |spoke up and said "Customers don't want interoperability, they want |integration." ... "Bzzzzzzzzzt! Wrong!" I thought. But then I |realized he was only half wrong; anyone going down this road needs |integration and interoperability. |--- | |What we have been discussing is "for customers". So it is about both |of these things: | |* Interoperability ("interop") |* Integration | |This made me realise that our OCCI discussions have correctly been |about both issues. But, incorrectly, we have been commingling the |two. For example a person will say "you need X for interop" and the |reply will be "but you need Y" when in fact Y is for integration. And |vice versa. | |This is a problem for us because it leads to confusion. But it's a |symptom of a larger problem which is that interop and integration have |opposite requirements. | |* Interop is about reducing the entropy in a general purpose system |down to some level of behavioural invariance. This minimises the cost |of meeting the defined behaviour correctly, and minimises the risk of |different behaviours breaking interop. This is in turn about having a |defined behaviour (in some format) and minimising its scope. | |* Integration is about minimising the frictions between (a) the |general purpose system and (b) any specific purpose external system |that may be coupled to it. It is also, often, about maximising the |number of specific systems (or "features") that may be connected. |Since we don't know ex ante what those are, this tends to maximise |scope (eg "feature creep"). Too much specificity is the same as |complexity. | |Because interop requires minimal scope, and integration pushes for |maximal scope, they are in tension. | |They are BOTH about simplicity. Simplicity cannot be invoked on its |own, as a reason for preferring interop over integration: | |* Interop is about simplicity of definition | |* Integration is about simplicity of end use | |BUT | |* Interop simplicity is ex ante | |* Integration simplicity is ex post | |We cannot predict all ex post issues, but we can try and make simple |definitions ex ante. I argue below this means we have to have ONE |definition. | |So let's look at interop first: | |It's really important that interop be based on defined behaviour. Or |it cannot be verified. Lack of verifiability is almost always a |symptom that something is opaque, complex and ambiguous, which will |break interop in subtle ways that cannot be seen in advance (due to |the opacity). This later leads to brittleness when interop and |integration are attempted in practice; and that leads to expensive |patching at all levels, ... which is one thing that WS-* exhibits. | |NOTE 1 on WS-* ---- IMO this was not accidental, and due to focussing |on solving integration before solving interop. IIRC, the first WS-* |was an Interop committee to fix minor vendor mismatches and versioning |issues in prior WS protocols such as WSDL and SOAP. The formalisms of |WSDL and SOAP were not precise enough to spot these issues upfront, so |they were left until it was too late. | |Now let's look at the implications of having a definition of behaviour: | |You cannot define a system using multiple definitions. You have to |have one definition, preferably in a formalism that admits of |automatic conformance testing. In the case of data formats, this |leads us to three possible choices: | |1) We remove data formats from the interop profile. They are not part |of interop. Data interop is excluded. Data payloads are opaque |blobs. | |OR |2) We have one data format that can be defined unambiguously. | |OR |3) We have multiple formats and derive them from a single unambiguous |definition using canonical, verifiable, automatable, mappings. This |definition can either be in a new format which we invent (option 3a) |or it can be in an existing format (option 3b). | |I am going to rule out option (3a) on grounds of time and due to the |prevalence of existing candidates. | |Also, I think choice (1) is a complete cop-out -- I don't see how we |can claim 'useful' interop without data interop. This leaves options |(2) and (3b). BOTH of these options require one format. | |By occam's razor, option 2 is preferable on grounds of simplicity - |option (3b) does not add anything except one mapping for every extra |format that is based on the core definition. Such complexity MUST be |deferred as long as possible, and MAY be pushed into integration use |cases. The latter is preferable. Complexity MAY be added later if |necessary and then, reluctantly. | |As a general observation if you have complexity (entropy) in a system, |it is very hard to take out. It may permeate the system at all |levels. Removal of the complexity is known as 'complete refactoring'. | This is a bad outcome. Even worse is when refactoring cannot be done |for commercial reasons, eg because the systems are in production. |Then the complexity must be encapsulated and hidden. Attempts to wrap |complex behaviours in simpler packaging usually fail. | |NOTE 2 on WS-* ---- This tried to do too much and ended up trying to |wrap the complexity in simple packaging. This usually fails to work |as we have seen. | |Another general observation: | |* Integration requires interop, ie. the use of a common general |interop model; otherwise it is piecemeal and pointless. Example - the |old 'integration brokers' that had N*N connections to maintain, and |which got replaced by N connections to a common system. | |* But you can have interop without integration - it just means 'you |have a smaller audience'. This is fine because you can always grow |your audience by integrating more case with the interoperating core. |It is easier to do that when the interoperating core is |programmatically simple (as in low entropy, small code blocks, easy to |test conformance to the definition). | |I would like to add some observations from the world of AMQP... | |AMQP-1) It is a protocol with one simple common data format - it gets |that right. We leave it to integration products and services to |support data formats at the edge (eg "ESBs"). OCCI should not be like |an ESB - that is for products and services in the integration |business. | |AMQP-2) That AMQP data format is not XML - see below for more thoughts |on that. XML can be carried as an attached payload (just as it can be |in the JSON case btw). | |AMQP-3) The 0-8 and 0-9 specs took 18 months of production use to show |up the many very tiny interop bugs. We used those to create 0-9-1 |which does have interop (we have tested this) and is only 40 pages |long. This would not have been possible with a complex spec. It |would not have been possible with multiple data formats. | |AMQP-4) The 0-10 spec was focussed on supporting a lot of use cases eg |"integrate with multiple transport formats including SCTP and TCP and |UDP" and adding lots of features at all levels (eg JMS, support for |hardware, ..). That spec is really complicated and nearly 300 pages |long. Some great new ideas are in it, but it's long and in my own |opinion not supportive of two interoperating implementations. | |AMQP-5) All these painful lessons have taken the AMQP working group to |a much happier place with AMQP 1.0 which tries to simplify everything |by taking stuff out that is not needed for interop, plus refactoring |(see above my comments on how removing entropy is hard) and clean-up. | |All of the above has taken time because we did not learn from WS-*. |We did too much too fast and confused interop with integration. We |are back on track now. | |Now to the issue of data formats. I have already argued that FOR |INTEROP, there must be one definition. I argued that the best way to |do this is via a suitable single format. We can support as many |recommended ways as we like FOR INTEGRATION ... and they can be |evolved over time. | |Here is my 2c on XML. | |XML-1) XML lets you do too much - because of namespaces and xsd it is |in effect multiple formats. This is bad - we want a single, testable, |constrained definition for data interop. | |XML-2) To enforce compliance with a simple XML definition, you need to |have an extra definition of a well formed 'small' document. But |creating a new definition of a data format in XML is equivalent to |defining a new data format, the same as 'option 3a' above. But that |option was ruled out above, on grounds of time constraint.. provided |that a suitable alternative exists already (see JSON claims below). | |NOTE 3 on WS-* ---- IMHO a third reason why WS-* failed to be simple |enough to get happy, quick and wide adoption, is that (XML-1) issue |left too much data integration in the hands of customers, because |vendors could not produce useful products when XML could be all things |to all people. By not delivering on data integration, it became hard |to deliver on the promise of integration at all. And recall that |lowering integration costs was the selling point... | |So let's get INTEROP right and then do INTEGRATION. | |Interop requires: | |* Model - yes |* Data format - one definition |* Metadata - ? tbd | |As an aside - I think that GData is the nuts but it is also really an |*integration techonology*. | |Now, here are some claims about JSON: | |JSON-1) Sun have demonstrated that it is plausible as a data model for |a cloud API. That makes it plausible for us to ask: can it be used as |the core definition for interop? | |JSON-2) It is lower entropy than XML. This makes it easy to test |conformance. | |JSON-3) This means we do things the right way round -- simpler 'ex |ante' choices make it EASIER for us to extend and enhance the protocol |on a by need basis. For example carrying OVF payloads, or other |integration points. Many will be done by users. | |So my recommendation is that the best way to avoid WS-* outcomes is | |A) Use one format for interop. Do interop first. Integration later. |B) *At least for now* and *during the working draft* stage, to use JSON |C) Other formats, for now, are "integration". But we do interop first. | |OK.... I have to run off. I wrote this down in one go and don't have |time to review it. Apologies for any mistakes in the presentation. | |What do you all think? | |alexis |_______________________________________________ |occi-wg mailing list |occi-wg@ogf.org |http://www.ogf.org/mailman/listinfo/occi-wg _______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
On 5/10/09, Krishna Sankar (ksankar) <ksankar@cisco.com> wrote:
Sam,
Relax. As Winston would say it, we should disagree agreeably (and in that spirit, am going to ignore the last two paragraphs (and signature) of your e-mail). (And might be surprising to you, but I am pretty sure that JSON is not the cause of plane flu ;o) - it only happens to XML fans !) BTW, the e-mail chain doesn't deserve a comprehensive retort at all.
a) Your work is not lost at all, the canonical model is required whether in JSON or XML or YAML or AOML.
I assure you the model is far easier than the implementation, for which many hours were spent simplifying - the renderer itself is only 2 LOC, and there's a template with 2 or 3 active tags each having about a dozen LOC. Strap on libvirt and you've got your cloud... good luck achieving the same with JSON (where you're basically just transporting data structures from one place to another - another good source of entropy).
b) Let us leave WS-* and the rest aside (pun intended!) BTW, if you care to notice, you will not see WS-* or JSON vs. XML in my previous e-mail.
Repeating history would constitute catastrophic failure so it makes sense for us to learn from our mistakes and at least try to avoid doing the same. Everyone working in a vacuum is a sure fire way to make the results unimplementable and that is essentially what has been proposed. Meanwhile I've been reaching out to Google PMs, SNIA, DMTF, etc. with a view to making sure this stuff all works together... since we're delivering first it is our responsibility to pave the road ahead, else it's not going to happen.
c) Our essential question is "What is the best format to capture the canonical cloud apis ?"
No, there was a strong consensus that our "essential question" is "What is the best *model* to capture canonical cloud APIs?". The format discussion is a distraction that runs the risk of derailing OCCI and missing the first major deadline for it will hurt us more than I think you might realise. What we need is format(s) that will satisfy the use cases, which certainly means multiples (a cron job, shell script or human won't talk json or XML and an enterprise will be far more comfortable talking XML, while web developer types love JSON, users want HTML, managers PDF and beancounters prefer CSV). That being the case the path of least resistance is to use a language that trivially supports cross platform, mechanical transforms. Ignoring use cases because they don't suit your $PREFERRED_FORMAT (ala dropping the text requirement and in doing so making the API far less approachable) is fraught with danger.
d) And in addition to Alexis's Interop and Integration we also need a third criteria namely "InterStandard". i.e. if SNIA and other cloud standards are using XML would we better off using XML ? If, as you say, signatures and encryption can be done easier, would XML be better ?
All the others (notably including Amazon, Google, VMware, Microsoft, etc.) are primarily using XML. Perhaps that will change if JSON eventually becomes the new pink, but for now there's no sign of that happening any time soon. A side effect of that is that implementations are, for the foreseeable future, going to have to talk XML anyway unless they operate in a vacuum. With XML we don't care because we can embed anything and most of the time it will be transparent (a huge advantage IMO).
e) BTW, if you think OCCI is supportable only if we use XML and if not OCCI is comparable to CCIF is pure bs(tm). OCCI is supportable because it solves a problem which we all care about.
If OCCI paints itself into a corner because of a poor format choice then we have failed miserably. If it also wreaks havoc on the industry in doing so by failing to step up to the plate in giving others a path to interstandard interop then we'll have done more damage than good. In the absence of large scale successful JSON-only APIs it's all far too experimental for my liking, not to mention arrogant to force others to eat our preferred brand of dogfood.
f) And about comparing to CCIF - Et tu Sam'e ?
There are enough similarities in terms of obvious conflicts of interest, process problems, etc. that the comparison not so far stretched. People do realize that I'm not talking about XML-RPC or SOAP, right? As such the "XML is noticably more complex than JSON" argument is bunk.
P.S : BTW, Happy Birthday - Live long and prosper (but of course, life would be easier if you support JSON ;o))
Thanks. Perhaps what you say will be true or OCCI 2.0 but while XML has well over a decade of extensive enterprise use, JSON has only been widely used (and almost always for wiring up web interfaces) for the last two years or so. It may well be the right choice later but for now the low risk option is XML as it has already been demonstrated it working in a similar context on a massive scale by Google. Sam on iPhone
From: Sam Johnston [mailto:samj@samj.net] Sent: Saturday, May 09, 2009 3:04 PM To: Krishna Sankar (ksankar) Cc: Alexis Richardson; occi-wg@ogf.org Subject: Re: [occi-wg] thought on interoperability vs. integration
Ok so since we're talking about trashing countless hours of my work and missing our deadlines for what appears to be (assuming perhaps a little too much good faith) nothing more than personal preference, and bearing in mind that I've been working on this since 4am this morning, on my birthday, which I've since had to cancel due to a nasty plane flu (thus the surly meter is already well and truly pegged), I'm of course going to have something to say - even if I lack the energy and inclination to give this the comprehensive retort that it deserves at this time.
For a start I know nothing about AMQP so I have nothing to say about the AMQP assertions, but the XML and JSON assertions are patently false and bordering on FUD (XML having extensibility and validation features when JSON doesn't is a drawback? JSON has less "entropy" so it's easier to test conformance even in the absence of schemas? XML is to blame for WS-*? Sun Cloud API, currently a work in progress, has proven something? Seriously, WTF?).
There is absolutely no question from anyone that XML is up to the task (Google have already proven it on a large scale) but significant concerns about whether JSON is - concerns that remain unanswered despite my repeated requests for answers and concerns that this proposal seeks to postpone or avoid entirely, thus making them someone else's problem. While most APIs currently support multiple formats (even GData offer a JSON option for convenience if not interop) I am yet to find a single successful JSON-only API so this all sounds a bit too experimental for my liking.
Furthermore, given that things that are trivial with XML (transforms, embedding, signatures, encryption, etc.) are currently impossible with JSON, this is the mother of all externalisations (that is, leaving our users up the creek without a paddle when we are in the best position to solve the problem by making practical choices). It's also a disappointing attempt to foist personal preferences on others when we can trivially cater for the entire spectrum of needs using XML & XSLT (work which I have alread done and demonstrated I might add - the absence of any volunteers to reinvent the wheel given I'm not about to is also disconcerting). Other difficult questions like the serialisation and serious performance problems of using hrefs (1 request for XML vs 1+n for JSON) have also been conveniently ignored thus far and the best answer I've got about the absence of embedding functionality in JSON is that as XML isn't perfect we're better off with nothing?!?
Most critically though, by telling SNIA, Google and pretty much everyone else in the cloud standard space that they can go fornicate themselves, and in doing so missing the [only] opportunity to standardise a well proven, risk-free, extensible protocol on which others can safely build, I think we may well be witnessing the inception of CC-* (that is, WS-*, JSON edition). More specifically, if we slam the door on the possibility of others building on our efforts then they will have little choice but to blaze their own trail, forcing clients to support a myriad standards (OCCI for trivial management tasks, OVF for compute resources, SNIA's stuff for storage and possibly networking, etc.).
I for one have negative interest in supporting something that I believe could do more damage than good and knowing what happened previously, learning from our mistakes and practicing the precautionary principle <http://en.wikipedia.org/wiki/Precautionary_principle> would be well advised. I'm currently OCCI's number one fan but if this attempt to pwn the process is successful then those who witnessed the CCIF goat rodeo well know that I have no problems saying it how I see it.
What I would suggest is that those who want to see JSON adopted get to work on cutting code to prove the concept as I have done already for XML - loose consensus and running code remember.
Sam (who can't help but to observe that the key JSON junkies all have significant business investment in same)
On Sat, May 9, 2009 at 10:25 PM, Krishna Sankar (ksankar) <ksankar@cisco.com> wrote:
Alexis et al, a) Wiki Yep, time to have a Wiki or some form of threaded discussion, other than e-mail. Having said that, let me add to the entropy ;o)
b) Interop vs Integration Agreed. Just as one more POV, interop tests the functionality while integration tests the ability to work with various systems. So interop first, with one format, is good. As you said, our first goal is to be functionally complete and have that canonical model, which when done, means, collectively we have understood the essential domain complexity (at least the current state of the art, as Tim pointed out a while ago)
c) "Transport" Interesting pointer to AMQP. What about XMPP as the substrate ? (Of course, it could mean XML ;o))
d) CDATA The CDATA issue pointed by Sam is an interesting one. I was also thinking of the implications and had the question whether we plan to chuck OVF through our APIs? I think, we can take a position like html - i.e. use href mechanism as far as possible and base64 for the rest.
e) OGF-25 Presentation Most probably we will not get to agreement on all the important points by OGF-25. We don't have to. At GGF-25, we present the state of our work - demark what we have agreed upon, and the alternatives we are working on. And until 1.0, things will be unstable anyway. Plus we can honestly represent the different camps and seek insights from the crowd. So I do not think GGF-25 is in critical path in anyway.
f) Disagreeing with Sam I also agree that we should not en-masse disagree with Sam. But as Alexis' points out, the minimal scope vs. maximum coverage is the key. We can restrict or curtail the APIs to a minimum set - without any reduction in functionality; for example use hrefs instead of embedding. Constraints are wonderful thongs !
Cheers & have a nice day <k/>
|-----Original Message----- |From: occi-wg-bounces@ogf.org [mailto:occi-wg-bounces@ogf.org] On Behalf |Of Alexis Richardson |Sent: Saturday, May 09, 2009 11:48 AM |To: occi-wg@ogf.org |Subject: [occi-wg] thought on interoperability vs. integration | |Hi all, | |Thanks for a thought-provoking week of emails on the OCCI-WG list. |Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of |the issues. | |One link that I found useful was this: |http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we |find the following statement: | |--- |if Cloud technology is going to take off, there'll have to be a |competitive ecosystem; so that when you bet on a service provider, if |the relationship doesn't work out there's a way to take your business |to another provider with relatively little operational pain. Put |another way: no lock-in. ... I got all excited about this back in |January at that Cloud Interop session. Anant Jhingran, an IBM VIP, |spoke up and said "Customers don't want interoperability, they want |integration." ... "Bzzzzzzzzzt! Wrong!" I thought. But then I |realized he was only half wrong; anyone going down this road needs |integration and interoperability. |--- | |What we have been discussing is "for customers". So it is about both |of these things: | |* Interoperability ("interop") |* Integration | |This made me realise that our OCCI discussions have correctly been |about both issues. But, incorrectly, we have been commingling the |two. For example a person will say "you need X for interop" and the |reply will be "but you need Y" when in fact Y is for integration. And |vice versa. | |This is a problem for us because it leads to confusion. But it's a |symptom of a larger problem which is that interop and integration have |opposite requirements. | |* Interop is about reducing the entropy in a general purpose system |down to some level of behavioural invariance. This minimises the cost |of meeting the defined behaviour correctly, and minimises the risk of |different behaviours breaking interop. This is in turn about having a |defined behaviour (in some format) and minimising its scope. | |* Integration is about minimising the frictions between (a) the |general purpose system and (b) any specific purpose external system |that may be coupled to it. It is also, often, about maximising the |number of specific systems (or "features") that may be connected. |Since we don't know ex ante what those are, this tends to maximise |scope (eg "feature creep"). Too much specificity is the same as |complexity. | |Because interop requires minimal scope, and integration pushes for |maximal scope, they are in tension. | |They are BOTH about simplicity. Simplicity cannot be invoked on its |own, as a reason for preferring interop over integration: | |* Interop is about simplicity of definition | |* Integration is about simplicity of end use | |BUT | |* Interop simplicity is ex ante | |* Integration simplicity is ex post | |We cannot predict all ex post issues, but we can try and make simple |definitions ex ante. I argue below this means we have to have ONE |definition. | |So let's look at interop first: | |It's really important that interop be based on defined behaviour. Or |it cannot be verified. Lack of verifiability is almost always a |symptom that something is opaque, complex and ambiguous, which will |break interop in subtle ways that cannot be seen in advance (due to |the opacity). This later leads to brittleness when interop and |integration are attempted in practice; and that leads to expensive |patching at all levels, ... which is one thing that WS-* exhibits. | |NOTE 1 on WS-* ---- IMO this was not accidental, and due to focussing |on solving integration before solving interop. IIRC, the first WS-* |was an Interop committee to fix minor vendor mismatches and versioning |issues in prior WS protocols such as WSDL and SOAP. The formalisms of |WSDL and SOAP were not precise enough to spot these issues upfront, so |they were left until it was too late. | |Now let's look at the implications of having a definition of behaviour: | |You cannot define a system using multiple definitions. You have to |have one definition, preferably in a formalism that admits of |automatic conformance testing. In the case of data formats, this |leads us to three possible choices: | |1) We remove data formats from the interop profile. They are not part |of interop. Data interop is excluded. Data payloads are opaque |blobs. | |OR |2) We have one data format that can be defined unambiguously. | |OR |3) We have multiple formats and derive them from a single unambiguous |definition using canonical, verifiable, automatable, mappings. This |definition can either be in a new format which we invent (option 3a) |or it can be in an existing format (option 3b). | |I am going to rule out option (3a) on grounds of time and due to the |prevalence of existing candidates. | |Also, I think choice (1) is a complete cop-out -- I don't see how we |can claim 'useful' interop without data interop. This leaves options |(2) and (3b). BOTH of these options require one format. | |By occam's razor, option 2 is preferable on grounds of simplicity - |option (3b) does not add anything except one mapping for every extra |format that is based on the core definition. Such complexity MUST be |deferred as long as possible, and MAY be pushed into integration use |cases. The latter is preferable. Complexity MAY be added later if |necessary and then, reluctantly. | |As a general observation if you have complexity (entropy) in a system, |it is very hard to take out. It may permeate the system at all |levels. Removal of the complexity is known as 'complete refactoring'. | This is a bad outcome. Even worse is when refactoring cannot be done |for commercial reasons, eg because the systems are in production. |Then the complexity must be encapsulated and hidden. Attempts to wrap |complex behaviours in simpler packaging usually fail. | |NOTE 2 on WS-* ---- This tried to do too much and ended up trying to |wrap the complexity in simple packaging. This usually fails to work |as we have seen. | |Another general observation: | |* Integration requires interop, ie. the use of a common general |interop model; otherwise it is piecemeal and pointless. Example - the |old 'integration brokers' that had N*N connections to maintain, and |which got replaced by N connections to a common system. | |* But you can have interop without integration - it just means 'you |have a smaller audience'. This is fine because you can always grow |your audience by integrating more case with the interoperating core. |It is easier to do that when the interoperating core is |programmatically simple (as in low entropy, small code blocks, easy to |test conformance to the definition). | |I would like to add some observations from the world of AMQP... | |AMQP-1) It is a protocol with one simple common data format - it gets |that right. We leave it to integration products and services to |support data formats at the edge (eg "ESBs"). OCCI should not be like |an ESB - that is for products and services in the integration |business. | |AMQP-2) That AMQP data format is not XML - see below for more thoughts |on that. XML can be carried as an attached payload (just as it can be |in the JSON case btw). | |AMQP-3) The 0-8 and 0-9 specs took 18 months of production use to show |up the many very tiny interop bugs. We used those to create 0-9-1 |which does have interop (we have tested this) and is only 40 pages |long. This would not have been possible with a complex spec. It |would not have been possible with multiple data formats. | |AMQP-4) The 0-10 spec was focussed on supporting a lot of use cases eg |"integrate with multiple transport formats including SCTP and TCP and |UDP" and adding lots of features at all levels (eg JMS, support for |hardware, ..). That spec is really complicated and nearly 300 pages |long. Some great new ideas are in it, but it's long and in my own |opinion not supportive of two interoperating implementations. | |AMQP-5) All these painful lessons have taken the AMQP working group to |a much happier place with AMQP 1.0 which tries to simplify everything |by taking stuff out that is not needed for interop, plus refactoring |(see above my comments on how removing entropy is hard) and clean-up. | |All of the above has taken time because we did not learn from WS-*. |We did too much too fast and confused interop with integration. We |are back on track now. | |Now to the issue of data formats. I have already argued that FOR |INTEROP, there must be one definition. I argued that the best way to |do this is via a suitable single format. We can support as many |recommended ways as we like FOR INTEGRATION ... and they can be |evolved over time. | |Here is my 2c on XML. | |XML-1) XML lets you do too much - because of namespaces and xsd it is |in effect multiple formats. This is bad - we want a single, testable, |constrained definition for data interop. | |XML-2) To enforce compliance with a simple XML definition, you need to |have an extra definition of a well formed 'small' document. But |creating a new definition of a data format in XML is equivalent to |defining a new data format, the same as 'option 3a' above. But that |option was ruled out above, on grounds of time constraint.. provided |that a suitable alternative exists already (see JSON claims below). | |NOTE 3 on WS-* ---- IMHO a third reason why WS-* failed to be simple |enough to get happy, quick and wide adoption, is that (XML-1) issue |left too much data integration in the hands of customers, because |vendors could not produce useful products when XML could be all things |to all people. By not delivering on data integration, it became hard |to deliver on the promise of integration at all. And recall that |lowering integration costs was the selling point... | |So let's get INTEROP right and then do INTEGRATION. | |Interop requires: | |* Model - yes |* Data format - one definition |* Metadata - ? tbd | |As an aside - I think that GData is the nuts but it is also really an |*integration techonology*. | |Now, here are some claims about JSON: | |JSON-1) Sun have demonstrated that it is plausible as a data model for |a cloud API. That makes it plausible for us to ask: can it be used as |the core definition for interop? | |JSON-2) It is lower entropy than XML. This makes it easy to test |conformance. | |JSON-3) This means we do things the right way round -- simpler 'ex |ante' choices make it EASIER for us to extend and enhance the protocol |on a by need basis. For example carrying OVF payloads, or other |integration points. Many will be done by users. | |So my recommendation is that the best way to avoid WS-* outcomes is | |A) Use one format for interop. Do interop first. Integration later. |B) *At least for now* and *during the working draft* stage, to use JSON |C) Other formats, for now, are "integration". But we do interop first. | |OK.... I have to run off. I wrote this down in one go and don't have |time to review it. Apologies for any mistakes in the presentation. | |What do you all think? | |alexis |_______________________________________________ |occi-wg mailing list |occi-wg@ogf.org |http://www.ogf.org/mailman/listinfo/occi-wg _______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
Krishna, On Sat, May 9, 2009 at 9:25 PM, Krishna Sankar (ksankar) <ksankar@cisco.com> wrote:
Alexis et al, a) Wiki Yep, time to have a Wiki or some form of threaded discussion, other than e-mail. Having said that, let me add to the entropy ;o)
:-)
b) Interop vs Integration Agreed. Just as one more POV, interop tests the functionality while integration tests the ability to work with various systems. So interop first, with one format, is good. As you said, our first goal is to be functionally complete and have that canonical model, which when done, means, collectively we have understood the essential domain complexity (at least the current state of the art, as Tim pointed out a while ago)
Thank-you.
c) "Transport" Interesting pointer to AMQP. What about XMPP as the substrate ? (Of course, it could mean XML ;o))
I was just using AMQP experiences as an example of dealing with interop issues in a standard. I don't think OCCI needs a substrate beyond HTTP. XMPP and AMQP are interesting technologies but way out of scope for anything I personally think OCCI needs to worry about in 2009.
d) CDATA The CDATA issue pointed by Sam is an interesting one. I was also thinking of the implications and had the question whether we plan to chuck OVF through our APIs? I think, we can take a position like html - i.e. use href mechanism as far as possible and base64 for the rest.
Yes, I think we need to think about OVF very carefully. I don't think OVF needs to be part of the interop profile because to do so would bring too much baggage. But I do think OVF should be integrated with and how to do that is well worth discussing. If our core format is 'flat' then OVF will have to be attached, regardless of whether we use JSON or XML.
e) OGF-25 Presentation Most probably we will not get to agreement on all the important points by OGF-25. We don't have to. At GGF-25, we present the state of our work - demark what we have agreed upon, and the alternatives we are working on. And until 1.0, things will be unstable anyway. Plus we can honestly represent the different camps and seek insights from the crowd. So I do not think GGF-25 is in critical path in anyway.
Cool. I would like to see a draft set of verbs and nouns, which is what I have been asking for from the start. I think the data model is next. We present what we have agreed on when asked to show a draft.
f) Disagreeing with Sam I also agree that we should not en-masse disagree with Sam. But as Alexis' points out, the minimal scope vs. maximum coverage is the key. We can restrict or curtail the APIs to a minimum set - without any reduction in functionality; for example use hrefs instead of embedding. Constraints are wonderful thongs !
Thongs? ;-) I don't think we are en masse disagreeing with Sam. I think Sam has done some good work and is defending several points concurrently, that are a mix of his own view, prior art, and stuff that others strongly agree on. I do think we need to separate out the strands though.
Cheers & have a nice day
You too! If you can help Richard and Chris, please do so :-) alexis
<k/>
|-----Original Message----- |From: occi-wg-bounces@ogf.org [mailto:occi-wg-bounces@ogf.org] On Behalf |Of Alexis Richardson |Sent: Saturday, May 09, 2009 11:48 AM |To: occi-wg@ogf.org |Subject: [occi-wg] thought on interoperability vs. integration | |Hi all, | |Thanks for a thought-provoking week of emails on the OCCI-WG list. |Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of |the issues. | |One link that I found useful was this: |http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we |find the following statement: | |--- |if Cloud technology is going to take off, there'll have to be a |competitive ecosystem; so that when you bet on a service provider, if |the relationship doesn't work out there's a way to take your business |to another provider with relatively little operational pain. Put |another way: no lock-in. ... I got all excited about this back in |January at that Cloud Interop session. Anant Jhingran, an IBM VIP, |spoke up and said "Customers don't want interoperability, they want |integration." ... "Bzzzzzzzzzt! Wrong!" I thought. But then I |realized he was only half wrong; anyone going down this road needs |integration and interoperability. |--- | |What we have been discussing is "for customers". So it is about both |of these things: | |* Interoperability ("interop") |* Integration | |This made me realise that our OCCI discussions have correctly been |about both issues. But, incorrectly, we have been commingling the |two. For example a person will say "you need X for interop" and the |reply will be "but you need Y" when in fact Y is for integration. And |vice versa. | |This is a problem for us because it leads to confusion. But it's a |symptom of a larger problem which is that interop and integration have |opposite requirements. | |* Interop is about reducing the entropy in a general purpose system |down to some level of behavioural invariance. This minimises the cost |of meeting the defined behaviour correctly, and minimises the risk of |different behaviours breaking interop. This is in turn about having a |defined behaviour (in some format) and minimising its scope. | |* Integration is about minimising the frictions between (a) the |general purpose system and (b) any specific purpose external system |that may be coupled to it. It is also, often, about maximising the |number of specific systems (or "features") that may be connected. |Since we don't know ex ante what those are, this tends to maximise |scope (eg "feature creep"). Too much specificity is the same as |complexity. | |Because interop requires minimal scope, and integration pushes for |maximal scope, they are in tension. | |They are BOTH about simplicity. Simplicity cannot be invoked on its |own, as a reason for preferring interop over integration: | |* Interop is about simplicity of definition | |* Integration is about simplicity of end use | |BUT | |* Interop simplicity is ex ante | |* Integration simplicity is ex post | |We cannot predict all ex post issues, but we can try and make simple |definitions ex ante. I argue below this means we have to have ONE |definition. | |So let's look at interop first: | |It's really important that interop be based on defined behaviour. Or |it cannot be verified. Lack of verifiability is almost always a |symptom that something is opaque, complex and ambiguous, which will |break interop in subtle ways that cannot be seen in advance (due to |the opacity). This later leads to brittleness when interop and |integration are attempted in practice; and that leads to expensive |patching at all levels, ... which is one thing that WS-* exhibits. | |NOTE 1 on WS-* ---- IMO this was not accidental, and due to focussing |on solving integration before solving interop. IIRC, the first WS-* |was an Interop committee to fix minor vendor mismatches and versioning |issues in prior WS protocols such as WSDL and SOAP. The formalisms of |WSDL and SOAP were not precise enough to spot these issues upfront, so |they were left until it was too late. | |Now let's look at the implications of having a definition of behaviour: | |You cannot define a system using multiple definitions. You have to |have one definition, preferably in a formalism that admits of |automatic conformance testing. In the case of data formats, this |leads us to three possible choices: | |1) We remove data formats from the interop profile. They are not part |of interop. Data interop is excluded. Data payloads are opaque |blobs. | |OR |2) We have one data format that can be defined unambiguously. | |OR |3) We have multiple formats and derive them from a single unambiguous |definition using canonical, verifiable, automatable, mappings. This |definition can either be in a new format which we invent (option 3a) |or it can be in an existing format (option 3b). | |I am going to rule out option (3a) on grounds of time and due to the |prevalence of existing candidates. | |Also, I think choice (1) is a complete cop-out -- I don't see how we |can claim 'useful' interop without data interop. This leaves options |(2) and (3b). BOTH of these options require one format. | |By occam's razor, option 2 is preferable on grounds of simplicity - |option (3b) does not add anything except one mapping for every extra |format that is based on the core definition. Such complexity MUST be |deferred as long as possible, and MAY be pushed into integration use |cases. The latter is preferable. Complexity MAY be added later if |necessary and then, reluctantly. | |As a general observation if you have complexity (entropy) in a system, |it is very hard to take out. It may permeate the system at all |levels. Removal of the complexity is known as 'complete refactoring'. | This is a bad outcome. Even worse is when refactoring cannot be done |for commercial reasons, eg because the systems are in production. |Then the complexity must be encapsulated and hidden. Attempts to wrap |complex behaviours in simpler packaging usually fail. | |NOTE 2 on WS-* ---- This tried to do too much and ended up trying to |wrap the complexity in simple packaging. This usually fails to work |as we have seen. | |Another general observation: | |* Integration requires interop, ie. the use of a common general |interop model; otherwise it is piecemeal and pointless. Example - the |old 'integration brokers' that had N*N connections to maintain, and |which got replaced by N connections to a common system. | |* But you can have interop without integration - it just means 'you |have a smaller audience'. This is fine because you can always grow |your audience by integrating more case with the interoperating core. |It is easier to do that when the interoperating core is |programmatically simple (as in low entropy, small code blocks, easy to |test conformance to the definition). | |I would like to add some observations from the world of AMQP... | |AMQP-1) It is a protocol with one simple common data format - it gets |that right. We leave it to integration products and services to |support data formats at the edge (eg "ESBs"). OCCI should not be like |an ESB - that is for products and services in the integration |business. | |AMQP-2) That AMQP data format is not XML - see below for more thoughts |on that. XML can be carried as an attached payload (just as it can be |in the JSON case btw). | |AMQP-3) The 0-8 and 0-9 specs took 18 months of production use to show |up the many very tiny interop bugs. We used those to create 0-9-1 |which does have interop (we have tested this) and is only 40 pages |long. This would not have been possible with a complex spec. It |would not have been possible with multiple data formats. | |AMQP-4) The 0-10 spec was focussed on supporting a lot of use cases eg |"integrate with multiple transport formats including SCTP and TCP and |UDP" and adding lots of features at all levels (eg JMS, support for |hardware, ..). That spec is really complicated and nearly 300 pages |long. Some great new ideas are in it, but it's long and in my own |opinion not supportive of two interoperating implementations. | |AMQP-5) All these painful lessons have taken the AMQP working group to |a much happier place with AMQP 1.0 which tries to simplify everything |by taking stuff out that is not needed for interop, plus refactoring |(see above my comments on how removing entropy is hard) and clean-up. | |All of the above has taken time because we did not learn from WS-*. |We did too much too fast and confused interop with integration. We |are back on track now. | |Now to the issue of data formats. I have already argued that FOR |INTEROP, there must be one definition. I argued that the best way to |do this is via a suitable single format. We can support as many |recommended ways as we like FOR INTEGRATION ... and they can be |evolved over time. | |Here is my 2c on XML. | |XML-1) XML lets you do too much - because of namespaces and xsd it is |in effect multiple formats. This is bad - we want a single, testable, |constrained definition for data interop. | |XML-2) To enforce compliance with a simple XML definition, you need to |have an extra definition of a well formed 'small' document. But |creating a new definition of a data format in XML is equivalent to |defining a new data format, the same as 'option 3a' above. But that |option was ruled out above, on grounds of time constraint.. provided |that a suitable alternative exists already (see JSON claims below). | |NOTE 3 on WS-* ---- IMHO a third reason why WS-* failed to be simple |enough to get happy, quick and wide adoption, is that (XML-1) issue |left too much data integration in the hands of customers, because |vendors could not produce useful products when XML could be all things |to all people. By not delivering on data integration, it became hard |to deliver on the promise of integration at all. And recall that |lowering integration costs was the selling point... | |So let's get INTEROP right and then do INTEGRATION. | |Interop requires: | |* Model - yes |* Data format - one definition |* Metadata - ? tbd | |As an aside - I think that GData is the nuts but it is also really an |*integration techonology*. | |Now, here are some claims about JSON: | |JSON-1) Sun have demonstrated that it is plausible as a data model for |a cloud API. That makes it plausible for us to ask: can it be used as |the core definition for interop? | |JSON-2) It is lower entropy than XML. This makes it easy to test |conformance. | |JSON-3) This means we do things the right way round -- simpler 'ex |ante' choices make it EASIER for us to extend and enhance the protocol |on a by need basis. For example carrying OVF payloads, or other |integration points. Many will be done by users. | |So my recommendation is that the best way to avoid WS-* outcomes is | |A) Use one format for interop. Do interop first. Integration later. |B) *At least for now* and *during the working draft* stage, to use JSON |C) Other formats, for now, are "integration". But we do interop first. | |OK.... I have to run off. I wrote this down in one go and don't have |time to review it. Apologies for any mistakes in the presentation. | |What do you all think? | |alexis |_______________________________________________ |occi-wg mailing list |occi-wg@ogf.org |http://www.ogf.org/mailman/listinfo/occi-wg
Alexis, I found this very enlightening. I think making the distinction between interop and integration is very important. In retrospect, I can now see that while many here have been talking about integration, I've solely been focused on interop. I agree that it is critical to get that piece functional ASAP and then begin working on the integration components. In fact, I'm even more excited now about the prospect of a very small core that could be adopted rapidly across many providers. This would go very far towards encouraging wider adoption much faster, I believe. GoGrid has folks approaching us now who are curious about who they should build to first and for many of them Amazon's APIs are the best bet because they are the de facto leader. Unfortunately, while I wish Amazon the best, as long as there is no standard this hurts everyone. We would rather have a simple interoperability API that would in effect be a "tide that lifts all boats", even Amazon. Thanks again. --Randy On 5/9/09 11:47 AM, "Alexis Richardson" <alexis.richardson@gmail.com> wrote:
Hi all,
Thanks for a thought-provoking week of emails on the OCCI-WG list. Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of the issues.
One link that I found useful was this: http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we find the following statement:
--- if Cloud technology is going to take off, there¹ll have to be a competitive ecosystem; so that when you bet on a service provider, if the relationship doesn¹t work out there¹s a way to take your business to another provider with relatively little operational pain. Put another way: no lock-in. ... I got all excited about this back in January at that Cloud Interop session. Anant Jhingran, an IBM VIP, spoke up and said ³Customers don¹t want interoperability, they want integration.² ... ³Bzzzzzzzzzt! Wrong!² I thought. But then I realized he was only half wrong; anyone going down this road needs integration and interoperability.
-- Randy Bias, VP Technology Strategy, GoGrid randyb@gogrid.com, (415) 939-8507 [mobile] BLOG: http://neotactics.com/blog, TWITTER: twitter.com/randybias
Randy, Thanks very much for your supportive words. It would be great if you could help the group by spelling out your use case and need a bit more, since you have users asking you for things right now. From what you say below, about a rising tide lifting all boats, my understanding is that you mean something like this use case: GoGrid API --- small interop core --- OtherVendor API If this is correct then how would this work in practice? It seems to me that there are three possibilities: 1. You adopt the OtherVendor API, which I assume you don't plan to do 2. You adopt the OCCI API, which would have commonality with other providers 3. You support the OCCI API for interop but provide, e.g., GG specific APIs and extensions, possibly in several styles In scenario 2 and 3, the customer can achieve interop with OtherVendor through the OCCI core. The big win comes when OtherVendor can be a large set of providers. Does this make sense? I would imagine that some folks would have a lot of extensions. For example the DMTF guys have a much broader scope than OCCI and if we integrated with them, we would not expect all their private cloud extensions to interoperate with, say, EH, who are somewhat at the other extreme of public cloud. alexis On Mon, May 11, 2009 at 12:49 AM, Randy Bias <randyb@gogrid.com> wrote:
Alexis,
I found this very enlightening. I think making the distinction between interop and integration is very important. In retrospect, I can now see that while many here have been talking about integration, I've solely been focused on interop. I agree that it is critical to get that piece functional ASAP and then begin working on the integration components.
In fact, I'm even more excited now about the prospect of a very small core that could be adopted rapidly across many providers. This would go very far towards encouraging wider adoption much faster, I believe. GoGrid has folks approaching us now who are curious about who they should build to first and for many of them Amazon's APIs are the best bet because they are the de facto leader.
Unfortunately, while I wish Amazon the best, as long as there is no standard this hurts everyone. We would rather have a simple interoperability API that would in effect be a "tide that lifts all boats", even Amazon.
Thanks again.
--Randy
On 5/9/09 11:47 AM, "Alexis Richardson" <alexis.richardson@gmail.com> wrote:
Hi all,
Thanks for a thought-provoking week of emails on the OCCI-WG list. Especially thanks to Sam, Richard, Ben and Tim for laying out a lot of the issues.
One link that I found useful was this: http://www.tbray.org/ongoing/When/200x/2009/03/16/Sun-Cloud where we find the following statement:
--- if Cloud technology is going to take off, there¹ll have to be a competitive ecosystem; so that when you bet on a service provider, if the relationship doesn¹t work out there¹s a way to take your business to another provider with relatively little operational pain. Put another way: no lock-in. ... I got all excited about this back in January at that Cloud Interop session. Anant Jhingran, an IBM VIP, spoke up and said ³Customers don¹t want interoperability, they want integration.² ... ³Bzzzzzzzzzt! Wrong!² I thought. But then I realized he was only half wrong; anyone going down this road needs integration and interoperability.
-- Randy Bias, VP Technology Strategy, GoGrid randyb@gogrid.com, (415) 939-8507 [mobile] BLOG: http://neotactics.com/blog, TWITTER: twitter.com/randybias
On 5/11/09 12:37 AM, "Alexis Richardson" <alexis.richardson@gmail.com> wrote:
1. You adopt the OtherVendor API, which I assume you don't plan to do Would love to, but only adopting Amazon's would make sense at this point given their market leadership. We've put our API under a license that allows other providers to adopt ours, but we also recognize it wasn't really designed for another provider's needs.
2. You adopt the OCCI API, which would have commonality with other providers 3. You support the OCCI API for interop but provide, e.g., GG specific APIs and extensions, possibly in several styles
Not sure about the difference between these two. It seems fairly subtle to me, but #3 is absolutely what I've been advocating all along. I think vendors still need to compete. If the core is easily extensible then we can adopt it, but extend it for our particular needs. Over time as extensions make sense to move to the core they can. A tools vendor building on top of us will get the benefits of being able to support the core easily across vendors. Implementing some support for extensions will then not be as onerous.
In scenario 2 and 3, the customer can achieve interop with OtherVendor through the OCCI core. The big win comes when OtherVendor can be a large set of providers.
Yes.
Does this make sense? I would imagine that some folks would have a
Yes, it does. --Randy -- Randy Bias, VP Technology Strategy, GoGrid randyb@gogrid.com, (415) 939-8507 [mobile] BLOG: http://neotactics.com/blog, TWITTER: twitter.com/randybias
Hi,
2. You adopt the OCCI API, which would have commonality with other providers 3. You support the OCCI API for interop but provide, e.g., GG specific APIs and extensions, possibly in several styles
Not sure about the difference between these two. It seems fairly subtle to me, but #3 is absolutely what I've been advocating all along. I think vendors still need to compete. If the core is easily extensible then we can adopt it, but extend it for our particular needs. Over time as extensions make sense to move to the core they can.
A tools vendor building on top of us will get the benefits of being able to support the core easily across vendors. Implementing some support for extensions will then not be as onerous.
Yes, Scenario #3 is our vision for OCCI API. SImilar scenarios have been achieved in the cotext if other OGF WGs. See for example DRMAA WG (http://www.drmaa.org), we created a specification and defined several bindings, and different technology providers adopted the new specification by implementing one or several bindings, reporting about their implementation in experience reports. These job management systems compete in the market and provide some extensions to access specific functionality not provided by others. In any case, several tools and applications were created on top of the new core interface (see http://www.drmaa.org/stories.php), emerging a new ecosystem around the open standard. See collection of documents (specifications, bindings and implementation reports) at http://www.drmaa.org/documents.php Let me add, that some of the main technology providers were very reluctant to implement the standard, buy when they realized that the rest of providers were implementing the API, they started to implement the bindings. See implementations at http://www.drmaa.org/implementations.php Cheers, Ignacio
Ignacio, Randy, Thanks! So this is good. We agree that *extensions* and other bindings can be defined at the edges of the system, where the provider controls the interface and has integration needs specific to them. Some of the integration and extension mechanisms could even be provided by common libraries (eg XML tooling if the core format is not XML, or vice versa). But we still keep to one interoperable core. Any other comments on this point? alexis On Wed, May 13, 2009 at 7:20 AM, Ignacio Martin Llorente <llorente@dacya.ucm.es> wrote:
Hi,
2. You adopt the OCCI API, which would have commonality with other providers 3. You support the OCCI API for interop but provide, e.g., GG specific APIs and extensions, possibly in several styles
Not sure about the difference between these two. It seems fairly subtle to me, but #3 is absolutely what I've been advocating all along. I think vendors still need to compete. If the core is easily extensible then we can adopt it, but extend it for our particular needs. Over time as extensions make sense to move to the core they can.
A tools vendor building on top of us will get the benefits of being able to support the core easily across vendors. Implementing some support for extensions will then not be as onerous.
Yes, Scenario #3 is our vision for OCCI API. SImilar scenarios have been achieved in the cotext if other OGF WGs. See for example DRMAA WG (http://www.drmaa.org), we created a specification and defined several bindings, and different technology providers adopted the new specification by implementing one or several bindings, reporting about their implementation in experience reports. These job management systems compete in the market and provide some extensions to access specific functionality not provided by others. In any case, several tools and applications were created on top of the new core interface (see http://www.drmaa.org/stories.php), emerging a new ecosystem around the open standard.
See collection of documents (specifications, bindings and implementation reports) at http://www.drmaa.org/documents.php
Let me add, that some of the main technology providers were very reluctant to implement the standard, buy when they realized that the rest of providers were implementing the API, they started to implement the bindings. See implementations at http://www.drmaa.org/implementations.php
Cheers,
Ignacio
Alexis, This is an important distinction, thank you for highlighting and explaining. Like Randy, my personal goals for OCCI are almost entirely for interop. Today every cloud provider invents their own API to cover essentially identical functionality, and every cloud customer or tools vendor integrates independently with every cloud provider (c.f. Alexis' old employer CohesiveFT integrating with Amazon, FlexiScale, ElasticHosts, Sun, etc.) If all we get out of OCCI is an interoperable API which multiple cloud providers actually adopt, then I would consider that a massive win for the customers and the ecosystem and hence for the cloud providers themselves. I believe that making this interoperable API as minimal as possible will speed adoption and make it more likely that different implementations actually interoperate. Cheers, Richard. Use case: http://forge.ogf.org/sf/wiki/do/viewPage/projects.occi-wg/wiki/SingleTechnic...
Richard, I think we should define more clearly, by examples, what it means for two implementations to interoperate and how they would use OCCI to do so, yet retain the right to extend their user APIs arbitrarily. Any thoughts on an example for - say - EH and EC2? a On Mon, May 11, 2009 at 6:12 PM, Richard Davies <richard@daviesmail.org> wrote:
Alexis,
This is an important distinction, thank you for highlighting and explaining.
Like Randy, my personal goals for OCCI are almost entirely for interop.
Today every cloud provider invents their own API to cover essentially identical functionality, and every cloud customer or tools vendor integrates independently with every cloud provider (c.f. Alexis' old employer CohesiveFT integrating with Amazon, FlexiScale, ElasticHosts, Sun, etc.)
If all we get out of OCCI is an interoperable API which multiple cloud providers actually adopt, then I would consider that a massive win for the customers and the ecosystem and hence for the cloud providers themselves.
I believe that making this interoperable API as minimal as possible will speed adoption and make it more likely that different implementations actually interoperate.
Cheers,
Richard.
Use case: http://forge.ogf.org/sf/wiki/do/viewPage/projects.occi-wg/wiki/SingleTechnic... _______________________________________________ occi-wg mailing list occi-wg@ogf.org http://www.ogf.org/mailman/listinfo/occi-wg
I think we should define more clearly, by examples, what it means for two implementations to interoperate and how they would use OCCI to do so, yet retain the right to extend their user APIs arbitrarily.
Any thoughts on an example for - say - EH and EC2?
EH and GG is probably the right example, given that Randy is on this list! What it would mean for me is that the same client code works against both clouds for core functionality. For example, if we take the Simple JSON Rendering which I just posted http://forge.ogf.org/sf/wiki/do/viewPage/projects.occi-wg/wiki/SimpleJSONRen... then I'd like to see a HTTP GET against our respective endpoints returning the same format JSON list of nouns, with the same names for the same core attributes (id, title, cores, etc.). I'd also like to see the same core verbs at the same actuator URLs - e.g. http://api.example.com/<id>/stop would stop a server on both clouds. This doesn't sound much, but it's a long way ahead of where we are today! As an example of vendor extensions, EH supports VNC access to servers, and GG supports load balancers. This likely means that EH would have a couple of extra attributes on a server (e.g. the VNC password), and GG would have a couple of extra attributes on a network (load balancer configuration) + possibly some verbs. We'd reserve part of the namespace for nouns, verbs and attributes for these vendor extensions. For example, our VNC attribute on a server might be: vx_com.elastichosts_vncpassword and Randy's load balancer attributes on a network might be within: vx_com.gogrid_XXXXX with any verbs he needs within http://api.example.com/<id>/vx/com.gogrid/XXXX Randy - would this kind of scheme work for you? Cheers, Richard.
On Mon, May 11, 2009 at 7:25 PM, Richard Davies <richard@daviesmail.org>wrote:
We'd reserve part of the namespace for nouns, verbs and attributes for these vendor extensions. For example, our VNC attribute on a server might be:
vx_com.elastichosts_vncpassword
and Randy's load balancer attributes on a network might be within:
vx_com.gogrid_XXXXX
with any verbs he needs within http://api.example.com/ <id>/vx/com.gogrid/XXXX
Randy - would this kind of scheme work for you?
The point of vendor extensions was for esoteric cruft that is unlikely to ever be needed by any two vendors (the example I gave was com.cisco.cdp). Attributes that are interesting to multiple vendors (e.g. vncpassword) can be added to the relevant registry after discussion... though for this specific example I was leaning towards a "console" link relation with credentials embedded in the url or link attributes. Sam
participants (7)
-
Alexis Richardson -
Gary Mazz -
Ignacio Martin Llorente -
Krishna Sankar (ksankar) -
Randy Bias -
Richard Davies -
Sam Johnston