Ok so for the record, here's a thread between Ben Black and myself discussing XML v JSON:

Sam,

I'm hoping there is more to the various decisions than is outlined in this post. "Everyone is doing it" is illustrative, but not convincing.

XML drags along an awful lot of baggage, which has resulted in many folks using lighter-weight formats like JSON. ATOM, in turns, lards still more baggage into the mix, again, without any clear advantage in this application. Finally, OAuth is very much in flux, as the recent security incident makes painfully clear, and, once again, its compelling advantage in this scenario is not covered above.

The Sun Cloud API work is extremely solid and goes towards simplicity in almost every choice. Why are you ignoring it?

Ben


G'day Ben,

Au contraire, we are paying *very* close attention to the Sun Cloud APIs and originally wanted to draw from them heavily were it not for a disagreement with the OGF powers that be over Creative Commons licensing. Nonetheless we have learnt a lot from them about high level concepts such as having a single entry point and "buttons" for state changes and the like. We are also looking at simple text based APIs like those used at ElasticHosts and GoGrid (and of course Amazon).

Sun's decision to use JSON no doubt stems from the fact that the API was previously consumed almost exclusively by the Q-Layer web interface - it is not so clear that it is the best choice when you stray from the web and native JavaScript environments, particularly when you want to (safely) support extensibility... XML allows users to extend the API as necessary in a fashion that can be validated and will not break other extensions.

Another reasonably firm requirement is that it be possible/trivial to script the interface for e.g. sysadmin tasks... for this we need a simple text based protocol anyway so we already have to support more than one output format (which is increasingly common today, even if not particularly elegant).

Finally, given that technology is only half the battle (the other half being marketing/politics), it's necessary to cater to the needs of enterprise users who are well used to horrors like WS-* and who will find a lightweight XML layer refreshingly simple in comparison to the status quo. More importantly though, enterprise requirements like validation, signing, encryption, etc. are natively supported by XML and not (easily) supported for the alternatives.

Oh and we are not at all wedded to OAuth - any HTTP authentication mechanism will do (though it may make sense for us to limit this somewhat for interoperability).

For more specific details check out the wiki and/or get involved yourself.

Sam


Sam,

The onus is on you to justify the additional complexity. Rejecting simpler solutions because they are unfamiliar is dangerous.

While JSON might have initially entered the protocol via Q-Layer, the current state of the work is miles from those origins. JSON has similarly moved well beyond its origins as something used by Javascript (see CouchDB for a great example). Finally, it is a simple, text-based system, far simpler than XML, and I have recent, painful experience in working in JSON and XML simultaneously for systems management. It's a no brainer: JSON wins in that space because it is easier to write, easier to read, and a lot harder to make poor, structural decisions using it.

Enterprises currently use a lot of XML because that is what vendors like Microsoft foisted on them in the name of "standards". If you adopt all the existing validation and security stack for XML, you are so close to WS-* as to be indistinguishable. This work will fail if all it does is produce "WS-Cloud", and that is very much the path on which you are placing it with these choices. You can do better.

Ben


Ben,

Thanks for your ongoing interest in this topic - your opinion is valued (and not just by me) so I would very much like us to be on the same wavelength.

I've since been looking at CouchDB which is a very interesting project given one of the design patterns I've been keeping in mind is a database driven controller (that is, OCCI being a direct interface to the database) with triggers firing off e.g. AMQP messages for state changes et al. Erlang is the obvious choice for such infrastructure and it seems like CouchDB has done a lot of the work already. Of course it's well out of scope for the spec itself but it's useful to consider how it might be realised.

In terms of the XML vs JSON bikeshed (and I say bikeshed because once you have to reach for library, as is the case for both XML and JSON, you may as well use ASN.1 on the wire for all it matters), I have many reasons for preferring XML, not the least of which is the ease at which it can be trivially converted to $PREFERRED_FORMAT using tools and expertise that already exists (the same cannot necessarily be said for the reverse, or between other formats). Very few enterprise developers have JSON exposure, let alone experience, and it's not surprising given that essential features such as schemas and validation are still in a state of flux. Incidentally one thing I *do* like about YAML (and to a lesser extent, JSON) is the relative ease at which even complex data structures can be represented - this does provide more rope with which to hang oneself though.

This brings us to the specifics of the proposal... XML can be [ab]used in so many different ways it's worth pointing out that the proposal uses it in the simplest way possible. Atom giving us just enough structure to link objects together, categorise them and provide metadata (such as owners, etags for caching, etc.) - as Andy observed it's more of a meta-model than anything else and we found that much of what we were trying to do was reinventing the [Atom] wheel. The actual content itself (cpu, memory, etc.) is simple, easily readable, and in all cases so far, flat... as such it's more a question of whether we use angle brackets or curly braces :)

So why go for angle brackets (XML) over curly braces (JSON) given most of the action is going on in the latter camp? In addition to the reasons given above, Google. That is, if we essentially rubber stamp GData (a well proven protocol) in the form of OCCI core then it's just a matter of changing (or supporting multiple) namespaces and we may well be able to get Google on board along with a raft of running code. I've already been discussing this with Google's API gurus and product managers this week and while there's no commitment there is some amount of interest.

As for WS-Cloud, I assure you this is nothing like it... I wonder when reading the WS-[death]* specs whether they weren't deliberately obfuscated in some cases and even the better ones like WS-Agreement are [over]complicated, in this case running out to dozens of pages.

Sam

On Tue, May 5, 2009 at 3:33 AM, Sam Johnston <samj@samj.net> wrote:
Morning all,

I'm going to break my own rules about reposting blog posts because this is very highly relevant, it's 03:30am already and I'm traveling again tomorrow. The next step for us is to work out what the protocol itself will look like on the wire, which is something I have been spending a good deal of time looking at over many months (both analysing existing efforts and thinking of "blue sky" possibilities).

I am now 100% convinced that the best results are to be had with a variant of XML over HTTP (as is the case with Amazon, Google, Sun and VMware) and that while Google's GData is by far the most successful cloud API in terms of implementations, users, disparate services, etc. Amazon's APIs are (at least for the foreseeable future) a legal minefield. I'm also very interested in the direction Sun and VMware are going and have of course been paying very close attention to existing public clouds like ElasticHosts and GoGrid (with a view to being essentially backwards compatible and sysadmin friendly).

I think the best strategy by a country mile is to standardise OCCI core protocol following Google's example (e.g. base it on Atom and/or AtomPub with additional specs for search, caching, etc.), build IaaS extensions in the spirit of Sun/VMware APIs and support alternative formats including HTML, JSON and TXT via XML Stylesheets (e.g. occi-to-html.xsl, occi-to-json.xsl and occi-to-text.xsl). You can see the basics in action thanks to my Google App Engine reference implementation at http://occitest.appspot.com/ (as well as HTML, JSON and TXT versions of same), KISS junkies bearing in mind that this weighs in under 200 lines of python code! Of particular interest is the ease at which arbitrarily complex [X]HTML interfaces can be built directly on top of OCCI (optionally rendered from raw XML in the browser itself) and the use of the hCard microformat as a simple demonstration of what is possible.

Anyway, without further ado:

Is OCCI the HTTP of Cloud Computing?
http://samj.net/2009/05/is-occi-http-of-cloud-computing.html

The Web is built on the Hypertext Transfer Protocol (HTTP), a client-server protocol that simply allows client user agents to retrieve and manipulate resources stored on a server. It follows that a single protocol could prove similarly critical for Cloud Computing, but what would that protocol look like?

The first place to look for the answer is limitations in HTTP itself. For a start the protocol doesn't care about the payload it carries (beyond its Internet media type, such as text/html), which doesn't bode well for realising the vision of the [Semantic] Web as a "universal medium for the exchange of data". Surely it should be possible to add some structure to that data in the simplest way possible, without having to resort to carrying complex, opaque file formats (as is the case today)?

Ideally any such scaffolding added would be as light as possible, providing key attributes common to all objects (such as updated time) as well as basic metadata such as contributors, categories, tags and links to alternative versions. The entire web is built on hyperlinks so it follows that the ability to link between resources would be key, and these links should be flexible such that we can describe relationships in some amount of detail. The protocol would also be capable of carrying opaque payloads (as HTTP does today) and for bonus points transparent ones that the server can seamlessly understand too.

Like HTTP this protocol would not impose restrictions on the type of data it could carry but it would be seamlessly (and safely) extensible so as to support everything from contacts to contracts, biographies to books (or entire libraries!). Messages should be able to be serialised for storage and/or queuing as well as signed and/or encrypted to ensure security. Furthermore, despite significant performance improvements introduced in HTTP 1.1 it would need to be able to stream many (possibly millions) of objects as efficiently as possible in a single request too. Already we're asking a lot from something that must be extremely simple and easy to understand.

XML

It doesn't take a rocket scientist to work out that this "new" protocol is going to be XML based, building on top of HTTP in order to take advantage of the extensive existing infrastructure. Those of us who know even a little about XML will be ready to point out that the "X" in XML means "eXtensible" so we need to be specific as to the schema for this assertion to mean anything. This is where things get interesting. We could of course go down the WS-* route and try to write our own but surely someone else has crossed this bridge before - after all, organising and manipulating objects is one of the primary tasks for computers.

Who better to turn to for inspiration than a company whose mission it is to "organize the world's information and make it universally accessible and useful", Google. They use a single protocol for almost all of their APIs, GData, and while people don't bother to look under the hood (no doubt thanks to the myriad client libraries made available under the permissive Apache 2.0 license), when you do you may be surprised at what you find: everything from contacts to calendar items, and pictures to videos is a feed (with some extensions for things like searching and caching).

OCCI

Enter the OGF's Open Cloud Computing Interface (OCCI) whose (initial) goal it is to provide an extensible interface to Cloud Infrastructure Services (IaaS). To do so it needs to allow clients to enumerate and manipulate an arbitrary number of server side "resources" (from one to many millions) all via a single entry point. These compute, network and storage resources need to be able to be created, retrieved, updated and deleted (CRUD) and links need to be able to be formed between them (e.g. virtual machines linking to storage devices and network interfaces). It is also necessary to manage state (start, stop, restart) and retrieve performance and billing information, among other things.

The OCCI working group basically has two options now in order to deliver an implementable draft this month as promised: follow Amazon or follow Google (the whole while keeping an eye on other players including Sun and VMware). Amazon use a simple but sprawling XML based API with a PHP style flat namespace and while there is growing momentum around it, it's not without its problems. Not only do I have my doubts about its scalability outside of a public cloud environment (calls like 'DescribeImages' would certainly choke with anything more than a modest number of objects and we're talking about potentially millions) but there are a raft of intellectual property issues as well:
While I wish the guys at Eucalyptus and Canonical well and don't have a bad word to say about Amazon Web Services, this is something I would be bearing in mind while actively seeking alternatives, especially as Amazon haven't worked out whether the interfaces are IP they should protect. Even if these issues were resolved via royalty free licensing it would be very hard as a single vendor to compete with truly open standards (RFC 4287: Atom Syndication Format and RFC 5023: Atom Publishing Protocol) which were developed at IETF by the community based on loose consensus and running code.

So what does all this have to do with an API for Cloud Infrastructure Services (IaaS)? In order to facilitate future extension my initial designs for OCCI have been as modular as possible. In fact the core protocol is completely generic, describing how to connect to a single entry point, authenticate, search, create, retrieve, update and delete resources, etc. all using existing standards including HTTP, TLS, OAuth and Atom. On top of this are extensions for compute, network and storage resources as well as state control (start, stop, restart), billing, performance, etc. in much the same way as Google have extensions for different data types (e.g. contacts vs YouTube movies).

Simply by standardising at this level OCCI may well become the HTTP of Cloud Computing.