And finally the conclusion: Is HTTP the HTTP of cloud computing?<http://samj.net/2009/05/is-http-http-of-cloud-computing.html>Ok so after asking Is OCCI the HTTP of cloud computing?<http://samj.net/2009/05/is-occi-http-of-cloud-computing.html>I realised that the position may have already been filled and that the question was more Is AtomPub already the HTTP of cloud computing?<http://samj.net/2009/05/is-atompub-already-http-of-cloud.html> After all my strategy for OCCI was to follow Google's example with GData by adding some necessary functionality (a search interface, caching directives, resource-specific attributes, etc.). Most of the heavy lifting was actually being done by AtomPub, thus avoiding a huge amount of tedious and error-prone protocol writing (around 20,000 words of it) - something which OGF and the OCCI working group isn't really geared up for anyway. This is clearly a workable and well-proven approach as it as been adopted strategically by both Microsoft and Google and also tactically by Salesforce and IBM, among others. Best of all adding things like queries<http://code.google.com/apis/gdata/docs/2.0/reference.html#Queries>and versioning<http://code.google.com/apis/gdata/docs/2.0/reference.html#ResourceVersioning>is a manageable workload while starting from scratch is most certainly not. But what if there were an easier way? Recall that the problem we are trying to solve is exposing a flexible interface to an arbitrarily large collection of interconnected compute, storage and network resources. We need to be able to describe and manipulate the resources (CRUD), associate them with each other via rich links (e.g. links with attributes like local identifiers - eth0, sda, etc.) and change their state (start, stop, restart, etc.), among other things. Representational State Transfer (REST) Actually we're not talking about exposing the resources themselves (that would be impossible) but various *representations* of those resources - like Plato's shadows on the cave walls<http://en.wikipedia.org/wiki/Allegory_of_the_cave>- this is the "REpresentational" in "REpresentational State Transfer (REST)". There's an infinite number of possible representations so it's impossible to try to capture them all now, but here's some examples: - An Open Virtualisation Format (OVF) serialisation of a compute resource - A platform-specific descriptor file (e.g. VMX) - A complete archive of the virtual machine with its dependencies (OVA) - A graphical image of the console at a given point in time ('snapshot') - A video stream of the console for archiving/audit purposes (ala Citrix's Project Iris<http://www.brianmadden.com/blogs/brianmadden/archive/2005/06/08/project-iris-becoming-a-reality-tech-preview-coming-next-month.aspx> ) - The console itself (e.g. SSH, ICA, RDP, VNC) - Build documentation (e.g. PDF, ODF) - Esoteric enterprise requirements (e.g. NMS configuration) It doesn't take a rocket scientist to spot the correlation between this and HTTP's existing content negotiation functionality (whereby a client can ask for a specific representation of a given resource - e.g. HTML vs PDF) so this is already pretty much solved for us (see HTTP's Accept: header<http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>for the details). For bonus points this information should be exposed in the URI as it's not always possible or convenient to set headers ala: - http://example.com/.atom (using filename extensions) - http://example.com/;content-type=text/html (using the full Internet media type) Web Linking But what about the links? As I explained yesterday<http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>the web is built on links embedded in HTML documents using the A tag. Atom also provides enhanced linking functionality via the LINK element, where it is also possible to specify content types, languages, etc. In this case however we want to allow resources to be arbitrary types and more often than not we won't have the ability to link within the payload itself. This leaves us with two options: put the links in the payload anyway by relying on a meta-model like Atom (or one we roll ourselves) or find some way to represent them within HTTP itself. Enter HTTP headers which are also extensible and, as it turns out, in the process of being extended (or at least refined) to handle this very requirement by fellow down under, Mark Nottingham <http://www.mnot.net/>. See the "Web Linking" IETF Internet-Draft (draft-nottingham-http-link-header<https://datatracker.ietf.org/drafts/draft-nottingham-http-link-header/>, at the time of writing version 05<http://www.ietf.org/internet-drafts/draft-nottingham-http-link-header-05.txt>) for the nitty gritty details and the ietf-http-wg list<http://lists.w3.org/Archives/Public/ietf-http-wg/>for some <http://lists.w3.org/Archives/Public/ietf-http-wg/2009AprJun/0211.html> current<http://lists.w3.org/Archives/Public/ietf-http-wg/2009AprJun/0196.html> discussions<http://lists.w3.org/Archives/Public/ietf-http-wg/2009AprJun/0103.html>. Basically it clarifies the existing Link: headers<http://tools.ietf.org/html/rfc2068#section-19.6.2.4>and the result looks something like this: Link: <http://example.com/TheBook/chapter2>; rel="previous"; title="previous chapter" The Link: header itself is also extensible so we can faithfully represent our model by adding e.g. the local device name when linking storage and network resources to compute resources and other requisite attributes. It would be helpful if the content-type were also specified (Atom allows for multiple links of the same relation provided the content-type differs for example) but language is already covered by HTTP (it doesn't seem useful to advertise French links to someone who already asked to speak English). It's also interesting to note that earlier versions of the HTTP RFCs actually [poorly] specified both the Link: headers<http://tools.ietf.org/html/rfc2068#section-19.6.2.4>as well as LINK <http://tools.ietf.org/html/rfc2068#section-19.6.1.2> and UNLINK<http://tools.ietf.org/html/rfc2068#section-19.6.1.3>methods for maintaining links between web resources. John Pritchard had a crack at clarification in the Efficient HyperLink Maintenance for HTTP<http://ftp.ics.uci.edu/pub/ietf/http/draft-pritchard-http-links-00.txt>I-D but like most I-Ds this one seems to have died after 6 months, and with it the methods themselves. It seems to me that adding HTTP methods at this time is a drastic (and almost certainly infeasible) action, especially for something that could just as easily be accomplished via headers ala Set-Cookie: (too bad the I-D doesn't specify how to add/delete/modify links!). In the simplest sense a Link: header appearing in a PUT or POST could replace the existing one(s) but something more elegant for acting on individual links would be nice - probably a discussion worth having on the ietf-http-wg list <http://lists.w3.org/Archives/Public/ietf-http-wg/>. Organisation of Information Looking back to Atom for a second we're still missing some key functionality: - Atom id -> HTTP URL - Atom updated -> HTTP Last-Modified: Header - Atom title and summary -> Atom/HTTP Slug: Header<http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.2>or equivalent - Atom link -> HTTP Link: Header - Atom category -> ??? Houston, we have a problem. OCCI use cases range from embedded hypervisors exposing a single resource to a single entry-point for an entire enterprise or the "Great Global Grid" - we need a way to organise, categories and search for the information, likely including: - Free text search via a Google-style "?q=firewall" syntax - Taxonomy <http://en.wikipedia.org/wiki/Taxonomy> via categories (already done <http://www.ibm.com/developerworks/xml/library/x-tipatom4.html> for Atom) for things like "Operating System" and "Data Center" - Folksonomy <http://en.wikipedia.org/wiki/Folksonomy> via [user] tags (already done <http://edward.oconnor.cx/2007/02/representing-tags-in-atom> for Atom and bearing in mind that tag spaces<http://microformats.org/wiki/rel-tag#Tag_Spaces>are cool) for things like "testlab" Fortunately the good work already done in this area for Atom would be realatively easy to port to a Category: HTTP header, following the Link:header example above. In the mean time a standard search interface (including category support) is trivial and thanks to Google, already done<http://code.google.com/apis/gdata/docs/2.0/reference.html#Queries> . Structured Data Formats HTML also resolves another pressing issue - what format to use for submitting key-value pairs (which constitutes a large part of what we need to do with OCCI). It gives us two options: - application/x-www-form-encoded<http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1>which is simple but quickly gets messy with encoding and non-ASCII characters - multipart/form-data<http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.2>which is less efficient but will handle pretty much whatever you throw at it (including large files) The advantages of being able to create a resource from a web form simply by POSTing to the collection of resources (e.g. http://example.com/compute), and with HTML 5 by PUTting the resource in place directly (e.g. http://example.com/compute/<uuid>) are immediately obvious. Not only does this help make the human and programmable web one and the same (which in turn makes it much easier for developers/users to kick the tyres and understand the API) but it means that scripting even advanced tasks with curl/wget would be trivial. Plus there's no place for time-wasting religious arguments about angle brackets (XML) over curly braces (JSON). RESTful State Machines Something else which has not sat well with me until I spent the weekend ingesting RESTful Web Services <http://oreilly.com/catalog/9780596529260/>book (by Leonard Richardson <http://www.crummy.com/> and Sam Ruby<http://intertwingly.net/blog/>) was the "actuator" concept we picked up from the Sun Cloud APIs. This breaks away from RESTful principles by exposing an RPC-style API for triggering state changes (e.g. start, stop, restart). Granted it's an improvement on the alternative (GETting a resource and PUTting it back with an updated state) as Tim Bray explains in RESTful Casuistry<http://www.tbray.org/ongoing/When/200x/2009/03/20/Rest-Casuistry>(to which Roy Fielding <http://roy.gbiv.com/untangled/2009/it-is-okay-to-use-post> and Bill de hÓra <http://www.dehora.net/journal/2009/02/03/just-use-post/>also responded), but it still "feels funky". Sure it doesn't make any sense to try to "force" a monitored status<http://roy.gbiv.com/untangled/2009/it-is-okay-to-use-post#comment-966>to some other value (for example setting a "state" attribute to "running"), especially when we can't be sure that's the state we'll get to (maybe there will be an error or the transition will be dependent on some outcome over which we have no control). Similarly it doesn't make much sense to treat states as nouns, for example adding a "running" state to a collection of states (even if a resource can be "running" and "backing up" concurrently). But is using URLs as "buttons" representing verbs/transitions the best answer? What makes more sense [to me] is to request a transition and check back for updates (e.g. by polling or HTTP server push<http://en.wikipedia.org/wiki/Push_technology>). If it's RESTful to POST comments to an article (which in addition to its own contents acts as a collection of zero or more comments) then POSTing a request to change state to a [sub]resource also makes sense. As a bonus these can be parametrised (for example a "resize" request can be accompanied with a "size" parameter and a "stop" request sent with clarification as to whether an "ACPI Off" or "Pull Cord" is required). Transitions that take a while, like "format" on a storage resource, can simply return HTTP 201 Accepted so we've got support for asynchronous actions as well - indeed some requests (e.g. "backup") may not even be started immediately. We may also want to consider using something like Post Once Exactly (POE)<http://www.mnot.net/drafts/draft-nottingham-http-poe-00.txt>to ensure that requests like "restart" aren't executed repeatedly and that we can cancel requests that the system hasn't had a chance to deal with yet. Exactly how this should look in terms of URL layout I'm not sure (perhaps http://example.com/<resource>/requests) but being able to enumerate the possible actions as well as acceptable parameters (e.g. an enum for variations on "stop" or a range for "resize") would be particularly useful for clients. Collections This is all well and good for individual resources, but collections are still a serious problem. There are many use cases which involve retrieving an arbitrarily large number of resources and making a HTTP request for each (as well as requests for enumeration etc.) doesn't make sense. More importantly, it doesn't scale - particularly in enterprise environments where requests via proxies and filters can suffer from high latency (if not low bandwidth). One potential solution is to strap multiple HTTP message entities together as a multipart document, but that's hardly clean and results in some hairy coding on the client side (e.g. manual manipulation of HTTP messages that would otherwise be fully automated). The best solution we currently have for this problem (as evidenced by widespread deployment) is AtomPub so I'm still fairly sure it's going to have to make an appearance somewhere, even if it doesn't wrap all of the resources by default.