June 2009 - occi-wg - lists.ogf.org

OCCI Conference Call has just started...
by Sam Johnston 24 Jun '09

24 Jun '09

Afternoon all, The weekly OCCI conf call<http://forge.ogf.org/sf/wiki/do/viewPage/projects.occi-wg/wiki/Teleconferen…>is starting and I wanted to share with you a start on the documentation of the API (attached). Cheers, Sam

1 0

Finished first pass over FeatureMatrix
by Sam Johnston 15 Jun '09

15 Jun '09

Morning all, I've just finished the first pass of the OCCI FeatureMatrix<http://forge.ogf.org/sf/wiki/do/viewPage/projects.occi-wg/wiki/FeatureMatrix>which compares APIs on a functional (rather than per-function) basis. If we want to be an implementation candidate for these APIs then we're going to have to be capable of exposing as much of the existing functionality as possible - which is to say that we're going to end up with something of a superset (hence the solid ticks in the OCCI column, except for a few things we haven't thought about yet). Some of the suggested APIs were dropped because they are not comparable or simply don't exist (e.g. Mosso CloudSites). We'll add new rows as interesting features are raised and new columns for new APIs. Cheers, Sam

7 13

Fwd: Clarifications on Web Linking with HTTP
by Sam Johnston 15 Jun '09

15 Jun '09

Here's a related discussion at IETF about Web Linking, which has allowed us to vastly simplify the descriptor format by moving linking meta-data to the HTTP headers (where it arguably belongs). Sam ---------- Forwarded message ---------- From: Sam Johnston <samj(a)samj.net> Date: Mon, Jun 15, 2009 at 5:27 AM Subject: Clarifications on Web Linking with HTTP To: apps-discuss(a)ietf.org Morning all, The HTTP Link: header enables web linking without hypermedia - that is, arbitrary content types can be linked (with attriubtes) out-of-band rather than within the payload (e.g. HTML) itself. This enables the use of HTTP as a meta-model (at least for individual resources) without having to resort to Atom, which is potentially great news for RESTful APIs. I am currently working on a real world application of Marks' Web Linking I-D[1] (OGF's Open Cloud Computing Interface - http://www.occi-wg.org/) and require clarification on a few points (which may want to end up in the I-D). - First and foremost, in the absence of the LINK and UNLINK verbs originally defined in RFC 2068[2] but specifically omitted from RFC 2616[3], what is the preferred mechanism for manipulating these links via HTTP? It appears that this header is intended for GET requests only, but presumably specifying it in POST and PUT requests would be one option that avoids the creation of [not so] "new" verbs (bearing in mind that short of accepting Link: headers from empty POST/PUT requests, it would be necessary to GET and then PUT the entire payload to update links - twice if they were reciprocal). While there was an attempt a dozen years ago to better define the relevant HTTP verbs[4], it strikes me as more sensible to follow the example of the Set-Cookie: header for this rather than WebDAV's example of creating new verbs (even if we've seen them before) but you guys are the experts. - Another concern with an arbitrary number of links is that arbitrary string length limits may be imposed by user agents, as they are with URLs. This should not be a problem where there is one link per header, but it may be where the headers are concatenated as described in RFC 2616[5]. This is a double edged sword however as some user agents have only recently added support for multiple headers of the same type[6] and it remains a problem for some[7]. - The introduction of a link relation registry at IANA makes a lot of sense, though it would be nice if these were common for HTTP, HTML, Atom and other places links appear. Perhaps namespaces (e.g. "atom:service" or "occi.state.restart") would be useful here so as to enable significantly more future extensibility. - It seems useful to be able to (optionally) specify the type (as in content type rather than relation type) of a given link, as is the case for Atom. That said, this also seems somewhat redundant with HTTP Content Negotiation, but implementations that choose to support the "type" attribute may gain some performance and usability advantages from doing so. The matter of whether this information belongs in URIs (and if so, which side of the '?') or in HTTP headers (or both) is still not clear to me as there are pros and cons of each - perhaps the relation type is more suitable (or both?) as it's often not possible to unambigously determine the relation type from the content type (consider modeling people where both fingerprint and portrait representations may exist in image/png format). To be more specific about the requirements, the API models cloud infrastructure services (IaaS) and has three main nouns (compute, network, storage) which need to be associated with each other with attributes on the links (e.g. a compute resource having a network resource associated with a local identifier attribute of "eth0"). Using Atom as the meta-model worked fine (as evidenced by GData) but it now seems possible - at least for individual resources - with HTTP. Cheers, Sam 1. http://tools.ietf.org/html/draft-nottingham-http-link-header-05 2. http://tools.ietf.org/html/rfc2068#section-19.6.1 3. http://tools.ietf.org/html/rfc2616#section-19.6.3 4. http://ftp.ics.uci.edu/pub/ietf/http/draft-pritchard-http-links-00.txt 5. http://tools.ietf.org/html/rfc2616#section-4.2 6. http://www.mail-archive.com/bug-wget@gnu.org/msg00076.html 7. http://bugs.python.org/issue1660009

1 0

Forget XML and JSON... Welcome to Avro!
by Sam Johnston 12 Jun '09

12 Jun '09

Only joking... though this could be useful for "internal" use. With the current thinking it's possible to use OCCI both for servicing clients and the nodes themselves - that is, machines can "PUT" their current status and so on, in which case you want the most efficient wire format possible. Optional of course, alongside TXT, JSON and XML. Sam ---------- Forwarded message ---------- From: Mark Nottingham <mnot(a)mnot.net> Date: Fri, Jun 12, 2009 at 3:11 PM Subject: FYI: Apache Avro To: apps-discuss(a)ietf.org Yet another data representation / schema / RPC: http://people.apache.org/~cutting/avro/<http://people.apache.org/%7Ecutting/avro/> -- Mark Nottingham http://www.mnot.net/ _______________________________________________ Apps-Discuss mailing list Apps-Discuss(a)ietf.org https://www.ietf.org/mailman/listinfo/apps-discuss

1 0

Fwd: [hybi] draft-loreto-http-bidirectional
by Sam Johnston 12 Jun '09

12 Jun '09

Some of us were talking about handling events, monitoring requests in progress, etc. the other day. This stuff could be useful for OCCI. Sam ---------- Forwarded message ---------- From: Salvatore Loreto <salvatore.loreto(a)ericsson.com> Date: Thu, Jun 11, 2009 at 10:39 PM Subject: Fwd: [hybi] draft-loreto-http-bidirectional To: apps-discuss(a)ietf.org -------- Original Message -------- Hi there, we have just submitted the following draft: "Best Practices for the Use of Long Polling and Streaming in Bidirectional HTTP" The draft is a first tentative to describe what does it means to use HTTP for Bidirectional communication and how to better use HTTP, as it exists today, to enable such "bidirectional HTTP" using "long polling" and "HTTP streaming" mechanisms. until it will appear on the IETF website you can find it here: http://xmpp.org/internet-drafts/draft-loreto-http-bidirectional-00.txt we are looking forward to receive your comments, suggestions and feedbacks on it we would like to remind that the preferred venue for discussion of this document is the hybi(a)ietf.org mailing list (visit <https://www.ietf.org/mailman/listinfo/hybi> for further information). best regards Sal -------- Original Message -------- Subject: New Version Notification for draft-loreto-http-bidirectional-00 Date: Thu, 11 Jun 2009 13:15:57 -0700 (PDT) From: IETF I-D Submission Tool <idsubmission(a)ietf.org> To: Salvatore.Loreto(a)ericsson.com CC: salvatore.loreto@ericsson.com,psaintan@cisco.com,gregw@webtide.com, stefano.salsano(a)uniroma2.it A new version of I-D, draft-loreto-http-bidirectional-00.txt has been successfuly submitted by Salvatore Loreto and posted to the IETF repository. Filename: draft-loreto-http-bidirectional Revision: 00 Title: Best Practices for the Use of Long Polling and Streaming in Bidirectional HTTP Creation_date: 2009-06-11 WG ID: Independent Submission Number_of_pages: 16 Abstract: There is widespread interest in using the Hypertext Transfer Protocol (HTTP) to enable asynchronous or server-initiated communication from a server to a client as well as from a client to a server. This document describes how to better use HTTP, as it exists today, to enable such "bidirectional HTTP" using "long polling" and "HTTP streaming" mechanisms. The IETF Secretariat. _______________________________________________ hybi mailing list hybi(a)ietf.org https://www.ietf.org/mailman/listinfo/hybi _______________________________________________ Apps-Discuss mailing list Apps-Discuss(a)ietf.org https://www.ietf.org/mailman/listinfo/apps-discuss

1 0

A note from the chairs
by Thijs Metsch 11 Jun '09

11 Jun '09

Hi all, As you know the goal of this working group is the creation of an API for cloud infrastructure services (IaaS). As you also might know over the last weeks the overall progress stalled a little bit. This was because of presentations, OGF26 and a number of people who are busy doing other work beside OCCI. Nevertheless we now want to work hard on achieving our goal to have an API by the next OGF meeting. This will be done in an open process where individuals as well as groups can participate. Within an open process, with many voices, it can happen that certain opinions, feelings and favors for certain technical solutions lead to a logjam. Therefore it is certain necessary to respect and appreciate all possibilities. The following process is defined by all chairs of the group and will support all options and still lead to a consensus based solution. Several steps need to be considered: 1.Operate on a requirements basis a)Categorize and compare existing APIs for Clouds. List what exists and look at the structure of their definition. b)Define and clearly state all requirements (which includes requirements on OCCI at large but as well as requirements on rendering formats) c)Define and clearly state all use cases which should be supported by the solution d)Prioritize all requirements and use those to look at technical solutions. Important is also a matching of the possible solution and the requirements. 2.Create a broad user base. a)Maintain a members register of people who are engaged in the process of the creation of the API b)Get statements who and when groups or individuals will implement the API 3.Create a clear representation and discussion base on the mailing- list and Wiki a)Create a clear structured Wiki where it is easy to find all information b)The representation of the API should be easy to read and implement. c)Include all use cases/requirements/comparison data in one place so they can be used for discussion (also see 1) 4.Collaborate with other standardization bodies a)Define and look at overlaps of working groups b)Assign people to act as liaisons to get an exchange of knowledge between groups We the chairs will support and look over this process. By doing so we hope to encourage all members to contribute to this open process and so get a broad user and member base. This process should lead to a broad consensus based on the requirements and opinions from the participant in this group. Please join the discussions and on particular issues feel free to contact the chairs. Overall the following assignments and action items will be made for now: * Work on requirements and use cases (includes the organization of an concall to prioritize): Thijs * Members recruitment and backup of their engagement: Alexis * Work on Wiki structure and API documentation: Sam and Andy * Coordinate collaboration efforts: Thijs Feel free to comment and discuss thinks. We will also discuss this a little bit during our concall tomorrow, Have a great day, -Thijs -- Thijs Metsch Tel: +49 (0)941 3075-122 (x60122) http://blogs.sun.com/intheclouds Software Engineer Grid Computing Sun Microsystems GmbH Dr.-Leo-Ritter-Str. 7 mailto:thijs.metsch@sun.com D-93049 Regensburg http://www.sun.com

5 4

Re: [occi-wg] OCCI members and implementors
by Kristoffer Sheather 11 Jun '09

11 Jun '09

Richard, CloudCentral will also be implementing OCCI. Regards, Kris ---------------------------------------- From: Richard Davies <richard.davies(a)elastichosts.com> Sent: Thursday, 11 June 2009 12:49 AM To: Thijs Metsch <thijs.metsch(a)sun.com>, Tim Bray <tim.bray(a)sun.com>, Randy Bias <randyb(a)neotactics.com>, Tony Lucas <tony(a)xcalibre.com>, Ignacio Martin Llorente <llorente(a)dacya.ucm.es>, Tino Vazquez <tinova(a)fdi.ucm.es> Subject: [occi-wg] OCCI members and implementors Thijs Metsch wrote: > 2.Create a broad user base. > a)Maintain a members register of people who are engaged in the process > of the creation of the API > b)Get statements who and when groups or individuals will implement the > API We should maintain these on the wiki at http://forge.ogf.org/sf/wiki/do/viewPage/projects.occi-wg/wiki/Members I've updated the page to add an Implementors section, and have listed the statements already made on the mailing list by ElasticHosts, GoGrid, FlexiScale, Sun and OpenNebula. If any of these above feel that I've misrepresented them, or anyone else would like to add a statement which I've missed, then please update that wiki page. Cheers, Richard. _______________________________________________ occi-wg mailing list occi-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/occi-wg

2 1

What next next on the apis ?
by Gary Mazz 11 Jun '09

11 Jun '09

Hi Since there's been so much feedback on the cloud model/responsibilities and the network attributes matrix, I was thinking about the next analysis. should it be storage, cpu/mem. or ancillary ? -gary

4 5

Re: [occi-wg] What next on the apis ?
by Kristoffer Sheather 11 Jun '09

11 Jun '09

+1 compute then storage. ---------------------------------------- From: Gary Mazz <garymazzaferro(a)gmail.com> Sent: Thursday, 11 June 2009 7:04 AM To: "Krishna Sankar (ksankar)" <ksankar(a)cisco.com> Subject: Re: [occi-wg] What next on the apis ? Thanks... Ancillary: 0 Computer : 1 Storage : 0 ============== Total : 1 Krishna Sankar (ksankar) wrote: > IMHO, compute might be the next step and then storage. > Cheers > > > |-----Original Message----- > |From: occi-wg-bounces(a)ogf.org [mailto:occi-wg-bounces@ogf.org] On > Behalf > |Of Gary Mazz > |Sent: Wednesday, June 10, 2009 1:24 PM > |To: Edmonds, AndrewX; occi-wg(a)ogf.org > |Subject: [occi-wg] What next next on the apis ? > | > |Hi > | > |Since there's been so much feedback on the cloud model/responsibilities > |and the network attributes matrix, I was thinking about the next > |analysis. should it be storage, cpu/mem. or ancillary ? > | > |-gary > |_______________________________________________ > |occi-wg mailing list > |occi-wg(a)ogf.org > |http://www.ogf.org/mailman/listinfo/occi-wg > > _______________________________________________ occi-wg mailing list occi-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/occi-wg

2 1

OCCI Walkthrough
by Sam Johnston 10 Jun '09

10 Jun '09

Hello again, To get the ball rolling I've spent some time over the weekend working on an OCCI Walkthrough<http://forge.ogf.org/sf/wiki/do/viewPage/projects.occi-wg/wiki/Walkthrough>(following in the footsteps of Sun's excellent HelloCloud <http://kenai.com/projects/suncloudapis/pages/HelloCloud>document). It captures the latest thinking and feedback received from many of you over the past few weeks and is intended as a simple (if technical) introduction to new users. As you can see we are about as close to raw HTTP as we'll ever get, and the use of HTML forms for many operations trivialises implementation on both the server and client side. Shifting further towards a document-based "Resource Oriented Architecture<http://www.infoq.com/resource/articles/richardson-ruby-restful-ws/en/resour…>" results in an API that "just makes sense" and is extremely (horizontally) scalable. Multiple representations also allow us to lock down what we need to for interoperability (e.g. the simple OCCI descriptor format) while enabling providers to innovate and differentiate (safely) by supporting various formats (e.g. VMD, VMDK, VMX, OVF, etc.) and features (e.g. auditing, change control, security, etc.). Anwyay without further ado, current version's copied below and latest version's in the wiki<http://forge.ogf.org/sf/wiki/do/viewPage/projects.occi-wg/wiki/Walkthrough> . Sam OCCI Walkthrough Overview The Open Cloud Computing Interface (OCCI) is an API for managing cloud infrastructure services (also known as Infrastructure as a Service or IaaS) which strictly adheres to REpresentational State Transfer (REST) principles and is closely tied to HyperText Tranfer Protocol (HTTP). For simplicity and scalability reasons it specifically avoids Remote Procedure Call (RPC) style interfaces and can essentially be implemented as a horizontally scalable document repository with which both nodes and clients interact. This document describes a step-by-step walkthrough of performing various tasks as at the time of writing. Getting started Connecting Each implementation has a single OCCI end-point URL (we'll use http://example.com/) and everything you need to know is linked from this point - configuring clients is just a case of providing this parameter. In the simplest case the end-point may contain only a single resource or type of resource (e.g. a hypervisor burnt into the BIOS of a motherboard exposing compute resources, a network switch/router exposing network resources or a SAN exposing storage resources) and at the other end of the spectrum it may provide access to a global cloud infrastructure (e.g. the "Great Global Grid" or GGG). You will only ever see those resources to which you have access to (typically all of them for a private cloud or a small subset for a public cloud) and flexible categorisation and search provide fine-grained control which resources are returned, allowing OCCI to handle the largest of installations. You will always connect to this end-point over HTTP(S) and given the simplicity of the interface most user-agents are suitable, including libraries (e.g. urllib2, LWP), command line tools (e.g. curl, wget) and full blown browsers (e.g. Firefox). AuthenticatingWhen you connect you will normally be challenged to authenticate via HTTP (this is not always the case - in secure/offline environments it may not be necessary) and will need to do so via the specified mechanism. It is anticipated that most implementations will require HTTP Basic Authentication over SSL/TLS so at the very least you should support this (fortunately almost all user-agents already do), but more advanced mechanisms such as NTLM or Kerberos may be deployed. Certain types of accesses (such as a compute resource querying OCCI for introspection and configuration) may be possible anonymously (having already been authenticated by interface and/or IP address). Should you be redirected by the API to a node, storage device, etc. (for example, to retrieve a large binary representation) then you should either be able to transparently authenticate or a signed URL should be provided. That is, a single set of credentials is all that is required to access the entire system from any point. Representations As the resource itself (e.g. a physical machine, storage array or network switch) cannot be transferred over HTTP (at least not yet!) we instead make available one or more representations of that resource. For example, an API modeling a person might return a picture, fingerprints, identity document(s) or even a digitised DNA sequence, but not the person themselves. A circle might be represented by SVG drawing primatives or any three distinct points on the curve. For cloud infrastructure there are many useful representations, and while OCCI standardises a number of them for interoperability purposes, an implementation is free to implement others in order to best serve the specific needs of their users and to differentiate from other offerings. Other examples include: - Open Cloud Computing Interface (OCCI) descriptor format (application/occi+xml) - Open Virtualisation Format (OVF) file (application/ovf+xml?) - Open Virtualisation Archive (OVA) file (application/x-ova?) - Screenshot of the console (image/png) - Access to the console (application/x-vnc) The client indicates which representation(s) it desires by way of the URL and/or HTTP Accept headers (e.g. HTTP Content Negotiation<http://www.w3.org/Protocols/rfc2616/rfc2616-sec12.html>) and if the server is unable to satisfy the request then it should return HTTP 406 Not Acceptable. Descriptors In addition to the protocol itself, OCCI defines a simple key/value based descriptor format for cloud infrastructure resources: computeProvides computational services, ranging from dedicated physical machines (e.g. Dedibox) to virtual machines (e.g. Amazon EC2) to slices/zones/containers (e.g. Mosso Cloud Servers). networkProvides connectivity between machines and the outside world. Usually virtual and may or may not be connected to a physical segment. storageProvides storage services, typically via magnetic mass storage devices (e.g. hard drives, RAID arrays, SANs). Given the simplicity of the format it is trivial to translate between wire formats including plain text, JSON, XML and others. For example: compute.cores 2 compute.speed 3200 compute.memory 2048 Identifiers Each resource is identified by its' dereferenceable URL which is by definition unique, giving information about the origin and type of the resource as well as a local identifier (the combination of which forms a globally unique compound key). The primary drawback is that the more information that goes into the key (and therefore the more transparent it is), the more likely it is to change. For example, if you migrate a resource from one implementation to another then its' identifier will change (though in this instance the source should provide a HTTP 301 Moved Permanentlyresponse along with the new location, assuming it is known, or HTTP 410 Gone otherwise). In order to realise the benefit of transparent, dereferenceable identifiers while still being able to track resources through their entire lifecycle an immutable UUID attribute should be allocated which will remain with the resource throughout its' life. This is particularly important where the same resource (e.g. a network) appears in multiple places. New implementations should use type 4 (random) UUIDs anyway, as these can be safely allocated by any node without consulting a register/sequence, but where existing identifiers are available they should be used instead (e.g. http://amazon.com/compute/ami-ef48af86) Operations Create To create a resource simply POST it to the appropropriate collection (e.g. /compute, /network or /storage) as an HTML form (supported by virtually all user agents) or in another supported format (e.g. OVF): POST /compute HTTP/1.1 Host: example.com Content-Length: 35 Content-Type: application/x-www-form-urlencoded compute.cores=2&compute.memory=2048 If this was successful the server will automatically allocate an identifier and return HTTP 200 OK along with a Location: header pointing at the newly created resource. You may also PUT the resource directly in place if you already know its' identifier or will be generating it on the client side: PUT /compute/b10fa926-41a6-4125-ae94-bfad2670ca87 HTTP/1.1 Host: example.com Content-Length: 35 Content-Type: application/occi+text compute.cores 2 compute.memory 2048 Rather than generating the new resource from scratch you may also be given the option to GET a template and POST or PUT it back (for example, where "small", "medium" and "large" instances or pre-configured appliances are offered). Retrieve The simplest command is to retrieve a single resource by conducting a HTTP GET on its' URL (which doubles as its' identifier): GET /compute/b10fa926-41a6-4125-ae94-bfad2670ca87 HTTP/1.1 Host: example.com This will return a HTTP 300 Multiple Choices response containing a list of available representations for the resource as well as a suggestion in the form of a HTTP Location: header of the default rendering, which should be HTML (thereby allowing standard browsers to access the API directly). An arbitrary number of alternatives may also be returned by way of HTTP Link:headers. If you just need to know what representations are available you should make a HEAD request instead of a GET - this will return the metadata in the headers without the default rendering. Collections Some requests (such as searches) will need to return a collection of resources. There are two options: Pass-by-reference A plain text or HTML list of links is provided but each needs to be retrieved separately, resulting in O(n+1) performance. Pass-by-value A wrapper format such as Atom is used to deliver [links to] the content as well as the metadata (e.g. links, associations, cahching information, etc.), resulting in O(1) performance. Update Updating resources is trivial - simply GET the resource, modify it as necessary and PUT it back where you found it. Delete Simply DELETE the resource: DELETE /compute/b10fa926-41a6-4125-ae94-bfad2670ca87 HTTP/1.1 Host: example.com Sub-resource Collections (For want of a better name) Each resource may expose collections for functions such as logging, auditing, change control, documentation and other operations (e.g. http://example.com/compute/123/log/456) in addition to any required by OCCI. As usual CRUD operations map to HTTP verbs (as above) and clients can either PUT entries directly if they know or will generate the identifiers, or POSTthem to the collection if this will be handled on the server side (using POST Once Exactly (POE)<http://tools.ietf.org/draft/draft-nottingham-http-poe/draft-nottingham-http…>to ensure idempotency). Requests Requests are used to trigger state changes and other operations such as backups, snapshots, migrations and invasive reconfigurations (such as storage resource resizing). Those that do not complete immediately (returning HTTP 200 OK or similar) must be handled asynchronously (returning HTTP 201 Accepted or similar). POST /compute/123/requests HTTP/1.1 Host: example.com Content-Length: 35 Content-Type: application/x-www-form-urlencoded state=shutdown&type=acpioff The actual operation may not start immediately (for example, backups which are only handled daily at midnight) and may take some time to complete (for example a secure erase which requires multiple passes over the disk). Clients can poll for status periodically or use server push<http://en.wikipedia.org/wiki/Push_technology>(or a non-HTTP technology such as XMPP) to monitor for events.

2 1