May 2009 - occi-wg - lists.ogf.org

"Approachability" of API
by Sam Johnston 20 May '09

20 May '09

Afternoon all, The reason we want a simple API is that it be approachable. It's easier for people write to SMTP, HTTP, etc. because they can kick the tyres and learn the ropes with telnet: *$ telnet www.google.com 80* > Trying 209.85.227.103... > Connected to www.l.google.com. > Escape character is '^]'. > *GET / HTTP/1.1* > *Host: www.google.com* > > HTTP/1.1 302 Found > Location: http://www.google.fr/ > <snip> > This is good, and a strong motivator for us to make it easy for implementors to provide a text rendering. That said it just dawned on me that with both Microsoft and Google (among others) strategically backing AtomPub, it will be impossible to claim with a straight face to be a cloud developer if you don't know it yourself. IOW, cloud developers will already necessarily have exposure to, and in many cases intimate knowledge of, AtomPub. Regardless of how simple our simple protocol is, everyone's going to be starting from scratch. Yet again oversimplification could well raise the bar... we need to be as simple as possible but no more. Whatever we do we need to make sure the very first step is as low as possible. Sam

2 5

Open Cloud Computing Interface Prague Presentation
by samj＠samj.net 19 May '09

19 May '09

I've shared a document with you called "Open Cloud Computing Interface Prague Presentation": http://docs.google.com/Presentation?id=ddqm27m2_298fsf9mqdg&invite=19887284… It's not an attachment -- it's stored online at Google Docs. To open this document, just click the link above. --- As promised, here is the presentation... you can join it to follow the slides automatically. Sam

3 2

Last minute dail-in details for Cloud Computing Expo (in 5 minutes!)
by Sam Johnston 19 May '09

19 May '09

Afternoon all, I'll shortly be presenting OCCI at the Cloud Computing Expo in Prague and it turns out we may be able to include you by way of a dial-in to the usual conference bridge as well as a Google Docs presentation which I'll share directly. Apologies for the late notice - we didn't have access to the cloud yesterday (at a cloud computing conference no less) and there's no guarantees Skype will work as intended but we'll soon see - anyway I've joined the call already so hope to see some of you soon. So it's from 6:30pm through 7:15pm and there'll be some time for questions at the end. Sam Passcode: Participant: 9032568 Dial in numbers: Country Toll Numbers Freephone/Toll Free Number ARGENTINA 0800-777-0463 AUSTRALIA ADELAIDE: 61-8-8121-4868 1-800-249-288 AUSTRALIA BRISBANE: 61-7-3102-0970 1-800-249-288 AUSTRALIA CANBERRA: 61-2-6100-1970 1-800-249-288 AUSTRALIA MELBOURNE: 61-3-9010-7739 1-800-249-288 AUSTRALIA PERTH: 61-8-9467-5249 1-800-249-288 AUSTRALIA SYDNEY: 61-2-8205-8125 1-800-249-288 AUSTRIA 43-1-92-86-506 0800-005-029 BELGIUM 32-1-150-0314 0800-4-8680 BRAZIL 0800-7610674 CHILE 1230-020-2867 CHINA* 86-400-810-4766 10800-712-1433 10800-120-1433 COLOMBIA 01800-9-156430 CZECH REPUBLIC 420-2-25-98-56-54 800-700-173 DENMARK 45-7014-0280 8088-6132 ESTONIA 800-011-1089 FINLAND Land Line: 106-33-146 0-800-1-10100 FINLAND Mobile: 09-106-33-146 0-800-1-10100 FRANCE LYON: 33-4-26-69-12-81 080-563-9647 FRANCE MARSEILLE: 33-4-86-06-00-81 080-563-9647 FRANCE PARIS: 33-1-70-70-74-20 080-563-9647 GERMANY 49-69-2222-2566 0800-000-3441 GREECE 30-80-1-100-0683 00800-12-6973 HONG KONG 852-2286-5731 800-930-705 HUNGARY 06-800-18013 INDIA 000-800-852-1266 INDONESIA 001-803-011-3787 IRELAND 353-1-247-5253 1800-932-145 ISRAEL 1-80-9214916 ITALY 39-02-3600-3642 800-986-570 JAPAN OSAKA: 81-6-7739-4769 0034-800-400828 JAPAN TOKYO: 81-3-5539-5189 0034-800-400828 LATVIA 8000-3025 LUXEMBOURG 352-27-000-1360 MALAYSIA 1-800-80-2812 MEXICO 001-866-627-0574 NETHERLANDS 31-20-718-8533 0800-020-1392 NEW ZEALAND 64-9-970-4769 0800-449-823 NORWAY 47-21-59-00-59 800-15414 PANAMA 011-001-800-5072129 PERU 0800-53733 PHILIPPINES 63-2-858-3715 POLAND 00-800-1212021 PORTUGAL 8008-14061 RUSSIA 8-10-8002-9683011 SINGAPORE 65-6883-9228 800-120-4662 SLOVAK REPUBLIC 421-2-322-422-21 SOUTH AFRICA 080-09-80416 SOUTH KOREA 82-2-6744-1081 00798-14800-6860 SPAIN 34-91-414-62-98 800-099-810 SWEDEN 46-8-505-78-524 0200-890-106 SWITZERLAND 41-44-580-4389 0800-001-523 TAIWAN 886-2-2795-7377 00801-137-766 THAILAND 001-800-1206-65656 UNITED KINGDOM BIRMINGHAM: 44-121-210-9021 0808-238-6025 UNITED KINGDOM GLASGOW: 44-141-202-3221 0808-238-6025 UNITED KINGDOM LEEDS: 44-113-301-2121 0808-238-6025 UNITED KINGDOM LONDON: 44-20-7075-3246 0808-238-6025 UNITED KINGDOM MANCHESTER: 44-161-601-1421 0808-238-6025 URUGUAY 000-413-598-3415 USA 1-203-418-3122 866-692-3163 VENEZUELA 0800-1-00-3733

1 0

OCCI-wg concall Wednesday May 20, 2009 @ 5pm CET
by Thijs Metsch 19 May '09

19 May '09

Hi guys, We meet again tomorrow for our weekly concall at 5pm CET. For other timezones and dialin details visit: http://forge.ogf.org/sf/go/wiki2128 The agenda is currently set to: 1. OGF preparation 2. first final discussion on formats in the API. Feel free to add more topics at: http://forge.ogf.org/sf/wiki/do/viewPage/projects.occi-wg/wiki/Concall20090… All the best, -Thijs -- Thijs Metsch Tel: +49 (0)941 3075-122 (x60122) http://blogs.sun.com/intheclouds Software Engineer Grid Computing Sun Microsystems GmbH Dr.-Leo-Ritter-Str. 7 mailto:thijs.metsch@sun.com D-93049 Regensburg http://www.sun.com

1 0

Future OCCI extensibility: compute, storage, network, service, application
by Sam Johnston 18 May '09

18 May '09

Evening all, According to NIST<http://csrc.nist.gov/groups/SNS/cloud-computing/index.html> : *Cloud computing is a pay-per-use model for enabling available, convenient, > on-demand > network access to a shared pool of configurable computing resources (e.g., > networks, > servers, storage, applications, services) that can be rapidly provisioned > and released with > minimal management effort or service provider interaction. This cloud model > promotes > availability and is comprised of five key characteristics, three delivery > models, and four > deployment models.* > Is it obvious to anyone else what the next resource extensions after compute, network and storage should be? Is there still someone who needs convincing that we need to have extensibility at the heart of the protocol and build incrementally? Sam

4 9

Is OCCI the HTTP of Cloud Computing?
by Sam Johnston 15 May '09

15 May '09

Morning all, I'm going to break my own rules about reposting blog posts because this is very highly relevant, it's 03:30am already and I'm traveling again tomorrow. The next step for us is to work out what the protocol itself will look like on the wire, which is something I have been spending a good deal of time looking at over many months (both analysing existing efforts and thinking of "blue sky" possibilities). I am now 100% convinced that the best results are to be had with a variant of XML over HTTP (as is the case with Amazon, Google, Sun and VMware) and that while Google's GData is by far the most successful cloud API in terms of implementations, users, disparate services, etc. Amazon's APIs are (at least for the foreseeable future) a legal minefield. I'm also very interested in the direction Sun and VMware are going and have of course been paying very close attention to existing public clouds like ElasticHosts and GoGrid (with a view to being essentially backwards compatible and sysadmin friendly). I think the best strategy by a country mile is to standardise OCCI core protocol following Google's example (e.g. base it on Atom and/or AtomPub with additional specs for search, caching, etc.), build IaaS extensions in the spirit of Sun/VMware APIs and support alternative formats including HTML, JSON and TXT via XML Stylesheets (e.g. occi-to-html.xsl<http://code.google.com/p/occi/source/browse/trunk/xml/occi-to-html.xsl>, occi-to-json.xsl<http://code.google.com/p/occi/source/browse/trunk/xml/occi-to-json.xsl>and occi-to-text.xsl<http://code.google.com/p/occi/source/browse/trunk/xml/occi-to-text.xsl>). You can see the basics in action thanks to my Google App Engine reference implementation<http://code.google.com/p/occi/source/browse/#svn/trunk/occitest>at http://occitest.appspot.com/ (as well as HTML<http://www.w3.org/2005/08/online_xslt/xslt?xslfile=http%3A%2F%2Focci.google…>, JSON<http://www.w3.org/2005/08/online_xslt/xslt?xslfile=http%3A%2F%2Focci.google…>and TXT<http://www.w3.org/2005/08/online_xslt/xslt?xslfile=http%3A%2F%2Focci.google…>versions of same), KISS junkies bearing in mind that this weighs in under 200 lines of python code! Of particular interest is the ease at which arbitrarily complex [X]HTML interfaces can be built directly on top of OCCI (optionally rendered from raw XML in the browser itself) and the use of the hCard microformat <http://microformats.org/> as a simple demonstration of what is possible. Anyway, without further ado: Is OCCI the HTTP of Cloud Computing? http://samj.net/2009/05/is-occi-http-of-cloud-computing.html The Web is built on the Hypertext Transfer Protocol (HTTP)<http://en.wikipedia.org/wiki/HTTP>, a client-server protocol that simply allows client user agents to retrieve and manipulate resources stored on a server. It follows that a single protocol could prove similarly critical for Cloud Computing<http://en.wikipedia.org/wiki/Cloud_computing>, but what would that protocol look like? The first place to look for the answer is limitations in HTTP itself. For a start the protocol doesn't care about the payload it carries (beyond its Internet media type <http://en.wikipedia.org/wiki/Internet_media_type>, such as text/html), which doesn't bode well for realising the vision<http://www.w3.org/2001/sw/Activity.html>of the [ Semantic <http://en.wikipedia.org/wiki/Semantic_Web>] Web<http://en.wikipedia.org/wiki/World_Wide_Web>as a "universal medium for the exchange of data". Surely it should be possible to add some structure to that data in the simplest way possible, without having to resort to carrying complex, opaque file formats (as is the case today)? Ideally any such scaffolding added would be as light as possible, providing key attributes common to all objects (such as updated time) as well as basic metadata such as contributors, categories, tags and links to alternative versions. The entire web is built on hyperlinks so it follows that the ability to link between resources would be key, and these links should be flexible such that we can describe relationships in some amount of detail. The protocol would also be capable of carrying opaque payloads (as HTTP does today) and for bonus points transparent ones that the server can seamlessly understand too. Like HTTP this protocol would not impose restrictions on the type of data it could carry but it would be seamlessly (and safely) extensible so as to support everything from contacts to contracts, biographies to books (or entire libraries!). Messages should be able to be serialised for storage and/or queuing as well as signed and/or encrypted to ensure security. Furthermore, despite significant performance improvements introduced in HTTP 1.1 it would need to be able to stream many (possibly millions) of objects as efficiently as possible in a single request too. Already we're asking a lot from something that must be extremely simple and easy to understand. XML It doesn't take a rocket scientist to work out that this "new" protocol is going to be XML based, building on top of HTTP in order to take advantage of the extensive existing infrastructure. Those of us who know even a little about XML will be ready to point out that the "X" in XML means "eXtensible" so we need to be specific as to the schema for this assertion to mean anything. This is where things get interesting. We could of course go down the WS-* route and try to write our own but surely someone else has crossed this bridge before - after all, organising and manipulating objects is one of the primary tasks for computers. Who better to turn to for inspiration than a company whose mission<http://www.google.com/corporate/>it is to "organize the world's information and make it universally accessible and useful", Google. They use a single protocol for almost all of their APIs, GData <http://code.google.com/apis/gdata/>, and while people don't bother to look under the hood (no doubt thanks to the myriad client libraries <http://code.google.com/apis/gdata/clientlibs.html> made available under the permissive Apache 2.0 license), when you do you may be surprised at what you find: everything from contacts to calendar items, and pictures to videos is a feed (with some extensions for things like searching<http://code.google.com/apis/gdata/docs/2.0/basics.html#Searching>and caching<http://code.google.com/apis/gdata/docs/2.0/reference.html#ResourceVersioning> ). OCCI Enter the OGF's Open Cloud Computing Interface (OCCI)<http://www.occi-wg.org/>whose (initial) goal it is to provide an extensible interface to Cloud Infrastructure Services (IaaS). To do so it needs to allow clients to enumerate and manipulate an arbitrary number of server side "resources" (from one to many millions) all via a single entry point. These compute, network and storage resources need to be able to be created, retrieved, updated and deleted (CRUD) and links need to be able to be formed between them (e.g. virtual machines linking to storage devices and network interfaces). It is also necessary to manage state (start, stop, restart) and retrieve performance and billing information, among other things. The OCCI working group basically has two options now in order to deliver an implementable draft this month as promised: follow Amazon or follow Google (the whole while keeping an eye on other players including Sun and VMware). Amazon use a simple but sprawling XML based API with a PHP style flat namespace and while there is growing momentum around it, it's not without its problems. Not only do I have my doubts about its scalability outside of a public cloud environment (calls like 'DescribeImages' would certainly choke with anything more than a modest number of objects and we're talking about potentially millions) but there are a raft of intellectual property issues as well: - *Copyrights* (specifically section 3.3 of the Amazon Software License<http://aws.amazon.com/asl/>) prevent the use of Amazon's "open source" clients with anything other than Amazon's own services. - *Patents* pending like #20070156842<http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p…>cover the Amazon Web Services APIs and we know that Amazon have been known to use patents offensively<http://en.wikipedia.org/wiki/1-Click#Barnes_.26_Noble>against competitors. - *Trademarks* like #3346899<http://tarr.uspto.gov/servlet/tarr?regser=serial&entry=77054011>prevent us from even referring to the Amazon APIs by name. While I wish the guys at Eucalyptus <http://open.eucalyptus.com/> and Canonical <http://news.zdnet.com/2100-9595_22-292296.html> well and don't have a bad word to say about Amazon Web Services, this is something I would be bearing in mind while actively seeking alternatives, especially as Amazon haven't worked out<http://www.tbray.org/ongoing/When/200x/2009/01/20/Cloud-Interop>whether the interfaces are IP they should protect. Even if these issues were resolved via royalty free licensing it would be very hard as a single vendor to compete with truly open standards (RFC 4287: Atom Syndication Format<http://tools.ietf.org/html/rfc4287>and RFC 5023: Atom Publishing Protocol <http://tools.ietf.org/html/rfc5023>) which were developed at IETF by the community based on loose consensus and running code. So what does all this have to do with an API for Cloud Infrastructure Services (IaaS)? In order to facilitate future extension my initial designs for OCCI have been as modular as possible. In fact the core protocol is completely generic, describing how to connect to a single entry point, authenticate, search, create, retrieve, update and delete resources, etc. all using existing standards including HTTP, TLS, OAuth and Atom. On top of this are extensions for compute, network and storage resources as well as state control (start, stop, restart), billing, performance, etc. in much the same way as Google have extensions for different data types (e.g. contacts vs YouTube movies). Simply by standardising at this level OCCI may well become the HTTP of Cloud Computing.

24 191

interop is hard
by Alexis Richardson 14 May '09

14 May '09

Example: http://twitter.com/crucially/statuses/1797948271

2 1

OCCI MC - State Machine Diagram
by Sam Johnston 14 May '09

14 May '09

Afternoon all, I have created a diagram (attached) of what I think the absolute minimum core states need to be... essentially boiling them down to "STOPPED" and "ACTIVE" with "START" and "STOP" being the only requisite actuators. Transitional "STOPPING" and "STARTING" states are optional. I believe states should be completely unambiguous so I don't particularly like the DMTF model ("stopped" and "active" machines are also "defined" but this is a separate state) and that also rules out vague terminology like "inactive" (which could mean both "stopped" and "suspended"). I've got an optional RESTART actuator which takes you from "ACTIVE" to "STARTING" as well as an optional "SUSPEND" and "RESUME" cycle which takes you via the transitional "SUSPENDING" and "RESUMING" states between "ACTIVE" and "SUSPENDED". DMTF have another "paused' state but I wonder whether this needs to be a state in its own right or if it's an attribute of "suspended". That's a rhetorical question - we could spend all week discussing nuances but for now I want to make sure we agree on the absolutely minimalist core functionality. Other states can be added via a live registry which we can pre-populate with a view to guiding innovation without stifling it. This has been uploaded to the wiki: http://forge.ogf.org/sf/wiki/do/viewPage/projects.occi-wg/wiki/StateModel Sam

17 89

(industry noise on CC protocols) cross post from a cloud group
by Alexis Richardson 14 May '09

14 May '09

I thought this might be of interest -- ---------- Forwarded message ---------- From: Gervas Douglas <gervas.douglas(a)gmail.com> Date: Thu, May 14, 2009 at 12:13 AM Subject: [cloudcomputing-tech] Brogden on protocols for CC services To: cloudcomputing-tech(a)yahoogroups.com <<Call it grid computing, cloud computing or software as a service, distributed computing is on a lot of developers minds recently. There are so many different ways in which people can use the "cloud" that there would appear to be no possibility of a single "framework" for cloud based services. This has not prevented a lot of developer discussion and fear that the "cloud" will somehow become a closed realm of proprietary protocols as opposed to an open-source utopia. (See for example the open cloud manifesto.) It seems to me that a discussion of the wide range of protocols and architecture in grid/cloud/distributed computing is in order. I am going to discuss some of them in order of increasing complexity. Bandwidth and storage with REST Little change in architecture is required when an enterprise uses cloud facillities to supplement owned server bandwidth to handle unusual demands. As I discussed in this article, making use of cloud bandwidth simply requires redirecting requests to a cloud URL. This follows the REST principle that when you want the current state of a resource, the internal details of how you get it are unimportant. No special software beyond a HTTP server is required on the cloud server side. The high bandwidth and storage capacities combined with low cost and simple application make this the simplest way to take advantage of the cloud. Remote Procedure Calls with XML-RPC and SOAP The concept of a Remote Procedure Call has been around for years and forms the basic technology for much network computing. The architecure gives the programmer an interface which lets you program as if the method will actually be executed on your computer but will actually run elsewhere. The technology known as XML-RPC was the pioneer for the idea of formatting Remote Procedure Calls in XML messages. This simple yet powerful concept for supplying variables to call a method on a remote computer is still in use. However, some developers wanted more, so we got SOAP. Originally SOAP stood for Simple Object Access Protocol but it certainly is not simple and has little requirement to use objects. The inescapable disadvantage of XML encoded messages is that they are very inefficient in terms of actual data transmission and CPU time required to parse the message. On the plus side, by using text XML documents, developers can paper over the incompatibilities between different hardware and software in distributed systems. Furthermore everything that is transmitted is human readable and it is easy to create or read messages with a variety of techniques. As the evolution of SOAP shows, it is easy to add capabilities to XML formatted messages without seriously impacting existing applications. Your SOAP or XML-RPC server on the cloud side will require custom programming but it will be a familiar environment, typically JAVA with an SQL server and huge huge disk capacity. Cloud vendors such as Amazon are rushing such capabilities into service. RPC between objects with CORBA The Common Object Requesting Broker Architecture or CORBA is an early effort at a standard to enable object interactions within a network. It uses an interface definition language (IDL) to define the face these objects present and an object request broker (ORB) to negotiate the exchange of information about objects. CORBA makes a major effort to ensure that the IDL can be supported in many languages and operating systems. The Internet Inter-Orb Protocol (IIOP) is the protocol by which ORBs communicate between heterogeneous mixes of computer systems using TCP/IP. It should not surprise you that object oriented languages like Java are easy to map to IDL. Support for IIOP has been part of the Java standard library since 1998. In spite of the fact that there have been many cycles of popular "latest thing" buzz-words since CORBA was introduced, it continues to be used in network computing with strong and mature industry support. The Object Management Group is the non-profit industry consortium responsible for maintaining the standard. Version 1.0 was released in 1991 and we are currently at version 3.1 released in 2004 so you can see that this is a refined and mature standard. You can get quite an argument among developers as to whether CORBA and/or SOAP is too complex for the future of cloud computing and SOA. Application specific protocols A recent news article indicated that Microsoft's "cloud services operating system" called Azure, will eventually allow the use of the "Tabular Data Streams" protocol in communicating with SQL Data Services, the cloud version of SQL. It appears that customers really wanted to enable cloud relational database operations similar to existing in-house applications rather than translating requests to REST style. I think it is likely we will see a lot more cloud services that use application specific protocols. Resources REST versus SOAP - the REST story REST versus SOAP - the SOAP story Home page for the "open cloud manifesto" Wikipedia history of Tabular Data Streams protocol>> You can find this at: http://searchsoa.techtarget.com/tip/0,289483,sid26_gci1355134_mem1,00.html?… Gervas __._,_.___ Messages in this topic (1) Reply (via web post) | Start a new topic Messages | Files | Photos | Links | Database | Polls | Members | Calendar Change settings via the Web (Yahoo! ID required) Change settings via email: Switch delivery to Daily Digest | Switch format to Traditional Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe Recent Activity 1 New Members Visit Your Group Give Back Yahoo! for Good Get inspired by a good cause. Y! Toolbar Get it Free! easy 1-click access to your groups. Yahoo! Groups Start a group in 3 easy steps. Connect with others. . __,_._,___

1 0

Networks: attributes and verbs
by Richard Davies 14 May '09

14 May '09

All, For me, the largest gap in the nouns, verbs and attributes is with regards to networks. Here are some thoughts on capabilities which I believe we need and options to implement these - feedback please! Basics ------ Networks will be a top level noun, along with servers and storage. Users will link a server to a network to specify that the server is attached to that network. There will be one special network presenting the public internet, and users can create additional private networks for themselves. The open questions are: Public static IPs ----------------- On all public clouds, customers can purchase static IPs (e.g. Amazon Elastic IP) for uses where a constant server location is helpful (e.g. typical web hosting). We'll need a means for customers to create, list and destroy the public static IPs which they own. Two options: a) Public static IPs are first-class nouns b) They are listed inside the public internet network I'd favour a), which is the case with most public clouds today. Active networks --------------- At its most basic, a network should behave like a plain ethernet switch - it provides no services at all, and simply connects all the servers which attach to it. Servers are free to chose their own IP addresses, etc. There are a number of active services which are possible on a network: - Central DHCP server - Bridge to a physical VLAN (e.g. containing physical dedicated servers in colocation with the cloud provider) - Load balancer between several web servers on a private network across to a single IP on the public internet. - etc. Two options: a) The network object itself can optionally provide these. They are configured using attributes and verbs on the network. b) There are separate 'appliance' objects which provide these services, and are linked onto the network just as a server would be (e.g. a 'DHCP server' appliance and or 'load balancer' appliance). a) feels lighter-weight, but I suspect b) is more powerful. As such the choice depends on how far OCCI wants to go down this route. Linking to a network -------------------- Finally, there are a few attributes which can specified on the link when a server is linked to a network: - The physical interface on the server (nic:0, nic:1, etc.) - Port firewalling rules (e.g. connect server to the internet network, but only allow port 80 inbound) - IP firewalling rules (e.g. connect server to the private network, and allow it to communicate on 192.168.0.0-23 but no others) - Local DHCP (e.g. the server operating system is internally configured to DHCP. I am connecting it to the public internet, and want it to appear with my assigned public static IP. Please send a DHCP response with that specific IP address). I think that all of these are best implemented as attributes on the link.

8 17