Yep, we couldn't define a GLUE 3 in a short amount of time. However, I'm not really for formalizing what you've described as GLUE 2.1. If we want to make it an official new version of GLUE, I would push to use the existing GLUE 2 entities wherever possible and just use the extension points in those entities instead of defining new entities (as I did in my approach). A better strategy may be to call what you are working on "EGI Cloud Extensions to GLUE 2.0" or something - a profile that describes how you use GLUE 2 to describe clouds in EGI. This would let you move forward with some input from the working group, but without needing approval from the working group. We could also see how your cloud profile works and how well my a cloud profile works, without trying to merge them now. We could use this information when we start work on GLUE 3. Warren ________________________________________ From: Salvatore Pinto [salvatore.pinto@egi.eu] Sent: Friday, May 02, 2014 10:04 AM To: Warren Smith; OGF GLUE Working Group Subject: Re: [glue-wg] glue2 cloud examples Hi Warren, I agree with you that the approach of making the existing Computing entities more abstract (by removing the references to Grid, like Job, Queue, Wallclock) and then specializing them to the Grid/Cloud particular use cases would be good, but I am worried that this will need a big change to the standard, who would lead to a GLUE 3.0 version. Since in EGI we are still struggling to abandon GLUE 1 for GLUE 2, producing a new GLUE3 not retro-compatible and asking all the sites to adapt seems to me a big effort many users will not be willing to take right now. Also, I guess it will be much more difficult to reach consensus with such major changes than with just the addition of new cloud entities. Maybe we can go step by step, first defining separate cloud entities (not touching the old grid ones), reaching consensus on what information should be displayed for the Cloud and what is the meaning of the entities, so finalizing a GLUE 2.1 retro-compatible with GLUE 2.0 who can be easily implemented by the FedCloud and existing Grid sites, then we can work on merging everting togheter in a GLUE 3.0 which abstracts the Computing entities. We can discuss about that more at the next teleconference. Cheers, Salvatore. On 08/04/2014 15:12, Warren Smith wrote:

One thing I forgot to mention is that going forward, I think a good approach for us to take would be to do something similar to what Salvatore is proposing, but do it across the board. What I mean is that we should abstract the existing entities a little bit and then add entities that are more specialized/concrete. This would allow us to include specialized information in the schema and also provide entities with names that others understand at a glance (if I have to explain what a ComputingActivity/Manager/Service/Share is one more time... :-P). It'll also allow us do any little cleanup changes we've come across in the current schema.

A few quick examples about how this could be done:

current ComputingShare: * create ClusterQueue entity that is a specialization of ComputingShare * create CloudAvailabilityZone entity that is a specialization of ComputingShare? * create CloudTenant entity that is a specialization of ComputingShare (and other entities) * change attribute names containing 'Jobs' to 'Activities' * move 'MappingQueue', '*Slots', '*WaitingTime' to ClusterQueue entity

current ComputingActivity: * create ClusterJob specialization * create GridJob specialization (e.g. a job managed by a grid resource manager/scheduler) * create CloudInstance specialization * move Local* attributes to GridJob * move WaitingPosition to ClusterJob and GridJob * delete ComputingManager* * move ProxyExpirationTime to GridJob

current ExecutionEnvironment: * create ClusterNode specialization (also used for physical nodes in clouds) * create CloudFlavor specialization * remove VirtualMachine attribute * move TotalInstances, UnavailableInstances to ClusterNode

I think this will be relatively straightforward to do for the computing and storage entities.

I guess what I'm proposing/asking is if we are talking about making nontrivial changes to the schema, why don't we just start work on GLUE 3.0?

Warren

________________________________________ From: Warren Smith Sent: Wednesday, March 26, 2014 12:49 PM To: OGF GLUE Working Group Subject: glue2 cloud examples

Hi all,

I put some examples of how I'm representing an IaaS cloud using GLUE 2 into https://github.com/OGF-GLUE/JSON/tree/master/examples. Look for the files ending in openstack.json and the file images.json. I use a few extensions, but not very many.

My approach is:

Instance/Virtual Machine - ComputingActivity

Physical Node - ExecutionEnvironment

Virtual Machine Image - ApplicationEnvironment/Handle

I haven't done anything with storage at this point, but my approach would be to handle an object or image store the same as a shared/parallel file system.

I also haven't done anything related to describing virtual networks - GLUE 2 doesn't have much support for describing networks in general. VLANs, overlay networks, etc. are used outside of cloud environments as well as in cloud environments, so I don't know if there would be any need for separate entities/schemas for cloud vs non-cloud environments.

Warren _______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg

-- Salvatore Pinto Cloud Technologist, EGI.eu e-mail: salvatore.pinto@egi.eu skype: salvatore.pinto0 Science Park 140, Amsterdam, The Netherlands

I don't really understand your comments below - you seem to be putting me in the position that I see you in: I'm suggesting that we use GLUE 2 as it is (and I described how I did that for IaaS clouds) while this proposal is to make some major additions to GLUE2 to support clouds. All I was wondering was that if we want to make such major additions, whether it would be better to rework the GLUE 2 model to better support them and similar situations. I'm perfectly fine not doing this. In fact, my last email suggested that "you just define your own schema" (with you being EGI) by specifying the cloud extensions to GLUE 2 that EGI wants to define. If we want to create a GLUE 2.1 to standardize some cloud extensions, then why don't we consider how I represent IaaS clouds as my counter proposal. We can see if we can reach consensus on some middle ground (which might be using existing GLUE 2 entities with Extensions where possible and adding a few new entities where there isn't any fit in the current model). Warren ________________________________________ From: stephen.burke@stfc.ac.uk [stephen.burke@stfc.ac.uk] Sent: Saturday, May 03, 2014 5:53 AM To: Warren Smith; salvatore.pinto@egi.eu; glue-wg@ogf.org Subject: RE: [glue-wg] glue2 cloud examples Warren Smith [mailto:wsmith@tacc.utexas.edu] said:

Yep, we couldn't define a GLUE 3 in a short amount of time.

I'd like to reiterate that I think any idea of a GLUE 3 is a red herring. For one thing I don't see any reason for it, in the sense of there being problems that can't be solved within the GLUE 2 framework. Secondly I think the timescale is prohibitive - GLUE 2 took 2.5 years to define and another 3-4 years to implement. Thirdly, at least EGI and WLCG no longer have any effort to engage with such a thing, so if other projects follow that route they'll lose the interoperability that was the main goal of GLUE. If you really don't want to use GLUE 2 I'd suggest that you just define your own schema, which you could presumably do a lot faster.

However, I'm not really for formalizing what you've described as GLUE 2.1. If we want to make it an official new version of GLUE, I would push to use the existing GLUE 2 entities wherever possible and just use the extension points in those entities instead of defining new entities (as I did in my approach).

Again I think the issue is interoperability. EGI could certainly go ahead and define its own extended GLUE 2 and use it within the project, but then you lose the standardisation that comes with the OGF. Conversely, if the WG wants to have something standard then it needs to be a GLUE 2.1 that goes through the official process. Stephen -- Scanned by iCritical.

I wouldn't really call the proposed cloud extensions backward compatible either: Will a GLUE 2.0 client be able to understand these extensions? No. Will a client that understands these extensions understand GLUE 2.0? Yes. I agree that a GLUE 3 would be less backward compatible - the answer to both of these questions would be no. Regarding Extensions or OtherInfo, they aren't standardized, but they can be documented in a profile and any GLUE 2.0 client would be able to parse them. I think your last paragraph is the main issue, and I agree with what you said. However, my view is that I didn't really find it harder to fit cloud descriptions into the current GLUE 2.0 than I had fitting cluster descriptions into GLUE 2.0. Warren ________________________________________ From: stephen.burke@stfc.ac.uk [stephen.burke@stfc.ac.uk] Sent: Saturday, May 03, 2014 9:39 AM To: Warren Smith; salvatore.pinto@egi.eu; glue-wg@ogf.org Subject: RE: [glue-wg] glue2 cloud examples Warren Smith [mailto:wsmith@tacc.utexas.edu] said:

I don't really understand your comments below - you seem to be putting me in the position that I see you in: I'm suggesting that we use GLUE 2 as it is (and I described how I did that for IaaS clouds) while this proposal is to make some major additions to GLUE2 to support clouds. All I was wondering was that if we want to make such major additions, whether it would be better to rework the GLUE 2 model to better support them and similar situations. I'm perfectly fine not doing this.

There's a world of difference between a GLUE 3 which would make backward-incompatible changes to existing objects, and a 2.1 revision which adds new objects and/or optional attributes to existing objects. The latter can be done as an incremental change with new code only required for things which explicitly know about the new things, while the former would need new code everywhere and a long and carefully-managed transition process as we've just had for GLUE 1 to GLUE 2.

If we want to create a GLUE 2.1 to standardize some cloud extensions, then why don't we consider how I represent IaaS clouds as my counter proposal. We can see if we can reach consensus on some middle ground (which might be using existing GLUE 2 entities with Extensions where possible and adding a few new entities where there isn't any fit in the current model).

Within the existing GLUE 2.0 anyone can use Extensions or OtherInfo attributes to publish extra attributes not defined in the schema, and indeed EGI already has quite a few of those. However, those are by definition not standardised, they're project-specific. If we want to standardise them they need to be turned into real named attributes and defined in a 2.1 revision. Whether we need new objects is a matter for discussion, it depends on how much overlap there is between the existing Computing Service objects and what's desired for cloud services. For example, ComputingShare has 43 attributes - if a cloud equivalent would use only 2 of those and would define another 35 cloud-specific attributes then it would seem more natural to have a new object. On the other hand, if most of the existing attributes still make sense and only a few new ones are needed then re-using the objects may be OK. However, if we're making a 2.1 revision anyway it isn't especially harder to create new objects than add attributes to existing ones. Also bear in mind that you need to consider object relations as well as attributes - if the existing objects are re-used you need to be happy with all the relations too. And trying to do a mixture, e.g. keeping most Computing Service objects but defining a new CloudShare object, is likely to be messy, the way we've done the derivations so far doesn't allow for that. Stephen -- Scanned by iCritical.

That would probably work for LDAP or SQL (aside from GLUE 2.0 clients not seeing any information about clouds). It may or may not work for XML with our XML Schema. I don't believe it would work for JSON with JSON schema (the validation would fail). Warren ________________________________________ From: stephen.burke@stfc.ac.uk [stephen.burke@stfc.ac.uk] Sent: Monday, May 05, 2014 8:20 AM To: Warren Smith; salvatore.pinto@egi.eu; glue-wg@ogf.org Subject: RE: [glue-wg] glue2 cloud examples Warren Smith [mailto:wsmith@tacc.utexas.edu] said:

I wouldn't really call the proposed cloud extensions backward compatible either: Will a GLUE 2.0 client be able to understand these extensions? No.

By backward-compatible I mean that changes don't break existing code, either publishers or clients. If new objects are introduced the answer is yes - publishers don't have to publish them, and clients won't query for them and hence won't be disturbed. Stephen -- Scanned by iCritical.

On May 2, 2014, at 2:29 PM, Warren Smith <wsmith@tacc.utexas.edu> wrote:

A better strategy may be to call what you are working on "EGI Cloud Extensions to GLUE 2.0" or something - a profile that describes how you use GLUE 2 to describe clouds in EGI. This would let you move forward with some input from the working group, but without needing approval from the working group. We could also see how your cloud profile works and how well my a cloud profile works, without trying to merge them now. We could use this information when we start work on GLUE 3.

The purpose of the GLUE WG group is to provide a forum for sharing diverse requirements and trying to find a common interoperable design. In my view, relinquishing the two current proposal as infrastructure specific profiles, that would not be interoperable, does not accomplish our purpose. I think clouds have been around long enough, and are understood enough, that we could agree on an approach. In fact, it sounds like there are ways to make both approaches work? If this is not the case, we should focus on what requirements can't be met by one of the approaches in tomorrow's meeting. It seems like both approach provide a mechanism to add necessary entities. It's less clear if both approach can describe the required relationships. I think we there is room to come to an agreement on a single GLUE 2 compatible and interoperable way to support clouds. I recommend we continue documenting the advantages and short comings of both approaches and try to come to a consensus. Can Salvator and Warren prepare brief list of advantages an disadvantages for both approaches that we could discuss tomorrow? In particular trying to highlight any use cases/requirements that are difficult to satisfy with either approach? Thanks, JP

Hi All, here my comments to be discussed in today's teleconferece. Cloud Profile for GLUE 2.0 Advantages: * No need to update the implementation. * No need to manage two separate group of entities. Disadvantages: * Big usage of extended attributes to represent all the needed additional attributes and many attributes are not needed. Eg. you add around 3-4 new attributes to each entities and you do not use more than 10 in average. * Some attributes names from the implementation (ApplicationEnvironment, ExecutionEnvironment) will not be self-explanatory and user will always need to read the profile to understand the meaning of the implementation. GLUE 2.1 Advantages: * Possibility to represent a different conceptual model, where the OS images (application environment) are directly linked to the VM (computing activity) and to the Share (Computing Share) so to the Policies (VM can be private and started only to a given share associated to given VOs) * Easier extension to future non-IaaS computing services (with its own set of resources, you can edit them without messing with the Grid entities) Disadvantages: * Need to update the implementation, adding the new cloud entities. * Cloud and Grid are seen separated, not as an unique computing resource. Cheers, Salvatore. On 05/05/2014 17:21, Navarro, John-Paul F. wrote:

On May 2, 2014, at 2:29 PM, Warren Smith <wsmith@tacc.utexas.edu> wrote:

...

A better strategy may be to call what you are working on "EGI Cloud Extensions to GLUE 2.0" or something - a profile that describes how you use GLUE 2 to describe clouds in EGI. This would let you move forward with some input from the working group, but without needing approval from the working group. We could also see how your cloud profile works and how well my a cloud profile works, without trying to merge them now. We could use this information when we start work on GLUE 3. The purpose of the GLUE WG group is to provide a forum for sharing diverse requirements and trying to find a common interoperable design. In my view, relinquishing the two current proposal as infrastructure specific profiles, that would not be interoperable, does not accomplish our purpose. I think clouds have been around long enough, and are understood enough, that we could agree on an approach. In fact, it sounds like there are ways to make both approaches work? If this is not the case, we should focus on what requirements can't be met by one of the approaches in tomorrow's meeting.

It seems like both approach provide a mechanism to add necessary entities. It's less clear if both approach can describe the required relationships.

I think we there is room to come to an agreement on a single GLUE 2 compatible and interoperable way to support clouds. I recommend we continue documenting the advantages and short comings of both approaches and try to come to a consensus. Can Salvator and Warren prepare brief list of advantages an disadvantages for both approaches that we could discuss tomorrow? In particular trying to highlight any use cases/requirements that are difficult to satisfy with either approach?

Thanks,

JP

-- Salvatore Pinto Cloud Technologist, EGI.eu e-mail: salvatore.pinto@egi.eu skype: salvatore.pinto0 Science Park 140, Amsterdam, The Netherlands

(Hi Warren, replies inline... On 17/06/2014 03:00, Warren Smith wrote:

Another disadvantage of adding new cloud entities to the model is that new renderings would also need to be defined.

Comments on your points:

* I've only had to use a couple extensions to represent clouds in the current GLUE2. I've used a few extensions to represent clusters, too (see https://github.com/OGF-GLUE/JSON/tree/master/examples). So, without some examples showing that a lot of extensions are needed, I don't agree with that point.

you are right, without examples it is difficult to make a decision. I have compared your implementations of JSON rendering for Compute_OpenStack with my 2.1 cloud entities and I come up with the attached draft of "profile". Considering the number of original attributes not used and the number of extensions, in my opinion, since most of the old attributes are not used and there are quite a few extensions to be added, there is a rational for defining new entities...

* I don't think the GLUE2 entity names are self-explanatory in general - I always have to explain what they mean to new users. I don't think this is much worse for a cloud infrastructure than for a cluster or grid.

for me users with a basic knowledge of Grid computing should understand the meaning, of course this is not valid for all the users, what I was saying is that users with basic knowledge of Cloud computing would probably not be able to understand all the meaning of the profiled attributes without reading the doc...

* I don't quite follow the first advantage you list under "GLUE 2.1". In GLUE 2.0, ComputingActivity (instance) has a RequestedApplicationEnvironment attribute (image). ComputingActivity also has a Share association (inherited from Activity), but I'm not sure what a Share represents here. a share in my view represented a zone (or sub site), like "Amazon Asia Pacific Site", with related quota, resources and templates associated

Warren Cheers, Salvatore.

________________________________________ From: Salvatore Pinto [salvatore.pinto@egi.eu] Sent: Tuesday, May 06, 2014 9:16 AM To: Navarro, John-Paul F.; Warren Smith Cc: OGF GLUE Working Group Subject: Re: [glue-wg] glue2 cloud examples

Hi All, here my comments to be discussed in today's teleconferece.

Cloud Profile for GLUE 2.0 Advantages: * No need to update the implementation. * No need to manage two separate group of entities. Disadvantages: * Big usage of extended attributes to represent all the needed additional attributes and many attributes are not needed. Eg. you add around 3-4 new attributes to each entities and you do not use more than 10 in average. * Some attributes names from the implementation (ApplicationEnvironment, ExecutionEnvironment) will not be self-explanatory and user will always need to read the profile to understand the meaning of the implementation.

GLUE 2.1 Advantages: * Possibility to represent a different conceptual model, where the OS images (application environment) are directly linked to the VM (computing activity) and to the Share (Computing Share) so to the Policies (VM can be private and started only to a given share associated to given VOs) * Easier extension to future non-IaaS computing services (with its own set of resources, you can edit them without messing with the Grid entities) Disadvantages: * Need to update the implementation, adding the new cloud entities. * Cloud and Grid are seen separated, not as an unique computing resource.

Cheers, Salvatore.

-- Salvatore Pinto Cloud Technologist, EGI.eu e-mail: salvatore.pinto@egi.eu skype: salvatore.pinto0 Science Park 140, Amsterdam, The Netherlands

Warren Smith [mailto:wsmith@tacc.utexas.edu] said:

Another disadvantage of adding new cloud entities to the model is that new renderings would also need to be defined.

I would hope that the rendering definitions are generic enough that extending them to include new objects would be straightforward.

* I don't think the GLUE2 entity names are self-explanatory in general - I always have to explain what they mean to new users. I don't think this is much worse for a cloud infrastructure than for a cluster or grid.

We deliberately chose fairly abstract names so that they could be used for a range of concepts - but still there are bound to be limits to how far that can be taken. I would say that the main concepts underlying the existing computing schema are: A job, being some computing task which is submitted as a request in terms of the required resources, is executed and which then terminates in a relatively short time and frees the resources. A scheduler, being some software which decides when jobs are executed and on what resources, according to some set of policies. A set of execution environments, which constitute the computing resources available to execute the jobs. They have defined properties, e.g. CPU type, memory and installed software, and are countable, i.e. you can say how many of each type of resource you have and how many are currently in use. My impression is that all of those are potentially relevant to clouds, but there may also be significant differences to a traditional cluster system. Probably the biggest potential pinch point is the use of VMs rather than physical hardware. VMs may be created to order so the properties may not be defined in advance, and it may also be hard to decide how to enumerate the size of the available resources. We could also consider how the information is used. In the grid the traditional uses would be to decide whether a system is capable of running a given job (does it have enough memory, the right OS etc) and to assess how long it might take before the job starts, e.g. queue length. There are also monitoring uses, e.g. to assess the total resources available and the current level of use. Stephen -- Scanned by iCritical.

Our OpenStack installs have flavors defined, but I didn't need to publish this information. I wanted to provide information about physical nodes so that consumers know how busy a cloud is. This lets us develop displays to show how busy both the available clusters and the available clouds are. In addition, we aren't running Amazon-scale clouds so we do run out of resources sometimes. What do you mean by 'EntityName'? I don't remember what that might refer to and a quick search didn't find it in the GLUE2 spec. Application Handle/Environment seemed the closest analogy to an image in GLUE2 to me. Not a great fit, of course, but they both identify environments to run in. I haven't thought of it, but I'd probably say that a contextualization of an image is like an input configuration file for an application. Yeah, I just don't see clouds and grids (and clusters) as all that different with hard lines between them. Jobs are pretty standard by whatever name (job, instance, task, workflow). Clusters often ask for a wall time limit for a task, but not always, while Clouds typically don't ask for a wall time limit for a task, but some do like Nimbus. Some clouds have queues (e.g. Amazon spot instances, Nimbus background instances), some clusters don't use queues (time shared nodes). Some of the traditional cluster scheduling vendors support virtual machines (e.g. HTCondor) and others are integrating themselves into IaaS cloud platforms (e.g. Moab into OpenStack). Grids can have multiple clusters accessed via common mechanisms and a cloud can have multiple availability zones. All of the XaaS services aren't necessarily provided atop IaaS - from what I've read about in the commercial world, some companies tend to do things in virtual machines and others just configure software on bare metal servers. Yep, the GLUE2 spec is subject to interpretation. I think the working group has the concept of profile documents to let different groups say how they are interpreting it in their situation. Warren ________________________________________ From: Salvatore Pinto [salvatore.pinto@egi.eu] Sent: Friday, May 02, 2014 9:57 AM To: Warren Smith; OGF GLUE Working Group Subject: Re: [glue-wg] glue2 cloud examples Hi Warren, I had a look at your examples and I have some comment: * It seems you are missing the information about the resource templates (OpenStack Flavors), aren't you using these? * Does the user really need the information about the physical node or is this only for administrators and other operational activities? If not, maybe the ExecutionEnvironment can be used to represent the OpenStack falvors. * I see you are using the ApplicationHandle to represent the image identifier, in my first tests I was thinking of using the EntityName for that, since it seems not needed to involved another entity and ApplicationHandle description is related to methods to setup the environment, so maybe more related to Cloud contextualization methods... Beside this particular comments, in general, what I see as reasons for updating GLUE are: 1. Cloud and Grid have different provisioning concepts. Of course, they both offer "Computing resources as a Service", but the overall provisioning concept of Jobs, Wallclock and Queues does not apply to the Cloud IaaS, where provisioning is more flexible and related only to Quotas per user. 2. Cloud is an environment with many services interconnected together and IaaS is the baseline for most of them. So if, in the future, we want to represent other services (like PaaS, iPaaS, VREaaS, LBaaS, and all the other aaS in the world :) ), with their interconnection to the IaaS, I think it could be better to have Cloud IaaS as a new separated set of entities. 3. Since the specification are quite Grid specific, the application to Cloud is subject to "interpretation". For example, to enable BDII in EGI Federated Cloud until the new specification comes out, we are using the old GLUE2.0 schema, but with a different interpretation of the entities than yours (eg. we are using the Execution Environment as VM flavours/resource templates). The fact that two people reading separately the specification interpreted them differently is a fault in the specifications, who needs to ensure interoperability. Cheers, Salvatore. On 26/03/2014 18:49, Warren Smith wrote:

Hi all,

I put some examples of how I'm representing an IaaS cloud using GLUE 2 into https://github.com/OGF-GLUE/JSON/tree/master/examples. Look for the files ending in openstack.json and the file images.json. I use a few extensions, but not very many.

My approach is:

Instance/Virtual Machine - ComputingActivity

Physical Node - ExecutionEnvironment

Virtual Machine Image - ApplicationEnvironment/Handle

I haven't done anything with storage at this point, but my approach would be to handle an object or image store the same as a shared/parallel file system.

I also haven't done anything related to describing virtual networks - GLUE 2 doesn't have much support for describing networks in general. VLANs, overlay networks, etc. are used outside of cloud environments as well as in cloud environments, so I don't know if there would be any need for separate entities/schemas for cloud vs non-cloud environments.

Warren _______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg

-- Salvatore Pinto Cloud Technologist, EGI.eu e-mail: salvatore.pinto@egi.eu skype: salvatore.pinto0 Science Park 140, Amsterdam, The Netherlands