Dear all, The only problem I see if we go for a 2.1 version is that we need to understand how to deploy all this in the hierarchical BDII model. What happens if a site BDII hasn´t upgraded the schema to 2.1, it only understands 2.0, and it queries a cloud resource which supports 2.1? will it fail? Or will it fail to publish only those new entities that do not exist yet in 2.0? The same scenario goes for a 2.0 top BDII trying to query 2.1 site BDIIs. I guess until the whole grid hasn´t moved to 2.1 we may have some inconsistent things published around. I haven´t tested this so I´m not sure what happens when you want to publish something in LDAP that is not declared in the LDAP schema file. I guess this is the reason why we had two different LDAP trees for GLUE 1.3 and GLUE 2.0, right? Regards, Maria

-----Original Message----- From: glue-wg-bounces@ogf.org [mailto:glue-wg-bounces@ogf.org] On Behalf Of david.meredith@stfc.ac.uk Sent: 29 July 2014 10:18 To: John-Paul Navarro; glue-wg@ogf.org Subject: Re: [glue-wg] VOTE by 8/1: GLUE 2.1 for describing cloud resources?

Hi all, I would be in favour of adding new 2.1 cloud entities provided:

* 2.1 only adds new (cloud) implementations that are derived from the 2.0 abstract entities. * There are no changes to the 2.0 entities, especially no changes to the core abstract entities.

This is important for the XSD rendering (and other renderings too I imagine). For an XSD rendering, the new cloud entities would be defined using a new namespace defined in a separate XSD. The new cloud entities would need to extend the existing abstract BaseTypes defined by the 2.0 element inheritance mechanism. An XML instance doc would still use the existing 2.0 '<GLUE2>' root element, and would import the new cloud entities to be nested under '<GLUE2>' using the existing SubstitutionGroup mechanism. When validating an instance doc, both the 2.0 XSD and 2.1 cloud extensions XSD would be needed for validation (quite normal).

Thanks, David

...

-----Original Message----- From: JP Navarro [mailto:navarro@mcs.anl.gov] Sent: 23 July 2014 16:36 To: OGF GLUE Working Group Subject: [glue-wg] VOTE by 8/1: GLUE 2.1 for describing cloud resources?

GLUE Working Group:

We have been discussing two options for using the GLUE schema to describe cloud services. You may review these options here: - http://redmine.ogf.org/dmsf_files/13294?download=

By August 1, please vote for one of the following options:

1) Use GLUE 2.0 with a Community Practice Profile that details how to describe cloud infrastructure using the existing 2.0 entities 2) Develop GLUE 2.1 adding new Cloud Computing entities (would be 100% backward compatible with GLUE 2.0 for non-Cloud entities)

For reference, a preliminary GLUE 2.1 Specification is available here: - http://www.ogf.org/pipermail/glue-wg/2014-January/001527.html

Regards,

JP _______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg -- Scanned by iCritical.

---

glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg

Dear Florido,

...

The only problem I see if we go for a 2.1 version is that we need to understand how to deploy all this in the hierarchical BDII model. What happens if a site BDII hasn´t upgraded the schema to 2.1, it only understands 2.0, and it queries a cloud resource which supports 2.1? will it fail? Or will it fail to publish only those new entities that do not exist yet in 2.0? The same scenario goes for a 2.0 top BDII trying to query 2.1 site BDIIs. I guess until the whole grid hasn´t moved to 2.1 we may have some inconsistent things published around. I haven´t tested this so I´m not sure what happens when you want to publish something in LDAP that is not declared in the LDAP schema file.

It is not possible to add ldap entities to an LDAP database if these entities do no refer to objects in the schema. In short, any BDII using the 2.0 schema will reject all 2.1-specific objects, that is, the cloud stuff.

OK, so this is going to be the case in the infrastructure until all resource, site and top BDIIs are updated to glue-schema 2.1. Regards, Maria

Dear Salvatore,

1. Top-BDII & Site-BDII s to update schema to 2.1 (this can be done via an update of the BDII software in the UMD/EMI and will not distrupt anything)

The relevant package is the glue-schema package that is part of all resource, site and top BDIIs. Once you release this in the usual repos, all BDII flavours will start upgrading when the site decides to do so, we don't have control over this.

2. Site-BDII provider update to publish cloud info in both 2.0 and 2.1 (this can be done in parallel with #1, since Top-BDIIs not already updated to 2.1 will just ignore the new 2.1 entities)

Which provider do you mean here? There is no need to change the ldap info provider. It will automatically publish GLUE 2.1 as long as the right version of the glue-schema package containing GLUE 2.1 is installed in the BDII. This means, a top BDII with the right version of the glue-schema will publish cloud objects of those site BDIIs with the right version of the glue-schema publishing cloud objects.

3. When 1&2 are complete, update clients to consume 2.1 information (at the time, only AppDB is consuming this info, the new clients we are going to develop will be already based on at least a draft of 2.1)

Which clients do you mean here? ginfo? This is the only client querying GLUE 2 information at the moment.

4. When 3 is complete, Site-BDII provider update to publish cloud info only in 2.1

As I said before, there is no need to do any changes to the ldap info provider, if this is what you mean here. Regards, Maria

Dear Salvatore,

...

...

2. Site-BDII provider update to publish cloud info in both 2.0 and 2.1 (this can be done in parallel with #1, since Top-BDIIs not already updated to 2.1 will just ignore the new 2.1 entities) Which provider do you mean here? There is no need to change the ldap info provider. It will automatically publish GLUE 2.1 as long as the right version of the glue-schema package containing GLUE 2.1 is installed in the BDII. This means, a top BDII with the right version of the glue-schema will publish cloud objects of those site BDIIs with the right version of the glue-schema publishing cloud objects. the cloud-info-provider, no need to change the others. Note that, for now, the cloud-info-provider is producing GLUE 2.0 entities (so a undocumented community profile for EGI to represent cloud sources).

But the cloud info provider is not running in the site BDII, this is running in the resource BDII of the cloud resources, right? What you have to decide is when to start publishing information in 2.1. There is no way to know whether BDIIs have upgraded or not. The only thing we currently publish in the BDII is the version of the bdii package, and this package is not affected here. So at some point you may start publishing GLUE 2.1 and depending on which BDII you query you may see this information or not. For a period of time the information published is going to be "chaotic": cloud resources published in both 2.0 and 2.1, BDIIs publishing cloud resources, BDIIs missing cloud resources...

AppDB is also queryiing the BDII to get the cloud information automatically, and there is plan to add COMPs and other cloud brokers too. I will push these latest to the 2.1 draft, also because they need information who is not published in our 2.0 profile right now.

This is something you can do at any time. Ideally you could query 2.1 and fallback to 2.0 for a while. How much are you using right now the cloud information published in 2.0? If you don't have a specific use case, it may be worth considering switching off 2.0 as soon as you start publishing 2.1, only if you could live with the fact that not all BDIIs will publish this info until they upgrade. Like this there is no mix of cloud resources in 2.0 and 2.1, although maybe this is not such a problem after all... Regards, Maria

Dear Maarten,

...

What you have to decide is when to start publishing information in 2.1. There is no way to know whether BDIIs have upgraded or not. The only thing we currently publish in the BDII is the version of the bdii package, and this package is not affected here. So at some point you may start publishing GLUE 2.1 and depending on which BDII you query you may see this information or not.

Therefore we need a new BDII release along _with_ the new schema.

As I said in my first reply to Salvatore, we need to release a new version of the glue-schema package, if this is what you mean by "a new BDII release". Regards, Maria

Dear Maarten,

...

...

...

What you have to decide is when to start publishing information in 2.1. There is no way to know whether BDIIs have upgraded or not. The only thing we currently publish in the BDII is the version of the bdii package, and this package is not affected here. So at some point you may start publishing GLUE 2.1 and depending on which BDII you query you may see this information or not.

Therefore we need a new BDII release along _with_ the new schema.

As I said in my first reply to Salvatore, we need to release a new version of the glue-schema package, if this is what you mean by "a new BDII release".

No, we also need to increase the BDII version.

To avoid the chaos that would otherwise ensue.

There is no chaos because of this, the chaos remains in any case no matter what you know about what´s installed out there. And in any case, it´s still an assumption to think that because a site upgraded the bdii package it also upgraded the glue-schema package. Maybe a fair assumption, but an assumption after all. Regards, Maria

Hi all, On 2014-07-31 17:38, Maarten.Litmaath@cern.ch wrote:

Hi,

[...]

Since only the BDII version is published today, the easiest way is to increase its version number, unfortunately. At the same time we may want to consider publishing the schema version explicitly.

I'd like to remind that since I took over the LDAP schema I introduced a version field in the file for the purposes detailed above:

grep -A12 '00-Version.schema' /etc/ldap/schema/GLUE20.schema # File: schema/00-Version.schema # URL: http://redmine.ogf.org/projects/glue-wg # Doc: GLUE Specification 2.0 (March 3, 2009) # Section: - Version file # Authors: Laurence Field (laurence.field@cern.ch), CERN # David Horat (david.horat@cern.ch), CERN # Florido Paganelli (florido.paganelli@hep.lu.se), Lund University # # # Schema Version: 2.0 # Last updated: 2014-07-02 #

ARC already used this information to do certain things with the LDAP tree. I read somewhere that there might be smarter ways to do this, for example telling the LDAP server itself and perform a special query on the schema, but these are not implemented and usually badly documented. Cheers, Florido -- ================================================== Florido Paganelli ARC Middleware Developer - NorduGrid Collaboration System Administrator Lund University Department of Physics Division of Particle Physics BOX118 221 00 Lund Office Location: Fysikum, Hus B, Rum B313 Office Tel: 046-2220272 Email: florido.paganelli@REMOVE_THIShep.lu.se Homepage: http://www.hep.lu.se/staff/paganelli ==================================================

Hi Salvatore, On 2014-08-01 09:46, Salvatore Pinto wrote:

Hi Florido, if ARC or other middleware providers uses this information, do you think there is the chance that 2.1 could be recognized as an invalid version and thus we will need to update the ARC and cloud middleware providers as well?

ARC will just do things in such a way that the information is always published according to the schema installed in the system. There will be no need for update. I am resposnible for that piece of code. So far we don't have any cloud extension to ARC, that is, there is no current plan to publish cloud objects. It might happen in the future, but this we discuss once we have a working schema. Obviously if the system where ARC runs ships GLUE2 LDAP schema 2.0, cloud objects will just be dropped. I don't know about the cloud middleware providers. It all depends on how they use this info. Cheers, Florido

Cheers, Salvatore.

On 31/07/2014 18:15, Florido Paganelli wrote:

...

Hi all,

On 2014-07-31 17:38, Maarten.Litmaath@cern.ch wrote:

...

Hi,

[...]

...

Since only the BDII version is published today, the easiest way is to increase its version number, unfortunately. At the same time we may want to consider publishing the schema version explicitly.

I'd like to remind that since I took over the LDAP schema I introduced a version field in the file for the purposes detailed above:

...

grep -A12 '00-Version.schema' /etc/ldap/schema/GLUE20.schema # File: schema/00-Version.schema # URL: http://redmine.ogf.org/projects/glue-wg # Doc: GLUE Specification 2.0 (March 3, 2009) # Section: - Version file # Authors: Laurence Field (laurence.field@cern.ch), CERN # David Horat (david.horat@cern.ch), CERN # Florido Paganelli (florido.paganelli@hep.lu.se), Lund University # # # Schema Version: 2.0 # Last updated: 2014-07-02 #

ARC already used this information to do certain things with the LDAP tree.

I read somewhere that there might be smarter ways to do this, for example telling the LDAP server itself and perform a special query on the schema, but these are not implemented and usually badly documented.

Cheers, Florido

-- ================================================== Florido Paganelli ARC Middleware Developer - NorduGrid Collaboration System Administrator Lund University Department of Physics Division of Particle Physics BOX118 221 00 Lund Office Location: Fysikum, Hus B, Rum B313 Office Tel: 046-2220272 Email: florido.paganelli@REMOVE_THIShep.lu.se Homepage: http://www.hep.lu.se/staff/paganelli ==================================================

Hi Salvatore, Then we don´t need to worry. The use cases are clear and as you have described, the applications or users who will make use of this information will know which top BDIIs they have to use. Apart from this use case in EGI, I´m not aware of any other use cases interested in using this information although I will check in WLCG. Regards, Maria

-----Original Message----- From: Salvatore Pinto [mailto:salvatore.pinto@egi.eu] Sent: 01 August 2014 09:45 To: Maarten Litmaath; Maria Alandes Pradillo Cc: Florido Paganelli; glue-wg@ogf.org Subject: Re: [glue-wg] VOTE by 8/1: GLUE 2.1 for describing cloud resources?

Hi, please consider that this issue will be only for the Top BDIIs and only for the cloud resources. For the site BDIIs, this will hot have this issue, since to publish cloud resources they will need to update to version 2.1 (and our fedcloud probes will check that), so update of the cloud-provider-info script will force update of the glue2-schema.

Regarding the Top-BDII, the problem is also mitigated. In the fedcloud, we have two kind of "users": the automatic systems (like AppDB and in the future COMPs, SlipStream, etc...), who will install and manage a Top-BDII locally, so they will ensure that the Top-BDII is updated when they will update their query to use 2.1 (and fall back to 2.0) and the "manual users", which uses the Top-BDIIs indicated by the FedCloud, reported into a wiki page, which we can easily check if they are updated to 2.1 or not.

Cheers, Salvatore.

On 31/07/2014 17:38, Maarten.Litmaath@cern.ch wrote:

...

Hi,

...

There is no chaos because of this, the chaos remains in any case no matter what you know about what´s installed out there. And in any case, it´s still an assumption to think that because a site upgraded the bdii package it also upgraded the glue-schema package. Maybe a fair assumption, but an assumption after all. I think we need a clear way to see which BDIIs supposedly support the use of GLUE 2.1. Else we may run into nasty operational issues, where things sometimes work and sometimes do not, while at the same time all BDIIs present themselves as equal... That would be a _disservice_ to the user communities and their support infrastructures.

Since only the BDII version is published today, the easiest way is to increase its version number, unfortunately. At the same time we may want to consider publishing the schema version explicitly.

-- Salvatore Pinto Cloud Technologist, EGI.eu e-mail: salvatore.pinto@egi.eu skype: salvatore.pinto0 Science Park 140, Amsterdam, The Netherlands

Dear Stephen,

Alternatively we could explicitly publish the schema version in the service information for the site and top BDIIs, e.g.

GLUE2EntityOtherInfo: SchemaVersion= 2.0.10

Yes, maybe this is something you can already implement in the update you are preparing for the service provider. Like this we start getting this information published. Regards, Maria

Hi Maria & Stephen, +1 from me to implement this, taking the info from the schema itself, as Florido noted. Cheers, Salvatore. On 31/07/2014 17:34, Maria Alandes Pradillo wrote:

Dear Stephen,

...

Alternatively we could explicitly publish the schema version in the service information for the site and top BDIIs, e.g.

GLUE2EntityOtherInfo: SchemaVersion= 2.0.10 Yes, maybe this is something you can already implement in the update you are preparing for the service provider. Like this we start getting this information published.

Regards, Maria _______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg

-- Salvatore Pinto Cloud Technologist, EGI.eu e-mail: salvatore.pinto@egi.eu skype: salvatore.pinto0 Science Park 140, Amsterdam, The Netherlands

Dear Salvatore,

...

But the cloud info provider is not running in the site BDII, this is running in the resource BDII of the cloud resources, right? ehm... no, it is running at the site BDII, contacting the resource remotely using the resource native APIs... see https://wiki.egi.eu/wiki/Fedclouds_BDII_instructions for more information.

But then you are putting a dependency on the site BDII. Or do you expect site BDIIs publishing cloud resources to do these "manual" steps? The model in the BDII is that resources run the info providers they need to publish the information that it is required and then this is propagated in a hierarchical way to site BDIIs and top BDIIs by using the ldap info providers. What you are now doing is to run the info provider in the site BDII. Is this to avoid installing a resource BDII in the cloud resources?

ok, if I understand well, you propose to go with the following phases: 1. Update the schema package 2. Update the Top-BDIIs (only the ones used by the FedCloud are required) and the clients to support 2.1 and then fall back to 2.0. 3. Produce an update of the cloud-info-provider script, which requires the GLUE2 schema update, and start publish the information in 4. (optional, in the end) Update the clients to support only 2.1

My only problem is how, during #2, we can test that everything works if we have site BDIIs publishing 2.1 (as per #3). But maybe we can define some test sites who will publish information in both 2.1 and 2.0 during phase #2.

I think this is really up to you. I think either case is fine (2.0 and 2.1 coexist vs only 2.1), as long as you know what your clients are consuming to avoid getting a duplication of resources because they are published in both 2.0 and 2.1. Regards, Maria

david.meredith@stfc.ac.uk [mailto:david.meredith@stfc.ac.uk] said:

I would be in favour of adding new 2.1 cloud entities provided:

* 2.1 only adds new (cloud) implementations that are derived from the 2.0 abstract entities. * There are no changes to the 2.0 entities, especially no changes to the core abstract entities.

Can I clarify exactly what you're saying: you want no completely new free-standing objects (analogous to e.g. Benchmark or StorageServiceCapacity)? And you don't want even optional attributes added to the abstract classes? Both of those would be OK in LDAP in the sense that if a server doesn't publish new objects/attributes it makes no difference whether the schema contains them or not. Stephen -- Scanned by iCritical.

Meredith, David (STFC,DL,SC) said:

What I'd like to try and avoid, if possible, is making changes to the 2.0 core abstract entities because this would require changes to the published 2.0 XSD - i.e. new attributes would need to be added to the base XSD. Conversely, only adding new attributes to new (cloud) entity sub-types would not require change to the base XSD (since we're just extending). Isn't this the whole point for inheritance and sub-type specialisation?

Well, not from my point of view, but XML may have different constraints. In LDAP there is only one schema in the server so changing existing objects is no harder than adding new ones, the problems with upgrading are to do with publishing and querying rather than the schema itself. The advantage of inheritance to me is somewhat the opposite, that if you do change a base type you automatically get it in any derived type. Anyway as far as clouds go it's probably a moot point since I guess you would only want new cloud-related things in the cloud objects. Stephen -- Scanned by iCritical.

Hi all, I obliviously vote for #2 too, with the constraints specified by both David and Stephen. Seeing the replies from the list, I suppose that the decision is to go for GLUE 2.1 . If so, I think the next steps would be these: 1. Update the latest 2.1 draft with track changes and including H2020 projects requirements (eg. representing cost associated to cloud services and additional policy-related info) 2. Request to the mailing list for collecting non disruptive changes to the document (eg. fixing of typos, including references to open enumeration procedure document) 3. Produce a draft update of the JSON, XSD & LDAP schema (check if it is needed to update SQL too), to be released together with the draft 2.1 Please, let me know your opinion about these next steps (especially if I am missing something). Cheers, Salvatore. On 29/07/2014 10:18, david.meredith@stfc.ac.uk wrote:

Hi all, I would be in favour of adding new 2.1 cloud entities provided:

* 2.1 only adds new (cloud) implementations that are derived from the 2.0 abstract entities. * There are no changes to the 2.0 entities, especially no changes to the core abstract entities.

This is important for the XSD rendering (and other renderings too I imagine). For an XSD rendering, the new cloud entities would be defined using a new namespace defined in a separate XSD. The new cloud entities would need to extend the existing abstract BaseTypes defined by the 2.0 element inheritance mechanism. An XML instance doc would still use the existing 2.0 '<GLUE2>' root element, and would import the new cloud entities to be nested under '<GLUE2>' using the existing SubstitutionGroup mechanism. When validating an instance doc, both the 2.0 XSD and 2.1 cloud extensions XSD would be needed for validation (quite normal).

Thanks, David

...

-----Original Message----- From: JP Navarro [mailto:navarro@mcs.anl.gov] Sent: 23 July 2014 16:36 To: OGF GLUE Working Group Subject: [glue-wg] VOTE by 8/1: GLUE 2.1 for describing cloud resources?

GLUE Working Group:

We have been discussing two options for using the GLUE schema to describe cloud services. You may review these options here: - http://redmine.ogf.org/dmsf_files/13294?download=

By August 1, please vote for one of the following options:

1) Use GLUE 2.0 with a Community Practice Profile that details how to describe cloud infrastructure using the existing 2.0 entities 2) Develop GLUE 2.1 adding new Cloud Computing entities (would be 100% backward compatible with GLUE 2.0 for non-Cloud entities)

For reference, a preliminary GLUE 2.1 Specification is available here: - http://www.ogf.org/pipermail/glue-wg/2014-January/001527.html

Regards,

JP _______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg

-- Salvatore Pinto Cloud Technologist, EGI.eu e-mail: salvatore.pinto@egi.eu skype: salvatore.pinto0 Science Park 140, Amsterdam, The Netherlands