- glue-wg - lists.ogf.org

Glue 2.1 draft as implemented in XSEDE
by Blau, Eric B. 27 Jun '18

27 Jun '18

This is a note by way of saying that XSEDE has just released a new version of IPF, our Information publishing framework that publishes data about XSEDE resources in a Glue2 schema. The new version, 1.4, publishes Accelerator information (at least for resources using SLURM—it hasn’t been implemented for all schedulers), based off of an earlier draft of the Glue2.1 draft spec. It appears to me that what has changed between that version of the 2.1 draft and the current 2.1 draft are cosmetic differences—some of the new entities have different names in the most recent draft. I will update IPF to publish under the 2.1 standard once it is final (IPF 1.4 was too far along in the testing process by the time I realized there were changes to make it practicable to make the schema changes for this release). Eric Blau

3 2

list of authors in glue2.1 draft
by Alessandro Paolini 15 Jun '18

15 Jun '18

dear all, we would need to update the list of authors and the related information in the draft. I cannot verify the correctness of the information about all the people that worked to the previous versions of the document: could you please have a look at it (first page, and then chapter 12) and update the information that you know are old? who should be marked as "editor" and "co-chair"? thanks in advance, Alessandro -- Dr. Alessandro Paolini Operations Officer - EGI Foundation Science Park 140 1098 XG Amsterdam The Netherlands skype: alessandro.paolini.egi ********************************* "I believe in the power of laughter and tears" "as an antidote to hatred and terror" "A day without laughter" "is a wasted day" >>> Charlie Chaplin

2 1

NOTES: GLUE WG teleconference, Tuesday, June 12, 2018
by Navarro, JP 15 Jun '18

15 Jun '18

Participants: Allesandro Paolini Baptiste Grenier Enol Fernandez Paolo Andreetto Stephen Burke JP Navarro Agenda notes: > 1) Discuss GLUE2.1 draft with cloud and accelerator entities > Lead: Alessandro Paolini and colleagues > https://docs.google.com/document/d/1BpeTBQfCn8xRo4Ff38esRhBwzoUHE4bVxba2rEA… <https://docs.google.com/document/d/1BpeTBQfCn8xRo4Ff38esRhBwzoUHE4bVxba2rEA…> > https://docs.google.com/document/d/1x2gUt2lJNevle8evQu6LcKr9xs5NHRd6FbNzb1y… <https://docs.google.com/document/d/1x2gUt2lJNevle8evQu6LcKr9xs5NHRd6FbNzb1y…> Meeting folder and slides: - https://redmine.ogf.org/dmsf/glue-wg?folder_id=6625 <https://redmine.ogf.org/dmsf/glue-wg?folder_id=6625> Schema discussion: - Describing hypervisor as an attribute rather than as a first order entity - Place a GLUE 2.1 Change Log in a new Appendix C IMPORTANT: All other proposed 2.1 schema changes should be sent to the list by June 22, 2018. Reference implementations: - OpenStack reference LDAP implementation that is a little behind the latest proposed schema: https://github.com/EGI-Foundation/cloud-info-provider <https://github.com/EGI-Foundation/cloud-info-provider> https://github.com/gwarf/cloud-info-provider/blob/glue_2.1_keystone/etc/tem… <https://github.com/gwarf/cloud-info-provider/blob/glue_2.1_keystone/etc/tem…> - XSEDE has an Accelerator JSON implementation (will send to the list next week any proposed schema adjustments) Next steps/action items: 1) Draft GLUE 2.1 document highlighting the changes versus 2.0 to be shard with the list by the end of June (Allesandro and company) - Continue to draft changes using Google doc 2) Internal GLUE-WG comment period thru the middle of July (~2 weeks) 3) Final cleanup, remove highlighted changes, and prepare for release to the council by the end of July (Allesandor, JP) - Final document can be an exported Word and/or PDF 4) The council can circulate for the required public comment by early August Meeting participants please reply/comment if any important discussion items are missing from these notes. Regards, JP

2 1

GLUE2.1 draft: request for final discussion and approval
by Alessandro Paolini 12 Jun '18

12 Jun '18

Dear all, over the last months it was discussed with the cloud providers the necessary network attributes to publish in the BDII and we arrived to a good compromise. So we would like to discuss with you the latest changes to the draft before getting the approval for implementing the new schema in production. This is the draft document: https://docs.google.com/document/d/1BpeTBQfCn8xRo4Ff38esRhBwzoUHE4bVxba2rEA… I still need to update the figures, which for the moment you can find here: https://docs.google.com/document/d/1x2gUt2lJNevle8evQu6LcKr9xs5NHRd6FbNzb1y… Here is the list of the latest classes and attributes added/modified: Classes: ComputingShareAcceleratorInfo ComputingManagerAcceleratorInfo AcceleratorEnvironment CloudComputingShareAcceleratorInfo CloudComputingManagerAcceleratorInfo CloudComputingImageAcceleratorInfo CloudComputingImageNetworkTraffic CloudComputingInstanceAcceleratorInfo Attributes: GLUE2EndpointAuthentication GLUE2ComputingEndpointAuthentication GLUE2CloudComputingEndpointAuthentication GLUE2CloudComputingShareDefaultNetwokType GLUE2CloudComputingSharePublicNetwokName GLUE2CloudComputingShareProjectID GLUE2CloudComputingVirtualAcceleratorComputeCapability GLUE2CloudComputingVirtualAcceleratorVirtualizationType GLUE2CloudComputingImageContextualizationName GLUE2CloudComputingImageContextualizationVersion GLUE2CloudComputingImageDescription GLUE2CloudComputingImageVersion Storage attributes: StorageShareNumberOfFiles (requested by ATLAS) Best regards, Alessandro -- Dr. Alessandro Paolini Operations Officer - EGI Foundation Science Park 140 1098 XG Amsterdam The Netherlands skype: alessandro.paolini.egi ********************************* "I believe in the power of laughter and tears" "as an antidote to hatred and terror" "A day without laughter" "is a wasted day" >>> Charlie Chaplin

7 26

GLUE WG teleconference, Tuesday, June 12, 2018
by Navarro, JP 12 Jun '18

12 Jun '18

Greetings GLUE WG, Our next GLUE WG teleconference will be: : Tuesday, June 12, 2018 : 15:30-17:00 Central Europe, 8:30-10:00 AM US Central We will use Zoom teleconference and screen sharing: : By Browser: https://zoom.us/j/451341881 : By Phone: +1 646 876 9923 or +1 669 900 6833 or +1 408 638 0968 : By iPhone one-tap: +16468769923,,451341881# or +16699006833,,451341881# : International numbers: https://zoom.us/zoomconference?m=7BCOXsCfNrZcQ9fxuSwFbJJY84d8IBfK : Conference ID: 451 341 881 IMPORTANT: Recommend that those that need to configure and test Zoom please try to connect 15 minutes early. Planned agenda: 1) Discuss GLUE2.1 draft with cloud and accelerator entities Lead: Alessandro Paolini and colleagues https://docs.google.com/document/d/1BpeTBQfCn8xRo4Ff38esRhBwzoUHE4bVxba2rEA… https://docs.google.com/document/d/1x2gUt2lJNevle8evQu6LcKr9xs5NHRd6FbNzb1y… Regards, JP

1 1

comments about cloud attributes in GLUE2.1 draft
by Alessandro Paolini 08 Jun '18

08 Jun '18

hi Stephen, On 3 June 2018 at 14:56, Stephen Burke - UKRI STFC <stephen.burke(a)stfc.ac.uk > wrote: > > I'll look at the Cloud part later. > > These are mostly just technical comments, I don't know enough about how > you use the cloud information to judge whether the attributes are the right > ones, but I assume you've done some prototyping by now. > > In the Service you say that the AUP should be a URL - if you want that to > be a hard restriction you should type it as a URL rather than a string so > it could be type-checked. > Ok for changing the type in URL > > I'm a bit surprised that there are no new attributes in Endpoint, > especially as the text above the table suggests that there should be - did > something get missed out? > originally there were two attributes regarding the contextualisation, but we moved them in another class > > For the Share relations you have 1..* for Endpoint and Resource which is > wrong, they should follow the inherited attributes and be *, i.e. the only > mandatory relation is Service. The diagram is correct. I won't necessarily > spot all mistakes in relations so someone should probably go through and > check systematically that everything is consistent. > Ok > > I would rename ShareAcceleratorInfo to AcceleratorLimit (lots of Info > classes!). > Ok >> I find the Share limits a bit confusing, i.e. the InstanceMaxXxx. Presumably you've done that to allow for different Shares allowing users to instantiate the same VMs with different limits, but I wonder if that's an important feature - the alternative would be to put limits in InstanceType and just have multiple objects if you have instances with different limits. Anyway if I look at InstanceType you have mandatory CPU and RAM attributes which suggest that the values are just fixed, so I don't understand where the limits come in - are they just defaults, or what? Enol, Baptiste, what do you think? > For Manager: MaxRAM seems a slightly strange thing to publish, is it > useful? I'm also not sure what you'd do with the InstanceMax/Min > attributes. Clients should be looking at the Share rather than the Manager, > so these would normally be for monitoring or accounting use cases, is there > a need to monitor those things? > in general we use these attributes for getting information about the hardware features of the cloud framework, independently from the Share > > InstanceType: I'd be inclined to call the ID attribute something like > InstanceID or TemplateID rather than IDForEndpoint, I don't think there's > any particular need to specify a use in the name. > Ok, TemplateID > > Again if MarketPlaceID is supposed to be a URL you should type it as such > and perhaps call it URL rather than ID. Also a trivial point, I'd call it > Marketplace rather than MarketPlace, it's a single word. > Ok, MarketplaceURL, type URL > > The vCPU attribute should probably just be called CPU, the virtual part is > implied by the context (and you don't have vRAM). And as above, I don't > really understand what these mean if the number can be variable. > Ok > > Perhaps my ignorance, but is the network connectivity really bound to the > image rather than being specified when you instantiate it? > the network connectivity may depend on the software installed in the image > > NetworkPortsIn/Out, do you really want an explicit list of every port > rather than allowing a range? > yes, the applications require specific ports, and we need to take into account the network configuration in the several sites that is quite diverse > > OtherHardware should probably be an open enumeration, otherwise it's > effectively free text and not useful for selection. > Ok > > The text says CloudComputingPrice but the class is actually called > CloudServicePrice. > thank you, corrected! > > VirtualAcclelerator to InstanceType is *-*, better to have it *-1 and just > duplicate it if necessary - logically it's part of the Instance. Also I'd > again ask whether this is a realistic use case - will you really have VMs > with multiple GPU types? Again ComputeCapability should probably be an open > enumeration. > Marco, Paolo, Do you agree? >>CloudComputingImage: conceptually this seems like another kind of Resource. I don't think we ever considered having two types of Resource in one Service and maybe it's better not to do it explicitly, but I'd expect it to be similar, and it is except for not having a relationship to Manager. Doesn't a Manager manage Images too? Enol, Baptiste, what do you think? >>There's also no relationship to InstanceType - does that mean that any VM can run any OS? In principle, yes. Enol, Baptiste? >>For the first two attributes I have the same comments as for InstanceType. Ok >> DefaultPassword is a bit surprising, should that be published? I remember this was already discussed and agreed, I will check the past email threads. >>I find ImageAcceleratorInfo a bit surprising, are there OS images that require GPUs? And if so, it seems unlikely that there are images that require more than one type? Paolo, Marco, could you please comment on that? >>NetworkTraffic doesn't seem like the right name, maybe NetworkConfiguration? And is it really part of the Image rather than the Instance? Maybe I just don't understand what it's for, the explanation isn't very detailed. Ok for changing the name. These attributes are related to the application. Baptiste? >>CloudComputingInstance: again I don't especially like the ID names, maybe ExternalID and InternalID? Ok for renaming (the first one could become VmID) Side question: the same names (IDFromEndpoint and LocalIDFromManager) are used in the ComputingActivity class. Do we change them as well? CEJobID and LocalJobID? >>It also seems like at least one of those should be mandatory so you can tell which job the object refers to. For Owner I'm not sure why you don't just make it an optional attribute and remove the CONFIDENTIAL value? Baptiste, Enol, what do you think? >>You have two attributes with ComputeManager in the name, should be ComputingManager. thank you, corrected! >>ToStorageService: again I'd call the IDForEndpoint attribute something different, maybe LocationID? Ok Cheers, Alessandro -- Dr. Alessandro Paolini Operations Officer - EGI Foundation Science Park 140 1098 XG Amsterdam The Netherlands skype: alessandro.paolini.egi ********************************* "I believe in the power of laughter and tears" "as an antidote to hatred and terror" "A day without laughter" "is a wasted day" >>> Charlie Chaplin

3 3

StorageSharePath multiplicity requested by Atlas
by Alessandro Paolini 08 Jun '18

08 Jun '18

Hi Paul, some time ago, after collecting the feedback from the Atlas community, you proposed to change the multiplicity of StorageSharePath from "0..1" to "0..*" This is the Stephen's comment about this change: "it still isn't clear to me that multiplicity * makes sense. This is supposed to be a default path, if there are several you don't know which to use. Also if there is any software using this (possibly not) it would expect a unique result and probably break if it got more than one. If this is really wanted I would say add a new multivalued attribute Paths or OtherPaths." I agree with him that adding a specific multivalued attribute would it be better than modifying the multiplicity of the existing one: would you agree as well? cheers, Alessandro -- Dr. Alessandro Paolini Operations Officer - EGI Foundation Science Park 140 1098 XG Amsterdam The Netherlands skype: alessandro.paolini.egi ********************************* "I believe in the power of laughter and tears" "as an antidote to hatred and terror" "A day without laughter" "is a wasted day" >>> Charlie Chaplin

5 9

New class and attributes for GLUE 2.1/cloud extensions
by Baptiste Grenier 30 Nov '17

30 Nov '17

Dear all, Please find here a proposition for the creation of a new clas and some attributes to replace the use of OtherInfo to represent some usefull cloud-related information. We would like to discuss this with you in order to integrate it in the GLUE2.1 draft as soon as possible. The names should be quite self explanatory, please let me know if you need any clarification. Best regards, Baptiste ## For CloudComputingEndpoint * GLUE2CloudComputingEndpointAuthentication * EndpointAuthentication_t * New type * Mandatory * Open enumeration * Default Values: X509-VOMS, OIDC ? * Replacing: * GLUE2EntityOtherInfo : Authn=X509-VOMS ## For CloudComputingImage * GLUE2CloudComputingImageDescription * New type * Mandatory * String * Replacing: * GLUE2EntityOtherInfo: description:Image for TinyCoreLinux * GLUE2CloudComputingImageVersion * New type * Mandatory * String * Replacing: * GLUE2EntityOtherInfo version:XXX * ContextualizationName_t * Existing type, just need to update default values * Open enumeration * Default Values: cloudinit (without a -), bash * Changing cloudinit to cloud-init (official name) * GLUE2CloudComputingImageNetworkInput / GLUE2CloudComputingImageNetworkOuput * Used to represent communication ports used/required/exposed by the image * Custom objectClass: NetworkTraffic * Optional * Can be specified multiple times * Replacing: * GLUE2EntityOtherInfo traffic-in:XXXXX * GLUE2EntityOtherInfo traffic-out:XXXXX * ObjectClass NetworkTraffic * New object class * inheriting from Entity * under/related to CloudCompoutingImage * Name * CloudCompoutingImageNetworkTraffic? * CloudCompoutingNetworkTraffic? * Attributes * NetworkTrafficProtocol_t: * Custom type * Mandatory * Closed enumeration * Values: all, tcp, udp, cmp, ipsec * NetworkTrafficType_t: * Custom type * Mandatory * Closed enumeration * Values: inbound, outbound * NetworkTrafficRange_t: * Custom type * Mandatory * String * Example (default?): 0.0.0.0/0 * NetworkTrafficPort_t: * Custom type * Mandatory * String * Example: 443 -- Baptiste Grenier EGI Foundation - Operations Officer Phone: +31 627 860 852 Skype: baptiste.grenier.egi

2 6

Handling VO specific information in Glue 2.1
by Enol Fernández 19 Oct '17

19 Oct '17

Hi *, as we start to implement v2.1 of the GlueSchema for the EGI federated cloud we have realised that we miss some VO specific information and we are unsure how/where is the best way to publish this: Users belong to several VOs and sites may support more than one of those. When users authenticate against a given endpoint, the endpoint will return a list of local projects/groups that the users is allowed to use. Each project/group supports a VO in our current implementation. While in the past this was not an issue since the user authenticated with a VOMS proxy that only contained information a single VO and therefore the endpoint would just return a single project. Now with the transition to federated identity, the endpoint will receive claims on every VO the user is member of and there is no way for the user to determine which local project/group to use. We would need a way to publish a site-defined identifier of the project/group that supports each VO at a given site, so user could just match the VO with that id and select the appropriate one during authentication. We have checked the current draft of the schema and haven't seen a clear place to publish this kind of information. Our current guess would be to include this in the AccessPolicy or MappingPolicy. Since in our implementation we are using shares as a way to express VO information probably for us the MappingPolicy is the best fit, but would like to get your input on the best way to proceed. PS: I tried to submit this email before being subscribed, so apologies if it gets sent twice. > -- Enol Fernández | Cloud Technologist | EGI Foundation enol.fernandez(a)egi.eu

4 6

new version of UML class diagrams for GLUE2.1
by Alessandro Paolini 02 Oct '17

02 Oct '17

Dear all, I've made a first version of the UML class diagrams (htc and cloud) comprising the new classes and other changes, for replacing the ones currently included in the GLue2.1 Specification draft document. Please have a look at them and let me know if you have any comment: https://docs.google.com/document/d/1x2gUt2lJNevle8evQu6LcKr9xs5NHRd6FbNzb1y… cheers, Alessandro -- Dr. Alessandro Paolini Operations Officer - EGI Foundation Science Park 140 1098 XG Amsterdam The Netherlands skype: alessandro.paolini.egi ********************************* "I believe in the power of laughter and tears" "as an antidote to hatred and terror" "A day without laughter" "is a wasted day" >>> Charlie Chaplin

1 0