- glue-wg - lists.ogf.org

OGF35 GLUE WG summary
by JP Navarro 19 Jun '12

19 Jun '12

Greetings GLUE WG, Summary of the discussions held in the GLUE WG session at OGF35 on Sunday, June 17, 2012. Many thanks for the strong participation with close to 20 present in person and 2 joining over Skype. Regards, JP and Shiraz Discussion Summary 1. XML Rendering ================ We discussed the pros and cons of hierarchical, hybrid, and flat. Flat rendering Pros ------------------- * We propose that elements can only appear in one place in a valid GLUE2 document ** Ensuring that elements can't be duplicated * Makes it easier to publish subsets of information * Element can be cross-referenced * Easier to generate and publish * Easier to cherry pick desired elements to import into external schemas Flat rendering Cons ------------------- * Harder to construct complex queries ** Parent/child relationships followed using nested queries Hierarchical Pros * Easier to construct complex queries * Naturally preserves parent/child relationships Hierarchical Cons ----------------- * Very hard to "slice" big document into pieces * Duplication of information may happen under different parents * Publishing subsets and merging is hard to implement * Must still use cross-references to content published in separate documents * The parent-child relationship mandated is sometimes too strict Hybrid Pros and Cons -------------------- * Inherited from both Flat and Hierarchical * Has information duplication Discussed that a flat to hierarchical translator could be developed once for the entire community if we determine that some consumers require the hierarchical rendering to perform certain queries. Adopted/Approved ---------------- # Flat rendering # Root “glue:Entities” element required for GLUE2 documents # Main elements are also global (can be Root) to enable inclusion by other schemas # Substitution groups for main element classes to facilitate consistency and adoption by other schemas Action Items ------------ # New XML rendering document in 4 weeks # New XML schema in 4 weeks # Follow OGF process to approve rendering and schema documents LDAP Rendering ============== Extensively discussed whether to publish a DIT or not. Universal agreement that we should not mandate a DIT. We will discuss in a future teleconference the pros and cons and decide whether to publish. Note that if we publish the DIT could be an implementor experience document. Naming Enumerations and Types ============================= We plan to discuss and implement a team based process for reviewing proposed Enumeration and Types for adoption and for publishing in a shared public location. Perhaps this could be part of a GLUE profile that applies to all the renderings. JSON Rendering ============== Shiraz presented the EMI Services JSON rendering. We will discuss JSON renderings in future teleconferences.

2 1

LDAP rendering document: new version as an outcome of Lund review
by Balazs Konya 18 Jun '12

18 Jun '12

Dear All, I've just uploaded a new version of the "GLUE v. 2.0 – Reference Realization to LDAP Schema" ldap rendering draft to the glue2 gridforge area. The uploaded new version contains comments and tracks all the changes we made in the document. Please find the files here: - word with all the changes tracked: https://forge.ogf.org/sf/go/doc15518?nav=1 - clean pdf: https://forge.ogf.org/sf/docman/do/downloadDocument/projects.glue-wg/docman… - pdf with all the changes tracked: https://forge.ogf.org/sf/docman/do/downloadDocument/projects.glue-wg/docman… During the last weeks (months) the NorduGrid/ARC team in Lund carried out a thorough review and major cleanup of the ldap rendering document. Basically we took the document and checked it against our and other LDAP implementations. The ldap rendering draft was created long time ago and since 07/01/2010 it was not touched, at many places it became obsolete. Furthermore, back then when the ldap rendering discussion took place there was only one ldap implementation (the glite-bdii), unfortunately ARC was busy with the xml glue2 rendering part and had no possibility to check/follow the ldap area. Furthermore, the ldap team did not follow the xml rendering discussions although there is quite similarity in the two data models. Now that ARC implements both an LDAP and XML rendering (i think we are the only one) we thought it was time to review and update the LDAP rendering draft. Here are some of the items we modified or run into (everything is tracked in the new version!): - The old document contained a proposed DIT that was incomplete and not followed by any of the actual implementations. We almost completely rewrote the section on DIT, introduced three-level information structuring and provided three detailed pictures that correspond to actual implementation apart from minor proposed changes. - while defining the proposed DIT we tried to keep it in sync with the XML rendering, this was most visible in the selection of the grouping elements - corrected the datatypes to match the current schema used by EMI - made a comment on the usage of structural vs. auxiliary types. The current limited usage of structural types are questionable. - made a comment on the strange and unjustified (for us) choice on the LDAP attributenames selected to form DNs - made a note on the unfortunate choice of GLUE2GRoupID attribute that is not an ID - followed the RFC4512 terminology (e.g. renamed ldap objects to ldap entries) - to be consistent with the xml and sql rendering documents changed "implementation" to "realization" all over the text - made a note that the used OID allocation mechanism is not extensible when it comes to adding attributes to entry. Furthermore, the choice is strange, it is not applied consistently and its benefits are unclear. Florido will attend the OGF Glue2 session this Sunday and prepares a short presentation about our LDAP draft rendering review including open questions and proposed changes. regards, Balazs Konya and Florido Paganelli

5 11

OGF 35 meeting and preparation reminder
by JP Navarro 17 Jun '12

17 Jun '12

Dear GLUE WG, Just a reminder of our OGF 35 GLUE2 working group meeting this coming Sunday June 17th 13:30-15:00 Central European Time. http://ogf.org/gf/event_schedule/index.php?id=2530 Our agenda: i) Substitution group discussion and agreement ii) EGI service names and types discussion iii) Globalized schema and cross-references iv) Future directions discussion: JSON rendering, etc. Recent action items to prepare for this meeting: Action Item: Provide a slide for OGF on how to implement the Glue Entities Bag high-level element, probably with example: Assigned to: David Action Item: A slide will be presented in next OGF to explain the problem in greater detail. Assigned to: Warren, JP We will post meeting call-in information when it's available. Regards, JP

12 22

XML rendering comments
by Warren Smith 15 Jun '12

15 Jun '12

Hi all, Below are some comments about the current hybrid XML rendering (we talked about a lot of them on the last call): * Not all elements can currently be at the top level. Examples are ComputingShare, ComputingEndpoint, and several storage ones. I think major elements like these should also be allowed at the top level for consistency as well as not needing placeholder parent elements to have a valid glue2 document with this information. * I think that allowing a generic top-level element ("glue" or whatever) would be good. It would make it possible to have documents with several somewhat unrelated entities without forcing a lot of dummy hierarchical structure. For example, a document that has ComputingActivities and ExecutionEnvironments. I don't know of anyone that needs this example specifically, but I can imagine someone wanting to publish documents that contain the more dynamic information like this. * A different question is whether there should only be a generic top-level element (in contrast to the current approach that allows many of the entities to be the top-level element). I think I prefer to have a single top-level element and allow any of the entities to be under that element (with minOccurs="0"), just so that we have a little more predictable document structure. Based on the call, I don't think we'd have a consensus about this. * Something to think about is how to represent associations that aren't represented hierarchically (or aren't always represented hierarchically). An example is ComputingActivity and ComputingShare. In the current schema, ComputingShare_t has a associations to ComputingActivities using ID_t. ComputingActivity_t has a reference to a ComputingShare using a LocalID_t (I'm not sure how a LocalID differs from an ID, actually). I think that it may be better if the many refer to the one than having the one refer to the many or to refer both ways. By this I mean that ComputingActivity_t should include the LocalID_t of its ComputingShare, but ComputingShare_t should not include the ID_t of its ComputingActivities. I prefer this approach because in general I think that the "many"s may be updated more often than the "one"s (e.g. ComputingActivities may be updated more frequently than ComputingShares) and if the "one"s refer to the "many"s, they are forced to be updated. * Related to the above, I saw that ComputingActivity_t doesn't have a ComputingEndpointID element. Perhaps it should, depending on which direction we're representing associations. * A minor question is why have XXXies in the complex types? For example, ComputingActivities in ComputingEndpoint_t? An alternative is to just have ComputingActivity with maxOccurs="unbounded". Thanks and sorry for having so many points in one email, Warren

3 2

GLUE WG teleconference reminder, Thursday, May 31, 2012
by JP Navarro 01 Jun '12

01 Jun '12

Greetings Glue WG, Coordinates and agenda for tomorrow's GLUE WG teleconference follow: *NOTE* 14:00-15:00 Central Europe and 8:00-9:00 AM US Eastern (versus 9:00-10:00 AM US Eastern) Coordinates: : Thursday, May 31, 2012 : 14:00-15:00 Central Europe, 8:00-9:00 AM US Eastern Teleconference and screen sharing: : https://webconf.vc.dfn.de/ogf-glue-wg/ (Adobe Connect) : Login as Guest using your real name Agenda: 1) Proposal to introduce Service and Endpoint substitution groups by David Meredith (20 minutes) Discussion points: a) Review draft changes circulated by David, both the approach and chosen elements b) Next steps: Vote to approve at OGF35? Upload to Grid Forge? Any other next steps? Rendering document token-pen: David 2) Finalizing an XML rendering agreement (20 minutes) Discussion points: a) Can XSEDE adopt the globalized schema? b) Do we have an agreeable set of globalized elements? c) How to represent cross-references/associations for globalized elements? Background material: - Globalized schema: https://github.com/OGF-GLUE/XSD/blob/master/schema/GLUE2.xsd - TeraGrid/XSEDE schema: https://github.com/OGF-GLUE/XSD/blob/master/schema/teragrid_glue_2.0_r01.xsd 3) Other proposed/outstanding GLUE2 changes (10 minutes) a) EGI service name and type values. Put the list of service names and type enumeration values on Google doc per our last meeting? Who? b) Status of LDAP proposed changes, who has them and can circulate them? 4) Plan OGF 35 meetings (10 minutes) OGF 35 in Delft, Sunday, June 17, 13:30-15:00, room 1.180 Call-in ability arrangements (Shiraz) Possible OGF 35 agenda: i) Substitution group discussion and agreement ii) EGI service names and types discussion iii) XSEDE and globalized schema iv) GLUE2 JSON rendering, scope, flat, etc. See you tomorrow, Shiraz Memon JP Navarro

3 2

entity references in XML
by david.meredith＠stfc.ac.uk 31 May '12

31 May '12

Hi everyone, In today's WG meeting we discussed the nested element approach VS the 'flat' element referencing approach to model associations in XML. While I don't want to introduce more complexity/options, I have worked on an unrelated schema in the past that may be worth a quick look by the WG. This schema gives you the choice of using both inline (nested) element definitions and/or element references to model relationships. Schema and sample doc at: http://tools.ngs.ac.uk/ngstools/glue2proposal/nestingAndElementReferencesSa… Brief description: The data copy schema defines the <DataCopy> document root that defines a single <common> element and one or more <copy> operations. Each <copy> either references or nests an implementation of an abstract data source and an abstract data sink. Importantly, the schema allows both inline (nested) element hierarchies and/or element references to model relationships. Having the option of both these approaches reduces element repetition since multiple <copy> operations can reference the same shared data source or sink. For increased document clarity, shared elements MAY be nested as children of the global <common> element. E.g. a single dataSource can be defined as a child of <common> and be subsequently referenced by multiple <copy> operations. This approach may be useful for the GLUE rendering: A single entity is never repeated. Rather, shared entities are referenced. This is similar to the teragrid <Entities> element (with "glue:ID_t" references). In addition, I think we may? need to introduce a new element bag akin to the teragrid <Entities> element so that all entities can be defined in a single document (as per Warren's use-case). For example, consider defining both a <UserDomain> and an unrelated <Service> in the same doc; currently this would not be possible without an abstract parent container element such as 'Entities[1]'. ([1]Note, the abstract element and substitution group proposal still holds, e.g. all the concrete elements listed as children of the 'Entities_t' could/should be replaced with corresponding abstract definitions which would facilitate future extension/profiling of that schema). Cheers David ========================== David Meredith Scientific Computing Technology eScience Daresbury Laboratory Tel: +44 (0) 1925 603762 Fax: +44 (0) 1925 603100 (Site) Email: david.meredith(a)stfc.ac.uk Skype name: davidismeredith -- Scanned by iCritical.

2 1

GLUE WG teleconference availability, May 29-June 5
by JP Navarro 29 May '12

29 May '12

Greetings GLUE WG, Thanks for the strong participation in today's GLUE WG teleconference. We agreed to attempt one more teleconference before OGF 35, if possible the week of May 29 thru June 1. I took the liberty of adding June 4 and 5 as possible teleconference dates. Please mark your available for a teleconference on following poll: http://www.doodle.com/srnuvtfug4tyfkvk We will continue discussing the same topics from today's call and plan our OGF 35 working group meeting. Please reply if you would like to propose a new topic. Regards, JP ------------------------------------------------------------------------------ John-Paul Navarro (630) 252-1233 navarro(a)mcs.anl.gov http://www.mcs.anl.gov/~navarro Argonne National Laboratory Mathematics & Computer Science Division TCS Building 240, Office 4154 Argonne, IL 60439 GPG: 4EA9 C86B C0F0 113D 6306 98B7 3649 D6CB EFA8 4133

1 1

abstract elements in XML rendering
by david.meredith＠stfc.ac.uk 25 May '12

25 May '12

Hi everyone, Following the last WG meeting, I had the action to update the GLUE XML rendering document for the abstract elements / substitution group proposal. Please find attached the first draft that includes those proposals. We also discussed whether this design should be applied to other relevant elements/entities beyond AbstractService and AbstractEndpoint. I have therefore listed all the relevant elements in Table 2. I still need to update the XSD to support the additional abstract elements but this should be simple enough to do. I'll hopefully do this before next week's phone meeting. Cheers, David ========================== David Meredith Scientific Computing Technology eScience Daresbury Laboratory Tel: +44 (0) 1925 603762 Fax: +44 (0) 1925 603100 (Site) Email: david.meredith(a)stfc.ac.uk Skype name: davidismeredith -- Scanned by iCritical.

1 0

GLUE WG teleconference reminder, Wednesday, May 16, 2012
by JP Navarro 17 May '12

17 May '12

Greetings Glue WG, Coordinates and agenda for tomorrow's GLUE WG teleconference follow: Coordinates: : Wednesday, May 16, 2012 : 15:00-16:00 PM Central European Time (CET), 9:00-10:00 AM US Eastern Time (EST) Teleconference and screen sharing: : https://webconf.vc.dfn.de/ogf-glue-wg/ (Adobe Connect) : Login as Guest using your real name Agenda: 1) Proposal by David Meredith to introduce Service and Endpoint substitution groups (20 minutes) Discussion points: a) What are the benefits for using substitution groups? b) Should we apply this design to Service and Endpoint elements only, or also to other key elements? c) Next steps: Integrate this proposal in the XML rendering document? Any other next steps? 2) Finalizing an XML rendering agreement (20 minutes) Discussion points: a) What are the benefits of globalizing elements? b) Which elements do we want to globalize, should we make all significant elements root elements? c) How to represent cross-references/associations for globalized elements? Background material: - Globalized schema: https://github.com/OGF-GLUE/XSD/blob/master/schema/GLUE2.xsd - TeraGrid/XSEDE schema: https://github.com/OGF-GLUE/XSD/blob/master/schema/teragrid_glue_2.0_r01.xsd 3) Other proposed/outstanding GLUE2 changes (10 minutes) a) EGI service name and type values. What is the goal? Where do we record agreements on Enumerated values? b) Are there other attributes that we need a consistent enumeration for? 4) Future meetings (10 minutes) OGF 35 in Delft, Sunday, June 17, 13:30-15:00, room 1.180 Other agenda items besides discussing each of the above 3 topics? Should we plan to hold more teleconferences? Teleconference to prepare for OGF 35 the week of May 29 thru June 1? See you tomorrow, Shiraz Memon JP Navarro

3 2

Re: [glue-wg] GLUE : Lifetime and persistence of the attributes of the 'Activity' entity
by Etienne URBAH 16 May '12

16 May '12

On Mon, 08/11/2010 18:32, Etienne URBAH wrote: > Balazs, Laurence, Sergio and all, > > > Inside the GLUE 2.0 information model, most entities are clearly > persistent : Once an entity has been created, it exists until it is > explicitly destroyed. > > > The exception is the 'Activity' entity, whose persistence is NOT clear. > In particular : > > > A) Chapter 5.11 'Activity' states : > For instance, a broker Service submits a Grid job to a selected > execution Service; upon completion the execution Service submits a > logging record to an accounting Service. Each of these Services may have > associated an instance of a Grid Activity related to the lifecycle of > the job within the service. All instances refer to the same conceptual > job submitted by the user. > > > B) Chapter 6.9 'ComputingActivity' does NOT mention persistence, but the > attributes of the 'ComputingActivity' entity have in fact various levels > of lifetime and persistence. For example : > > B.a) 'IDFromEndpoint', 'State', 'Error', 'Owner' and 'OtherMessages' > have a meaning as soon as the ComputingActivity has been accepted by the > endpoint, and should be kept in logging records with NO time limit. > > B.b) 'WaitingPosition' has a meaning ONLY during the time when the > ComputingActivity has been accepted by the ComputingManager. As soon as > execution is finished, this attribute has NO meaning anymore. > > B.c) 'LocalIDFromManager', 'LocalOwner' and 'ExecutionNode' have a > direct meaning only during the time when the ComputingActivity is really > executed by the ComputingManager. After the end of the execution, they > only have an historical meaning, permitting to retrieve logging records > inside the logs of the ComputingManager if needed. > > B.d) 'UsedTotalWallTime', 'UsedTotalCPUTime' and 'UsedMainMemory' have a > meaning as soon as the ComputingActivity has been accepted by the > ComputingManager, and should be kept in accounting records (following UR > GFD.98) with NO time limit. > > B.e) 'ExitCode' and 'ComputingManagerExitCode' have a meaning only at > the END of the execution of the ComputingActivity, and should be kept in > accounting records (following the 'Activity Instance Documents > Specification' proposal) with NO time limit. > > > C) Inside some implementations of grid middleware (notably gLite), the > Information System has been designed to optimally manage a limited > amount (a few MB) of information about the grid infrastructure, but NOT > all logging and accounting records generated by Activities. So, the > above mentioned attributes are NOT managed by the Information System, > but by separate services, with adequate lifetime and persistence : > > C.a) All attributes are managed directly by the ComputingService, but > only during the execution of the ComputingActivity. As soon as execution > is finished, the ComputingService forgets everything about the > ComputingActivity. > > C.b) All attributes which need persistence (as required for security > audits) are saved on due time by the ComputingService inside logging > and/or accounting records. Even after execution is finished, these > logging and/or accounting records can be queried through dedicated > logging and accounting services. > > > Finally, the current definition of the 'Activity' entity does NOT > satisfactorily address the question of the lifetime and persistence of > its attributes. > > > I imagine 2 solutions for addressing this question of lifetime and > persistence (as required for security audits) of the attributes of the > 'Activity' entity : > > > 1) Keep the current 'Activity' entity unchanged > ------------------------------------------------ > The first solution is to keep the current 'Activity' entity unchanged, > but clearly indicate that during its lifetime, any activity may be > managed by various services, with various persistence. In particular, > besides the service dedicated to its execution, each Activity should > also managed by logging and accounting services for the persistence of > its attributes. > > This first solution keeps the current information model unchanged, but > requires very precise coordination between the various services in order > to simultaneously manage the SAME entity. > > > 2) Split the current 'Activity' entity > --------------------------------------- > The second solution is to split the current 'Activity' entity into : > > - A 'TransientActivity' entity managed during the adequate time span by > the service dedicated to its execution, > > - An 'ActivityRecord' entity whose instances are all attributes > requiring persistence, with 2 attributes : > - Type : Activity description (JSDL), Log record, Accounting record, ... > - Value > > This second solution breaks the current information model, but handles > more clearly the lifetime of attributes, and is more flexible. > > > In both solutions, the 'Activity' entity is already managed at any time > by at least one service. Then, is it really necessary that the > Information Service ALSO has any knowledge of the ComputingActivity ? > This is an open question. > > > Thank you in advance for : > - studying this question of lifetime and persistence of the attributes > of the 'Activity' entity, > - providing explanations on points which I could misunderstand, > - studying my 2 proposed solutions, > - expressing your opinions. > > > Best regards. > > ----------------------------------------------------- > Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS > Bat 200 91898 ORSAY France > Tel: +33 1 64 46 84 87 Skype: etienne.urbah > Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr > ----------------------------------------------------- >

2 1