v1.3 documentation, please!
Hi all, Sorry, this is a bit of rant. I'm in the process overhauling how dCache information is published into GLUE: a complete rewrite of how information is propagated. So I'm looking for definitive information about how to publish Glue through GIP. ... and there's the rub. I can't find any good references for how to provide this information. I've started a dCache wiki page[1] that holds various nuggets of information, but none of them seem complete or authoritative. [1] http://trac.dcache.org/trac.cgi/wiki/GLUE I understand that the current dCache system, is functional through Owen's sterling effort, with various trial-and-error attempts and information conveyed through emails. I'm looking to avoid this process as much as possible; having accurate information would be a good start! First off, the only GLUE v1.3 documentation I could find is from CNAF CVS [2]. The most recent version is "Draft 3---16 Jan 2007". I think Sergio was going to make this the final version, but could this process be clarified? BTW, there's a typo in the StorageArea, UsedNearlineSize description; there are other sections that look somewhat incomplete (e.g., grep for "TO BE ADDED") [2] https://forge.cnaf.infn.it/plugins/scmsvn/viewcvs.php/*checkout*/v_1_3/spec/pdf/GLUESchema.pdf?rev=32&root=glueschema Second, I could also find no authoritative source of information on Glue/LDAP binding for v1.3. There are some CNAF notes[3], but these mention, in bold, that "[t]his version is for early evaluation and is not meant to be deployed yet". Can someone say what is incorrect on this page? More importantly, can someone update it so the page is correct? [3] http://glueschema.forge.cnaf.infn.it/SpecV13/LDAP The CNAF notes also mention that certain attributes are "deprecated and their use should be removed from any software". Does this mean info-provides "must" (or "should", or "may", see RFC 2119) refrain from publishing deprecated attributes? For example, should no info-provider be publishing GlueChunkKey anymore? For publishing SRM spaces, I found a *proposal* for how this should be done[4]. Confusingly, this contradicts the GLUE/LDAP notes[3] as it stipulates that "mds-vo-name = local" must be used, whereas the notes state that mds-vo-name has been removed from the DIT; in practise "mds-vo-name = resource" (as a primary document, for feeding into GIP) is what seems to work. [4] https://twiki.cern.ch/twiki/bin/view/LCG/GSSDGLUEProposal Finally, (prodding Laurence) I could not find *any* statement about what format GIP supports as primary input (although I might have missed this). I'd assume this is roughly GLUE v1.3/LDAP, but apparently there's some magic happens with DNs with RDF of "mds-vo-name=resource". Is this (still) true? Is it required? What other mappings and translations are possible? Sorry, I'm sure this info is well known by others, but it doesn't appear to be documented. Can some knowledgable person take on responsibility for ensuring this is all properly documented? Cheers, Paul.
Hi Paul,
I'm in the process overhauling how dCache information is published into GLUE: a complete rewrite of how information is propagated. So I'm looking for definitive information about how to publish Glue through GIP.
... and there's the rub. I can't find any good references for how to provide this information. I've started a dCache wiki page[1] that holds various nuggets of information, but none of them seem complete or authoritative.
[1] http://trac.dcache.org/trac.cgi/wiki/GLUE
I understand that the current dCache system, is functional through Owen's sterling effort, with various trial-and-error attempts and information conveyed through emails. I'm looking to avoid this process as much as possible; having accurate information would be a good start!
First off, the only GLUE v1.3 documentation I could find is from CNAF CVS [2]. The most recent version is "Draft 3---16 Jan 2007". I think Sergio was going to make this the final version, but could this process be clarified? BTW, there's a typo in the StorageArea, UsedNearlineSize description; there are other sections that look somewhat incomplete (e.g., grep for "TO BE ADDED")
Second, I could also find no authoritative source of information on Glue/LDAP
Might this be a start: http://jra1mw.cvs.cern.ch:8180/cgi-bin/jra1mw.cgi/glue-schema/
binding for v1.3. There are some CNAF notes[3], but these mention, in bold, that "[t]his version is for early evaluation and is not meant to be deployed yet". Can someone say what is incorrect on this page? More importantly, can someone update it so the page is correct?
[3] http://glueschema.forge.cnaf.infn.it/SpecV13/LDAP
The CNAF notes also mention that certain attributes are "deprecated and their use should be removed from any software". Does this mean info-provides "must" (or "should", or "may", see RFC 2119) refrain from publishing deprecated attributes? For example, should no info-provider be publishing GlueChunkKey anymore?
That cannot be right! The GlueVOInfo object needs GlueChunkKey attributes to allow it to be linked to both the SE and the SA to which it refers! Examples here: https://twiki.cern.ch/twiki/bin/view/LCG/GSSDGLUEProposal
For publishing SRM spaces, I found a *proposal* for how this should be done[4]. Confusingly, this contradicts the GLUE/LDAP notes[3] as it stipulates that "mds-vo-name = local" must be used, whereas the notes state that mds-vo-name has been removed from the DIT; in practise "mds-vo-name =
Indeed... Well, we did not do that. I suppose it would have caused too many problems for just a minor version increment. Others to comment.
resource" (as a primary document, for feeding into GIP) is what seems to work.
Yes, it should be "mds-vo-name=resource" now. I have fixed the page.
[4] https://twiki.cern.ch/twiki/bin/view/LCG/GSSDGLUEProposal
Finally, (prodding Laurence) I could not find *any* statement about what format GIP supports as primary input (although I might have missed this). I'd assume this is roughly GLUE v1.3/LDAP, but apparently there's some magic happens with DNs with RDF of "mds-vo-name=resource". Is this (still) true? Is it required? What other mappings and translations are possible?
A site BDII queries site resources that have "mds-vo-name=resource" (BDII) or "mds-vo-name=local" (MDS GRIS), and substitutes "resource" or "local" with the site name. A top-level BDII _inserts_ "mds-vo-name=local" right before "o=grid" for various reasons. That is another matter to improve.
Hi Maarten, others, On Tuesday 06 May 2008 23:37:30 Maarten.Litmaath@cern.ch wrote: [...]
I could also find no authoritative source of information on Glue/LDAP
Might this be a start: http://jra1mw.cvs.cern.ch:8180/cgi-bin/jra1mw.cgi/glue-schema/
Thanks, it's a good start (although not really "enough").
The CNAF notes also mention that certain attributes are "deprecated and their use should be removed from any software". Does this mean info-provides "must" (or "should", or "may", see RFC 2119) refrain from publishing deprecated attributes? For example, should no info-provider be publishing GlueChunkKey anymore?
That cannot be right! The GlueVOInfo object needs GlueChunkKey attributes to allow it to be linked to both the SE and the SA to which it refers!
OK, do we know who has write access for that wiki? Could someone take on responsibility to either update or delete that page? [...]
resource" (as a primary document, for feeding into GIP) is what seems to work.
Yes, it should be "mds-vo-name=resource" now. I have fixed the page.
Thanks! (which page did you correct?)
[4] https://twiki.cern.ch/twiki/bin/view/LCG/GSSDGLUEProposal
Finally, (prodding Laurence) I could not find *any* statement about what format GIP supports as primary input (although I might have missed this. [...] A site BDII queries site resources that have "mds-vo-name=resource" (BDII) or "mds-vo-name=local" (MDS GRIS), and substitutes "resource" or "local" with the site name. A top-level BDII _inserts_ "mds-vo-name=local" right before "o=grid" for various reasons. That is another matter to improve.
Ta for the info. (Could this be documented, perhaps in a wiki?) Cheers, Paul.
glue-wg-bounces@ogf.org
[mailto:glue-wg-bounces@ogf.org] On Behalf Of Paul Millar said: I've started a dCache wiki page[1] that holds various nuggets of information, but none of them seem complete or authoritative.
Bear in mind that GLUE and the gip live in different domains, so nothing is authoritative for both (and the LDAP schema is different again). The GSSD wiki is at yet another level, it's only for LCG and is only a recommendation - and personally I think is wrong in at least one respect (see below). For Glue 1.3, which is presumably what you're currently implementing, OGF is not directly relevant although the web site does have various useful documents.
Second, I could also find no authoritative source of information on Glue/LDAP binding for v1.3.
Fundamentally there is only one authoritative source, namely the actual LDAP schema in cvs! (as Maarten already pointed out.) There are some CNAF notes[3], but these
mention, in bold, that "[t]his version is for early evaluation and is not meant to be deployed yet". Can someone say what is incorrect on this page? More importantly, can someone update it so the page is correct?
Persoanally I think that whole page should be deprecated ... most of it is not relevant to the deployed schema, and as I argued a while back the ChunkKey at least is important and we should keep it in Glue 2. Most of the other things are in the schema document anyway - specifically, the old GlueService URI and URL attributes have been removed from my service publisher, which is just in the process of being released, and the CE renaming of CPUs to Slots is true but we still publish CPUs as well.
For publishing SRM spaces, I found a *proposal* for how this should be done[4]. Confusingly, this contradicts the GLUE/LDAP notes[3] as it stipulates that "mds-vo-name = local" must be used, whereas the notes state that mds-vo-name has been removed from the DIT; in practise "mds-vo-name = resource" (as a primary document, for feeding into GIP) is what seems to work.
The mds-vo-name part varies depending on where you query: = resource at the lowest level (and in the info provider), = <site-name> from a site BDII, and = local from a top-level BDII. BDIIs all have their root DN at o=grid so you can always query from there, but the mds-vo-name is still used as a second-level filter even though MDS itself is now completely gone. As for the proposal, it suggests that all space tokens of a given type should be merged into a single SA, but personally I still think that we should publish one SA per token - this is e.g. how Jens' castor provider works and I find it extremely useful to keep track of free and used space. However, we should discuss this more offline as this is not an OGF/Glue 2 issue.
Finally, (prodding Laurence) I could not find *any* statement about what format GIP supports as primary input (although I might have missed this). I'd assume this is roughly GLUE v1.3/LDAP,
Well, it's LDIF conforming to the 1.3 schema, basically what you get from ldapsearch. As above, info providers need to publish their objects with DNs ending "mds-vo-name=resource,o=grid". It may also be worth pointing out that you can publish in two different ways: info providers publish entire objects, whereas gip plugins just publish the changes relative to a static file created by YAIM. Laurence will no doubt say more, but this is the wiki page: https://twiki.cern.ch/twiki//bin/view/EGEE/InformationSystem Stephen
On Wed, 7 May 2008, Burke, S (Stephen) wrote:
[...] MDS itself is now completely gone.
In gLite 3.1. There are still quite a few 3.0 services out there.
As for the proposal, it suggests that all space tokens of a given type should be merged into a single SA, but personally I still think that we should publish one SA per token - this is e.g. how Jens' castor provider works and I find it extremely useful to keep track of free and used space. However, we should discuss this more offline as this is not an OGF/Glue 2 issue.
I believe everyone agrees with that now. At the time there were reservations from various developers, so we cooked up a compromise.
Finally, (prodding Laurence) I could not find *any* statement about what format GIP supports as primary input (although I might have missed this). I'd assume this is roughly GLUE v1.3/LDAP,
Well, it's LDIF conforming to the 1.3 schema, basically what you get from ldapsearch. As above, info providers need to publish their objects with DNs ending "mds-vo-name=resource,o=grid". It may also be worth pointing out that you can publish in two different ways: info providers publish entire objects, whereas gip plugins just publish the changes relative to a static file created by YAIM. [...]
And relative to the provider output.
Hi Stephen, Thanks for the comments. My comments are interleaved: On Wednesday 07 May 2008 19:09:27 Burke, S (Stephen) wrote:
[...] nuggets of information, but none of them seem complete or authoritative.
Bear in mind that GLUE and the gip live in different domains, so nothing is authoritative for both (and the LDAP schema is different again).
Yes, indeed. To my mind there are four layers that should be documentation: a. GLUE, b. GLUE/LDAP binding, c. EGEE/(W)LCG's usage of GLUE, d. how to get information into GIP. Each layer builds on the previous one, so GLUE/LDAP documentation should (naturally) refer the reader to the relevant GLUE documentation for definitions of attribute values. Because of this, the GLUE/LDAP documentation can (and should) be terse, but I think it is still needed. I guess not all of this can be fixed in this forum: the last two are not really OSG GLUE WG issues; so perhaps documenting WLCG's usage and publishing through GIP should be an off-line discussion (from this mailing list).
The GSSD wiki is at yet another level, it's only for LCG and is only a recommendation - and personally I think is wrong in at least one respect (see below).
This is a management issue within WLCG. It's probably not helpful discussing this further here, although it's something I feel should be discussed.
For Glue 1.3, which is presumably what you're currently implementing, OGF is not directly relevant although the web site does have various useful documents.
True, but OGF-GLUE-WG does provide the GLUE/LDAP binding, which I feel currently is lacking some documentation.
Second, I could also find no authoritative source of information on Glue/LDAP binding for v1.3.
Fundamentally there is only one authoritative source, namely the actual LDAP schema in cvs! (as Maarten already pointed out.)
Ah! At the risk of pointing out (at length) the obvious: validity != correctness (disclaimer, my usage of the term "correct" here). A document (e.g., an LDIF file) is valid iff it satisfies a set of validation criteria, which are usually written in some validation language. With XML there are several validation languages (DTD, XML Schema, Relax NG, Schematron, ...), for LDAP there's the LDAP Schema language, etc. A document is correct iff it contains all the necessary information in the prescribed fashion, allowing the information to be accessed. I believe we are striving for correctness here, so people can use the information published; this is a much stronger requirement than validity, but also one that's harder to check, beyond applying validation checks. In an ideal world, all valid documents would also be structurally correct; however, due to limitations in the validation languages, one cannot always write validation tests to fully check a document is structurally correct. A specific example: GlueChunkKey is described in the LDAP schema as taking attribute values with the 1.3.6.1.4.1.1466.115.121.1.26 format, which is IA5 (e.k.a ISO 646 or IRA5): a 7-bit-clean sequence of 8-bit bytes (roughly ASCII without $ or ~). So a GlueChunkKey can take (more or less) any ASCII text as a valid attribute value. However, from the notes, this is clearly not what is expected, hence validity != correctness. So, for these reasons, one cannot (reasonably) claim that the schema is the single authoritative source as schemata can only validate a document, which is a lesser requirement than correctness.
There are some CNAF notes[3] [...]
Persoanally I think that whole page should be deprecated
Personally, I think the page should be deleted. It contains incorrect information and (apparently) isn't being maintained (sorry Sergio, that's how it looks.) However, as a proposal, I feel this WG should have an authoritative document (which could be a wiki page) that describes GLUE/LDAP for v1.3 GLUE. This document should refer the reader to the LDAP-Schema in CVS, but is should contain the extra "additional" information, like how to construct a valid GlueChunkKey, where they might be necessary, etc.
[...] Most of the other things are in the schema document anyway -
Most things are, but the GlueChunkKey (as one example) isn't. There may be other information that is LDAP-specific but not captured by the validation tests (LDAP Schema).
specifically, the old GlueService URI and URL attributes have been removed from my service publisher, which is just in the process of being released, and the CE renaming of CPUs to Slots is true but we still publish CPUs as well.
Yes, but these are EGEE/WLCG issues, right? [As an aside: does EGEE/WLCG document that GlueService -URI / -URL are no longer required? Is there anyway of being notification when these requirements change?]
The mds-vo-name part varies depending on where you query: = resource at [...]
Ta!
As for the proposal, it suggests that all space tokens of a given type should be merged into a single SA, but personally I still think that we should publish one SA per token - this is e.g. how Jens' castor provider works and I find it extremely useful to keep track of free and used space.
Again, I guess this is a EGEE/WLCG issue; not that it isn't interesting, but I'm trying to limit myself to just OGF-GLUE-WG issues.
However, we should discuss this more offline as this is not an OGF/Glue 2 issue.
Agreed. Which forum within EGEE (or WLCG) discusses its adoption of GLUE?
[How to publish to GIP] Well, it's LDIF conforming to the 1.3 schema, basically what you get from ldapsearch. As above, info providers need to publish their objects with DNs ending "mds-vo-name=resource,o=grid".
Aye, so "any old" LDAP isn't sufficient, it must be child objects under the "mds-vo-name=resource,o=grid" object.
It may also be worth pointing out that you can publish in two different ways: info providers publish entire objects, whereas gip plugins just publish the changes relative to a static file created by YAIM.
OK.
Laurence will no doubt say more, but this is the wiki page:
https://twiki.cern.ch/twiki//bin/view/EGEE/InformationSystem
Thanks! That page helps greatly. Cheers, Paul.
glue-wg-bounces@ogf.org
[mailto:glue-wg-bounces@ogf.org] On Behalf Of Paul Millar said: True, but OGF-GLUE-WG does provide the GLUE/LDAP binding, which I feel currently is lacking some documentation.
GLUE is only an OGF WG for the Glue 2.0 spec, so while it will produce an LDAP binding for 2.0 it doesn't have any direct responsibility for 1.3. Indeed for 1.2 and 1.3 there was no GLUE project as such, the original GLUE wound up several years back.
So, for these reasons, one cannot (reasonably) claim that the schema is the single authoritative source as schemata can only validate a document, which is a lesser requirement than correctness.
Being picky, the schema is in fact the only authoritative source - while it might be nice to have some additional definitive document we don't actually have one!
Most things are, but the GlueChunkKey (as one example) isn't.
That's because it's only in the LDAP implementation - but anyway, as I say I don't think it should be obsoleted, and it certainly isn't for 1.3.
specifically, the old GlueService URI and URL attributes have been removed from my service publisher, which is just in the process of being released, and the CE renaming of CPUs to Slots is true but we still publish CPUs as well.
Yes, but these are EGEE/WLCG issues, right?
The GlueService things are LCG-specific because those attributes were never officially part of GLUE at all (LCG had its own Service object at one time). CPUs -> Slots is a general schema issue.
[As an aside: does EGEE/WLCG document that GlueService -URI / -URL are no longer required? Is there anyway of being notification when these requirements change?]
Not really, because it's a question of fixing individual bugs - those attributes should never have been used in the first place, but they were published because there was code that did in fact use them. Unfortunately we quite often hit problems like that, where we can't in practice publish according to the schema because there is broken code making invalid assumptions and it takes time to get things fixed.
As for the proposal, it suggests that all space tokens of a given type should be merged into a single SA, but personally I still think that we should publish one SA per token - this is e.g. how Jens' castor provider works and I find it extremely useful to keep track of free and used space. Again, I guess this is a EGEE/WLCG issue; not that it isn't interesting, but I'm trying to limit myself to just OGF-GLUE-WG issues.
Well, yes and no - SRM is a general standard and now also an OGF WG so the general question of how SRM v2 should be reflected in GLUE goes beyond WLCG, it just happens to be the case that WLCG is currently the main user.
Agreed. Which forum within EGEE (or WLCG) discusses its adoption of GLUE?
It was the GSSD group, but I'm not sure if that still exists (Maarten?). Stephen
Hi Stephen, Thanks for the comments and info. My comments are... On Thursday 08 May 2008 16:37:57 Burke, S (Stephen) wrote:
glue-wg-bounces@ogf.org
[mailto:glue-wg-bounces@ogf.org] On Behalf Of Paul Millar said: True, but OGF-GLUE-WG does provide the GLUE/LDAP binding, which I feel currently is lacking some documentation.
GLUE is only an OGF WG for the Glue 2.0 spec, so while it will produce an LDAP binding for 2.0 it doesn't have any direct responsibility for 1.3. Indeed for 1.2 and 1.3 there was no GLUE project as such, the original GLUE wound up several years back.
OK, fair enough, but I'd mark this one down as "inherited debt": someone should be "maintaining" (in the ISO meaning of the word) the GLUE v1.3 standard since it's in active use. Perhaps this should be the OGF GLUE WG? Besides, many of the people in GLUE WG are among the authors of the GLUE v1.3 specification and adding a wiki page isn't such an onerous task.
So, for these reasons, one cannot (reasonably) claim that the schema is the single authoritative source as schemata can only validate a document, which is a lesser requirement than correctness.
Being picky, the schema is in fact the only authoritative source
Yes, with the schema one can make an authoritative statement, but only in answer to one question: is a document is valid? Ensuring documents are valid is very useful, but it really isn't sufficient: we want documents (i.e., LDIF information) that are going to be *useful* to people. Not all valid documents are useful. Here's an explicit example. (I believe) one can publish the following object: dn: GlueSEAccessProtocolLocalID=gridftp@example.org,GlueSEUniqueID=example.org,mds-vo-name=resource,o=grid objectClass: GlueSETop objectClass: GlueSEAccessProtocol objectClass: GlueKey GlueSEAccessProtocolLocalID: gridftp@example.org GlueSEAccessProtocolType: gridftp GlueSEAccessProtocolEndpoint: gridftp://milliways.example.org:2811 GlueSEAccessProtocolCapability: file transfer GlueSEAccessProtocolVersion: 1.0 GlueSEAccessProtocolMaxStreams: 20 GlueSEAccessProtocolPort: 2811 GlueSEAccessProtocolSupportedSecurity: GSI GlueChunkKey: GlueSEUniqueID=bogusValue (I believe) this will pass the LDAP Schema checks (the attribute value is a valid IA5String) so the object is valid; however, it's certainly not "correct" as the GlueChunkKey points the end-user to a GlueSEUniqueID that doesn't match the corresponding RDN within the DN. Another example would be the same object, but published without the GlueKey objectClass and without any GlueChunkKey attribute. This, too, would be valid LDIF (I believe) and also not "correct": Glue-v1.3 specifies a link from AccessProtocol to StorageElement (see Glue-v1.3, page 14). This link would be missing on LDAPv2 implementations (with LDAPv3 it is available through the :dn: extension to the query language, but I believe we're leaving that out of the current discussion). This is the difference between a document being valid and (what I mean by) "correct": valid documents pass the schema, correct documents mean no one can (reasonably) complain the information is in some way wrong.
while it might be nice to have some additional definitive document we don't actually have one!
Exactly: someone should create this additional definitive document; a wiki page would be sufficient. (IMHO the absence, so-far, of documentation is not a great argument for its continued absence!)
Most things are, but the GlueChunkKey (as one example) isn't.
That's because it's only in the LDAP implementation
Exactly! GLUE/LDAP has some information that should be document that are both: i) specific to the GLUE/LDAP binding, ii) for technical reasons, are not covered by the LDAP Schema. The format of GlueChunkKey is one, there may be other things.
but anyway, as I say I don't think [GlueChunkKey] should be obsoleted, and it certainly isn't for 1.3.
Yup, fair enough. Personally, I don't care one way or the other. If it's needed by people querying the info-system, we should provide it. However, by the same token, if it *is* needed, there should be a document describing how to generate the values, for which objectClasses it is needed, etc...
[GlueService -URI and -URL attributes being removed] The GlueService things are LCG-specific because those attributes were never officially part of GLUE at all
Err... so, out of idle curiosity, if these attributes are LCG extensions, why did the attributes being with "Glue"? Would something else (for example, "LCG") not have been a better prefix? There's a more general issue here: should GLUE reserve some part of the name-space? (This could be more of a "Gentleman's agreement" than a rigid partitioning.) Apart from avoiding confusion, Glue reserving some part of the name-space might help prevent future problems; for example, if a version 1.4 of GLUE was introduced that defines a new attribute "GlueServiceURI", but with different semantics to LCG's usage (e.g., LCG allowed GlueServiceURI attribute values that are not a URI, but GLUE requires all values to be a URI), this would cause problems.
(LCG had its own Service object at one time).
Fair enough. I guess EGEE, LCG or any group should feel free to publish additional information, provided it doesn't conflict with Glue information. [...]
[As an aside: does EGEE/WLCG document that GlueService -URI / -URL are no longer required? Is there anyway of being notification when these requirements change?]
Not really, because it's a question of fixing individual bugs - those attributes should never have been used in the first place, but they were published because there was code that did in fact use them.
The correctness or otherwise of publishing some attributes aside, if there are changes in what data the info-providers should provide then the people writing those info-providers *need* to know. When there are changes, there must be timely dissemination of these changes within the developer community.
[changes to GLUE schema] Unfortunately we quite often hit problems like that, where we can't in practice publish according to the schema because there is broken code making invalid assumptions and it takes time to get things fixed.
Hmm... I'm not sure if, with "code making invalid assumptions", you're referring to the info-providers or the underlying systems. I first read this as referring to the underlying services and a little warning bell (marked "horse before cart") just went off in my head. If so, to me this sounds more like the schema is making invalid assumptions about the services so the schema is broken; requiring the underlying services to change so the correct information can be publish seems bizarre. Of course, if you're referring to the info-providers, then that's fair enough (although it does hint that "correct" is insufficiently defined).
[publishing GLUE (v1.3 in particular) as a EGEE issue]
Well, yes and no - SRM is a general standard and now also an OGF WG so the general question of how SRM v2 should be reflected in GLUE goes beyond WLCG, it just happens to be the case that WLCG is currently the main user.
This sounds like a good idea to me. Does this mean that (someone within) GLUE is going to provide documentation that describes how to map SRM v2.2 concepts to GLUE v1.3? A wiki page would be good.
Agreed. Which forum within EGEE (or WLCG) discusses its adoption of GLUE?
It was the GSSD group, but I'm not sure if that still exists (Maarten?).
Aye, but I believe GSSD is specifically storage and not specifically GLUE; so, although the should (IMHO) have a strong influence on EGEE's usage of GLUE for storage, they're not the forum that decides how EGEE should use GLUE. Cheers, Paul. (who often gets confused due to the plethora of groups, boards and panels our community seems to develop!)
Paul Millar [mailto:paul.millar@desy.de] said:
There's a more general issue here: should GLUE reserve some part of the name-space? (This could be more of a "Gentleman's agreement" than a rigid partitioning.)
In fact we've gone the other way and decided that we won't have "Glue" in any names, the only scoping is the schema as a whole. Sergio or Laurence may remember the arguments better than me.
Fair enough. I guess EGEE, LCG or any group should feel free to publish additional information, provided it doesn't conflict with Glue information.
In theory yes, in practice it hasn't happened apart from Service, partly because LDAP schemas are so difficult to extend.
When there are changes, there must be timely dissemination of these changes within the developer community.
As I just pointed out in another mail, within EGEE/LCG that's done via savannah.
Hmm... I'm not sure if, with "code making invalid assumptions", you're referring to the info-providers or the underlying systems.
Neither, if I've understood your question! I'm referring to client code, e.g. lcg-utils or the WMS, that makes use of the GLUE attributes. For example, lcg-utils used to expect the type of SE (classic or SRM) to be encoded in the SEName - like many things that was a quick hack to solve a short-term problem which took several years to get rid of. Or there's the assumption that there would be only one GlueSA per VO, which was never formally true in the schema but used to be true in practice so it got hard-wired as an assumption - we're still trying to deal with that one.
Aye, but I believe GSSD is specifically storage and not specifically GLUE; so, although the should (IMHO) have a strong influence on EGEE's usage of GLUE for storage, they're not the forum that decides how EGEE should use GLUE.
Perhaps not in an ideal world, but it's the best we've got ... Stephen
dear all, Paul Millar wrote:
OK, fair enough, but I'd mark this one down as "inherited debt": someone should be "maintaining" (in the ISO meaning of the word) the GLUE v1.3 standard since it's in active use. Perhaps this should be the OGF GLUE WG?
Besides, many of the people in GLUE WG are among the authors of the GLUE v1.3 specification and adding a wiki page isn't such an onerous task.
I agree with Paul that the OGF Glue WG should take care of the v1.3 of glue as well, at least on the collecting/maintaining the documentation level. I suggest that once we pushed the v2.0 into public comment we dedicate some time cleaning up the v1.3 documentation and moving everything to the gridforge. I hope by that time gridforge will recover.
Yes, indeed. To my mind there are four layers that should be documentation: a. GLUE, b. GLUE/LDAP binding, c. EGEE/(W)LCG's usage of GLUE, d. how to get information into GIP.
agree, the above bits & pieces are essential information. btw, one of the main problem with glue 1.x was that the ldap part never really defined the DIT, that was only "encoded in implementations". In 2.0 we try to avoid it and that is why we'll have the proper data model rendering documents for xml, ldap, relational db. bye, Balazs
Burke, S (Stephen) wrote:
Well, yes and no - SRM is a general standard and now also an OGF WG so the general question of how SRM v2 should be reflected in GLUE goes beyond WLCG, it just happens to be the case that WLCG is currently the main user.
Agreed. Which forum within EGEE (or WLCG) discusses its adoption of GLUE?
It was the GSSD group, but I'm not sure if that still exists (Maarten?).
It has been succeeded by the Storage Solutions Working Group. For GLUE vs. SRM matters we have this list: glue20-se@cern.ch I think 1.3 issues may best be discussed there as well.
Hi Paul et Al. for the context of discussion about GLUE 1.3, at the time of closing the old GLUE list it was agreed that this list should be the place for both GLUE 1.x and GLUE 2.0: http://acs.lbl.gov/pipermail/glue-schema/2007-February/000168.html if this is not anymore the wish, we should create a different list. Paul Millar wrote:
There are some CNAF notes[3] [...]
[3] http://glueschema.forge.cnaf.infn.it/SpecV13/LDAP Persoanally I think that whole page should be deprecated
Personally, I think the page should be deleted. It contains incorrect information and (apparently) isn't being maintained (sorry Sergio, that's how it looks.)
the state of the page reflects the last time when the activity on GLUE 1.3 stopped as regards discussing schema and LDAP rendering. The tentative restyling of the LDAP DIT was experimental. Later, EGEE decided to maintain the old DIT for easier deployment and made the implementation/deployment. We should decide: 1. email list to discuss GLUE 1.3 issue 2. if the list is this one, then we could decide to move the GLUE 1.3 content to OGF GLUE WG wiki and I can close the glueschema.forge.cnaf.infn.it wiki; another option is: different list and update glueschema.forge.cnaf.infn.it wiki Cheers, Sergio
However, as a proposal, I feel this WG should have an authoritative document (which could be a wiki page) that describes GLUE/LDAP for v1.3 GLUE. This document should refer the reader to the LDAP-Schema in CVS, but is should contain the extra "additional" information, like how to construct a valid GlueChunkKey, where they might be necessary, etc.
[...] Most of the other things are in the schema document anyway -
Most things are, but the GlueChunkKey (as one example) isn't. There may be other information that is LDAP-specific but not captured by the validation tests (LDAP Schema).
specifically, the old GlueService URI and URL attributes have been removed from my service publisher, which is just in the process of being released, and the CE renaming of CPUs to Slots is true but we still publish CPUs as well.
Yes, but these are EGEE/WLCG issues, right?
[As an aside: does EGEE/WLCG document that GlueService -URI / -URL are no longer required? Is there anyway of being notification when these requirements change?]
The mds-vo-name part varies depending on where you query: = resource at [...]
Ta!
As for the proposal, it suggests that all space tokens of a given type should be merged into a single SA, but personally I still think that we should publish one SA per token - this is e.g. how Jens' castor provider works and I find it extremely useful to keep track of free and used space.
Again, I guess this is a EGEE/WLCG issue; not that it isn't interesting, but I'm trying to limit myself to just OGF-GLUE-WG issues.
However, we should discuss this more offline as this is not an OGF/Glue 2 issue.
Agreed. Which forum within EGEE (or WLCG) discusses its adoption of GLUE?
[How to publish to GIP] Well, it's LDIF conforming to the 1.3 schema, basically what you get from ldapsearch. As above, info providers need to publish their objects with DNs ending "mds-vo-name=resource,o=grid".
Aye, so "any old" LDAP isn't sufficient, it must be child objects under the "mds-vo-name=resource,o=grid" object.
It may also be worth pointing out that you can publish in two different ways: info providers publish entire objects, whereas gip plugins just publish the changes relative to a static file created by YAIM.
OK.
Laurence will no doubt say more, but this is the wiki page:
https://twiki.cern.ch/twiki//bin/view/EGEE/InformationSystem
Thanks! That page helps greatly.
Cheers,
Paul.
_______________________________________________ glue-wg mailing list glue-wg@ogf.org http://www.ogf.org/mailman/listinfo/glue-wg
-- Sergio Andreozzi INFN-CNAF, Tel: +39 051 609 2860 Viale Berti Pichat, 6/2 Fax: +39 051 609 2746 40126 Bologna (Italy) Web: http://www.cnaf.infn.it/~andreozzi
Hi Sergio, I think that we should just keep this list as the same people will be involved in both discussions. I would just recommend that out of courtesy for those who are only interested in Glue 2.0 that any 1.3 discussions should have 1.3 in the subject. We should probably make more use of the discussion board for certain topics. I have created two forums, one on Glue 1.3 and one for FAQs. Our priority over the next week should be to get the document finalized and postpone any project specific discussions until the document is ready. Laurence
We should decide:
1. email list to discuss GLUE 1.3 issue
2. if the list is this one, then we could decide to move the GLUE 1.3 content to OGF GLUE WG wiki and I can close the glueschema.forge.cnaf.infn.it wiki;
another option is:
different list and update glueschema.forge.cnaf.infn.it wiki
Cheers, Sergio
Sergio Andreozzi [mailto:sergio.andreozzi@cnaf.infn.it] said:
2. if the list is this one, then we could decide to move the GLUE 1.3 content to OGF GLUE WG wiki and I can close the glueschema.forge.cnaf.infn.it wiki;
I hope you would keep the old site as an archive, it has quite a lot of information, but it may be worth moving the official documents to the OGF site and marking the cnaf site as obsolete. One other point while we're discussing this is that the enumerated type lists (e.g. Service types) only exist on the 1.2 page: http://glueschema.forge.cnaf.infn.it/Spec/V12 Stephen
One other point while we're discussing this is that the enumerated type lists (e.g. Service types) only exist on the 1.2 page:
This information should be moved to the OGF wiki. Laurence
Hi Paul, It think that the issue you are facing is who is authoritative depends on your perspective. The Glue schema document is authoritative on the information model. but the projects are authoritative on their implementations. Although the Glue document can be found on Sergios web page, I prefer to take the one in the OGF repository. http://forge.gridforum.org/sf/go/doc14185?nav=1 For the LDAP schema it is these files in the SVN repository. http://forge.cnaf.infn.it/plugins/scmsvn/viewcvs.php/v_1_3/mapping/ldap/schema/?rev=29&root=glueschema However, within EGEE the files used are found in the following CVS directory which is a copy of the SVN one. http://glite.cvs.cern.ch:8180/cgi-bin/glite.cgi/glue-schema/ Within EGEE the LDAP model is used for the information and the information system itself. For more details on the information system please visit this page. http://twiki.cern.ch/twiki/bin/view/EGEE/InformationSystem For how Glue is used in EGEE you can take a look at this page. http://twiki.cern.ch/twiki/bin/view/EGEE/GlueUse If you are packaging a service for use within EGEE, you can find more information about adding a resource BDII here. http://twiki.cern.ch/twiki/bin/view/EGEE/ResourceBDII Within EGEE the Generic Information Provider should be used to publish information. This aims to minimize the work required by the use of templates so only the dynamic values need to be found. The glue templates can be found here and should be used to either create LDIF use this LDIF as a guide. http://glite.cvs.cern.ch:8180/cgi-bin/glite.cgi/glite-info-templates/ All of the above information is linked from the information system section in this page. http://twiki.cern.ch/twiki/bin/view/EGEE/EGEEMiddlewareSupport If you have any more questions related to the implementation within EGEE you should contact is-grid-support@cern.ch Laurence
I can't find any good references for how to provide this information.
I've started a dCache wiki page[1] that holds various nuggets of information, but none of them seem complete or authoritative.
[1] http://trac.dcache.org/trac.cgi/wiki/GLUE
I understand that the current dCache system, is functional through Owen's sterling effort, with various trial-and-error attempts and information conveyed through emails. I'm looking to avoid this process as much as possible; having accurate information would be a good start!
First off, the only GLUE v1.3 documentation I could find is from CNAF CVS [2]. The most recent version is "Draft 3---16 Jan 2007". I think Sergio was going to make this the final version, but could this process be clarified? BTW, there's a typo in the StorageArea, UsedNearlineSize description; there are other sections that look somewhat incomplete (e.g., grep for "TO BE ADDED")
Second, I could also find no authoritative source of information on Glue/LDAP binding for v1.3. There are some CNAF notes[3], but these mention, in bold, that "[t]his version is for early evaluation and is not meant to be deployed yet". Can someone say what is incorrect on this page? More importantly, can someone update it so the page is correct?
[3] http://glueschema.forge.cnaf.infn.it/SpecV13/LDAP
The CNAF notes also mention that certain attributes are "deprecated and their use should be removed from any software". Does this mean info-provides "must" (or "should", or "may", see RFC 2119) refrain from publishing deprecated attributes? For example, should no info-provider be publishing GlueChunkKey anymore?
For publishing SRM spaces, I found a *proposal* for how this should be done[4]. Confusingly, this contradicts the GLUE/LDAP notes[3] as it stipulates that "mds-vo-name = local" must be used, whereas the notes state that mds-vo-name has been removed from the DIT; in practise "mds-vo-name = resource" (as a primary document, for feeding into GIP) is what seems to work.
[4] https://twiki.cern.ch/twiki/bin/view/LCG/GSSDGLUEProposal
Finally, (prodding Laurence) I could not find *any* statement about what format GIP supports as primary input (although I might have missed this). I'd assume this is roughly GLUE v1.3/LDAP, but apparently there's some magic happens with DNs with RDF of "mds-vo-name=resource". Is this (still) true? Is it required? What other mappings and translations are possible?
Sorry, I'm sure this info is well known by others, but it doesn't appear to be documented. Can some knowledgable person take on responsibility for ensuring this is all properly documented?
Cheers,
Paul. _______________________________________________ glue-wg mailing list glue-wg@ogf.org http://www.ogf.org/mailman/listinfo/glue-wg
Hi Laurence, Thanks for all the information: all very useful. Some comments interleaved below (I've edited your email for brevity): On Thursday 08 May 2008 10:16:30 Laurence Field wrote:
Although the Glue document can be found on Sergios web page, I prefer to take the one in the OGF repository.
Ta. A very minor technical point: the MIME type looks like it's chosen to force a download ("application/force-download"); is it possible to use the correct MIME type instead? I think all modern web-browsers support a "save as" option in their link-context menus. [...]
However, within EGEE the files used are found in the following CVS directory which is a copy of the SVN one.
http://glite.cvs.cern.ch:8180/cgi-bin/glite.cgi/glue-schema/
Are these schema files identical to the ones at the CNAF SVN repository? [...]
Very useful, ta!
For how Glue is used in EGEE you can take a look at this page. http://twiki.cern.ch/twiki/bin/view/EGEE/GlueUse
This is the documentation I thought was missing. There's several points I feel would benefit from more information (i.e., making more precise), but this page has great virtue in existing! [Many useful links]
If you have any more questions related to the implementation within EGEE you should contact is-grid-support@cern.ch
Ta! Is it better to email grid-support@cern.ch, fire off a GGUS ticket or do both (mentioning the GGUS ticket in the email)? Cheers, Paul.
Hi Paul,
A very minor technical point: the MIME type looks like it's chosen to force a download ("application/force-download"); is it possible to use the correct MIME type instead? I think all modern web-browsers support a "save as" option in their link-context menus.
???
Are these schema files identical to the ones at the CNAF SVN repository?
They should be. But by defacto for EGEE the ones in the glite repository are authoritative as they are the ones that are deployed.
Is it better to email grid-support@cern.ch, fire off a GGUS ticket or do both (mentioning the GGUS ticket in the email)?
It depends on the question. If it is a deployment issue then GGUS. If it is more of a question then you will get a better response from information support list. The most effective way is to either email me directly or phone me :) http://consult.cern.ch/xwho Laurence
Hi Laurence, On Thursday 08 May 2008 16:29:56 Laurence Field wrote:
A very minor technical point: the MIME type looks like it's chosen to force a download ("application/force-download"); is it possible to use the correct MIME type instead? I think all modern web-browsers support a "save as" option in their link-context menus.
???
OK, this is just that when I click on the link, it forces me to download the file somewhere, which I can then load up in my favourite PDF viewer. I'd rather use the (in fact, same) PDF view but embedded within the web-browser. This works fine if the web-server reports the correct MIME type (application/pdf). For example, the "GLUE Specification v. 2.0 (latest draft) - PDF" document[1] (somehow) knows it's a PDF file and reports the correct MIME type, whereas the GLUE Schema 1.3 document[2] apparently doesn't. [1] http://forge.ogf.org/sf/go/doc15023 [2] http://forge.ogf.org/sf/go/doc14185
Are these schema files identical to the ones at the CNAF SVN repository?
They should be. But by defacto for EGEE the ones in the glite repository are authoritative as they are the ones that are deployed.
OK, fair enough.
Is it better to email grid-support@cern.ch, fire off a GGUS ticket or do both (mentioning the GGUS ticket in the email)?
It depends on the question. If it is a deployment issue then GGUS. If it is more of a question then you will get a better response from information support list. The most effective way is to either email me directly or phone me :)
'k. I might give you a phone tomorrow, if that's convenient. Cheers, Paul.
participants (7)
-
Balazs Konya -
Burke, S (Stephen) -
Laurence Field -
Maarten Litmaath -
Maarten.Litmaath＠cern.ch
-
Paul Millar -
Sergio Andreozzi