September 2006 - ur-wg - lists.ogf.org

LAST CALL - Usage Record Format Recommendation - Version 1
by Laura F McGinnis 16 Jan '07

16 Jan '07

Please review Version 1 of the Usage Record Format Recommendation. It is on our gridforge repository at https://forge.gridforum.org/sf/go/doc13864?nav=1, and also on the usage record website maintained at http://www.psc.edu/~lfm/PSC/Grid/UR-WG/. Please send all comments to the mailing list by 1 October 2006. After that, the document will be submitted to the OGF Editor. Thx LM ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Laura F. McGinnis, Project Coordinator Data & Information Resource Services Pittsburgh Supercomputing Center email: lfm(a)psc.edu 300 South Craig St, #313 phone: 412-268-5642 Pittsburgh, PA 15213 fax: 412-268-5832 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

6 10

Re: [UR-WG] LAST CALL - Usage Record Format Recommendation - Version 1 (fwd)
by Rosario Piro 29 Sep '06

29 Sep '06

(Since I'm having problems with our mail server I send a second copy of my mail hoping that at least one will make it :o) Sorry for an eventual suplication, Rosario) ---------- Hi all! I'm sorry for sending my comments on the UR spec for version 1 this late (but then, I still respect the deadline :o), but I was quite busy lately (as always) ... The list is quite long. Most points are just minor corrections, such as wrong references between Sections of the document, but there are some very important observations as well, so please read carefully through them! I'd appreciate some comments above all to A1 (proposal to have a version attribute for UR instances, and to also specify the version in the schema) since I think it is important. Cheers, Rosario. Here are my comments: ------------------------------------------------------- Comments on version 1 of the UR spec: IMPORTANT PROPOSAL: A1) Since we are currently finishing version 1, but are also thinking about version 2 we should make sure that different UR versions can be easily distinguished by processing software (e.g. accounting systems), this means the version should be specified in both, the schema itself (as part of the annotation) and each instance of a UR document. For example: <JobUsageRecord version="1.0"> (or <UsageRecord version="1.0">) ... </JobUsageRecord> Otherwise announting tools will have a hard time figuering out against what version of the UR spec they should validate a specific instance. The specification of a version should me _required_ for each instance. I am aware that this might imply problems for already existing implementations based on earlier drafts, but it will avoid a lot of trouble in the future (think about major changes in version 2, 3, 4, wheverever we will end up). INCONSISTENCIES IN THE DOCUMENT: B1) While the abstract talks says that "This format must encompass both job level accounting and aggregate accounting" and §6 briefly explains how to do aggregation, §1.1.2 explicitly states that the "document does not adress aggregation, summary records ... or anything other than an atomic resource.". Actually, the "aggregate" property mentioned in §6 is not part of the UR schema. This is more than confusing and will lead to problems. I suggest to remove all references on aggregation from the document (abstract, §1.2.4, §6, ...) and just leave the note in $1.1.2 to make clear that the current spec is not supposed to be used for aggregated accounting data. B2) §2.3.6: "Timestamp" should actually be called "dateTime". B3) §3: What is called "Basic" Properties here is called "Common" properties in §11. B4) Are §3 and §4 necessary since they correspond to §11 and §12, respectively? It would be more appropriate to unite Sections §3 and §11, and §4 and §12, since it is confusing if information on the same property has to be looked up in different parts of the document. B5) §3.1 says the RecordIdentity property MUST have data of type string, while this element actually contains no data. The identity of the record is instead specified in its "recordId" attribute. B6) §3.1 says that the create time of the record is required, while it actually is optional according to the schema B7) §3.2, §3.3 and §3.4: The "description" attributes mentioned here are not in the schema. This seems to be an artifact from previous UR versions where GlobalJobId, LocalJobId and ProcessId where distinct resource properties and not part of the JobIdentity property. B8) §3.4 says ProcessId is of type integer, while the schema says it's a string B9) §3.12 and §3.13 mention data type "timestamp", it is preferable to use "dateTime". B10) §4.1, §4.2, and §4.3 say that "Units MAY be specified", it might be better to directly refer to intervallicVolume B11) §4.4: "metric", "type" and "intervallicVolume" (or units) are not listed. B12) §4.7 and §4.8: "description" and "type" are not listed. B13) §4.9: "type" is not listed. B14) §4.10 seems to be obsolete be cause it talks about an "Extension" property while now we have "Resource", "ConsumableResource", ... B15) §5 says that each record must contain at least one of LocalJobId or GlobalJobId while their parent property (JobIdentity) is optional according to the schema. B16) The schema does contain the old enumeration for storageUnit and not the new one described in Appendix C (prefixes for binary multiples and SI prefixes for negative exponents are missing int the schema) B17) not all schema excertps in §11 and §12 correspond to the complete schema in Appendix D. Please update the exceprts to the final schema version. (e.g. excerpt in §11.1 contains "createDate" while schema and description contain "createTime") B18) §11.3.2: should be "GlobalUserName" instead of "GlobalUsername" B19) §11.6: The values that have to be supported are listed with all letters being lowercase, while the annotation in the schema in Appendix D describes them with the first letter being uppercase. This is only a slight error, but may cause problems for implementations that are case-sensitive. B20) §12.1, §12.2, §12.3 and §12.4: "intervallicVolume" is missing in the list of attributes B21) §12.7.1 and §12.8.1: the schema in App. D does not contain a "description" attribute for TimeDuration and TimeInstant! Either add it to the schema or remove it from the description. B22) §13.1.2.2: "unit" should be "units" according to the schema B23) The last note in App. B (that the Appendices are included for historical reference) should better be on top of the Appendix instead of being at the end (where it can easily be overlooked) WRONG REFERENCES: The follwing observations refer to the current numbering of the Sections. If §3 and §4 should be removed (see comment no. B4) the references change accordingly: C1) in §3, 1st paragraph: the reference to §8 since that Sections doesn't contain basic data tye definition. C2) in §2, 2nd paragraph: reference to §8, should be: §9 C3) in §7.2.6: reference to §10.1, should be: §11.1 C4) in §7.2.8: reference to §12, should be: §13 C5) in §9.5, on page 20: reference to §11.2.2, should be: §12.2.2 C6) in §9.5, on page 21: references to §10 and §11, should be: §11 and §12, respectively C7) in §9.6: references to §11.2.3, §10 and §11, should be: §12.2.3, §11 and §12, respectively C8) in §10.1: references to §10.1 and §10.6, should be: §11.1 and §11.6, respectively C6) in §11.1.2: reference to §6.2.5, should be: §7.2.5 C7) in §11.1.3 on page 25: reference to §6.2.7, should be: §7.2.7 C8) in §11.4: reference to §10.2, should be: §11.2 ------------------------------------------------------- On Sun, 17 Sep 2006, Laura F McGinnis wrote: > Please review Version 1 of the Usage Record Format Recommendation. It is on > our gridforge repository at > https://forge.gridforum.org/sf/go/doc13864?nav=1, and also on the usage > record website maintained at http://www.psc.edu/~lfm/PSC/Grid/UR-WG/. > > Please send all comments to the mailing list by 1 October 2006. After that, > the document will be submitted to the OGF Editor. > > Thx > LM > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Laura F. McGinnis, Project Coordinator > Data & Information Resource Services > Pittsburgh Supercomputing Center email: lfm(a)psc.edu > 300 South Craig St, #313 phone: 412-268-5642 > Pittsburgh, PA 15213 fax: 412-268-5832 > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > -- > ur-wg mailing list > ur-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/ur-wg >

1 0

Comments/Questsions on usage#draft3 schema
by Matt Ford 27 Sep '06

27 Sep '06

Hi Donal, I have some comments/questions on your latest release of the proposed 2.0 spec. - It's much more legible now; your effort in this is much appreciated by me at least! :-) - I like the much cleaner type derivations. - With regard to the AtomicActivityUsageRecord type: * I don't like how we call it an AtomicActivity and then use "JobName" as an element name. The activity may have nothing to do with jobs; I still feel as though we have a mix of the specific and generic in this type. * In a similar line of thinking I wonder if elements such as "Status" should not be global and instead be local to a derived type such as "AtomicActivity" i.e., declared instead of referenced. Not all Usage records will necessarily have "Status" and it's meaning can in principle vary wildly. - With regard to the CompoundUsageRecord: * It's abstract as you note; so how's it made specific? (the following questions may be due to my ignorance of XSD's). I guess I could extend the schema but how then do we make it so that people don't go off and arbitrarily extend it? How do we capture this if they do? For example if I naively defined a "grid-job" - treat this name as arbitrary ;-) - as the collection of AtomicActivity UsageRecords spawned by my Resource Broker (never more than ten) what do I do? Should I even attempt this: my aim is to extend the schema to allow only compound records of 0 to 10 Atomic Activity records only? <xsd:complexType name="GridJob"> <xsd:complexContent> <xsd:restriction base="usage:CompoundUsageRecord"> <xsd:element ref="usage:AtomicActivity" minOccurs="0" maxOccurs="10" maxOccurs="unbounded" /> <xsd:any namespace="#other" processContents="lax" minOccurs="0" maxOccurs="unbounded" /> </xsd:sequence> </xsd:restriction> </xsd:complexContent> </xsd:complexType> <xsd:element name="GridJob" substitutionGroup="usage:UsageRecord" type="usage:GridJob"> </xsd:element> Or should I be thinking of CompoundUsageRecord as plumbing for as yet undefined parts of the UR-WG schema? Cheers, Matt.

2 3

Comment on duration format for WallDuration and CpuDuration
by Bart Heupers 21 Sep '06

21 Sep '06

Hello, I would like to comment on the proposed format for the WallDuration and CpuDuration properties in the Usage Record - Format Recommandation. "The lexical representation for duration is the [ISO 8601] extended format PnYn MnDTnH nMnS, where nY represents the number of years, nM the number of months, nD the number of days, 'T' is the date/time separator, nH the number of hours, nM the number of minutes and nS the number of seconds. The number of seconds can include up to 6 decimal digits." If you want to convert this into seconds, or add this kind of duration to a grand total it is unclear how long a month is or a year. Is a month 30 ,31 or 28 days ? And a year is that 365 or 366 days? The use of years and months in this representation only give rise to needless uncertainties. Therefore I think it is better to use as duration format the number of seconds, with six decimal digits, or of you still want to use the ISO8601 standard, use it with the restriction that years and months will never be used. In that case it is always possible to convert it unambiguously to seconds. Bart Heupers, HPC Advisor SARA, Kruislaan 415, 1098 SJ Amsterdam

6 18

GFD.58 Compliance
by Laura F McGinnis 13 Sep '06

13 Sep '06

Thanks to Chris Baumbauer at SDSC, I have posted an updated version of the UR recommendation on the website. Chris made the adjustments to the schema and text to make the document GFD.58 compliant. Thx LM ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Laura F. McGinnis, Project Coordinator Data & Information Resource Services Pittsburgh Supercomputing Center email: lfm(a)psc.edu 300 South Craig St, #313 phone: 412-268-5642 Pittsburgh, PA 15213 fax: 412-268-5832 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

2 1

Second draft of a strawman of the UR 2.0 schema
by Donal K. Fellows 11 Sep '06

11 Sep '06

Hi everyone! Please find attached the second version (confusingly named ur-draft-3) of my UR 2.0 strawman schema. This version is approximately complete, except for Disk and Network usage, which I realized I truly did not understand at all. :-) Enjoy... Donal.

1 0

Re: [UR-WG] [ur-wg] Phone Conf.
by Laura F McGinnis 03 Sep '06

03 Sep '06

We won't be able to set up a conference call ofr GGF/OGF18, but we will collect and post notes and presentations. If there is interest, we may restart the conference calls that we were holding earlier this summer. Please keep an eye on the mailing list and website for further information. Thx LM ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Laura F. McGinnis, Project Coordinator Data & Information Resource Services Pittsburgh Supercomputing Center email: lfm(a)psc.edu 300 South Craig St, #313 phone: 412-268-5642 Pittsburgh, PA 15213 fax: 412-268-5832 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > -----Original Message----- > From: Akram Khan [mailto:akram@slac.stanford.edu] > Sent: Sunday, September 03, 2006 11:02 AM > To: Laura F McGinnis > Subject: [ur-wg] Phone Conf. > > Hello Laura, > > I sorry but we will not be able to make it to this GGF > meeting for we have > alot of University commitment! Would it be possible to arrange a phone > conference, whereby we can phone in and join the dicussion. > For it sounds > like it would be a fun few sessions. > > cheers > akram >

1 0

Updates mailing list address
by Laura F McGinnis 03 Sep '06

03 Sep '06

Open Grid Forum has migrated our mailing list from ggf.org to ogf.org. I have made the appropriate changes on our website. If you see any problems, please let me know. Thx LM ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Laura F. McGinnis, Project Coordinator Data & Information Resource Services Pittsburgh Supercomputing Center email: lfm(a)psc.edu 300 South Craig St, #313 phone: 412-268-5642 Pittsburgh, PA 15213 fax: 412-268-5832 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

1 0

[wg-all] Mailing List Changes
by Andre Merzky 03 Sep '06

03 Sep '06

Sorry for duplicates ------------------------------------------------------------ As of yesterday, all GGF/OGF mailing lists are migrated to the ogf domain. They are now managed by mailman, the web interface is available at http://www.ogf.org/mailman/listinfo With that migration, all list can be now reached via <list>@ogf.org The old @ggf.org addresses will be available for some time, but are bound to disappear eventually - please start using the ogf addresses! A number of lists have changed their names: ggf-it --> ogf-it ggf-office --> ogf-office ggf-proc-wg --> ogf-proc-wg ggf-sponsors --> ogf-sponsors ggf-tutorials --> ogf-tutorials ggf-compute --> ogf-compute ggflistowners --> ogflistowners ggfstudentmembers --> ogfstudentmembers Postings to the old list names get forwarded to the new ones. With the migration we have also been able to fix the archives - you may have noticed that list archiving stopped sometime in mid June. Well, now its working again, and all mails from June to now have been added to the archive. The list archives can be found at: http://www.ogf.org/archives/<list> If you are an list administrator, and would like to receive the list admin password, please contact postmaster(a)ogf.org. In general, however, we tried to map the old majordomo settings to the mailman settings, in terms of privacy etc, so no or little changes should be required from your side. The spam avoidance policy has been changed to some extent, and is somewhat stricter than before. Please let us know if you experience any trouble with that. On all questions in respect to mailing lists, or mails in general, please contact postmaster(a)ogf.org. Best regards, Andre. -- "So much time, so little to do..." -- Garfield -- wg-all mailing list wg-all(a)ogf.org http://www.ogf.org/mailman/listinfo/wg-all

1 0

Version 1 Done
by Laura F McGinnis 01 Sep '06

01 Sep '06

(I hope.) I have uploaded Msword and PDF versions to the working group website at http://www.psc.edu/~lfm/PSC/Grid/UR-WG/. I have also put a PDF copy in the GridForge Usage Records WG Documents repository. I have two outstanding questions that would be nice to have answered, but I don't want to hold the document up any longer waiting for answers: 1) Could someone verify that the material in Appendix D, the actual schema, is GFD.58 compliant, please. Donal was going to work on this, but I know he's been busy with other obligations this summer. If there are others in the WG who are conversant with XML and GFD.58, please step up and review this for us. 2) The Status attribute is of type token, with a minimal set of values, but it is extensible. There are no required values listed in the schema (Appendix D). Should there be, or will this negate the ability of the field to be extensible? Having naught but a smattering of XML comprehension, this is beyond my ability to answer confidently (or probably correctly). There, that's it. Just two open items, but neither of them show-stoppers to getting this document back to the editor. Thanks for all your input and I look forward to seeing how V2 develops. LM ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Laura F. McGinnis, Project Coordinator Data & Information Resource Services Pittsburgh Supercomputing Center email: lfm(a)psc.edu 300 South Craig St, #313 phone: 412-268-5642 Pittsburgh, PA 15213 fax: 412-268-5832 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

1 0