May 2006 - ur-wg - lists.ogf.org

Aggregate Accounting
by Matthew Ford 09 May '06

09 May '06

Hi All, I hope you have (or have had) a good face-to-face meeting. On the conference call previously we have discussed aggregate accounting and made moves to move it out of scope. I attach below, a valid and well formed record, using the definitions from section 10 of the spec. <UsageRecord> <RecordId urwg:recordId="foo"/> <UserIdentity> <ds:KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#" xmlns:ds="http://www.w3.org/2000/09/xmldsig#"> <X509Data> <X509SubjectName>cn=matt ford</X509SubjectName> </X509Data> </ds:KeyInfo> </UserIdentity> <StartTime urwg:description="accountperiod">2003-06-16T08:24:32Z</StartTime> <EndTime urwg:description="accountperiod">2006-05-08T10:34:58Z</EndTime> <CPUDuration urwg:description="sum over accountperiod for UserIdentity">234234324325</CPUDuration> <WallDuration urwg:description="sum over accountperiod for UserIdentity">23423434</WallDuration> <Status urwg:description="all complete jobs over accounting period for UserIdentity">completed</Status> </UsageRecord> This I think certainly constitutes a valid aggregate record. It would require a farily significant rewrite to make this _not_ be allowed. I'm thinking of using something like this to report total cpuduration for some of my users. Is this against the spirit of the clarification we are trying to make, should this be something that is allowed? The move away from aggregation is a big one...or have I been too liberal in my interpretation (but I'd argue against this) Matt.

2 2

UR-WG at GGF17 in Tokyo
by Laura F McGinnis 09 May '06

09 May '06

The Usage Records Working Group will have one face-to-face session at GGF17 in Tokyo. We are currently scheduled as follows: Thursday, 11 May, 11am-12:30 Room G401 See you there! LM ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Laura F. McGinnis, Project Coordinator Data & Information Resource Services Pittsburgh Supercomputing Center email: lfm(a)psc.edu 300 South Craig Street, #313 voice: 412-268-5642 Pittsburgh, PA 15213 fax: 412-268-8200 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

2 1

Usage vs Job accounting and the UR framework
by Matthew Ford 05 May '06

05 May '06

Donal Wrote: >This point has two distinct aspects. > 1) The StartTime and EndTime values refer to The time period to which > the Usage Record itself applies. For a job (whether grid, batch or > flipping hamburgers for minimum wage) one natural interpretation is > that the start and end indicate an accounting period that matches > when the job is "consuming resources" in some appropriate sense. But > strictly the start and end indicate the accounting period, not the > job. Whether this is true or not depends on which part of the spec you read! Compare 10.10 to 3.13. The former defines StartTime in terms of a "job" and the latter defines in terms of "usage". It is the lack of a clear definition of a "job" and the intermingling of "job" and "usage" in the definitions that are leading, I think, to a *lot* of confusion with the schema. I use job from now on to mean _batch system job_. It goes to the heart of the framework. Regarding the framework ----------------------- Donal's email of the 27th noted the confusion of UsageRecord and JobUsageRecord and talks about the dividing of terms(elements) into generic and computational groups. The division of elements is alluded to in the current spec: base and job level accounting - see the first sentence of Secs. 3 and 4 respectively. It just doesn't seem to be consistently applied. Further, Donal's composite suggestion, also sort-of exists in the current spec: the UsageRecords element (note: it does not allow Sven's nested requirement) - see section 7, Global Element Definitions. Below is a summary of what we have now: Record A: <UsageRecord> <Base Properties..../> </UsageRecord> Record B: <JobUsageRecord> <Base Properties..../> </JobUsageRecord> Record C: <UsageRecords> <UsageRecord> </UsageRecord> ... <JobUsageRecord> </JobUsageRecord> ... </UsageRecords> Records A and B are at the moment equivalent. Record C illustrates the composite structure. It is a *syntactical change of name only*, between Records A and B. The issue is that the Base properties in records A and B are an identical set. The JobUsageRecord whilst inheriting the UsageRecord type does not extend it in anyway (but this may be intentional, I would like to understand the original authors original intent at this point - see questions at end). Without the extension or the separation of the current base properties there is little point to having the two elements UsageRecord and JobUsageRecord. Personally I would like to see the spec use a combination of types and sub-elements to capture usage and to split the current base elements accordingly. I second Donal's proposal. For example; <UsageRecord> <Base Properties> <JobUsageRecord> <CPUDuration/> <Processors/> </JobUsageRecord> </UsageRecord> This allows consistency of data format through types and extensibility through additional elements. It will provide a sensible place for future work on aggregation and allow grid-jobs for example to be much more easily defined. For example: <UsageRecord> <Base Properties> <GlobalJobId/> <Accounting Period/> <Aggregate Info/> </Base Properties> <Grid Job> <JobUsageRecords> <JobUsageRecord/> ... </JobUsageRecords> <DataRecord/> </Grid Job> </UsageRecord> I'm also painfully aware that such a change will mean changes/breaks for existing accounting solutions in place. For example the NGS and it's current RUS service....it may be too big an ask given the current spec. But I believe it's necessary in order to make the spec something more that job accounting. Further it is not feasible for a small group like this to define the elements such as DataRecord and Grid Job (we have a head start on JobUsage). I think the schema should be made extensible to allow other groups to "plug-in" to this structure and that the group should aim to act somehow in a arbitrators role. Notes ----- Historical Questions: Is it that the UsageRecordType should be inherited by elements such as JobUsage or, hypothetically DataUsage? And where possible appropriate elements are mapped onto the global UsageRecord type - this would ensure consistency of data (Duration fields for example through-out the record) but require a certain amount of shoe horning (and potentially a very large global type) to make things work. Why was this approach taken? The extensibility talked about in section 7 - how was that envisaged working? Thanks, Matt. -- Grid Support, Manchester Computing.

1 1

Mailing List Discussions: Comment #5b
by Laura F McGinnis 05 May '06

05 May '06

3 5

XML Changes/Questions: Comment #6c & #9
by Laura F McGinnis 05 May '06

05 May '06

3 2

XML Changes/Questions: Comment #1
by Laura F McGinnis 04 May '06

04 May '06

1 0

Editor Document Scope: Comment #6i-1
by Laura F McGinnis 04 May '06

04 May '06

1 0

Editor Document Scope: Comment #5a
by Laura F McGinnis 04 May '06

04 May '06

1 0

Editor Document Scope: Comment #5f
by Laura F McGinnis 04 May '06

04 May '06

1 0

Editor Attribute Clarification/Correction: Comment #6b
by Laura F McGinnis 04 May '06

04 May '06

1 0