- ur-wg - lists.ogf.org

Re: [UR-WG] [emi-jra1-data-sar] Fwd: [wg-all] OGF31 Call for Session Proposals extended until 25 January, 2011
by Andrea Cristofori 04 Feb '11

04 Feb '11

Dear Morris, Up to now I think there have been only two people from EMI that answered. Do you already have a scheduled session? I do not know how many from UR-WG will be there. In fact I also do not know if I will be there. Anyway if you think it can be a good idea I can request a session before tomorrow's deadline. Andrea On 01/22/2011 08:25 AM, Morris Riedel wrote: > > Dear SAR team, > > I encourage you to proceed with your planned session request. > > Our good work in EMI should be reported there and there will be surely > someone from EMI. > > Nevertheless, EMI has allocated travel money for these purposes so > please check again with your supervisors mentioning this. > > Harmonization via standardization is one of the key goals of EMI and > as such OGF is a major event being essentially our major > standardization body. > > If there is finally nobody from your team I will do my best to check > whether me or other EMI folks there can perhaps jump-in and present a > little on your behalf. > > Finally, I would like to thank you all since the progress made and > your move towards standardization is a very fruitful and sustainable > approach! > > Have a nice weekend, > > Morris > > *Von:* ur-wg-bounces(a)ogf.org [mailto:ur-wg-bounces@ogf.org] *Im > Auftrag von *Jon Kerr Nilsen > *Gesendet:* Donnerstag, 20. Januar 2011 16:26 > *An:* Andrea Cristofori > *Cc:* ur-wg(a)ogf.org; EMI JRA1 DATA SAR Mailing list > *Betreff:* Re: [UR-WG] [emi-jra1-data-sar] Fwd: [wg-all] OGF31 Call > for Session Proposals extended until 25 January, 2011 > > Hi, > > I think it would be good with a UR session at this OGF as we will have > a finished proposal of a storage accounting definition at that time. > Unfortunately, I don't think I have the finances to get to Taipei at > that time. Hopefully someone else from EMI can get there. > > cheers, > > Jon > > On 20. jan. 2011, at 16.01, Andrea Cristofori wrote: > > > > > Dear UR-WG and EMI members, > > I received this email from the OGF organization. Should I proceed with > a session request? > Who will be there? > > Andrea > > > > -------- Original Message -------- > > *Subject: * > > > > [wg-all] OGF31 Call for Session Proposals extended until 25 January, 2011 > > *Date: * > > > > Tue, 18 Jan 2011 23:16:25 -0500 > > *From: * > > > > Joel Replogle <replogle(a)ogf.org> <mailto:replogle@ogf.org> > > *To: * > > > > wg-all(a)ogf.org <mailto:wg-all@ogf.org> > > > > OGF Community - The OGF31 Call for Session Proposal has been extended > until Tuesday, 25 January 2011. Please make session requests and > other proposals by this deadline. > > Call Details: > > OGF is pleased to partner with the ASGC (Academia Sinica Grid > Computing Centre) to host OGF31 co-located with ISGC 2011 > (International Symposium on Grids and Clouds) in Taipei, Taiwan from > March 21-25, 2011. > > In addition to the ISGC call for papers, OGF is seeking active > participation and proposals for sessions that will further explore and > develop the state of the art in distributed computing. > > Topics of interest may include (but are not limited to): > > * Campus Grids and Clouds > > * Data Management in Grids and Clouds > > * Federated Disriibuted Computing Infrastructures > > * Identity Management and Virtual Organizations > > * Inter-cloud interoperability and SLA management > > * Interoperability of Grids and Clouds > > * Middleware and Interoperability > > * Networking for Distributed Computing Infrastructures > > * Performance measurement of Grids and Clouds > > * Performance of security implementations. > > * Science Clouds across North America, Europe, and Asia > > * Grid and Cloud Security > > * Distributed Desktop Grid and Volunteer Computing > > Chartered Group sessions, Area Meetings, and Birds of a Feather > sessions also may be requested. > > Important Dates > > Deadline for workshop, BoF and group session submissions: > > Tuesday, January 25. > > Acceptance notification: > > Friday, January 28. > > OGF31 Call for Session Proposals: > > http://www.ogf.org/OGF31/cfp.php > > Thanks, > > Joel > > -------------------------------------------------------------------- > > Joel Replogle Executive Director, OGF > > replogle(a)ogf.org <mailto:replogle@ogf.org> http://www.ogf.org > <http://www.ogf.org/> > > PO Box 1738 T: +1-765-228-9068 > > Muncie, IN 47308 USA F: +1-815-927-6284 > > Upcoming Events: > > OGF31 * March 21-25 * Taipei, Taiwan * www.ogf.org/OGF31 > <http://www.ogf.org/OGF31> > > <Attached Message Part.txt>_______________________________________________ > emi-jra1-data-sar mailing list > emi-jra1-data-sar(a)eu-emi.eu <mailto:emi-jra1-data-sar@eu-emi.eu> > http://mail.eu-emi.eu/mailman/listinfo/emi-jra1-data-sar > -- Andrea Cristofori INFN-CNAF Viale Berti Pichat 6/2 40127 Bologna Italy Tel. : +39-051-6092920 Skype: andrea-cnaf

3 2

Fwd: [wg-all] OGF31 Call for Session Proposals extended until 25 January, 2011
by Andrea Cristofori 20 Jan '11

20 Jan '11

Dear UR-WG and EMI members, I received this email from the OGF organization. Should I proceed with a session request? Who will be there? Andrea -------- Original Message -------- Subject: [wg-all] OGF31 Call for Session Proposals extended until 25 January, 2011 Date: Tue, 18 Jan 2011 23:16:25 -0500 From: Joel Replogle <replogle(a)ogf.org> To: wg-all(a)ogf.org OGF Community - The OGF31 Call for Session Proposal has been extended until Tuesday, 25 January 2011. Please make session requests and other proposals by this deadline. Call Details: OGF is pleased to partner with the ASGC (Academia Sinica Grid Computing Centre) to host OGF31 co-located with ISGC 2011 (International Symposium on Grids and Clouds) in Taipei, Taiwan from March 21-25, 2011. In addition to the ISGC call for papers, OGF is seeking active participation and proposals for sessions that will further explore and develop the state of the art in distributed computing. Topics of interest may include (but are not limited to): * Campus Grids and Clouds * Data Management in Grids and Clouds * Federated Disriibuted Computing Infrastructures * Identity Management and Virtual Organizations * Inter-cloud interoperability and SLA management * Interoperability of Grids and Clouds * Middleware and Interoperability * Networking for Distributed Computing Infrastructures * Performance measurement of Grids and Clouds * Performance of security implementations. * Science Clouds across North America, Europe, and Asia * Grid and Cloud Security * Distributed Desktop Grid and Volunteer Computing Chartered Group sessions, Area Meetings, and Birds of a Feather sessions also may be requested. Important Dates Deadline for workshop, BoF and group session submissions: Tuesday, January 25. Acceptance notification: Friday, January 28. OGF31 Call for Session Proposals: http://www.ogf.org/OGF31/cfp.php Thanks, Joel -------------------------------------------------------------------- Joel Replogle Executive Director, OGF replogle(a)ogf.org <mailto:replogle@ogf.org> http://www.ogf.org PO Box 1738 T: +1-765-228-9068 Muncie, IN 47308 USA F: +1-815-927-6284 Upcoming Events: OGF31 * March 21-25 * Taipei, Taiwan * www.ogf.org/OGF31 <http://www.ogf.org/OGF31>

2 1

Summary of Storage Record session @ OGF30
by Jon Kerr Nilsen 01 Nov '10

01 Nov '10

Hi, I've tried to summarize the storage accounting session at OGF30 in Brussels, here: http://twiki.cern.ch/twiki/bin/view/EMI/EmiDataStorageAccountingSummary2010…, repeated below. Please check if I have forgot misunderstood something important. cheers, Jon ---++ SAR @ OGF30 (brief and selected summary) *Jon presented some slides.* * EMI has signed up to deliver a storage record definition and implement it. * Disclaimer: EMI will not make OGF UR2, but will contribute to it. * Definition needed within January 2011 - making an OGF document available for public hearing should be enough for EMI. * EMI has sent out requests for storage accounting, got a little feedback - some feedback may be out of scope (per-file access to assess need for new storage space). * OSG has some extensions to OGF UR (!StorageElement, !StorageElementRecord), but that may be related to CPU accounting, not SR *Henrik presented some slides.* *What to measure:* * We need used space * Reserved and available space not so clear * Tricky to measure * They're optional anyway * Should perhaps have total space as well * Forget about transfer statistics if you want to meet deliverable deadline (nice to have as a future item) * Focus on what is actually used - !SpaceUsed, !SpaceAllocated and !SpaceReserved, drop !SpaceAvailable *Measuring Space* * use is continous, records are discrete -> need some way of conversion * should probably replace start+end with timestamp+duration *Measuring Transfers* * somewhat tricky to measure? * many use cases where you only want to measure storage or only transfers -> should be separate from storage part (storage transfer?) * the owner of a file and the identity performing a transfer, may not be the same, so the owner of the file should not be accountable for the transfer. Hence a seperate record for transfer is probably preferable. * transfers out of of scope for SR *Concepts* *Where* * Maybe too complex? !StorageElement is a bad word! GLUE uses !StorageShare(?) * Structure of where-block should be flat - hierarchical is messy *What* * Filenames should be probably be avoided - privacy issues (filename may contain person id) * Per-file records should be possible - if someone wants to publish millions of records, they shouldn't be stopped * NDFG is medium sized WLCG site and has 11M files (not all files will be changed every day though) * Can always aggregate down the size anyway *Who* * Should adhere to CPU accounting part * CPU accounting currently has only Global ID and Local ID * More or less everyone using CPU accounting has extended Global ID with VO data * SR *needs* VO info, though with better name, like project or group, to appeal to non-grid infrastructures * Solution: SR includes project data, UR follows up and adds project data *UR2 or separate format* * UR2 will not be ready for EMI deadline * Make a standard for ID block and use it in SAR *Some points from discussions* EMI delivarable can at best be "putting definition out on public commenting" !SpaceUsed / !SpaceReserved / !SpaceAvailable looks messy need to adhere to GLUE definition from OGF perspective it is important to have something anyone can use, not just grid people We cannot do peak measurements at the moment, but maybe something for the future? Need good argument why storage accounting is taken out of UR * this is a usage record driven by a storage accounting need. (Does anyone remember the great argument that was proposed?) If we do need political correctness, we could reuse the top-level element in the UR standard, and have a subelement below it. The SR format definition is a very nice step forward. Now we need to write it up in a clear and concise way, discuss it on mailing list and put it on public hearing as soon as possible.

1 0

Storage Record slides from OGF30 Session
by Henrik Thostrup Jensen 29 Oct '10

29 Oct '10

Hi As requested, here are the two presentations from the OGF30 Session. Best regards, Henrik Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org

1 0

Storage Accounting - Requirements Capture
by john alan kennedy 30 Sep '10

30 Sep '10

Hi Jon, I know some type of requirements capture is under way in EMI since I was passed questions from a DEISA contact. I have asked similar questions to IN2P3 who deploy a large iRODS instance and also to iRODS developers. Although EMI may not have iRODS on the roadmap I thought it would make sense for me to forward the responses I get before we meet in Brussels so you can correlate requirements - Would this be OK? The iRODS responses raised a few issues, it shows that different people see things in a slightly different light. I'd simple post to the list but if we start posting all the responses it'll get quite noisy. I will also ask for input from OSG if we still need a better description of their storage accounting. From my point of view it would be nice if people could try to break the elements down into 3 brackets. * Must Have - I really need this * Very Useful - Can I please have * Useful - but I can live without We can ask - people generally don't though. Just some though cheers johnk -- +------------------------------------------------------------+ |Dr. John Alan Kennedy Rechenzentrum Garching (RZG) | |Mail:jkennedy@rzg.mpg.de Boltzmannstrasse 2 | |Phone: +49 89 3299 2694 85748 Garching | |Fax: +49 89 3299 1301 | +------------------------------------------------------------+

2 1

Ideas for storage accounting record
by Henrik Thostrup Jensen 29 Sep '10

29 Sep '10

Hi Here are some preliminary ideas for fields in a storage accounting record. The first target for NDGF is dCache, however we also want be able to do storage accounting on plain unix directories (SGAS has a special client called Bart, which does job accounting on LRMS-level, and we have users who's primary usage for SGAS is regular (non-grid) HPC). * record metainformation: recordId unique createTime iso timestamp for the record * identity block: global uid global user identity (properly optional, vo is often more interesting) vo_issuer vo_type vo_name (or just vo) vo_group vo_role ## complex vo data? (multiple groups/roles) ## having multiple groups/roles for data is somewhat messy * storage information: sitename FQDN / Unique sitename poolname? A site will often have several places to store (should probably have another name) storagesystem which storage system is used storagetype ssd / platter disk / tape storageclass pinned / deletable / precious / archival / whatever storageurl url where the storage is located at * time period information: starttime startime for when the record holds endtime endtime for when the record holds * space usage spaceused space used by the identity (bytes is probably best) spacefree optional (number is not always available) spacereserved space reserved for the identity block (optional) Design issues: should everything be split into seperate records (say one for disk, one for tape, and so on), or should be possible to specify multiple usage sections in storage accounting record for different classes of stored data. AFAIK EMI is very interested in having something usuable very soon (this year) for performing storage accounting, so this is a fairly urgent topic if OGF wants to have something to say. However there is likely to be a certain overlap between the two groups. Best regards, Henrik Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org

4 7

Re: [UR-WG] Ideas for storage accounting record
by Gordon, John (STFC,RAL,ESC) 29 Sep '10

29 Sep '10

Here was a link on the EMI page that someone sent round this list recently. John Sent from my phone. -----Original Message----- From: john alan kennedy <jkennedy(a)rzg.mpg.de> Sent: 29 September 2010 10:03 To: john.gordon(a)stfc.ac.uk <john.gordon(a)stfc.ac.uk> Cc: htj(a)ndgf.org <htj(a)ndgf.org>; j.k.nilsen(a)fys.uio.no <j.k.nilsen(a)fys.uio.no>; ur-wg(a)ogf.org <ur-wg(a)ogf.org> Subject: Re: [UR-WG] Ideas for storage accounting record Hi John, That's a good point. Do you know where we can find it or do you have a contact person who we can talk to? Google showed me that they just had a Storage Forum but I didn't find anything on their accounting as of yet. osg-storage(a)opensciencegrid.org seems to be a possible starting point. Did anyone else chase this up? cheers johnk On 09/27/2010 01:33 PM, john.gordon(a)stfc.ac.uk wrote: > Perhaps people should look at the OSG one and comment? > > John > > >> -----Original Message----- >> From: ur-wg-bounces(a)ogf.org [mailto:ur-wg-bounces@ogf.org] On Behalf >> > Of > >> Henrik Thostrup Jensen >> Sent: 27 September 2010 11:55 >> To: Jon Kerr Nilsen >> Cc: ur-wg(a)ogf.org >> Subject: Re: [UR-WG] Ideas for storage accounting record >> >> Hi Jon >> >> On Thu, 23 Sep 2010, Jon Kerr Nilsen wrote: >> >> >>> Just to confirm that EMI is indeed interested in having a usable >>> definition of a storage accounting record before January... >>> >>> The EMI data group is currently in the process of collecting >>> requirements from user communities and storage providers, so not >>> > much > >>> input from us yet, but I'm certainly hoping for overlap between OGF >>> >> and >> >>> EMI work on this topic. >>> >> Given the short timespand of getting a record definition ready, I >> > would > >> strongly urge towards getting a rudementary format ready very fast and >> then iterating over the format multiple times (as requirements get in >> and >> showing the formats to the interested parties to see if it can fulfill >> their requirements). >> >> If you wait to start defining the format until requirements have been >> collected there will be very little time to work with the record >> format. >> >> >> Best regards, Henrik >> >> Software Developer, Henrik Thostrup Jensen<htj at ndgf.org> >> Nordic Data Grid Facility. WWW: www.ndgf.org >> -- >> ur-wg mailing list >> ur-wg(a)ogf.org >> http://www.ogf.org/mailman/listinfo/ur-wg >> -- +------------------------------------------------------------+ |Dr. John Alan Kennedy Rechenzentrum Garching (RZG) | |Mail: jkennedy(a)rzg.mpg.de Boltzmannstrasse 2 | |Phone: +49 89 3299 2694 85748 Garching | |Fax: +49 89 3299 1301 | +------------------------------------------------------------+ -- Scanned by iCritical.

1 0

OGF-30
by Andrea Cristofori 28 Sep '10

28 Sep '10

Dear all, I have requested a session of 1.5 hours at next OGF in Brussels asking preferably for the 28th or 29th. I hope we can all make it there and continue the discussion. In the meanwhile we could try to collect some request from the users and try to put all the informations together. Cheers, Andrea -- Andrea Cristofori INFN-CNAF Viale Berti Pichat 6/2 40127 Bologna Italy Tel. : +39-051-6092920 Skype: andrea-cnaf

4 6

Aggregated usage records in SGAS
by Henrik Thostrup Jensen 23 Sep '10

23 Sep '10

Hi Here is an actual use case for how aggregation can be done. In SGAS we aggregate usage records into the followign format: execution_date | date insert_date | date machine_name | string user_identity | string vo_issuer | string vo_name | string vo_group | string vo_role | string n_jobs | integer cputime | numeric walltime | numeric generate_time | timestamp The combination of the following fields is considered unique: execution_date, insert_date, machine_name, user_identity, vo_issuer, vo_name, vo_group, vo_role. Some of these can be null/non-existing (vo_group and vo_role). The reason for seperating insert and execution date as that some records arrive late when the registration process fails for some reason. Most admins care more about registrations, where as usage data usually uses execution time for statistics. The three following fields are aggregated number of jobs, summed cputime, summed walltime The final field us when the record was generated. This format aggregates quite well. 3.8M records aggregated into 18K records in the NDGF accounting database. I'm an in no way implying that all aggregations should look like that and that all queries can be answered by such an aggregation. But for us, in can answer most of the common queries, and time resolution per day is typical enough. Currently we don't use AUR in SGAS (we do use UR for all job records), but might in the future (currently the methods for querying data is somewhat limited). Best regards, Henrik Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org

2 2

SGAS extensions to usage record
by Henrik Thostrup Jensen 23 Sep '10

23 Sep '10

Hi Here is a list of the UR extensions used in SGAS. It is provided in the hope to be usefull in the UR2 creation process. Arguably the most important extension the ability to describe VO affiliation, including groups and roles. It looks like this: <vo:VO vo:type="voms" xmlns:vo="http://www.sgas.se/namespaces/2009/05/ur/vo"> <vo:Name>bio.ndgf.org</vo:Name> <vo:Issuer>/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org</vo:Issuer> <vo:Attribute> <vo:Group>bio.ndgf.org/test</vo:Group> <vo:Role>VO-dev</vo:Role> </vo:Attribute> <vo:Attribute> <vo:Group>bio.ndgf.org</vo:Group> </vo:Attribute> <vo:Attribute> <vo:Group>bio.ndgf.org/test</vo:Group> </vo:Attribute> </vo:VO> It is placed in the user identity block. It is somewhat closely modelled after VOMS (which is what we needed to model), but has been used for several other systems (usually one would just the Name and type elements). Descibing the type VO information a rather important feature IMO. Futhermore we have a range of small simple extensions. We share submit time with the DEISA project (well we just reused what they had made), though there is no data exchange between NDGF and DEISA. The following lists the extensions: Submit time (shared with Deisa) <deisa:SubmitTime xmlns:deisa="http://rmis.deisa.org/acct">2009-07-07T09:06:37Z</deisa:SubmitTime> Submission time can be used to tell how long time a job spend in queue, which is an often-asked-for metric. SGAS UR extensions: KSI2KCpuDuration (scaled cpu duration) <sgas:KSI2KCpuDuration>PT150.4S</sgas:KSI2KCpuDuration> KSI2KWallDuration (scaled wall duration) <sgas:KSI2KWallDuration>PT8488.44S</sgas:KSI2KWallDuration> The two ksi2k elements are no longer used. If there should be the possibility for scaled values in ur2, it should not be limited to a specific scaling method. insertTime (when the record was inserted into the database. this field only exists when extracting a record from a database <ur:RecordIdentity sgas:insertTime="2009-07-07T09:06:52Z" /> UserTime (time spend in userland) <sgas:UserTime>PT84.3S</sgas:UserTime> KernelTime (time spend in kernel time) <sgas:KernelTime>PT13.2S</sgas:KernelTime> ExitCode (job exit code, integer value) <sgas:ExitCode>0</sgas:ExitCode> MajorPageFaults (times a major page fault occured) <sgas:MajorPageFaults>3</sgas:MajorPageFaults> RuntimeEnvironment (requested ARC runtime environments, allows per-application accounting, the element can be repeated) <sgas:RuntimeEnvironment>APPS/HEP/ATLAS-13.0.40</sgas:RuntimeEnvironment> The latter five where actually requested by CSC (Finish HPC centre), which also uses SGAS (there are at least five deployments of the SGAS accounting service AFAIK). Futhermore we are thinking about adding fields for how many files the job down/up-loaded and how much data was transferred. Furthermore how many files was used from the cache (Input files, which existed locally and does not need to be downloaded. This might be a bit ARC specific, but the concept could still be usefull for others). We also include information about what software produced the record, and optionally which version of the software. This information does not describe the job in any way, but can be practical for identifying the source of the record and detect incompatabilities. logger = namespace http://www.sgas.se/namespaces/2010/08/logger LoggerName (name of the logger, can have optional version element) <logger:LoggerName>ARC0-URLogger</logger:LoggerName> <logger:LoggerName logger:version="0.8.2.2">ARC0-URLogger</logger:LoggerName> Other issues: The specification text for the machine name, host names, etc. could be clearer in the standard. In the future a sample set of records for compliancy testing could be provided. Best regards, Henrik Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org

1 0