September 2010 - ur-wg - lists.ogf.org

Storage Accounting - Requirements Capture
by john alan kennedy 30 Sep '10

30 Sep '10

Hi Jon, I know some type of requirements capture is under way in EMI since I was passed questions from a DEISA contact. I have asked similar questions to IN2P3 who deploy a large iRODS instance and also to iRODS developers. Although EMI may not have iRODS on the roadmap I thought it would make sense for me to forward the responses I get before we meet in Brussels so you can correlate requirements - Would this be OK? The iRODS responses raised a few issues, it shows that different people see things in a slightly different light. I'd simple post to the list but if we start posting all the responses it'll get quite noisy. I will also ask for input from OSG if we still need a better description of their storage accounting. From my point of view it would be nice if people could try to break the elements down into 3 brackets. * Must Have - I really need this * Very Useful - Can I please have * Useful - but I can live without We can ask - people generally don't though. Just some though cheers johnk -- +------------------------------------------------------------+ |Dr. John Alan Kennedy Rechenzentrum Garching (RZG) | |Mail:jkennedy@rzg.mpg.de Boltzmannstrasse 2 | |Phone: +49 89 3299 2694 85748 Garching | |Fax: +49 89 3299 1301 | +------------------------------------------------------------+

2 1

Ideas for storage accounting record
by Henrik Thostrup Jensen 29 Sep '10

29 Sep '10

Hi Here are some preliminary ideas for fields in a storage accounting record. The first target for NDGF is dCache, however we also want be able to do storage accounting on plain unix directories (SGAS has a special client called Bart, which does job accounting on LRMS-level, and we have users who's primary usage for SGAS is regular (non-grid) HPC). * record metainformation: recordId unique createTime iso timestamp for the record * identity block: global uid global user identity (properly optional, vo is often more interesting) vo_issuer vo_type vo_name (or just vo) vo_group vo_role ## complex vo data? (multiple groups/roles) ## having multiple groups/roles for data is somewhat messy * storage information: sitename FQDN / Unique sitename poolname? A site will often have several places to store (should probably have another name) storagesystem which storage system is used storagetype ssd / platter disk / tape storageclass pinned / deletable / precious / archival / whatever storageurl url where the storage is located at * time period information: starttime startime for when the record holds endtime endtime for when the record holds * space usage spaceused space used by the identity (bytes is probably best) spacefree optional (number is not always available) spacereserved space reserved for the identity block (optional) Design issues: should everything be split into seperate records (say one for disk, one for tape, and so on), or should be possible to specify multiple usage sections in storage accounting record for different classes of stored data. AFAIK EMI is very interested in having something usuable very soon (this year) for performing storage accounting, so this is a fairly urgent topic if OGF wants to have something to say. However there is likely to be a certain overlap between the two groups. Best regards, Henrik Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org

4 7

Re: [UR-WG] Ideas for storage accounting record
by Gordon, John (STFC,RAL,ESC) 29 Sep '10

29 Sep '10

Here was a link on the EMI page that someone sent round this list recently. John Sent from my phone. -----Original Message----- From: john alan kennedy <jkennedy(a)rzg.mpg.de> Sent: 29 September 2010 10:03 To: john.gordon(a)stfc.ac.uk <john.gordon(a)stfc.ac.uk> Cc: htj(a)ndgf.org <htj(a)ndgf.org>; j.k.nilsen(a)fys.uio.no <j.k.nilsen(a)fys.uio.no>; ur-wg(a)ogf.org <ur-wg(a)ogf.org> Subject: Re: [UR-WG] Ideas for storage accounting record Hi John, That's a good point. Do you know where we can find it or do you have a contact person who we can talk to? Google showed me that they just had a Storage Forum but I didn't find anything on their accounting as of yet. osg-storage(a)opensciencegrid.org seems to be a possible starting point. Did anyone else chase this up? cheers johnk On 09/27/2010 01:33 PM, john.gordon(a)stfc.ac.uk wrote: > Perhaps people should look at the OSG one and comment? > > John > > >> -----Original Message----- >> From: ur-wg-bounces(a)ogf.org [mailto:ur-wg-bounces@ogf.org] On Behalf >> > Of > >> Henrik Thostrup Jensen >> Sent: 27 September 2010 11:55 >> To: Jon Kerr Nilsen >> Cc: ur-wg(a)ogf.org >> Subject: Re: [UR-WG] Ideas for storage accounting record >> >> Hi Jon >> >> On Thu, 23 Sep 2010, Jon Kerr Nilsen wrote: >> >> >>> Just to confirm that EMI is indeed interested in having a usable >>> definition of a storage accounting record before January... >>> >>> The EMI data group is currently in the process of collecting >>> requirements from user communities and storage providers, so not >>> > much > >>> input from us yet, but I'm certainly hoping for overlap between OGF >>> >> and >> >>> EMI work on this topic. >>> >> Given the short timespand of getting a record definition ready, I >> > would > >> strongly urge towards getting a rudementary format ready very fast and >> then iterating over the format multiple times (as requirements get in >> and >> showing the formats to the interested parties to see if it can fulfill >> their requirements). >> >> If you wait to start defining the format until requirements have been >> collected there will be very little time to work with the record >> format. >> >> >> Best regards, Henrik >> >> Software Developer, Henrik Thostrup Jensen<htj at ndgf.org> >> Nordic Data Grid Facility. WWW: www.ndgf.org >> -- >> ur-wg mailing list >> ur-wg(a)ogf.org >> http://www.ogf.org/mailman/listinfo/ur-wg >> -- +------------------------------------------------------------+ |Dr. John Alan Kennedy Rechenzentrum Garching (RZG) | |Mail: jkennedy(a)rzg.mpg.de Boltzmannstrasse 2 | |Phone: +49 89 3299 2694 85748 Garching | |Fax: +49 89 3299 1301 | +------------------------------------------------------------+ -- Scanned by iCritical.

1 0

OGF-30
by Andrea Cristofori 28 Sep '10

28 Sep '10

Dear all, I have requested a session of 1.5 hours at next OGF in Brussels asking preferably for the 28th or 29th. I hope we can all make it there and continue the discussion. In the meanwhile we could try to collect some request from the users and try to put all the informations together. Cheers, Andrea -- Andrea Cristofori INFN-CNAF Viale Berti Pichat 6/2 40127 Bologna Italy Tel. : +39-051-6092920 Skype: andrea-cnaf

4 6

Aggregated usage records in SGAS
by Henrik Thostrup Jensen 23 Sep '10

23 Sep '10

Hi Here is an actual use case for how aggregation can be done. In SGAS we aggregate usage records into the followign format: execution_date | date insert_date | date machine_name | string user_identity | string vo_issuer | string vo_name | string vo_group | string vo_role | string n_jobs | integer cputime | numeric walltime | numeric generate_time | timestamp The combination of the following fields is considered unique: execution_date, insert_date, machine_name, user_identity, vo_issuer, vo_name, vo_group, vo_role. Some of these can be null/non-existing (vo_group and vo_role). The reason for seperating insert and execution date as that some records arrive late when the registration process fails for some reason. Most admins care more about registrations, where as usage data usually uses execution time for statistics. The three following fields are aggregated number of jobs, summed cputime, summed walltime The final field us when the record was generated. This format aggregates quite well. 3.8M records aggregated into 18K records in the NDGF accounting database. I'm an in no way implying that all aggregations should look like that and that all queries can be answered by such an aggregation. But for us, in can answer most of the common queries, and time resolution per day is typical enough. Currently we don't use AUR in SGAS (we do use UR for all job records), but might in the future (currently the methods for querying data is somewhat limited). Best regards, Henrik Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org

2 2

SGAS extensions to usage record
by Henrik Thostrup Jensen 23 Sep '10

23 Sep '10

Hi Here is a list of the UR extensions used in SGAS. It is provided in the hope to be usefull in the UR2 creation process. Arguably the most important extension the ability to describe VO affiliation, including groups and roles. It looks like this: <vo:VO vo:type="voms" xmlns:vo="http://www.sgas.se/namespaces/2009/05/ur/vo"> <vo:Name>bio.ndgf.org</vo:Name> <vo:Issuer>/O=Grid/O=NorduGrid/CN=host/voms.ndgf.org</vo:Issuer> <vo:Attribute> <vo:Group>bio.ndgf.org/test</vo:Group> <vo:Role>VO-dev</vo:Role> </vo:Attribute> <vo:Attribute> <vo:Group>bio.ndgf.org</vo:Group> </vo:Attribute> <vo:Attribute> <vo:Group>bio.ndgf.org/test</vo:Group> </vo:Attribute> </vo:VO> It is placed in the user identity block. It is somewhat closely modelled after VOMS (which is what we needed to model), but has been used for several other systems (usually one would just the Name and type elements). Descibing the type VO information a rather important feature IMO. Futhermore we have a range of small simple extensions. We share submit time with the DEISA project (well we just reused what they had made), though there is no data exchange between NDGF and DEISA. The following lists the extensions: Submit time (shared with Deisa) <deisa:SubmitTime xmlns:deisa="http://rmis.deisa.org/acct">2009-07-07T09:06:37Z</deisa:SubmitTime> Submission time can be used to tell how long time a job spend in queue, which is an often-asked-for metric. SGAS UR extensions: KSI2KCpuDuration (scaled cpu duration) <sgas:KSI2KCpuDuration>PT150.4S</sgas:KSI2KCpuDuration> KSI2KWallDuration (scaled wall duration) <sgas:KSI2KWallDuration>PT8488.44S</sgas:KSI2KWallDuration> The two ksi2k elements are no longer used. If there should be the possibility for scaled values in ur2, it should not be limited to a specific scaling method. insertTime (when the record was inserted into the database. this field only exists when extracting a record from a database <ur:RecordIdentity sgas:insertTime="2009-07-07T09:06:52Z" /> UserTime (time spend in userland) <sgas:UserTime>PT84.3S</sgas:UserTime> KernelTime (time spend in kernel time) <sgas:KernelTime>PT13.2S</sgas:KernelTime> ExitCode (job exit code, integer value) <sgas:ExitCode>0</sgas:ExitCode> MajorPageFaults (times a major page fault occured) <sgas:MajorPageFaults>3</sgas:MajorPageFaults> RuntimeEnvironment (requested ARC runtime environments, allows per-application accounting, the element can be repeated) <sgas:RuntimeEnvironment>APPS/HEP/ATLAS-13.0.40</sgas:RuntimeEnvironment> The latter five where actually requested by CSC (Finish HPC centre), which also uses SGAS (there are at least five deployments of the SGAS accounting service AFAIK). Futhermore we are thinking about adding fields for how many files the job down/up-loaded and how much data was transferred. Furthermore how many files was used from the cache (Input files, which existed locally and does not need to be downloaded. This might be a bit ARC specific, but the concept could still be usefull for others). We also include information about what software produced the record, and optionally which version of the software. This information does not describe the job in any way, but can be practical for identifying the source of the record and detect incompatabilities. logger = namespace http://www.sgas.se/namespaces/2010/08/logger LoggerName (name of the logger, can have optional version element) <logger:LoggerName>ARC0-URLogger</logger:LoggerName> <logger:LoggerName logger:version="0.8.2.2">ARC0-URLogger</logger:LoggerName> Other issues: The specification text for the machine name, host names, etc. could be clearer in the standard. In the future a sample set of records for compliancy testing could be provided. Best regards, Henrik Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org

1 0

Re: [UR-WG] ur-wg Digest, Vol 25, Issue 1
by Michael Brenner 20 Sep '10

20 Sep '10

Andrea, the 28th is preferred. Cheers Michael Am 17.09.10 19:00, schrieb ur-wg-request(a)ogf.org: > Send ur-wg mailing list submissions to > ur-wg(a)ogf.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://www.ogf.org/mailman/listinfo/ur-wg > or, via email, send a message with subject or body 'help' to > ur-wg-request(a)ogf.org > > You can reach the person managing the list at > ur-wg-owner(a)ogf.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ur-wg digest..." > > > Today's Topics: > > 1. OGF-30 (Andrea Cristofori) > 2. Re: OGF-30 (john.gordon(a)stfc.ac.uk) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 17 Sep 2010 11:21:25 +0200 > From: Andrea Cristofori <andrea.cristofori(a)cnaf.infn.it> > Subject: [UR-WG] OGF-30 > To: "ur-wg(a)ogf.org" <ur-wg(a)ogf.org> > Message-ID: <4C933315.5070800(a)cnaf.infn.it> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Dear all, > > I have requested a session of 1.5 hours at next OGF in Brussels asking > preferably for the 28th or 29th. > I hope we can all make it there and continue the discussion. > In the meanwhile we could try to collect some request from the users and > try to put all the informations together. > > Cheers, > Andrea > -- M. Sc. Michael Brenner Lehrgebiet Rechnernetze Regionales Rechenzentrum fuer Niedersachsen (RRZN) Forschungszentrum L3S Leibniz Universitaet Hannover Schlosswender Strasse 5, 30159 Hannover Tel. +49 511 762-796242 mailto:brenner@rvs.uni-hannover.de

1 0