Ideas for storage accounting record
Hi Here are some preliminary ideas for fields in a storage accounting record. The first target for NDGF is dCache, however we also want be able to do storage accounting on plain unix directories (SGAS has a special client called Bart, which does job accounting on LRMS-level, and we have users who's primary usage for SGAS is regular (non-grid) HPC). * record metainformation: recordId unique createTime iso timestamp for the record * identity block: global uid global user identity (properly optional, vo is often more interesting) vo_issuer vo_type vo_name (or just vo) vo_group vo_role ## complex vo data? (multiple groups/roles) ## having multiple groups/roles for data is somewhat messy * storage information: sitename FQDN / Unique sitename poolname? A site will often have several places to store (should probably have another name) storagesystem which storage system is used storagetype ssd / platter disk / tape storageclass pinned / deletable / precious / archival / whatever storageurl url where the storage is located at * time period information: starttime startime for when the record holds endtime endtime for when the record holds * space usage spaceused space used by the identity (bytes is probably best) spacefree optional (number is not always available) spacereserved space reserved for the identity block (optional) Design issues: should everything be split into seperate records (say one for disk, one for tape, and so on), or should be possible to specify multiple usage sections in storage accounting record for different classes of stored data. AFAIK EMI is very interested in having something usuable very soon (this year) for performing storage accounting, so this is a fairly urgent topic if OGF wants to have something to say. However there is likely to be a certain overlap between the two groups. Best regards, Henrik Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org
Hi,
AFAIK EMI is very interested in having something usuable very soon (this year) for performing storage accounting, so this is a fairly urgent topic if OGF wants to have something to say. However there is likely to be a certain overlap between the two groups.
Just to confirm that EMI is indeed interested in having a usable definition of a storage accounting record before January as we have promised EU to have definition ready by then. This definition, "reflecting practical, financial and legal requirements of storage location, usage and space and data flow", is then planned to be implemented in DPM, StoRM, dCache, gLite FTS and UNICORE FTS before the end of EMI. The EMI data group is currently in the process of collecting requirements from user communities and storage providers, so not much input from us yet, but I'm certainly hoping for overlap between OGF and EMI work on this topic. best regards, Jon Storage accounting task leader EMI data group On 23. sep. 2010, at 13.16, Henrik Thostrup Jensen wrote:
Hi
Here are some preliminary ideas for fields in a storage accounting record. The first target for NDGF is dCache, however we also want be able to do storage accounting on plain unix directories (SGAS has a special client called Bart, which does job accounting on LRMS-level, and we have users who's primary usage for SGAS is regular (non-grid) HPC).
* record metainformation:
recordId unique createTime iso timestamp for the record
* identity block:
global uid global user identity (properly optional, vo is often more interesting) vo_issuer vo_type vo_name (or just vo) vo_group vo_role ## complex vo data? (multiple groups/roles) ## having multiple groups/roles for data is somewhat messy
* storage information:
sitename FQDN / Unique sitename poolname? A site will often have several places to store (should probably have another name) storagesystem which storage system is used storagetype ssd / platter disk / tape storageclass pinned / deletable / precious / archival / whatever storageurl url where the storage is located at
* time period information:
starttime startime for when the record holds endtime endtime for when the record holds
* space usage
spaceused space used by the identity (bytes is probably best) spacefree optional (number is not always available) spacereserved space reserved for the identity block (optional)
Design issues:
should everything be split into seperate records (say one for disk, one for tape, and so on), or should be possible to specify multiple usage sections in storage accounting record for different classes of stored data.
AFAIK EMI is very interested in having something usuable very soon (this year) for performing storage accounting, so this is a fairly urgent topic if OGF wants to have something to say. However there is likely to be a certain overlap between the two groups.
Best regards, Henrik
Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org -- ur-wg mailing list ur-wg@ogf.org http://www.ogf.org/mailman/listinfo/ur-wg
Hi Jon On Thu, 23 Sep 2010, Jon Kerr Nilsen wrote:
Just to confirm that EMI is indeed interested in having a usable definition of a storage accounting record before January...
The EMI data group is currently in the process of collecting requirements from user communities and storage providers, so not much input from us yet, but I'm certainly hoping for overlap between OGF and EMI work on this topic.
Given the short timespand of getting a record definition ready, I would strongly urge towards getting a rudementary format ready very fast and then iterating over the format multiple times (as requirements get in and showing the formats to the interested parties to see if it can fulfill their requirements). If you wait to start defining the format until requirements have been collected there will be very little time to work with the record format. Best regards, Henrik Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org
Perhaps people should look at the OSG one and comment? John
-----Original Message----- From: ur-wg-bounces@ogf.org [mailto:ur-wg-bounces@ogf.org] On Behalf Of Henrik Thostrup Jensen Sent: 27 September 2010 11:55 To: Jon Kerr Nilsen Cc: ur-wg@ogf.org Subject: Re: [UR-WG] Ideas for storage accounting record
Hi Jon
On Thu, 23 Sep 2010, Jon Kerr Nilsen wrote:
Just to confirm that EMI is indeed interested in having a usable definition of a storage accounting record before January...
The EMI data group is currently in the process of collecting requirements from user communities and storage providers, so not much input from us yet, but I'm certainly hoping for overlap between OGF and EMI work on this topic.
Given the short timespand of getting a record definition ready, I would strongly urge towards getting a rudementary format ready very fast and then iterating over the format multiple times (as requirements get in and showing the formats to the interested parties to see if it can fulfill their requirements).
If you wait to start defining the format until requirements have been collected there will be very little time to work with the record format.
Best regards, Henrik
Software Developer, Henrik Thostrup Jensen <htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org -- ur-wg mailing list ur-wg@ogf.org http://www.ogf.org/mailman/listinfo/ur-wg -- Scanned by iCritical.
Any news on our request for a session? The Early Bird registration closes this week. John -- Scanned by iCritical.
Hi John, That's a good point. Do you know where we can find it or do you have a contact person who we can talk to? Google showed me that they just had a Storage Forum but I didn't find anything on their accounting as of yet. osg-storage@opensciencegrid.org seems to be a possible starting point. Did anyone else chase this up? cheers johnk On 09/27/2010 01:33 PM, john.gordon@stfc.ac.uk wrote:
Perhaps people should look at the OSG one and comment?
John
-----Original Message----- From: ur-wg-bounces@ogf.org [mailto:ur-wg-bounces@ogf.org] On Behalf
Of
Henrik Thostrup Jensen Sent: 27 September 2010 11:55 To: Jon Kerr Nilsen Cc: ur-wg@ogf.org Subject: Re: [UR-WG] Ideas for storage accounting record
Hi Jon
On Thu, 23 Sep 2010, Jon Kerr Nilsen wrote:
Just to confirm that EMI is indeed interested in having a usable definition of a storage accounting record before January...
The EMI data group is currently in the process of collecting requirements from user communities and storage providers, so not
much
input from us yet, but I'm certainly hoping for overlap between OGF
and
EMI work on this topic.
Given the short timespand of getting a record definition ready, I
would
strongly urge towards getting a rudementary format ready very fast and then iterating over the format multiple times (as requirements get in and showing the formats to the interested parties to see if it can fulfill their requirements).
If you wait to start defining the format until requirements have been collected there will be very little time to work with the record format.
Best regards, Henrik
Software Developer, Henrik Thostrup Jensen<htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org -- ur-wg mailing list ur-wg@ogf.org http://www.ogf.org/mailman/listinfo/ur-wg
-- +------------------------------------------------------------+ |Dr. John Alan Kennedy Rechenzentrum Garching (RZG) | |Mail: jkennedy@rzg.mpg.de Boltzmannstrasse 2 | |Phone: +49 89 3299 2694 85748 Garching | |Fax: +49 89 3299 1301 | +------------------------------------------------------------+
Hi John, We have done a bit of chasing in EMI. OSG has a software project called Gratia (https://twiki.grid.iu.edu/bin/view/Accounting/WebHome) which, according to Brian Bockelman, implements the OGF UR with two storage accounting records, StorageElement (path, space token name, space type, etc.) and StorageElementRecord (dealing with measurements). Philippe Canal is head of the software project, he may be a good contact person. cheers, Jon On 29. sep. 2010, at 11.02, john alan kennedy wrote:
Hi John,
That's a good point.
Do you know where we can find it or do you have a contact person who we can talk to?
Google showed me that they just had a Storage Forum but I didn't find anything on their accounting as of yet.
osg-storage@opensciencegrid.org seems to be a possible starting point.
Did anyone else chase this up?
cheers johnk
On 09/27/2010 01:33 PM, john.gordon@stfc.ac.uk wrote:
Perhaps people should look at the OSG one and comment?
John
-----Original Message----- From: ur-wg-bounces@ogf.org [mailto:ur-wg-bounces@ogf.org] On Behalf
Of
Henrik Thostrup Jensen Sent: 27 September 2010 11:55 To: Jon Kerr Nilsen Cc: ur-wg@ogf.org Subject: Re: [UR-WG] Ideas for storage accounting record
Hi Jon
On Thu, 23 Sep 2010, Jon Kerr Nilsen wrote:
Just to confirm that EMI is indeed interested in having a usable definition of a storage accounting record before January...
The EMI data group is currently in the process of collecting requirements from user communities and storage providers, so not
much
input from us yet, but I'm certainly hoping for overlap between OGF
and
EMI work on this topic.
Given the short timespand of getting a record definition ready, I
would
strongly urge towards getting a rudementary format ready very fast and then iterating over the format multiple times (as requirements get in and showing the formats to the interested parties to see if it can fulfill their requirements).
If you wait to start defining the format until requirements have been collected there will be very little time to work with the record format.
Best regards, Henrik
Software Developer, Henrik Thostrup Jensen<htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org -- ur-wg mailing list ur-wg@ogf.org http://www.ogf.org/mailman/listinfo/ur-wg
-- +------------------------------------------------------------+ |Dr. John Alan Kennedy Rechenzentrum Garching (RZG) | |Mail: jkennedy@rzg.mpg.de Boltzmannstrasse 2 | |Phone: +49 89 3299 2694 85748 Garching | |Fax: +49 89 3299 1301 | +------------------------------------------------------------+
I know Philiipe (he may be on this list). OSG sends cpu UR from Gratia to the APEL repository for the LHC VOs work which runs on OSG. I haven't seen any mention of their storage records in OGF. John From: Jon Kerr Nilsen [mailto:j.k.nilsen@fys.uio.no] Sent: 29 September 2010 11:34 To: john alan kennedy Cc: Gordon, John (STFC,RAL,ESC); htj@ndgf.org; ur-wg@ogf.org Subject: Re: [UR-WG] Ideas for storage accounting record Hi John, We have done a bit of chasing in EMI. OSG has a software project called Gratia (https://twiki.grid.iu.edu/bin/view/Accounting/WebHome) which, according to Brian Bockelman, implements the OGF UR with two storage accounting records, StorageElement (path, space token name, space type, etc.) and StorageElementRecord (dealing with measurements). Philippe Canal is head of the software project, he may be a good contact person. cheers, Jon On 29. sep. 2010, at 11.02, john alan kennedy wrote: Hi John, That's a good point. Do you know where we can find it or do you have a contact person who we can talk to? Google showed me that they just had a Storage Forum but I didn't find anything on their accounting as of yet. osg-storage@opensciencegrid.org seems to be a possible starting point. Did anyone else chase this up? cheers johnk On 09/27/2010 01:33 PM, john.gordon@stfc.ac.uk wrote: Perhaps people should look at the OSG one and comment? John -----Original Message----- From: ur-wg-bounces@ogf.org [mailto:ur-wg-bounces@ogf.org] On Behalf Of Henrik Thostrup Jensen Sent: 27 September 2010 11:55 To: Jon Kerr Nilsen Cc: ur-wg@ogf.org Subject: Re: [UR-WG] Ideas for storage accounting record Hi Jon On Thu, 23 Sep 2010, Jon Kerr Nilsen wrote: Just to confirm that EMI is indeed interested in having a usable definition of a storage accounting record before January... The EMI data group is currently in the process of collecting requirements from user communities and storage providers, so not much input from us yet, but I'm certainly hoping for overlap between OGF and EMI work on this topic. Given the short timespand of getting a record definition ready, I would strongly urge towards getting a rudementary format ready very fast and then iterating over the format multiple times (as requirements get in and showing the formats to the interested parties to see if it can fulfill their requirements). If you wait to start defining the format until requirements have been collected there will be very little time to work with the record format. Best regards, Henrik Software Developer, Henrik Thostrup Jensen<htj at ndgf.org> Nordic Data Grid Facility. WWW: www.ndgf.org -- ur-wg mailing list ur-wg@ogf.org http://www.ogf.org/mailman/listinfo/ur-wg -- +------------------------------------------------------------+ |Dr. John Alan Kennedy Rechenzentrum Garching (RZG) | |Mail: jkennedy@rzg.mpg.de Boltzmannstrasse 2 | |Phone: +49 89 3299 2694 85748 Garching | |Fax: +49 89 3299 1301 | +------------------------------------------------------------+ -- Scanned by iCritical.
participants (4)
-
Henrik Thostrup Jensen -
john alan kennedy -
john.gordon＠stfc.ac.uk -
Jon Kerr Nilsen