- ur-wg - lists.ogf.org

Next phone meeting on 30th of April at 15:00
by Andrea Cristofori 09 May '13

09 May '13

Dear all, Looks like today many of us could not join the phone meeting so I think it is better to postpone it. Next one will be in two weeks, the 30th of April at 15:00. In the meanwhile if you have any idea or thought about what John Gordon proposed please send an email to the list so we can start the discussion via email and think about it before the meeting. Regards Andrea -- Andrea Cristofori INFN-CNAF Viale Berti Pichat 6/2 40127 Bologna Italy Tel. : +39-051-6092920 Skype: andrea-cnaf

1 2

Draft/suggestion for Storage Accounting Record (2010/10/07)
by Henrik Thostrup Jensen 11 Apr '13

11 Apr '13

Hi (this is send both to the ogf ur list and the emi sar list, sorry for duplicates). Jon and me happened to be at the same meeting this week, so we took out a couple of hours of the programme to create a draft suggestion for a storage accounting record. The suggestion is mainly meant as a way to kickoff the meeting in Brussels. 1. Discrete vs continuous First we discussed how storage differs from jobs in the sense that a job is a discrete unit whereas storage has a more continuous nature, where the usage more or less constantly varies (bytes used), but typically in relatively small scale. A storage record can however only describe something discrete, so the continuous nature of storage will have to be split into something discrete (similar to integration). Of course the granularity of this process should not be dictated be the standard. The first suggestions in then to have a start and an end time for the storage measurement where a measurement is taken. This allows to create per-day, per-hour resolution or something third, depending on the needs. Result: "StartTime" and "EndTime" element for describing the time interval for when the record is valid. Both of them are DateTime values. 2. What to measure The most obvious metric here is the amount of used space. Another metric is the amount of reserved space, i.e., space not taken by actual files/objects, but which cannot be used by other parties. A third metric, is the amount of space which is not allocated, and can be used. Initially we called this "free space", but this term is not exactly accurate as it refers more to a file system metric, which can be very different from what can actually be used due to quotas. Instead we choose to have an element describing how much unallocated space there is, i.e., how space one can be expected to be able to use. The actual free space is a metric, which can be very tricky to measure, and is typically only of interest to the storage system administrator. A secondary issue is how to report the numbers. We quickly decided on using bytes, as it is the fundamental unit, and saves us from deciding on weather to use 1000 or 1024 bytes as a base. Having multiple options (say KB, MB) for reporting would complicate the standard unnecessarily without any real benefits, so we suggest to keep it to bytes, and bytes only. This would also ensure that the number is always an integer (when using KB and up, floats could be used to be exact) and saves silly conversion routines when parsing the record. We briefly considered if reporting should be per file, but this was quite quickly shoot down, as it would make the records unreasonably large, without providing any real value. We did end with an element describing the number of files, which are using the space reported. Result: "UsedSpace", "ReservedSpace", "UnAllocatedSpace" metrics for describing how space is used, reserved, and available for use. Reserved and unallocated are probably not overlapping (as reserved space is technically used). The measurement is in bytes. "FileCount" for describing the number of files using the space. 3. Site and Storage Concepts The issue of how to describe the site and what storage the accounted data is stored is perhaps the most complex issue in defining this format. The discussion to achieve this was very non-linear, so I will just the describe the result: A site is considered a top level container for storage. The site name should globally unique (which probably means an FQDN). A storage system is an independent system on a site. A storage system partition is a part of storage system (similar to dCache pools) A storage type describes the storage type where the data is stored (disk, tape) None of these elements are mandatory (though we couldn't find a valid use case for a record without a site). E.g., is perfectly valid to just use site, site and storage type, or site and storage system. If multiple of the elements exist, they are considered to have hierarchical tree-like structure, i.e., site -> storage system -> storage system partition -> storage type. How to structure this is described in the "Record Structure" section. This allows a site with a simple setup to reporting just for the site, where as more complex installations can report per storage system, and how much is placed on tape and disk respectively for different storage systems. A single institution can easily report multiple sites, we do not interfere in this, but it is very likely that a site would have several storage element / systems and would like to able to aggregate it under a single site. Finally we also add the possibility for a storage class, which describes the class of storage accounted for. This could "precious", "deletable", "pinned", etc. It is not considered a part of the hierarchy described above. 4. Identity To describe who is using the space, an identity block or group, similar to the one in usage record must be supported. As with usage record it must be possible to specify both a local user name and global user identity. However the common case is likely to be a VO or a VO group, so being able to describe this is extremely important. Another use case is a local group at the site who owns the data (not everyone uses grid). How exactly to describe the virtual organization is not quite clear, but the following elements are needed as a base: VO name, VO issuer (DN of the VO issuer, somewhat VOMS specific), VO group, and VO role. There might be use cases for being able to have multiple VO blocks (though I suspect that will be messy). Resulting elements: VO Name VO Issuer VO group VO role Global User Identity Local User Name Local User Group All element are strings. The Global User Identity is to be considered global unique. Furthermore the combination of VO issuer and VO name should be globally unique. 5. Record Structure Having identified the identity block and the site/storage structure, we tried to find a good way to structure the individual records. We considered the possibility of having a tree structure in the storage accounting record, with the site, storage system, storage system partition, and storage type as possible levels in the tree. The leaves would then each have an identity and space usage block. This however would be quite complicated to construct and parsed, so a simpler - more flat - structure was quickly preferred. Here each record would maximum have a one site, storage system, storage system partition, and storage type (but all still optional). If a system would need to describe for multiple storage system / storage system partitions / storage type several records would have to be generated. This format should still be easy to aggregate together into site or storage system to get complete numbers. The flat format is also much easier to explain which probably means that it should be preferred. The two structure can describe exactly the same, so there is no limitation by choosing the flat format. 6. Record Overview Given the previous section, we can now present an overview of the elements in a storage accounting record. We reuse the recordId and createTime elements from the usage record standard (just the element names, not the namespaces). StorageAccountingRecord recordId (considered globally unique) createTime (timestamp when the record was created) StartTime (datetime value) EndTime (datetime value) Site StorageSystem StorageSystemPartition StorageType StorageClass IdentityBlock VO Block VO Name VO Issuer VO Group VO Role GlobalUserIdentity LocalUserName LocalGroupName UsedSpace ReservedSpace UnallocatedSpace FileCount The only enforced element is the recordId. 7. Name Issues Someone brought up that SAR is not the most fortunate name. It sounds a bit like SARS, and the phonetic sound is apparently close to a non-to-fortunate Hungarian word. We could consider SR (storage record) or SUR (storage usage record) This really doesn't matter. 8. Sharing Elements with Usage Record Standard Some of the elements are identical in both name and semantics to the ones in usage record. We do not suggest to share the elements as such (same namespace), as it would make the standard rely on the UR standard, and hence make it less self-contained. The UR standard is only used in a few systems, and is likely to be replaced with a new standard sometime. Furthermore the implementation gains of sharing the names are very small, if they even exist. We have definitely missed something in this, but we hope this can be a start for the discussion in Brussels. If you see problems or issues with this record please let us know. Best regards, Henrik Thostrup Jensen & Jon Kerr Nilsen

5 9

No meeting today and summary of OGF37
by Jon Kerr Nilsen 19 Mar '13

19 Mar '13

Hi all, Since Andrea is still in US on a well deserved vacation we will have no meeting today (sorry about the short notice). Next meeting will be in two weeks on 2 April. While this week's meeting is cancelled and the UR-2.0 has entered the public comment period, our work is not yet completed. As an outcome of the UR-WG session the following tasks should be carried out in the near future: - Present UR-2.0 on WG page. - short summary of how UR-2.0 can be used (to attract potential users). - Create a document on how to make a profile. - Take some real-life use-cases (NIST, clouds, ...) and make suggestions how UR-2.0 can cover them. - Ping potential users (at least the same we did for StAR) and ask for comments and encourage them to try to implement. These topics will be discussed on the next meeting, but if there are already volunteers for helping out on any of them there's no need to hesitate :) cheers, Jon

2 1

Fwd: [wg-all] New documents in public comment: Usage Record 2.0 and NML Base Schema
by Andrea Cristofori 06 Mar '13

06 Mar '13

FYI -------- Original Message -------- Subject: [wg-all] New documents in public comment: Usage Record 2.0 and NML Base Schema Date: Tue, 5 Mar 2013 11:35:56 -0900 From: Greg Newby <gbnewby(a)alaska.edu> To: All WG <wg-all(a)ogf.org> Dear OGF members and affiliates: Two new documents have begun public comment. Please consider reading and providing feedback. Brief affirmations are very useful to demonstrate community input and support. More detailed analysis is also, of course, welcome. All documents in public comment are accessible here: http://redmine.ogf.org/projects/editor-pubcom/boards Note that this is an updated location, using the new Redmine tracker system. Anyone may register for a Redmine username. These documents hav been newly added for public comment: * Usage Record 2.0, recommendations track document, via Usage Record WG * NML Base Schema version 1, recommendations track document, via Network Markup Language WG -- Greg Newby, OGF Standards Editor Dr. Gregory Newby, Director of the Arctic Region Supercomputing Center Univ of Alaska Fairbanks-909 Koyukuk Dr-PO Box 756020-Fairbanks-AK 99775-6020 e: gbnewby(a)alaska.edu v: 907-450-8663 f: 907-450-8603 w: people.arsc.edu/~newby -- The wg-all mailing list is an umbrella list for all OGF mailing lists. If you are subscribed to any OGF mailing list, you will receive mails to wg-all. wg-all mailing list wg-all(a)ogf.org https://www.ogf.org/mailman/listinfo/wg-all

1 0

No meeting today and update on our document
by Andrea Cristofori 05 Mar '13

05 Mar '13

Dear all, We have some good news and some better news. Today we will not have our usual meeting. Our document has just passed the scrutiny of the internal review and is now ready for public comment. I believe that soon we should receive an official email with all the detail for the comment procedure. For this reasons I do not think that there is much to discuss today. The plan now is to present the work done to the next OGF and discuss the possible comments. We will resume the phone meetings after OGF. Cheers, Andrea and Jon -- Andrea Cristofori INFN-CNAF Viale Berti Pichat 6/2 40127 Bologna Italy Tel. : +39-051-6092920 Skype: andrea-cnaf

1 0

Usage Record 2.0 - P-REC
by Andrea Cristofori 22 Feb '13

22 Feb '13

Dear all, as planned we just sent the latest version of our document to OGF: Usage Record 2.0 - P-REC (Proposed Recommendation). The pdf is available here: http://redmine.ogf.org/dmsf_files/13010?download= It contains some minor corrections. If you want to check the detail you can have a look to the diff in the repository. Have a nice weekend. Cheers, Andrea and Jon -- Andrea Cristofori INFN-CNAF Viale Berti Pichat 6/2 40127 Bologna Italy Tel. : +39-051-6092920 Skype: andrea-cnaf

1 0

Final draft ready(?) for public comment
by Andrea Cristofori 21 Feb '13

21 Feb '13

Dear all, as discussed during last phone meeting Ralph made some corrections and I and Jon applied other correction suggested by John Alan Kennedy and discussed last Tuesday. You might find the latest version here: https://redmine.ogf.org/dmsf_files/13009?download= If there are no objection by tomorrow evening we will send it for public comments. After that if there are comments, I am thinking especially to the Cloud block, they can be sent as public comments. Cheers, Andrea P.S.: for those who were not present during last phone meeting: next phone meeting was supposed to be the 5th of March. We think that if there are no big issues to discuss we can just skip it and fix one after OGF37. -- Andrea Cristofori INFN-CNAF Viale Berti Pichat 6/2 40127 Bologna Italy Tel. : +39-051-6092920 Skype: andrea-cnaf

1 0

profile
by Ralph Mueller-Pfefferkorn 19 Feb '13

19 Feb '13

Hi there, just checked in some minor changes including a sentence about a profile for SubjectIdentityBlock. Regards, Ralph

1 0

Next phone meeting will be today the 19th of February at 15:00 CET
by Andrea Cristofori 19 Feb '13

19 Feb '13

Dear all, A reminder for today's meeting at 15:00 CET. John Alan Kennedy provided several correction and suggestion on how to improve the document. I already applied those that did not required to be discussed. For the others we can have a look during today's meeting. You might find the latest version of the document here: https://redmine.ogf.org/dmsf_files/13008?download= Cheers, Andrea -- Andrea Cristofori INFN-CNAF Viale Berti Pichat 6/2 40127 Bologna Italy Tel. : +39-051-6092920 Skype: andrea-cnaf

1 0

minor changes
by Ralph Mueller-Pfefferkorn 31 Jan '13

31 Jan '13

Hi, I checked in some minor phrasing changes. See attached diffs. Regards, Ralph

2 1