- ogsa-d-wg - lists.ogf.org

FW: Data Management Interface Standardization BOF working list is up
by Allen Luniewski 13 Oct '05

13 Oct '05

Bill Allcock's note bounced... Allen ----- Forwarded by Allen Luniewski/Almaden/IBM on 10/13/2005 09:35 AM ----- To: dmis-bof(a)ggf.org, gridftp-wg(a)ggf.org, gsm-wg(a)ggf.org, ogsa-d-wg(a)ggf.org Subject: Data Management Interface Standardization BOF working list is up This is the last time I will SPAM the other lists. If you wish to subscribe, please send mail to majordomo(a)ggf.org with subscribe dmis-bof END in the body. On Monday, I will send the first draft of the charter out to the dmis-bof list for comment. Please subscribe by then if you are interested. If we become an officially chartered WG, we will get a new list dmis-wg. Bill -- William E. Allcock Argonne National Laboratory Bldg 221, Office C-115A 9700 South Cass Ave Argonne, IL 60439-4844 Office Phone: +1-630-252-7573 Office Fax: +1-630-252-1997 Cell Phone: +1-630-854-2842

1 0

Telcon Agenda, THURSDAY October 20th
by Dave Berry 12 Oct '05

12 Oct '05

Dear all, The next OGSA-D telcon will be on an unusual day, Thursday 20th October, at the usual time of 8:00am Pacific / 11:00am Eastern / 4:00pm UK / 5:00pm Europe. It will also have a DIFFERENT PIN: UK Dial In: 0845 245 0224 US Dial In: 1 866 838 0046 International Dial In: +44 1452 542309 PIN Number: 275 359 Agenda for the October 20th teleconference: 1) Early discussion (5 minutes) Note taker assignment Roll call Minutes approval from October 12th teleconference (http://tinyurl.com/9bkxm) Agenda bashing 2) Action report (5 minutes) - Dave to put slides and reports from GGF on GridForge. - Dave to forward OGSA-DAI internal memo to Allen - Allen to comment on OGSA 1.5 draft; Dave to revise and send to list - Dave to send F2F plans to mailing list. - Peter to rewrite storage managment section to reflect changes from GSM-WG. - Allen to write text about policy. - Allen to write an overview section on security - Section authors to add security notes to their sections, once the overview is ready. - Dave to revamp section 3 of the architecture document - Everyone to talk with your contacts to get more participation in this WG. - Allen/Dave to contact David Martin & Hiro Kishimoto to get reviewers & expert participation (after internal reviews). 3) Planning update (5 minutes) Update on plans for telcon, F2F and document writing 4) OGSA 1.5 Discussion (45 minutes) Review of the OGSA 1.5 draft 5) Wrap up (5 minutes) AOB

1 0

Telcon Agenda, 11th October: Security and planning our first drafts
by Dave Berry 11 Oct '05

11 Oct '05

Dear All, The next teleconference of the OGSA Data Architecture working group will be held on Wednesday, October 11th, 2005 at the usual time of 8:00am Pacific / 11:00am Eastern / 4:00pm UK / 5:00pm Europe. As a reminder, here is the teleconference information: UK dialin: 0845 245 0224 US dialin: 1 866 838 0046 Intl dialin: +44 1452 542309 Passcode: 369786 Agenda for the October 11th teleconference: 1) Early discussion (5 minutes) Note taker assignment Roll call Minutes approval from September 28th teleconference (http://tinyurl.com/72v6w) Agenda bashing 2) Action report (5 minutes) - Various actions relating to GGF15 [Done] - Dave to chase section authors re dates for their review sessions - Mario to check whether internal OGSA-DAI notes on security could be released, possibly just to Allen and Dave [Done] - Dave to redraft data sections for OGSA 1.5 document [Ongoing] - Allen to write an overview section on security [Ongoing] - Section authors to add security notes to their sections, once the overview is ready. - Everyone to talk with your contacts to get more participation in this WG. - Dave to revamp section 3 of the architecture document [Ongoing] - Allen/Dave to contact David Martin & Hiro Kishimoto to get reviewers & expert participation (after internal reviews). 3) Planning our first drafts (15 minutes) Allen & Dave have drafted a timetable for completing the first draft of both documents. This includes telcon schedules and F2F meetings. We will present this and discuss. 4) Security Discussion (15 minutes) Assimilating the information we've received from GGF. 5) Wrap up (5 minutes) AOB DONM - THURSDAY 20th: OGSA 1.5 revision Dave Berry Research Manager, National e-Science Centre 15 South College St., Edinburgh Tel: +44 131 6514039

1 0

Telcon number and schedule: Oct 12 - Nov 30
by Dave Berry 08 Oct '05

08 Oct '05

Dear all, Here is a provisional timetable for OGSA-D telcons for the next two months. Please note that the reviews have been rescheduled to ensure that key people will be present. On the four weeks from Oct 26th to Nov 16th we will review four sections of the services document along with their associated scenarios. With the exception of the Oct 20th call, all telcons will be on Wednesdays at the usual time of 8:00am Pacific / 11:00am Eastern / 4:00pm UK / 5:00pm Europe. Oct 12th: Security Oct 20th: Review of OGSA 1.5 document (THURSDAY, at the usual time) Oct 26th: Section review Nov 2nd: Section review Nov 9th: Section review Nov 16th: Section review Nov 23rd: No telcon (Thanksgiving) Nov 30th: Review of scenarios document Here is the dialin information for the Wednesday calls: UK dialin: 0845 245 0224 US dialin: 1 866 838 0046 Intl dialin: +44 1452 542309 Passcode: 369786 Best wishes, Dave Berry Research Manager, National e-Science Centre 15 South College St., Edinburgh Tel: +44 131 6514039

1 0

schedule change
by Osamu Tatebe 04 Oct '05

04 Oct '05

Hi all, We would like to swap session topics #1 and #2. Now, * GFS-WG #2 (GFS-ARCH) Tuesday, October 4 2:00 pm - 3:30 pm Location: White Hill Grid File System Use Case (45 min) Grid File System Diagram (45 min) * GFS-WG #1 (RNS/VFDS) Wednesday, October 5 6:00 pm - 7:30 pm Location: Whitter RNS specification (30 min) VFDS specification (60 min) Sorry for this, Osamu >> On Tue, 27 Sep 2005 10:56:24 +0900 (JST), Osamu Tatebe <o.tatebe(a)aist.go.jp> said: >> Hi all, >> We will have two sessions at GGF15; one is for RNS/VFDS discussion, >> and the other is for architecture of Grid file system. >> * GFS-WG #1 (RNS/VFDS) >> Tuesday, October 4 2:00 pm - 3:30 pm Location: White Hill >> RNS specification (30 min) >> VFDS specification (60 min) >> * GFS-WG #2 (GFS-ARCH) >> Wednesday, October 5 6:00 pm - 7:30 pm Location: Whitter >> Grid File System Use Case (45 min) >> Grid File System Diagram (45 min) >> Regarding RNS specification, the real final spec that reflects >> comments at August OGSA-WG F2F meeting will be available soon. >> Regarding Architecture session, we will start detailed and concrete >> discussion. At GGF12, we discussed the service diagram, see the >> picture of the blackboard; >> https://forge.gridforum.org/projects/gfs-wg/document/GFS_Architecture_Black… >> The goal is to identify >> * necessary services and their functionality, and >> * interaction among these services >> to provide Grid file system functionality (or file system use case). >> For this purpose, we would like to invite speakers to share current >> solutions. Main points are what kind of services are needed and how >> these services are related each other to provide Grid file system >> functionality. Please respond to this email if you can give a talk. >> EGEE, distributed storage tank, and other efforts are quite welcome. >> Thanks, >> Osamu >> --- >> Osamu Tatebe, Ph.D. Tel: +81-29-861-5844 FAX: +81-29-862-6601 >> Grid Technology Research Center, >> National Institute of Advanced Industrial Science and Technology (AIST) >> AIST Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 3058568 JAPAN >> E-mail: o.tatebe(a)aist.go.jp http://phase.hpcc.jp/people/tatebe/

1 0

Stepping back: the high-level view
by Dave Berry 03 Oct '05

03 Oct '05

All, I've been tasked with rewriting the sections of the main OGSA document that deal with data services. This is providing interesting as a check on the work we've been doing on the data architecture document itself. The good news is that much of the existing description in the OGSA document remains accurate. This suggests we did a good job early on. The OGSA document does describe some higher-level features that we haven't developed sufficiently in the data architecture document. I think the most important is the specification of policy, QoS, etc. We've always been aware of this during our discussions but we haven't really developed it, at least not across the whole architecture. I think we need to make this the next cross-cutting issue that we address (analogous to the way we have addressed transactions and security). I also think we need to raise this with the OGSA WG, as we will need to use the same mechanisms as the rest of OGSA. (It may be that we force the pace of this, as I suspect QoS is more important in data services). We also need to think about how to provide performance information, e.g. latency, bandwicth, network location, etc. This is clearly linked to the QoS question. Another cross-cutting concern is graceful degradation of availability. The OGSA document says, "The OGSA data services provide features for graceful degradation in the event of network or other failures. For example, query services may be configured to return partial results when only a subset of sources is available". I suspect we need to address this as another cross-cutting feature. Another major feature from the OGSA document that we haven't addressed is that of transformations and derived data services. I think we took an explicit decision to leave these to a later version of the architecture - does this match your memories? If we agree this is the case, I'll discuss with the OGSA WG how best to reflect this in the OGSA document. The OGSA document also lists provenance as a particular feature that data services support. Provenance is clearly important but I doubt we will address it explicitly in the current version of the data architecture. We made need to address it in the longer term. It seems to me that provenance should be just another form of metadata - but I do know of projects that have developed particular architectures for storing provenance data (e.g. the PASOA project in the UK). Best wishes, Dave Berry Research Manager, National e-Science Centre 15 South College St., Edinburgh Tel: +44 131 6514039

1 0

Minutes & list of GGF sessions
by Dave Berry 28 Sep '05

28 Sep '05

Dear All, Minutes for today's telcon can be found at http://tinyurl.com/72v6w. Here is a reminder of OGSA-D related sessions at GGF15: Monday 2pm: Data Movement Interface Standardisation BOF. Monday 6pm: Data Streams ad-hoc BOF Tuesday 10.45am: Data architecture scenarios (45 mins) Wednesday 2pm: Data architecture: security and transfer (60 mins/30 mins) And the session at the OGSA F2F: Friday 10.30am: OGSA-D session (focus on names for data) Dave.

1 0

Ad-hoc BOF on Streams at GGF15
by Dave Berry 23 Sep '05

23 Sep '05

All, I apologise for cross-posting so widely; we are trying to reach people who are work with data streams in a Grid setting and there is no obvious single place to post. We are organising an ad-hoc BOF at GGF15 on the subject of data streams in Grid architectures. Our aim is to bring together people working on various aspects of data streams. Our personal goals are to determine how best to fit streams into the OGSA Data architecture but we also see this session as an opportunity for people to share experiences and meet others working in overlapping areas. The BOF will be from 6.00-7.30pm on Monday Oct. 3rd. The first half of the session will see presenters addressing some of the questions below. We will keep the second half of the session for discussion. Issues to consider: - What do we mean by a "data stream"? - Grids have to consider attributes such as security, persistence, and quality of service. Do these raise any particular questions for streams? - What are the important use cases? Are there unusual or non-obvious cases that we need to consider? - What is the natural programming model for streams in a distributed environment? - How do we connect one stream to another? - How do streams fit into a service-oriented architecture? - The OGSA Data Architecture currently has sections on Provisioning, Querying, Replication, Caching, etc. Do these apply to streams as well as static data? Are there any particular aspects of streams that affect any of these sections? We look forward to seeing you there. Dave Berry Research Manager, National e-Science Centre 15 South College St., Edinburgh Tel: +44 131 6514039

1 0

Telcon agenda, 28th Sep: streams, security, transfer & use cases
by Dave Berry 23 Sep '05

23 Sep '05

All, The next teleconference of the OGSA Data Architecture working group will be held on Wednesday, September 28, 2005 at the usual time of 8:00am PDT / 11:00am EDT / 4:00pm BST / 5:00pm CEST. As a reminder, here is the teleconference information: UK Dial In: 0845 245 0224 US Dial In: 1 866 838 0046 International Dial In: +44 1452 542309 PIN: 369 786 Agenda for the September 28th teleconference: 1) Early discussion (5 minutes) Note taker assignment Roll call Minutes approval from September 14th teleconference (http://tinyurl.com/d38sz) Agenda bashing 2) Action report (5 minutes) - Dave to ask ADs to approve the streams BOF [Done]. - Dave to mail Joel and Bill to suggest moving the data transfer BOF [Done]. - Dave to propose dates to section authors for their review sessions. - Everyone to talk with your contacts to get more participation in this WG/ - Allen to write an overview section on security. - Section authors to add security notes to their sections. - Mario to ask an OGSA-DAI staff member to write something on the security issues around OGSA-DAI - Dave to revamp section 3 of the architecture document - Allen/Dave to contact David Martin & Hiro Kishimoto to get reviewers & expert participation (after internal reviews). 3) Planning the GGF15 Data Streams BOF (10 minutes) - Speakers - Review of issues - Invitees 4) Planning the GGF15 Use Cases presentation (10 minutes) - Initial presentation - Invitees? 5) Planning the GGF15 Security session (10 minutes) - Presentations? - Issues 6) Planning the GGF15 High-level transfer discussion (10 minutes) - Initial presentation? - Issues 7) Wrap up (5 minutes) AOB Dave Berry Research Manager, National e-Science Centre 15 South College St., Edinburgh Tel: +44 131 6514039

1 0

RE: [ogsa-d-wg] Draft Charter for Data Movement Interface Standardization WG
by Malcolm Atkinson 20 Sep '05

20 Sep '05

Dave this is a good way of posing the question. One can also relate it to the proposal a couple of years ago by Peter Kunszt for grid data handles as a generic concept for naming data. I think the problem arises when the selected data is an arbitrary subset of the stored data or a derivative of the stored data derived via any of a wide range of languages: Xquery, Xpath, SQL, LDAP, semi-structured QLs, statistical languages, FFT, datacutter, .... These all make sense. It is just hard to understand how to compose them. It is hard to make general rules. The derived data has to be evaluated to some point where it can be moved - a RAM buffer - can the movement be part of the standard and the derivation processes be strictly corraled in some other specs? Malcolm >-----Original Message----- >From: Dave Berry >Sent: 14 September 2005 21:00 >To: Malcolm Atkinson; William E. Allcock; ogsa-d-wg(a)ggf.org; >gsm-wg(a)ggf.org; byte-io-wg(a)ggf.org; Peter Kunszt; James >Casey; Ravi Madduri >Subject: RE: [ogsa-d-wg] Draft Charter for Data Movement >Interface Standardization WG > >Hi Malcolm, > >I'd rather ask, what are the characteristics of a file that >makes these file transfer mechanisms tractable? Then we can >ask, to what extent can we generalise the mechanism? > >For example, if the key characteristics are that a file can >be named and supports random access, then we might generalise >the mechanism to include data in RAM (which would avoid >unnecessary copying to disk). This case would be analogous >to some operating systems which allow entities in RAM to be >addressed as part of the file system. > >Conversely, if the mechanism can handle any named sequence of >bytes, then it could presumably handle streaming data as >well. Or if it requires other operations that are specific >to the location of bytes on a disk (or tape), then the WG >will restrict its attention to those cases. > >I would expect this group to place a strong requirement on >the OGSA WG to provide a naming system that can specify >whatever data sets this WG wants to move. > >Dave. > > >-----Original Message----- >From: owner-ogsa-d-wg(a)ggf.org >[mailto:owner-ogsa-d-wg@ggf.org] On Behalf Of Malcolm Atkinson >Sent: 14 September 2005 17:20 >To: William E. Allcock; ogsa-d-wg(a)ggf.org; gsm-wg(a)ggf.org; >byte-io-wg(a)ggf.org; Peter Kunszt; James Casey; Ravi Madduri >Subject: RE: [ogsa-d-wg] Draft Charter for Data Movement >Interface Standardization WG > > >Hi Bill > >I agree that such a standard interface is needed. >When you look at files I presume you consider files where >ever they are, secondary or tertiary storage at least. > >When you say any dataa, then there is the possibility of >trivial or large amounts of data between RAM, as well as data >from files and databases with an enormous set of >possibilities of the way it may be selected and identified. >Eventually it gets close to Byte-IO, Streams >(BoF) and InfoD etc. > >So I'm agreeing with you that if yu go beyond files then >scope control is difficult. > >Would it be better to do the standardisation of file movement >first and look at other forms of adta movement later? > >Malcolm > > > >-----Original Message----- > >From: owner-ogsa-d-wg(a)ggf.org > >[mailto:owner-ogsa-d-wg@ggf.org] On Behalf Of William E. Allcock > >Sent: 14 September 2005 17:04 > >To: ogsa-d-wg(a)ggf.org; gsm-wg(a)ggf.org; byte-io-wg(a)ggf.org; > >Peter Kunszt; James Casey; Ravi Madduri > >Subject: [ogsa-d-wg] Draft Charter for Data Movement > >Interface Standardization WG > > > >Sorry for the re-send, I typoed the byte-io mail list. > > > >All, > > > >Sorry for the SPAM, but I sent this to the "likely >suspects" who might >be interested. I have a proposed BOF >(waiting for AD approval) to >discuss standardizing an >interface for invoking data movement. There >are several of >them out there already. CERN has the File Transfer >System >(FTS), the gsm-wg has SRM copy, Globus has the Reliable File >>Transfer (RFT) service, etc.. I don't think there will be > >any argument > >that there is a need for such standardization, the hard >part will be >scoping the extent of what we will work on. >For instance, all the >examples above are file based, but >ideally, this interface would work >for any data that can be >addressed. > >I expect that that the BOF will be centered >around scoping the working >group, but I think we should >(and approval of the BOF depends on) >getting some initial >discussion around the scope. So... here it goes: > >I >think the obvious thing is that it needs to be able to have > >the basic > >functionality presented by FTS, RFT, and SRM-copy, however > >the devil is > >in the details, so I will break this up into "blocks of > >functionality": > > > >Lets start with naming. What will this service accept as >valid names >for entities that it will move? URLs? EPRs? >Will logical > >file names be > >accepted or should they be translated outside this service? > > > >Related to the naming is what type of data will this >service move? >Files? video streams? the output of >simulations? the output > >of database > >queries? Can we make this a service that any service that > >wants to move > >data can simply invoke it? Note that I am differentiating >data from >messages. You would not use this to send the >result from a > >service that > >summed a bunch of numbers, that would simply be a SOAP > >response... IMHO :-). > > > >Can we make a generic module that would allow this >functionality to be >applied to any service that exposes the >byte-io interface? Does that >affect the interface or is it >just an implementation issue? > >Can we make this service >transport mechanism agnostic? both > >application > >transport (GridFTP vs HTTP vs ...) as well as network > >transport (TCP vs > >UDP vs UDT vs ...). My concern here is that I am not sure > >SOAP has the > >functionality we need. To do this, I wonder if we need the >equivalent >of a union in C, so that the parameters >specified are based on the > >transport(s) chosen. For instance, if you use TCP you need > >to specify a > >buffer size, but not for UDP. GridFTP specifies streams >and data >channel authentication, but HTTP does not. > >>What about security / authorization. This is a broad >category and we >should push as much as possible outside of >scope via callouts > >and Policy > >Enforcement Points (PEPs), but what about delivery guarantees > >such as AT > >MOST ONCE, AT LEAST ONCE, EXACTLY ONCE, non-repudiation, >etc.? I know >Dieter has a set of use cases that require >some of this type delivery >guarantee functionality. > >A >potentially contentious issue is whether or not these >services will >use WSRF and notifications to expose (push >from the service) > >or methods > >to query the state (pull from the service). Hopefully, we >can find a >way to make each optional. > >If we start >making many optional parts to the interface, it will make >>what is exposed as service metadata for brokering will >become more >important. I would propose that we should make >at a minimum a >recommendation for what facts about the >service should be exposed. > >All of the existing services >accept "bulk" inputs, i.e., move > >these 100 > >files. This can be a problem when the requests become very > >large due to > >de-serialization. Should we provide a "chunking" interface >so that >requests can be of unlimited size? > >Please feel >free to make comments on the above and more importantly >>suggest other important issues we need to address. > >btw, >once we have a mail list of our own we will quit > >spamming the other > >lists :-). > > > >Bill > >-- > >William E. Allcock > >Argonne National Laboratory > >Bldg 221, Office C-115A > >9700 South Cass Ave > >Argonne, IL 60439-4844 > >Office Phone: +1-630-252-7573 > >Office Fax: +1-630-252-1997 > >Cell Phone: +1-630-854-2842 > > > > > > > > > >

3 3