January 2006 - ogsa-d-wg

Notes from ByteIO F2F 20060119 Sunnyvale
by neil p chue hong 19 Jan '06

19 Jan '06

The presentations and minutes are available on GridForge: http://forge.gridforum.org/docman2/ViewCategory.php?group_id=154&category_id =1210&filtertype=basic ByteIO F2F Minutes 19/1/06 - Fujitsu, Sunnyvale ------------------ Mark Morgan Neil Chue Hong Andrew Grimshaw Steve McGough Dave Berry Fred Brisard? Darren Pulsipher Andreas Savva Dave Snelling Hiro Kishimoto Allen Lunieski Started session by screening www.405themovie.com - played using ByteIO (DIME/FTP) to access the file. Cool! Dave: Does the interface simplifies if you remove strided read? - Yes Andreas: Does long mean 64 bits? - Yes Hiro: Why is the return type of seekWrite void? What if there is an error? - An error would throw a fault ?: Why is seekOrigin a URI? - Essentially have defined three qnames as URIs, rather than a URI as a generic locator. Could do as an enum, but noone has complained with this approach. Hiro: what happens if someone seekWrites backwards. - This can fail, if the stream does not allow it, but may work if the stream supports it. Andreas: Do you write zeroes if seekWriting backwards? - You pass over the data unchanged, if going past end of stream. Is implemented similar to UNIX, i.e. bytes to pad are undefined. - Does the document specify the semantics for this clearly? Check this. Currently in GFSG review. Will revise the docs as go through PC comments. DEMO: demonstrated FTP interface using ByteIO (easy to get Windows to interface with it) and a set of command line utilities. EPCC addresses uses lowlevel Java functions to a particular URL but will write a different address in the <To> field which means the MS client cannot use this. Looking at <To> field and using it as a despatcher. Service is at http://foobar/blah/byteio <EPR> <To>http://byteio/?/ResourceName</To> </EPR> There is no information in the To field to point it to the service. We'll look at this and try for interop at GGF16. GFSG is working on a template for interop documents. We should work with them (Steve Pickles?) Does GFSG have a template for experience documents - what should they contain? Dave B: you're using URIs to define transfer mechanisms. Does OGSA as a whole have a policy on the use of URIs? e.g. what happens if we use URIs to identify transfer messages elsewhere? Dave S: If you are using ByteIO as a mechanism, then you should use the ByteIO one. If not, there is no problem with having a plethora of mechanism definitions. Dave B: recommend OGSA create a central list of mechanisms Neil: just now, think it's better to just let groups own the lists, when other groups need them refer back to OGSA. Only problem is when two groups overlap. This needs coordination. Dave B: e.g. DMIS should refer to byteio mechanisms and add a GridFTP transfer mechanism. Andreas: which version are you using of e.g. MTOM, DIME. We need to define this properly in the document, and also support this in the URI defining the transfer mechanism. Final words: - Give comments on the spec - next is interop between the three implementations - what do we need to write in an interop and experiences document? -- Neil P Chue Hong | T: [+44] (0)131 650 5957 Project Manager, EPCC | F: [+44] (0)131 650 6555 Rm 2409, JCMB, Mayfield Rd. | E: N.ChueHong(a)epcc.ed.ac.uk Edinburgh, EH9 3JZ, UK | W: http://www.epcc.ed.ac.uk BT MeetMe: http://tinyurl.com/8mwhd - Code: 14712935# "A film is like a battleground. It's love, hate, action, violence, death - in a word, emotion." - Sam Fuller

1 0

FW: HPDC Workshop on Next-Generation Distributed Data Management
by Dave Berry 15 Jan '06

15 Jan '06

FYI. _____ From: Ann Chervenak [mailto:annc@ISI.EDU] Sent: 13 January 2006 23:24 To: undisclosed-recipients Subject: CFP: HPDC Workshop on Next-Generation Distributed Data Management HPDC Workshop on Next-Generation Distributed Data Management http://www.isi.edu/~annc/distributedDataWorkshop.html <http://www.isi.edu/%7Eannc/distributedDataWorkshop.html> June 20, 2006 Paris, France Distributed data management is a challenging problem in high performance computing environments, and the difficulty of managing scientific and commercial data sets will increase as they scale to terabytes and petabytes in the coming years. In conjunction with HPDC, we will hold a one-day workshop that will gather together researchers and industry participants to discuss future trends for distributed data management in HPDC environments. We invite short papers (maximum 6 pages) of two types: position papers that discuss the authors' vision for trends or developments in data management in the three to five year timeframe or papers that describe innovative algorithms, tools or services for distributed data management. Topics of interest include, but are not limited to: * Peer-to-peer approaches to data discovery and data storage * Distributed Metadata management * Services for generating and maintaining provenance information * Techniques for replica management, including consistency of replicas * Algorithms for data placement, prefetching, staging, proactive replication * Federation of diverse databases, data mediation services * Distributed query processing * Virtualization of data resources * Interaction of data management with execution environments, including workflow management services, virtual machines * Validation of correctness of distributed data * Publish/subscribe systems * Advanced data transfer techniques * Namespace management, globally unique naming * Support for streaming data Important dates: * Position papers due: February 28, 2006 (Tuesday) * Program committee discussion: March 23, 2006 (Thursday) * Notification of acceptance of papers: March 27, 2006 (Monday) * Final papers due: April 14, 2006 (Friday) Submission: We will be using the HPDC conference submission site. Submission instructions to follow on workshop web site. Workshop organizers/co-chairs: Ann Chervenak, USC Information Sciences Institute (annc(a)isi.edu) Dave Berry, National UK e-Science Center (daveb(a)nesc.ac.uk) Neil P. Chue Hong, EPCC, University of Edinburgh (N.ChueHong(a)epcc.ed.ac.uk) Program committee: Gagan Agrawal, Ohio State Bill Allcock, Argonne National Laboratory Chaitan Baru, SDSC Karan Bhatia, SDSC Dieter Gawlick, Oracle Pilar Herrero, UPM Tahsin Kurc, Ohio State Allen Luniewski, IBM Andre Merzky, Zuse Institute Berlin Reagan Moore, SDSC Beth Plale, Indiana University Osamu Tatebe, AIST Kurt Stockinger, Lawrence Berkeley National Laboratory Stratis Viglas, U. Edinburgh

1 0

FW: [dais-wg] Request for Comments and Volunteers. (fwd)
by Dave Berry 13 Jan '06

13 Jan '06

Reagan Moore has given us a list of all the SRB access commands. These will be a useful check as to whether our architecture provides all the necessary functionality. (Although I expect these commands to map to various different parts of our architecture, not just our data access section). Dave. -----Original Message----- From: Mario Antonioletti [mailto:mario@epcc.ed.ac.uk] Sent: 12 January 2006 07:41 To: Dave Berry; Allen Luniewski Subject: Re: [dais-wg] Request for Comments and Volunteers. (fwd) Of course it helps if you guys get a copy of what Regan has added. +----------------------------------------------------------------------- + |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ || +----------------------------------------------------------------------- + ---------- Forwarded message ---------- Date: Wed, 11 Jan 2006 14:36:02 -0800 From: Reagan Moore <moore(a)sdsc.edu> To: Mario Antonioletti <mario(a)epcc.ed.ac.uk> Subject: Re: [dais-wg] Request for Comments and Volunteers. Mario: I added to the document the set of data access commands (74 in total) used by the Storage Resource Broker data grid to manage distributed shared collections. These commands go beyond simple data access, and include commands needed to minimize network latencies, manage distributed data, and federate independent data grids. The desire is to broaden the goals of the proposed data access mechanisms to make them relevant to a large number of established international data grids. A standard interface that is not able to support the operations provided by existing data grids will artificially restrict itself. Note that the functionality of the commands that are listed is required to support a diverse set of applications ranging from data grids to digital libraries to preservation environments. Reagan Moore > Hi, > Please find enclosed an updated (but not yet finished) version of a > Data > Access setion that is being prepared as a contribution to the OGSA Data > Archtecture document. The document is still a bit rough so my apologies for > that. I would like people who might want to help write > this section (and contribute to the OGSA Data document in general). > In particular: > > - I need someone to put some proper content in the ByteIO section. > - Folks to review and put forward their requirements/comments. > - DAIS folks to read the DAIS part and make sure it makes sense. > > This is a work in progress. Thanks in advance for any input you may > have or any bits that you may wish to contribute. > > Mario > > +----------------------------------------------------------------------- + > |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | > |Tel:0131 650 > |5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | > +----------------------------------------------------------------------- + > > Content-Type: APPLICATION/msword; name=DataAccess2.doc > Content-ID: <Pine.SOC.4.63.0601111855110.27083(a)garnet.epcc.ed.ac.uk> > Content-Description: > Content-Disposition: attachment; filename=DataAccess2.doc > > Attachment converted: Macintosh HD:DataAccess2.doc (WDBN/<IC>) > (0040EF21)

1 0

RE: [ogsa-d-wg] Telcon details for Monday's F2F - with passcode
by Dave Berry 13 Jan '06

13 Jan '06

I forgot to include the passcode. USA: 1 866 838 0046 UK: 0845 245 0224 International: +44 1452 542309 Passcode: 275359 The meeting is scheduled for 9-12.30 and 1.30-5.00, US Pacific time. Dave.

1 0

Telcon details for Monday's F2F
by Dave Berry 13 Jan '06

13 Jan '06

All, We have arranged a phone bridge for Monday's OGSA Data F2F. To join, please call the following telcon number: USA: 1 866 838 0046 UK: 0845 245 0224 International: +44 1452 542309 The meeting is scheduled for 9-12.30 and 1.30-5.00, US Pacific time. Dave Berry Research Manager, National e-Science Centre 15 South College St., Edinburgh Tel: +44 131 6514039

1 0

Request for Comments and Volunteers.
by Mario Antonioletti 11 Jan '06

11 Jan '06

Hi, Please find enclosed an updated (but not yet finished) version of a Data Access setion that is being prepared as a contribution to the OGSA Data Archtecture document. The document is still a bit rough so my apologies for that. I would like people who might want to help write this section (and contribute to the OGSA Data document in general). In particular: - I need someone to put some proper content in the ByteIO section. - Folks to review and put forward their requirements/comments. - DAIS folks to read the DAIS part and make sure it makes sense. This is a work in progress. Thanks in advance for any input you may have or any bits that you may wish to contribute. Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+

1 0

RE: [ogsa-d-wg] Format for interface descriptions in OGSA-D
by Stephen Davey 09 Jan '06

09 Jan '06

Hi Michael, I prefer your version for the format for interface descriptions. Though it is perhaps not ideal to include the actual operation and property names, it does make the interfaces much more readable. Adding the table lines will also make them stand out more, which will help me when I try to match up the scenario steps with appropriate interfaces - whether we want to emphasize the interfaces in the data architecture document is another matter I guess. Cheers, Stephen. -----Original Message----- From: owner-ogsa-d-wg(a)ggf.org [mailto:owner-ogsa-d-wg@ggf.org] On Behalf Of Michael Behrens Sent: 23 December 2005 02:08 To: Dave Berry Cc: ogsa-d-wg(a)ggf.org Subject: Re: [ogsa-d-wg] Format for interface descriptions in OGSA-D I attempted to rearange the text a bit and added lines, thinking that might be easier for the reader when confronting that pattern? It's attached - not sure if you'll like it. Its a little more UML-ish. BTW: The strawman architecture (v601) is looking good - covers a lot of ground! Good concepts for global access to information via caching, naming, etc. Dave Berry wrote: >Folks, > >On yesterday's telcon we discussed how to describe interfaces in the >OGSA Data Architecture document. I undertook to draft a straw man >description of the ByteIO interface as an example. I attach this draft >for comment. (Note: it is based on the ByteIO interface described in >our document, which is significantly out of date. What we're interested >in here is the format, not the content). > >On the telcon we agreed that we wanted to abstract away the details of >the operations such as their names and the names and types of their >parameters. This is so that (a) we don't swamp our readers with detail, >and (b) we don't have to keep changing our document to track changes in >the detailed specification. On the other hand, we do want to describe >the key operations, so that (a) readers can better understand what the >interface does, and (b) we show how the interface will interact with >other parts of the system. > >I'm not wedded to the format in the attachment. It is purely intended >as a starting point for discussion. > >Best wishes, > >Dave. > > -- Michael Behrens R2AD, LLC (571) 594-3008 (cell) *new* (703) 714-0442 (land)

1 0

IVOA work on Grid storage and personal data management
by Dave Berry 09 Jan '06

09 Jan '06

Dear All, The International Virtual Observatory Alliance is a group of astronomers who are working on shared Grid software for astronomy. I asked them about their work on managing data in a distributed environment and they pointed me at the following document: http://www.ivoa.net/internal/IVOA/IvoaGridAndWebServices/vostore-0.18.pd f. At a quick glance, this seems to describe requirements for one interface to abstract storage provision and another to abstract data distribution (such as grid file systems). I suggest that this is worth a look to see what overlap, if any, exists with the work of the OGSA-D and GFS WGs (and possibly also GSM?). The IVOA are actively working on this, so the document referenced above will be superseded in due course. Best wishes, Dave Berry Research Manager, National e-Science Centre 15 South College St., Edinburgh Tel: +44 131 6514039

1 0

OGSA-D telcon agenda for January 11th - formats and structure
by Dave Berry 09 Jan '06

09 Jan '06

Dear All, Please let me know in advance if you plan to attend this week's telcon. For those of you calling in, please note the new passcode. The next teleconference of the OGSA Data Architecture working group will be held on Wednesday, January 11th, 2006 at the usual time of 8:00am Pacific / 11:00am Eastern / 4:00pm UK / 5:00pm Europe. In this telcon we will discuss plans for the F2F & GGF, what format to use for describing interfaces, and how to link the scenarios and architecture documents. Here is the updated teleconference information: UK dialin: 0845 245 0224 US dialin: 1 866 838 0046 Intl dialin: +44 1452 542309 NEW Passcode: 275359 Agenda for the January 11th teleconference: 1) Early discussion (5 minutes) Note taker assignment Roll call Minutes Approval (http://tinyurl.com/ace5k) Agenda bashing 2) Action report (5 minutes) - Allen to work on an integration pass, including a scoping section. - Dave to iterate the description of the ByteIO interface (and let Mario know). - Dave to investigate Skype or phone for OGSA Data session at F2F. - Stephen to investigate whether transient data is supported by RNS. - Allen & Dave to draft message to GFS co-chair & ADs, saying RNS is too complicated. - Ann to check the Access section for suitability for file replication. - Mario to update the Access section and make text available for comment. - Section authors to add security notes to their sections. - Everyone to talk with your contacts to get more participation in this WG. - Allen/Dave to contact David Martin & Hiro Kishimoto to get reviewers & expert participation (after internal reviews). 3) Progess & planning update (10 minutes) Update on document writing, links with other WGs, plans for F2F and telcons up to GGF16. 4) Format for documenting interfaces (15 minutes) See the messages from myself and Michael Behrens on 22nd/23rd Dec (attached) 5) Linking the scenarios and architecture documents (15 minutes) Right now, the scenarios document is fairly loosely tied to the architecture document. Do we want to go into greater detail about how the diagrammatic solutions in the scenarios document tie to specific sections of the architecture document? Do we want to create interaction diagrams that show how and when the architectural components interact to effect a solution? Should the architecture document also refer to the scenarios document? 6) Wrap up (5 minutes) Next meetings: Jan 16th-20th: OGSA F2F in Sunnyvale Jan 25th: Report from F2F; review of documents? Feb 1st: Feb 8th: Feb 13th-16th: GGF16 in Athens Dave Berry Research Manager, National e-Science Centre 15 South College St., Edinburgh Tel: +44 131 6514039

1 0