June 2009 - ogsa-dmi-wg - lists.ogf.org

Updated Minutes
by Mario Antonioletti 25 Jun '09

25 Jun '09

Hi, Peter has passed on some addendums and corrections to my minutes. Have put a new version up on Grid Forge: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.ogsa-dmi-… Thanks Peter, Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+ -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

DMI Telcon Notes
by Mario Antonioletti 22 Jun '09

22 Jun '09

Hi, Ok, here are my notes from today's call. Please feel free to supplement/correct. Not a great idea to chair and take notes at the same time when more than 2 or 3 people ... at least that is my excuse, Mario --- DMI Telcon 22nd June 2009 ========================= Present: Mario Antonioletti, EPCC David Meredith, STFC Shahbaz Memon, FZJ Gerson Galang, ARCS Steven Crouch, OMII-UK Allen Luniewski, IBM Alex Arana, Intersect Peter Turner, University of Sidney Agenda: - Introductions - Agenda bashing - DTS progress update - Relationship of progress to JSDL and DMI specs - Consequential anticipated extension requests for JSDL and DMI - Planning ---- Introduction ------------ Mario Antonioletti is one of the chairs of OGSA-DMI. Took the position after Michel Drescher had to step down. Keen to see adoption of DMI to take place. Dave Meredith STFC Daresbury Laboratory - interested in DMI from the Data Minx project and NGS perspectives. Gerson Galang is a developer based in Melbourne, working in Data-MINX, interested in DMI and JSDL and has been looking at the schemas. Steve Crouch, from OMII-UK, pursuing this work from the JSDL route. Have been involved with Peter, Gerson and David for a while now. Allen Luniewski, has been with OGF for many years now. One of the instigators for this work going on within OGF. Co-chaired the data architecture working group that motivated DMI and has been involved in this ever since. Alex Arana is looking at this work from the Data-MINX perspective - has been looking at the specs. Peter Turner from the University of Sidney, is a Crystallographer - but now providing data management and access for open neutron source and the Australian Synchrotron. Shahbaz Memon is from FZJ and is a co-chair of the DMI working group. He has been implementing a WSRF implementation of DMI. DTS Progress Update ------------------- Alex - was in the UK in late April. Attended the OMII-UK Collaborations Workshop - got a design for a heavy lifting data moving that will become part of the data-MINX project (http://tinyurl.com/ppt6gr) Looked at various technologies. Came up with the draft version of the current schema from a JSDL spec. Doing the bits of the architecture. Have defined 3 major components: o client - responsible for sending requests to a DTS web service o DTS web service for managing data transfers, use JMS o DTS worker node(s) - receive and process messages. Have implemented a "thin red line" - an end-to-end implementation that satisfies the most basic use case. Thus have a 3 tier architecture: DTS client ---> DTS WS ---> DTS worker node Have JMS queues and topics for transmitting/submitting job requests. DTS worker node are at the end - each does a unique job, DTS worker node cloud is responsible for picking up messages. When one arrives the DTS worker node works on this. Fulfills request. Provisions for management of requests. JMS good for this as it scales - can move data between different types of protocol - SRB, ftp, etc ... add worker to process messages. Thus JMS - allows for horizontal scaling. Lots of flexibility in how you confer the jobs through JMS. Protocol between client and server is SOAP. Interested in a document/literal approach. Want to be able to express multiple data transfers within one requests and manage these from one service. David Meredith has put in a proposal to use a wrapped data transfer request that could encapsulate multiple pairwise sources and sinks as a starting position to be able to do this: http://www.ogf.org/pipermail/ogsa-dmi-wg/2009-June/000488.html At the moment DMI cannot handle multiple transfers from one service instance. This would be an extension to the existing functionality but probably needs some refinement. There was some question as to the nature of these. If one transaction fails will the whole data transfer fail? Is it best effort? How do you cope with the lack of transactional support for file operations in most OSs? There are a large number of questions that need to be addressed. The other proposal within the mail message to add extensibility points to various elements seemed acceptable. Mario was in favour of trying to modify the spec accordingly if it would increase uptake/adoption. Need to think carefully about what the implication of any changes will be though and see what the processes are. Planning -------- Mario will try to contact with the OGF editor to see how easy it will be to put changes through. Allen will check to see whether a number of distinct directories can be transferred from one source using a single request to a DTF (and thus managed by a single DTI). All to look at and comment on David's proposal and see how/whether they can be improved. Date of next call: 8am UK time (GMT+1), 13 July 2009 +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+ -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Moving DMI stuff forward
by Meredith, DJ (David) 22 Jun '09

22 Jun '09

Hi folks, (apologies if you have already received this, I seem to be having trouble posting to the list). A bit more material in preparation for Mondays phone meeting, items 1) and 2). 1) Catering for moving of multiple source/sinks within a single atomic operation. Basically, an important requirement for us is to define a bulk data transfer that could accommodate the transfer of multiple files/dirs within a single atomic operation. We realize that the way to do this in the DMI world is probably to instantiate a number of DTI instances, where each instance caters for a single source DEPR and an associated sink DEPR. However, ideally we would like to be able to define multiple transfers in a single atomic request, and importantly, be able to define this within a single xml doc. As Gerson has shown in his previous email, we have also looked at the JSDL HPC file staging spec that facilitates the definition of multiple DataStaging elements (each containing a srcURI and an associated targetURI and much like DMI, embeds required credentials). At present however, neither JSDL File staging or DMI completely satisfy our requirements hence the current JSDL-DMI hybrid ( http://code.google.com/p/dtsproject/source/browse/trunk/dts-jaxb/src/mai n/resources/minx-dts.xsd). Of course, ideally we would prefer to adopt a spec. We were wondering if something like the following would be considered 'in-scope' / suitable for DMI ? (please see the schema and sample document in the attached examples.zip). The attached schema only adds the following extra element and complex type to the dmi-messages.xsd (below). In doing this, it becomes possible to wrap multiple source-and-sink DEPR combinations within the same doc. Thus, rather than define a single source and sink such as: GetDataTransferInstance( [source DEPR], [sink DEPR], [transfer requirements] ); Maybe define: GetDataTransferInstance( [GetWrappedDataTransferInstanceRequestMessage] );  <element name="GetWrappedDataTransferInstanceRequestMessage" type="dmi-plain:GetWrappedDataTransferInstanceRequestType"/> <complexType name="GetWrappedDataTransferInstanceRequestType"> <sequence> <element name="WrappedSourceSinkDEPR" type="dmi-plain:GetDataTransferInstanceRequestType" minOccurs="1" maxOccurs="unbounded"/> </sequence> </complexType> 2) xsd:any extension points Would the working group also consider the addition of some xsd:any extension points within the dmi.xsd ? (e.g. within the DataType complex type). This is necessary so that we can define extra connection properties such as the MCATZone and MdasDomain for SRB /iRODS. This is also shown in the attached example. I think there a some other items but I forget. Cheers, Dave From: Gerson Galang [mailto:gerson.galang@arcs.org.au] Sent: 19 June 2009 13:11 To: Mario Antonioletti; OGSA-DMI Cc: Peter Turner; Stephen Crouch; Meredith, DJ (David); alexa(a)intersect.org.au; Mohammad Shahbaz Memon; Ravi Madduri; Allen Luniewski; Carlos Aya; Christopher Mendes; Paul Coddington Subject: Re: Moving DMI stuff forward Hi guys, Just thought of updating you on where we are at in terms of the DTS development and schema specification. At the moment, we've finished implementing the modules for the DTS and are starting to integrate bits we implemented so we could demonstrate a very basic type of transfer (ie ftp to smb). We've also started planning on how we could complicate stuff by integrating Shibb authentication. So here's the stuff that you might be interested in.. the approach we've taken is take bits and pieces off JSDL and OGSA-DMI. See the current schema we have.. http://code.google.com/p/dtsproject/source/browse/trunk/dts-jaxb/src/mai n/resources/minx-dts.xsd You probably might be wondering why we decided to replicate stuff from JSDL and OGSA-DMI instead of just importing from them. The reason for it is, we just wanted something quick and dirty to play with. The 'quick and dirty' start will then allow us to properly identify extension requirements for both JSDL and DMI and ultimately for implementing with respect to either JSDL and DMI. The plan is to import from standard schemata and use them instead of reinventing the wheel. At the moment, not all we needed are implemented by the schemata so we've gone for the approach of just using standard when it has the stuff we need. The stuff we're working on is a prototype only and modifying it to use the standard won't be that hard to do. We quite like the JSDL way of specifying the job definition so we've taken the JSDL schema and wrapped it with web service operations that use bits and pieces from DMI. We also use the DMI standards for the messages that will be exchanged by the WS and its client. Regards, Gerson On Thu, Jun 18, 2009 at 9:44 PM, Mario Antonioletti < mario(a)epcc.ed.ac.uk> wrote: Hi, Hi Mario, think 8am GMT Monday June 22 would be 5pm AEST Monday June 22. This seems generous on your part, we could certainly accommodate a later start - if we say 8:30am GMT you get a little more sleep at little cost to us :-) I'm usually in well before 8 so let's leave it at that. My UK colleagues saidthey could make it.... Could also do later. I've added little to the briefing doc since we last communicated - had no time. Happy to provide it as it currently is - does not include anything of the work undertaken since the UK visit. Gerson and Alex have been making progress. I'll ask then how best to provide access to their stuff. Could you guys come up with an agenda as to what needs to be discussed and I'll email the list - hopefully today or tomorrow. If the briefing doc is fit to show to the public then we could just distribute that. Let me know, Mario +----------------------------------------------------------------------- + |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ <http://www.epcc.ed.ac.uk/%7Emario/> | +----------------------------------------------------------------------- + -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- Scanned by iCritical.

2 1

DMI Telcon
by Mario Antonioletti 18 Jun '09

18 Jun '09

Hi, Time we had a telcon. We've had interest in DMI from Australia so will try to interface between europe and Australia this time. If there is interest from the Americas can try to have a second call or find someway of accommodating everyone or at least sharing the pain. Time: 8am UK time (GMT+1), Monday June 22 (please work out your own time zone equivalent as I'll get it wrong). Let me know if you would like to be involved but can't make the suggested time. Suggested agenda: - Introductions - Agenda bashing - DTS progress update - Relationship of progress to JSDL and DMI specs - Consequential anticipated extension requests for JSDL and DMI - Planning Please let me know if anything else should be added to the list. Call details: Participant access code: 23057332 then # Primary dial-in number: 0870 2407821 or if dialling internationally +44 2078193600 Or local access alternatives: USA West Coast 4089616553 USA East Coast 7183541169 Germany (0)6950070811 Finland (0)969379595 Australia 0282239343 If you would like a local dial number let me know and I'll see if one can be provided. Let me know if there are any questions, Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+ -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 1