September 2007 - ogsa-dmi-wg

Fwd: Materials/Description for OGF21 Session: OGSA DMI Specification Discussion
by Ravi Madduri 19 Sep '07

19 Sep '07

FYI ---------- Forwarded message ---------- From: replogle(a)ogf.org <replogle(a)ogf.org> Date: Sep 18, 2007 3:11 PM Subject: Materials/Description for OGF21 Session: OGSA DMI Specification Discussion To: madduri(a)mcs.anl.gov Cc: replogle(a)ogf.org Dear OGF21 Session Organizer: As a session organizer, you are able to upload materials (slides, etc) to the OGF website. Please use the link below to upload any pertinent session materials for the OGF21 session titled: "OGSA DMI Specification Discussion" http://www.ogf.org/gf/event_schedule/manage_session.php?id=1002&key=1d585df9 Please also update your session information (title, description, agenda, etc) so that printed materials (finalized in the next week or so) will reflect most current information: http://www.ogf.org/gf/session_request/editreq.php?id=1002 Please let me know if you have any problems. Thanks, Joel -- -- Ravi Ravi K Madduri The Globus Alliance | Argonne National Laboratory http://www-unix.mcs.anl.gov/~madduri

1 0

WSDL and XML Schema
by Mario Antonioletti 12 Sep '07

12 Sep '07

Hi, Michel has produced a first version of the WSDL and XML Schema of the DMI Spec, see: http://forge.ogf.org/sf/go/doc14760?nav=1 Looking for suggestions and comments and especially for volunteers to try and produce implementations. If you think you might be interested in this then please get in touch. Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+

1 0

Telcon Minutes
by Mario Antonioletti 12 Sep '07

12 Sep '07

DMI Telcon - 12 September ========================= Attendees: Steve Newhouse, Microsoft Mario Antonioletti, EPCC Michel Drescher, Fujitsu Allen Luniewski, IBM Agenda: 1. IPR Notice 2. Previous Minutes and Action Review (no actions from last meeting) 3. Agenda Bashing 4. Review comments on the spec made by Allen, Steven and Michel 5. WSDL and XML Schema Review and Implementation 6. Socialising the Spec 7. AOB +-- 3. Review of comments Allen sent an email with comments on the spec (draft 35). This elicited further comment from Steven and Allen. Most of these were good catches and are being addressed (new version will go up soon after telcon). Only noting salient points: Issue between the created and the scheduled state of the DTI. Steven advocates that a change of state can take place if the user invokes the start operation or if the scheduled start time is reached. If a DTI is created without a scheduled state then it will only migrate to a scheduled state if it is done manually (start operation is invoked). This needs to be clarified in the spec document. The text in the document is wrong/unclear which is causing the confusion. Issue as to whether Activate and Start should be swapped around in terms of their semantics. For section 5.1.1 need to add text to say that the DTI state will migrate only if moved manually if the start operation is used. The start not before time would equate to the scheduled time of the DTI. There is an issue as to how you represent a "start as soon as possible" or "start NOW" time for the service - needs to be addressed off-line. Need to be very clear about the start and created time and what the semantics are. It is not very clear in the document just now. Question about the DEPRs. Defining a PortType that extracts the contents of the metadata section (source and sink have to be able to generate this metadata). How much do we specify this. One option is to leave as is now and see if we get any comments about this when we go to public comment. The URL attribute should probably be renamed. It should be a URI, the name, as it stands, is misleading. This data type for the XML Schema reflects this already. The spec will be updated by Michel and some of the points uncovered by Michel will be discussed at a future telcon (Michel will send out an email to the list with this). 5. WSDL and XML Schema Review and Implementation Michel produced the WSDL and XML Schema currently included in the spec. Steven took the WSDL/XML Schema produced by Michel and run it through Microsoft tooling. No issues thrown up. Michel used eclipse to generate the XML Schema/WSDL. No surprise then that Mario had no problems with eclipse. The HPC Profile looking at supporting a restriction on the JSDL data staging elements. Will require re-engineering of the Microsoft HPCP C# web service implementation - need to check that transfers have been successful - this capability could be implemented using part of DMI. Steven will be thinking about this. Michel will be looking at trying out a DMI implementation later. Can try to look for other volunteers to try and do implementations. Any volunteers out there? When Michel was doing the schema for the the start, stop, activate, etc operations noted that response elements had to be defined in the WSDL. Currently have define empty response messages - without these you can't throw a fault. only the status element currently has content. Probably need to work these back into the spec - they need to be synchronous. Michel will have a look. Others should think about this too. 6. Socialising the Specs Michel has been sending version the spec to people. No comments thus far. Steven will talk to Andrew Grimshaw about the spec (both are currently attending the UK AHM). 7. AOB Michel has some thoughts of the WSRF rendering but will talk about this later. Michel will do the changes to the spec discussed during the telcon. One for Ravi: as DMI chair we need your feedback on the current version of the spec. Need to discuss planning for OGF - produce a slide set. Michel noted things to discuss might be the position of DMI within the OGF landscape - how it relates and is distinct from gridFTP, ByteIO, etc. Possibly have some hypothetical client pseudo code to show easy it would be to use. +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+

2 1

[Fwd: Re: Latest DMI Spec.]
by Michel Drescher 12 Sep '07

12 Sep '07

-- REPOST -- Sorry for the spam, but earlier today, the OGF mailing list manager choked on mails sent to OGSA-DMI. Here's a reply to a mail from Allen regarding the DataEPR element we define in OGSA-DMI. Cheers, Michel -------- Original Message -------- Subject: Re: Latest DMI Spec. Date: Wed, 12 Sep 2007 11:38:23 +0100 From: Michel Drescher <Michel.Drescher(a)uk.fujitsu.com> To: Steven Newhouse <Steven.Newhouse(a)microsoft.com> CC: Allen Luniewski <luniew(a)us.ibm.com>, "mario(a)epcc.ed.ac.uk" <mario(a)epcc.ed.ac.uk>, OGSA-DMI ML <ogsa-dmi-wg(a)ogf.org> References: <OF75CBA36B.23241ED0-ON8825734E.0058DC3A-8825734E.005CB8BA(a)us.ibm.com> <F75A5628125A9C4EA64981CC4D412B94273558D977(a)NA-EXMSG-C102.redmond.corp.microsoft.com> Folks, this is going to be a quite lengthy mail, but I hope I can make that point clear: Steven Newhouse wrote: > Philosophical discussion. > > 4.2.1.2.1 XML Representation. Just an opinion, and not one I will fall on my > sword about, but I have never liked, and still do not like, putting all of this > information into an EPR. I strongly believe that it is simpler, and more > appropriate, to get this information via a call to the source/sink of the transfer > via an architected interface. > > I’d agree this is one of our weak areas. We need to get this information… but we > don’t quite know from where. I would be nervous about defining (and getting > implemented) another port type. For now I think we should keep it weak and abuse the EPR as a mere data container. Later on we can still bang our heads on that other port type (do we really have to define that port type? Or can we trust implementors being intelligent and program an intelligent DMI system that acts as an intelligent client that can programmatically analyze an EPR metadata, find a link to the EPR's live WSDL, analyse that and act appropriately? > Later in this section, @dmi:url. I am rather surprised that this information is > being carried in a URL. Yes, one can put almost anything into a URL > but it would seem more natural to admit that this information is basically > source/sink specific and thus passing it back as an "any" would > seem logical. > > Its more protocol specific in my view. But I see what you mean. If the protocol > was (say) SQL this would be an SQL statement on a particular data base table. The datatype of @dmi:url is already xsd:anyURI so we are safe there as there is no doubt a URI scheme "sql:" that models an SQL query as a URI. IF not, who stops people creating one? But I still think we should rename that attribute to something more accurate, e.g. "dmi:dataDescriptor". > Again later in this section, @dmi:sourceMode. As written this section could be > interpreted to require that a DTI implementation always work in a > synchronous mode with the source/sink. That is, do a push (pull) and then wait > for an acknowledgement. I do not believe that we want to > impose that kind of restriction. Somehow this description should make it clear > that a DTI implementation may have many outstanding > push/pull messages outstanding at any given time. > > In this same area, the description of 3rd party transfer is not very clear. Yes, > I understand what it is but since this is a specification I think > that this document needs to make it quite clear what is intended. > > Michel – You led the definition of this section… care to comment? Okay, here's the lengthy part of this specification: Remember, a DataEPR without changing can be used as either a "Source DataEPR" or a "Sink DataEPR", but it SHOULD NOT be used for both in the same data transfer request to the DTF (results are undefined and DMI implementation specific). Hence a DataEPR, for each supported protocol, provides information on how the data described by this EPR should be accessed when used in a particular context (i.e. either as source or as sink). The following boolean table illustrates which combinations are feasible and can be used to initiate a data transfer. Note that this table should be viewed in a fixed font (e.g. Courier). Source DataEPR's || Sink DataEPR's || @dmi:sourceMode value || @dmi:sinkMode value || -------------------------------||--------------------------------|| Data | | pull | 3rd || | | pull | 3rd || Transfer pull | push | and push | party || pull | push | and push | party || possible -----|------|----------|-------||------|------|----------|-------||---------- X | | | || X | | | || - -----|------|----------|-------||------|------|----------|-------||---------- X | | | || | X | | || X -----|------|----------|-------||------|------|----------|-------||---------- X | | | || | | X | || X -----|------|----------|-------||------|------|----------|-------||---------- X | | | || | | | X || - -----|------|----------|-------||------|------|----------|-------||---------- | X | | || X | | | || - -----|------|----------|-------||------|------|----------|-------||---------- | X | | || | X | | || - -----|------|----------|-------||------|------|----------|-------||---------- | X | | || | | X | || - -----|------|----------|-------||------|------|----------|-------||---------- | X | | || | | | X || - -----|------|----------|-------||------|------|----------|-------||---------- | | X | || X | | | || - -----|------|----------|-------||------|------|----------|-------||---------- | | X | || | X | | || X -----|------|----------|-------||------|------|----------|-------||---------- | | X | || | | X | || X -----|------|----------|-------||------|------|----------|-------||---------- | | X | || | | | X || - -----|------|----------|-------||------|------|----------|-------||---------- | | | X || X | | | || - -----|------|----------|-------||------|------|----------|-------||---------- | | | X || | X | | || - -----|------|----------|-------||------|------|----------|-------||---------- | | | X || | | X | || - -----|------|----------|-------||------|------|----------|-------||---------- | | | X || | | | X || X The table should be read like this. "pull" means: The data can be only be pulled from the (source | sink) using the given protocol. "push" means: The data can be only pushed into the (source | sink) using the given protocol. "pull and push" means: The data can either be pulled from or pushed into the (source | sink) using the given protocol. "3rd party" means: The given protocol supports true 3rd party data transfers that do not involve an intermediate to cache the data. The table does not specify protocols as those combinations can occur for any protocol that is supported by the source and/or sink. It is also clear that except for 3rd party data transfer that the DTI always has to cache the data in transit, i.e. the DTI "pull"s the data from the source and "push"es it to the sink. The table does *not* cover the use case where a particular implementation of a data transfer protocol (or an extension to it) enables the DTI to: a) instruct the source to push the data to the sink, or b) instruct the sink to pull the data from the source. Those use cases would require the definition of additional values for the @dmi:sourceMode and @dmi:sinkMode attributes. Those scenarios are possible, but I don't know of any data transfer protocol standard that actually explicitly allows that. This ominous "ParallelHTTP" I was working on is based on HTTP but explicitly allowed entities such as the DTI to instruct a ParallelHTTP-enabled source or sink to perform the described actions. However, the work on this stalled and I don't know whether it is worth mentioning it in OGSA-DMI or even worth pursuing this programming effort at all. Instead I am inclined to add a ByteIO identifier as a very last common resort to the list of identified data transfer standard. I know it's slow, but it's in the OGF track, and there are several implementations available in UNICORE, GenesisII, EPCC's OGSA-DAI effort. The UNICORE implementation also provides Globus integration if I am not wrong. I hope that this makes it a lot clearer what I intend to express with those values. Cheers, Michel

1 0

New DMI spec, draft 36
by Michel Drescher 12 Sep '07

12 Sep '07

Folks, here's draft 36 of the spec, containing the latest changes we agreed on at today's conference call: http://forge.gridforum.org/sf/go/doc14200?nav=1 Cheers, Michel

1 0

DMI Telcon
by Mario Antonioletti 12 Sep '07

12 Sep '07

Hi, A short reminder (apologies for the lateness but there were some issues with the mailing list) that we will be having a DMI telcon today (with the right times this time) - once again thanks to Steven for providing the telcon numbers, Date: 12th September 2007 When: 9am (Pacific) 5pm (UK) PIN: 2719140 USA: +1 866 500 6738 (Toll: +1-203-480-8000) UK: +44 118 909 2000 a plausible agenda for this meeting will be: Agenda: 1. IPR Notice 2. Previous Minutes and Action Review (no actions from last meeting) 3. Agenda Bashing 4. Current Document/WSDL and XML Review 5. Socialising the Spec 6. AOB Can change the above agenda on the call as required. Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+

1 0

New document version 35
by Michel Drescher 11 Sep '07

11 Sep '07

Folks, I am happy to announce a new document version 35 that includes the complete WS-I BP 1.1 compliant WSDL rendering with XML Schema. The WSRF 1.2 compliant rendering (possibly OGSA BP 1.0 compliant, too?) will follow later - I see this fit best in a separate document. Please find the new document version here: http://forge.gridforum.org/sf/go/doc14200?nav=1 Cheers, Michel

1 0

DMI Telcon Minutes
by Mario Antonioletti 05 Sep '07

05 Sep '07

OGSA DMI Telcon - Telcon 5th September ====================================== Attendees: Steve Newhouse, Microsoft Mario Antonioletti, EPCC Michel Drescher, Fujitsu Allen Luniewski, IBM Reviewed changes done to the document: Steven did made changes to the faults, security section and the get attributes operations for the factory and the data transfer instance. Added getFactoryAttributesDocument() operation in section 4.1. Michel thought that an xs:any should be added in case there are additional attributes that could be returned to the client [for the XML representation in section 4.2.1.1]. Lowered the case of the first letter of the start, activate, stop, restart and suspend operations for consistency. Also added a getInstanceAttributesDocument operation. Added failed-{clean,unclean, unknown} failed faults (same as the states. getStatus operation defined in section 5.2.6. The xs:any should be at the top of the complex type, before the defined elements else there are parsing problems. In section 5.3 have two new faults: IncorrectStateFault and FailedStateTransitionFault. Semantics are as in the document. States can change without user intervention - the text in the document currently seems to suggest that all state changes are user driven. Text to be changed to indicate this should be changed. Sections 5.4.1.6-8 has new text on the failed states. Section 6 contains information about the security. Section 12 (Renderings) will have a section for the XML relating to the spec, common XML Schema and WSDL for the DTF and DTI. Additional renderings may be placed in other documents. SN will make some more changes and accept changes. MD then to create the XML Schema and WSDL to be done for next week. Group will then work on a slide set for OGF. Michel and Steven will be at OGF - Mario will try to call in, Allen is not sure yet if he can make it to OGF. Start socialising the spec and trying to get key people and groups to look over it. +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+

2 1

DMI Telcon
by Mario Antonioletti 05 Sep '07

05 Sep '07

Hi, A short reminder that we will be having a DMI telcon - thanks to Steven for providing the telcon numbers, Date: 5th September 2007 When: 8am (Pacific) 4pm (UK) PIN: 2719140 USA: +1 866 500 6738 (Toll: +1-203-480-8000) UK: +44 118 909 2000 a plausible agenda for this meeting will be: Agenda: 1. IPR Notice 2. Previous Minutes and Action Review 3. Agenda Bashing 4. Current Document Review 5. Socialising the Spec 6. AOB Can change the above agenda on the call as required. Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+

2 2

New version uploaded
by Steven Newhouse 05 Sep '07

05 Sep '07

All, New version uploaded - http://forge.ggf.org/sf/docman/do/downloadDocument/projects.ogsa-dmi-wg/doc… Contains the changes discussed in the last call. If there is a call on Wednesday I'd suggest we step through and review/approve the changes. The regular numbers will be valid if there is a call. Steven

1 0