November 2007 - ogsa-dmi-wg

Proposal on fault communication for the DTI
by Michel Drescher 06 Dec '07

06 Dec '07

Folks, please find attached my proposal on how DMI could model the fault communication for the DTI. I deliberately did not fully write the proposal as a finished text for the specification. I first wanted to make clear why I designed the proposal as it is. If we agree to go ahead with whatever solution, we then still can add the solution to the spec, and bang our heads together on the words. :) Cheers, Michel

4 7

Telecon Minutes - Next Meeting 5th December 2007 - 0830 Pacific
by Steven Newhouse 27 Nov '07

27 Nov '07

Attendance: Steven Newhouse Michel Drescher Allen Luniewski Mario Antonioletti Ravi Madduri Agenda 1. One Interface - Three uses * How sufficiently well described are the DEPRs for the three usage scenarios? * Is there an issue around interoperability? Review the current draft to ensure that there are sufficiently well-defined faults to enable the three different scenarios to be handled. Proposal: SJNs to put document from 29th October 2007 into the specification. 2. Selected Protocol in the DEPR Fails To Transfer * Several scenarios to resolve this: + Cleverness in the Server. When a transfer protocol within the DEPR fails, this protocol is removed from the DEPR and the new DEPR is returned to the client. It may then use this new DEPR to start a new protocol selection process by again calling the DMI. + Cleverness in the Client. When a transfer protocol within the DEPR fails this is reported to the client. It may then remove the failed protocol from DPER and start a new protocol selection process by calling the DMI again. + Intermediate Agent or Port Type. Adds complexity. Proposal: When a transfer fails within the DMI it will be reported as a fault. Within the fault will be an optional element(s) containing revised DEPRs which MAY be used by the client in a retry. MD to draft a separate document for discussion. 3. Data Aggregation [Ravi] Separate document to follow for discussion next week. 4. Retry [Ravi] Separate document to follow for discussion next week. Next Meeting: 5th December 2007 - 8.30AM PACIFIC

1 0

Next Telecon - 27th November 2007
by Steven Newhouse 20 Nov '07

20 Nov '07

Call cancelled as unable to make progress as action item owners were not on the call. Same time next week. Date: 27th November 2007 When: 9am (Pacific) 11am (CENTRAL) 5pm (UK) PIN: 2719140 USA: +1 866 500 6738 (Toll: +1-203-480-8000) UK: +44 118 909 2000 Agenda: 1. IPR Notice 2. Previous Minutes and Action Review 3. Agenda Bashing 4. Discussion on the Data EPRs and scenarios 5. Discussion on resolving of DTI transport failures 6. Other Issues Arising 7. Spec Progress 8. AOB Outstanding Actions: [Ravi] Resolve minor comments on the spec. [Ravi] Come up with a proposal on how to represent data aggregates (e.g. multiple files) with Data EPRs to the mailing list. [Ravi] Come up with a proposal for a multiple retries property to the mailing list.

1 0

DMI Telcon
by Mario Antonioletti 20 Nov '07

20 Nov '07

Hi, A short reminder that we will be having our DMI telcon today in advance of thanks giving. I'm assuming that Steven's numbers are ok and I'm not sure whether I can make the call due to phone connection woes - if it is set up I can try and call: Date: 20th November 2007 When: 9am (Pacific) 11am (Eastern) 5pm (UK) PIN: 2719140 USA: +1 866 500 6738 (Toll: +1-203-480-8000) UK: +44 118 909 2000 a plausible agenda for this meeting will be: Agenda: 1. IPR Notice 2. Previous Minutes and Action Review 3. Agenda Bashing 4. Discussion on the Data EPRs and scenarios 5. Discussion on resolving of DTI transport failures 6. Other Issues Arising 7. Spec Progress 8. AOB Outstanding Actions: [Ravi] Resolve minor comments on the spec. [Ravi] Come up with a proposal on how to represent data aggregates (e.g. multiple files) with Data EPRs to the mailing list. [Ravi] Come up with a proposal for a multiple retries property to the mailing list. Can change the above agenda on the call as required. Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+

4 6

DMI Telcon Minutes
by Mario Antonioletti 20 Nov '07

20 Nov '07

Note: No call next week and the next call is on the 20th at the usual time (not the 21st!). OGSA-DMI Telcon - 07/11/07 ========================== Attendees: Steve Newhouse, Microsoft Mario Antonioletti, EPCC Allen Luniewski, IBM Michel Drescher, Fujitsu Agenda: 1. IPR Notice 2. Previous Minutes and Action Review 3. Agenda Bashing 4. Issues Arising 5. Spec Progress 6. AOB Actions: [Ravi] Resolve minor comments on the spec. [Ravi] Come up with a proposal on how to represent data aggregates (e.g. multiple files) with Data EPRs to the mailing list. [Ravi] Come up with a proposal for a multiple retries property to the mailing list. [Steve/Michel] Come up with a proposal for Data EPRs that can deal with the three scenarios proposed below. http://www.ogf.org/pipermail/ogsa-dmi-wg/2007-October/000293.html +--- Discussion on the Data EPRs and scenarios was postponed until Ravi becomes available or comments on the suggestions proposed by Michel: http://www.ogf.org/pipermail/ogsa-dmi-wg/2007-October/000293.html and Steven: http://www.ogf.org/pipermail/ogsa-dmi-wg/2007-October/000294.html Discussion thus proceeded on how to address transfer failures by the DMI architecture. DTF transport negotiation is done by protocol matching but, at run time, other factors may come into play that prevent the transfer from succeeding, e.g. fire walls. At the moment this means that the DTI will report the failure to the client but the client has no current way to communicate this information back to the DTF - the client can talk to the DTF again to effect the transfer but this will probably lead to the same protocols being chosen and thus reproduce the same failure. Allen suggested three possible solutions. http://www.ogf.org/pipermail/ogsa-dmi-wg/2007-November/000301.html Neither 1 and 2 are preferred, 3 requires active negotiation which is not in scope for this version of the spec. Not clear that it would not be able to do the transfer unless it performed a small test. The following solutions were discussed all starting from the point where the data transfer has began with the DTI but then failed: 1. The DTI returns the failure to the client with modified DEPRs to the client - effectively new ones with the failed protocol removed from the list of supported protocols. The client can then use these DEPRs to retry the transfer through the DTF. 2. The DTI returns the failed protocol to the client, the client modifies the data EPRs (as in 1 by removing the failed transport protocol) and resubmits these to the DEPRs. 3. The client passes on the DEPRs to the DTF as well as any failure messages returned by the DTI to the DTF which is then informed of the protocols that do not work (this might mean aggregating multiple failure messages which does not seem desirable). 4. The DTI is able to communicate an outcome of a transfer to the DTF which is then informed of what protocols do not work and it may be able to act on this. 5. There is a user agent between the DTF and DTI that maintains state and is able to apply some re-try policy on behalf of the client. This would maintain the clean interfaces and state models already in the spec. Michel is not at all keen on 1 as the minting of the DEPRs should be done by third parties. Allen not keen on 2 as this is making clients do stuff that should not really be in their scope. Idea now is that interested parties will flesh out the use case above (or some other) that most appeals to them noting the pros and cons. It was felt that we should not produce a first version of this spec that does not address this problem. Other factor addressed in the call was when the DTF can match more than one protocol - how does it chose what protocol to use? The fastest? The cheapest? etc? It was thought that for version 1 of the scope this would not be addressed - i.e. the client does not provide hints or QoS parameters BUT the DTF should publish what algorithm it will use to choose a transport protocol when there is more than one valid choice. There will be no DMI call next week in part to SC07 and other commitments. The next call on the 20th of November at the same time (this is so as to not clash with the night before thanks giving). +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+

3 2

Telecon notes
by Steven Newhouse 08 Nov '07

08 Nov '07

Glenn Wasson, UVa Marty Humphrey, UVa Steven Newhouse, Microsoft Shabaz , FZJ Chris Smith, Platform Steve McGough, Imperial College London John Ehrig, OGF Vesselin Novov, Imperial College London Paolo Andreetto, INFN Joel Replogle, OGF Gabor Robbei, KnowARC Steve Crumb, OGF Bill Nitzberg, Altair SC07 Activity Flyer : Update awaited from FZJ. Needed by 11:00 Pacific time TODAY, i.e. NOW!!! Press Release: Chris and Bill have initiated getting quotes from their respective organizations. Quotes from MS and OGF in place. Press release will probably come out Wednesday morning. Discussion will be off list due to it having embargoed contents. Demo Updates * UVa: FileStaging and ActivityCredential is now implemented in the compliance tester. Available for interop. * Microsoft: FileStaging, ActivityCredential and AdvancedFilter. Available for interop. * FZJ: Endpoint up and running and working with CREAM BES. Has just got username & password for UVa * Platform: Service that is mostly compliant for username & password. Contact Chris Smith. * Imperial College: See wiki page for basic service details. Different service for file staging. Problems with Platform and UVa endpoints with the GridSAM client. * INFN: Service & FTP server available. Problems with gLite and the certificate in the compliance tester - unable to workaround. OK with NorduGrid and GridSAM. * KnowARC: Service details up on wiki page. Only certificate. * Altair: Just demoing client side. BES++ activity at UVa: Deploy BES++ on SGE and PBS. Will modify the wiki. Marty will be touring SC07 getting feedback on your implementation experiences for the document. Please talk to him directly or contribute thoughts to the list. Good experiences are as important as identifying issues that needed resolution. Chris raised an issue as to how to handle the faults that may be generated on the operations taking a vector of activity identifiers as input. For HPCBP where the vector length can be constrained to be 1 a normal SOAP fault can be generated. For BES compliant operations this SOAP fault need to become (potentially) a vector of SOAP faults. This support is proving difficult to achieve in gSOAP tooling. In WCF it was also hard to get the correct classes generated from the XML declaration.

3 4

Apologies
by Ravi Madduri 07 Nov '07

07 Nov '07

Sorry for the late notice but with SC so close I don't think I can make it to the call today -- --Ravi Ravi K Madduri Argonne National Laboratory | University of Chicago | Globus Alliace http://www-unix.mcs.anl.gov/~madduri

1 0

DMI Telcon
by Mario Antonioletti 07 Nov '07

07 Nov '07

Hi, A short reminder that we will be having a DMI telcon today: Date: 7th November 2007 When: 9am (Pacific) 11am (Eastern) 5pm (UK) PIN: 2719140 USA: +1 866 500 6738 (Toll: +1-203-480-8000) UK: +44 118 909 2000 a plausible agenda for this meeting will be: Agenda: 1. IPR Notice 2. Previous Minutes and Action Review 3. Agenda Bashing 4. Issues Arising 5. Spec Progress 6. AOB Can change the above agenda on the call as required. Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+

1 0

Potential problem
by Mario Antonioletti 05 Nov '07

05 Nov '07

Hi, Last week we were discussing comments to the OGSA Data document and something came up that is pertinent to DMI (Alle pitch in). Basically, suppose that the source and sink publish a set of supported transport protocols. Then, if client wants to effect a transfer between the source and sink, they would talk to a DTF. The DTF would negotiate a protocol supported at both ends and create a DTI to manage the transfer for the user. The issue is that there may be other factors that preclude that transfer from taking place - for instance as a simple example, if active ftp was chosen as the protocol of choice a firewall could stop this transfer from successfully completing. The DTI would not be able to communicate this back to the DTF as the two are logically dissociated. Is there an expectation that the DTF, would not only protocol match but also test out that the transfer protocol actually works between source and sink before creating the DTI? I'm not sure if this is an implementation thing or whether there is a deeper potential problem here - thoughts? Mario +-----------------------------------------------------------------------+ |Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ. | |Tel:0131 650 5141|mario@epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ | +-----------------------------------------------------------------------+

3 3