October 2008 - ogsa-dmi-wg

UNICORE DMI implementation ready for Interop
by Mohammad Shahbaz Memon 05 Nov '08

05 Nov '08

Hi Folks, I am pleased to announce that UNICORE has completed the first initial DMI implementation based on its WRSF based hosting environment. The endpoint and certificate information is available on the DMI Wiki: http://forge.ogf.org/sf/wiki/do/viewPage/projects.ogsa-dmi-wg/wiki/PlainWSI… The implementation is a first release, therefore I would expect some failures in the first test pass. Feel free to contact me for further questions. Cheers, -- ------------------ Mohammad Shahbaz Memon Distributed Systems and Grid Computing Jülich Supercomputing Center Forschungszentrum Jülich GmbH Jülich Germany Office: +49 (0)2461 61 6567 Fax: +49 (0)2461 61 6656 http://www.fz-juelich.de/jsc Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), Dr. Ulrich Krafft (stellv. Vorsitzender)

3 2

New draft (10) of the Plain WS Rendering specification
by Michel Drescher 17 Oct '08

17 Oct '08

Folks, I have found some time and took your comments on the previously published draft. Please find draft 10 attached and on Gridforge () for further comments. Cheers, Michel -- Michel Drescher Fujitsu Laboratories of Europe, Ltd. Hayes Park Central Hayes End Road Hayes, Middlesex UB4 8FE Reg. No. 4153469 +44 20 8606 4834 Michel.Drescher(a)uk.fujitsu.com

3 6

Re: [ogsa-dmi-wg] StartNotBefore element in Data Transfer
by Michel Drescher 16 Oct '08

16 Oct '08

Hi Shahbaz, of cure, you are right - well spotted. I double-checked my implementation, and again the Functional Specification (draft 64). The Functional Specification is correct and clear when describing the Start operation and the [start time] attribute of the DTI. However, the Functional Specification is ambiguous when describing the [start not before] transfer requirement. From discussions with Steve around the same topic I remember what we resolved: If no StartNotBefore (SNB) is given in the transfer requirements, then the DTI stays in the Created state until the EndNoLaterThan requirement has passed by (or until the system cleans up because of some policy), or until the user invoked the Start operation. If a SNB has been specified in the transfer requirements, then the DTI MUST have a [start time] attribute, and it will automatically advance to the Scheduled state once the [start time] has passed. In this use case the user MAY try and start the transfer *before* the [start time] has passed by, ut the DTI MAY reject. For the group: I think we need to slightly reword the Functional Specification better describing the StartNotBefore transfer requirement. It currently reads as: "This represents the [startNotBefore] information element. The data transfer encapsulated by the DTI MUST NOT start before the date, time and time zone specified by this element. If not specified its default value will be the date and time that the DTI was created. The DTI MUST NOT enter the transferring state before this time unless the “Start” operation on the DTI is invoked." I propose to replace that with the following paragraph: "This represents the [start not before] information element. The data transfer encapsulated by the DTI MUST NOT start before the date, time and time zone specified by this element. If not specified then the DTI MUST wait for a client to invoke the Start operation before it can enter the Scheduled state." By standards process we are safe as we are allowed to submit a changed version of the Functional Specification along with the Interop Experiences document. Cheers, Michel On 15 Oct 2008, at 15:18, Mohammad Shahbaz Memon wrote: > Thanks Michel. > > I have another question pertaining to the plain-ws interop details > posted on dmi wiki. > > There is a test case No. 5 (if its still valid) > > which says : > > ------------------------------------------------------ > Description: > The Client asks the Factory to create a Data Transfer Instance. > The source and sink EPRs, and the transfer requirements must be > suitable to create a DTI instance. The TransferRequirements MUST be > empty. > After obtaining the EPR to the DTI instance the client then asks the > DTI instance for its attributes. > > Expected outcome: > The DTI instance returns its attributes without faults. > > Success Requirements: > - The instance attributes list MUST contain a valid set of attributes. > - The Status attribute MUST have the value "Created". > - The instance attribute list MUST NOT have a value for StartTime. > ---------------------------------------- > > If you see the success requirements, it says the status must have a > Created value and instance attributes with the start time. There is a > contradiction between this success req. and the way you described in > the last email, because it says if SNB is not specified then transfer > will immediately start which could possibly lead this transfer to the > scheduled/transferring/done state depends upon the job execution at > the backend. > > > Thanks > > Shahbaz > > > > > > On Wed, Oct 15, 2008 at 12:12 PM, Michel Drescher > <Michel.Drescher(a)uk.fujitsu.com> wrote: >> Hi Mohammad, >> >> for your reference, I am referring to draft 64 of the functional >> specification. >> >> The main use case for the SNB requirement lies in resource >> allocation and >> reservation. While either is out of scope for DMI they nonetheless >> must be >> accommodated for. So consider a request for transferring 10 TB of >> data. Sure >> enough you need to check and reserve disk space at the sink, and most >> certainly also some bandwidth for the transfer itself. >> Implementations of >> DMI my do reservations transparently, but often it is not. Hence a >> client to >> DMI (not necessarily the user) reserves some bandwidth and storage >> space, >> and makes sure that the transfer itself starts not before the >> reservation is >> activated. >> >> The most important thing to understand is that the events and >> operations in >> the state diagram for the DTI may happen explicitly, e.g. by >> sending the >> corresponding WS message to the DTI, or implicitly or >> automatically. For >> example, a client to a DTI may invoke the "stop" operation, >> triggering the >> FAILED event causing a running transfer to fail. But equally the >> DTI itself >> may trigger the FAILED event, possibly because the EndNoLaterThat >> (ENLT) >> requirement must be observed. >> >> Having said that, the semantics for a DTI are as follows: >> In any circumstance, when a DTI is created it is always in the >> "Created" >> state. >> From there on, if a SNB is *not* given, then the implicit SNB is >> set to the >> time the DTI is created. Then the DTI waits until the SNB has >> passed before >> triggering the STARTED event by itself. During that time the DTI >> MUST stay >> in state "Created". That implicitly means that, if no SNB has been >> given >> beforehand, that the DTI more or less immediately triggers the >> STARTED >> event, because the implicit SNB has already passed. In other words, >> if the >> client does not specify a SNB then the transfer is supposed to start >> immediately. >> It is slightly different though in the case when the SNB indeed >> lies in the >> future. As said before, the DTI sits and waits until the configured >> SNB >> passes by. In that time, a client to the DTI is allowed to invoke >> the DTI's >> start() operation to attempt to start the transfer even though a >> SNB has >> been previously given. The DTI however, may decide to either accept >> or >> decline the start request, depending on the individual situation. >> If it >> declines the request the DTI MUST send back a >> FailedStateTransitionFault >> with the appropriate explanation. If it accepts the start request, >> it will >> send back the correct response, and advance to the SCHEDULED state. >> >> It is important to note that the "Scheduled" state is named quite >> ambiguously. This "Scheduled" state has nothing to do with the SNB >> in the >> DMI sense. It is present in the state machine simply because in most >> situations, there is a time gap between that point in time of >> telling the >> underlying transfer protocol to move bytes, and the time of the bytes >> actually hitting the wire. During this time gap the DTI MUST be in >> state >> "Scheduled". Again, the proper state for the STI to be in while >> waiting for >> the SNB to pass by is the state "Created". >> >> -- >> >> I hope that, though quite long, the explanation gives you enough >> information >> to proceed. >> >> Cheers, >> Michel >> >> On 15 Oct 2008, at 10:01, Mohammad Shahbaz Memon wrote: >> >>> Hi Michel, >>> >>> in DMI, I am bit skeptical in the use of startnotbefore element of >>> transfer requirement. >>> >>> Considering a scenario where client creates a transfer by specifying >>> startNotBefore(SNB) element in the transfer requirements. Shall the >>> transfer initialization be started automatically just after the >>> client >>> calls DataTransferFactory.getDataTransferInstance() or the >>> scheduling >>> will start after client calls the DataTransferInstance.Start()? >>> >>> If I guess, the scheduling starts after the GetDatatransferInstance >>> then client doesn't need to send Start method call, however client >>> can >>> send start calls which will result in a throw incorrect statefault. >>> >>> Thanks, >>> -- >>> ------------------ >>> Mohammad Shahbaz Memon >>> Distributed Systems and Grid Computing >>> Jülich Supercomputing Center >>> Forschungszentrum Jülich GmbH >>> Jülich Germany >>> >>> Office: +49 (0)2461 61 6567 >>> Fax: +49 (0)2461 61 6656 >>> http://www.fz-juelich.de/jsc >>> >>> Sitz der Gesellschaft: Jülich >>> Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 >>> Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe >>> Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), >>> Dr. Ulrich Krafft (stellv. Vorsitzender) >> >> -- >> Michel Drescher >> Fujitsu Laboratories of Europe, Ltd. >> Hayes Park Central >> Hayes End Road >> Hayes, Middlesex UB4 8FE >> Reg. No. 4153469 >> >> +44 20 8606 4834 >> Michel.Drescher(a)uk.fujitsu.com >> >> >> >> >> > > > > -- > ------------------ > Mohammad Shahbaz Memon > Distributed Systems and Grid Computing > Jülich Supercomputing Center > Forschungszentrum Jülich GmbH > Jülich Germany > > Office: +49 (0)2461 61 6567 > Fax: +49 (0)2461 61 6656 > http://www.fz-juelich.de/jsc > > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzende des Aufsichtsrats: MinDirig'in Bärbel Brumme-Bothe > Vorstand: Prof. Dr. Achim Bachem (Vorsitzender), > Dr. Ulrich Krafft (stellv. Vorsitzender) -- Michel Drescher Fujitsu Laboratories of Europe, Ltd. Hayes Park Central Hayes End Road Hayes, Middlesex UB4 8FE Reg. No. 4153469 +44 20 8606 4834 Michel.Drescher(a)uk.fujitsu.com

2 2