Hi folks,

(apologies if you have already received this, I seem to be having trouble posting to the list).

 

 

A bit more material in preparation for Mondays phone meeting, items 1) and 2).

 

1)  Catering for moving of multiple source/sinks within a single atomic operation.

Basically, an important requirement for us is to define a bulk data transfer that could accommodate the transfer of multiple files/dirs within a single atomic operation. We realize that the way to do this in the DMI world is probably to instantiate a number of DTI instances, where each instance caters for a single source DEPR and an associated sink DEPR. However, ideally we would like to be able to define multiple transfers in a single atomic request, and importantly, be able to define this within a single xml doc.

 

As Gerson has shown in his previous email, we have also looked at the JSDL HPC file staging spec that facilitates the definition of multiple DataStaging elements (each containing a srcURI and an associated targetURI and much like DMI, embeds required credentials). At present however, neither JSDL File staging or DMI completely satisfy our requirements hence the current JSDL-DMI hybrid (http://code.google.com/p/dtsproject/source/browse/trunk/dts-jaxb/src/main/resources/minx-dts.xsd). Of course, ideally we would prefer to adopt a spec.

 

We were wondering if something like the following would be considered ‘in-scope’ / suitable for DMI ? (please see the schema and sample document in the attached examples.zip).  The attached schema only adds the following extra element and complex type to the dmi-messages.xsd (below). In doing this, it becomes possible to wrap multiple source-and-sink DEPR combinations within the same doc.

 

       Thus, rather than define a single source and sink such as:

           GetDataTransferInstance( [source DEPR], [sink DEPR], [transfer requirements] );

       Maybe define:

           GetDataTransferInstance( [GetWrappedDataTransferInstanceRequestMessage] );

 

 

  <!--

Proposed additional Request Message element for moving multiple files/dirs in one atomic unit:

  -->

  <element name="GetWrappedDataTransferInstanceRequestMessage" type="dmi-plain:GetWrappedDataTransferInstanceRequestType"/>

  <complexType name="GetWrappedDataTransferInstanceRequestType">

    <sequence>

      <element name="WrappedSourceSinkDEPR" type="dmi-plain:GetDataTransferInstanceRequestType" minOccurs="1" maxOccurs="unbounded"/>

    </sequence>

  </complexType>

 

 

2) xsd:any extension points

Would the working group also consider the addition of some xsd:any extension points within the dmi.xsd ? (e.g. within the DataType complex type). This is necessary so that we can define extra connection properties such as the MCATZone and MdasDomain for SRB /iRODS. This is also shown in the attached example. I think there a some other items but I forget.

 

Cheers,

Dave

 

 

 

 

 

 

Hi guys,

Just thought of updating you on where we are at in terms of the DTS development and schema specification. At the moment, we've finished implementing the modules for the DTS and are starting to integrate bits we implemented so we could demonstrate a very basic type of transfer (ie ftp to smb). We've also started planning on how we could complicate stuff by integrating Shibb authentication.

So here's the stuff that you might be interested in.. the approach we've taken is take bits and pieces off JSDL and OGSA-DMI. See the current schema we have..

http://code.google.com/p/dtsproject/source/browse/trunk/dts-jaxb/src/main/resources/minx-dts.xsd

You probably might be wondering why we decided to replicate stuff from JSDL and OGSA-DMI instead of just importing from them. The reason for it is, we just wanted something quick and dirty to play with. The 'quick and dirty' start will then allow us to properly identify extension requirements for both JSDL and DMI and ultimately for implementing with respect to either JSDL and DMI. The plan is to import from standard schemata and use them instead of reinventing the wheel. At the moment, not all we needed are implemented by the schemata so we've gone for the approach of just using standard when it has the stuff we need. The stuff we're working on is a prototype only and modifying it to use  the standard won't be that hard to do.

We quite like the JSDL way of specifying the job definition so we've taken the JSDL schema and wrapped it with web service operations that use bits and pieces from DMI. We also use the DMI standards for the messages that will be exchanged by the WS and its client.

Regards,
Gerson