Dear Massimo, Many thanks for your interest and your comparison of CREAM JDL with JSDL. Before I address the issues that you raise I feel it best to first set out the scope of what we feel the JSDL group is aiming for and indeed where the group should progress to from this point on. It is probably best to define JSDL 1.0 as a "minimum standard". There are many Grid job submission systems (and consumers) already in existence. The main problem that we have had in defining JSDL 1.0 is that these different systems define what a job is and what attributes you can specify about them vary dramatically. Systems such as Condor allow for workflows through the Condor DAGMAN system with the ability to specify to an intricate level of detail the type of resource preferred for execution while other systems such as (Sun) Grid Engine simply allow single shell scripts to be deployed onto a computational queue. In order to come up with a specification that all potential consumers were willing to consider we have had to scope back the functionality of JSDL 1.0. Hence this has become effectively a "minimum standard requirement" for job submission. We don't have the authority (or desire) to force (Sun) Grid Engine to support Condor DAGMAN workflows. Likewise we can't force Condor to support the (Sun) Grid Engine ticketing system - which I'm sure they would argue is far superior to the Condor rank & requirements system. With this said JSDL 1.0 is not the end of the story by any means. We see it rather being a core set which further standardization can be built on. Functionality which is not core to all JSDL consumers can be standardized as "add-ons" to the core specification. This is how we feel that we can support the high level functionality which is really needed to make good use of the Grid. If I can now move over to the points that you raised in your email. For no particular reason I'd like to cover them in reverse order: Support for matchmaking: I for one believe the current functionality in JSDL for matching with resources is minimal - though having said this it matches the minimum functionality that we can expect to find on current job submission systems. At this point we need to convince potential JSDL consumer vendors to adopt JSDL. If this can be achieved by a simple mapping to their existing system then we have a good chance to get this adoption. However, if they are required to add in vast amounts of functionality then they are less likely to adopt. With this said, I am working with a group at University College London (UCL) who are using JSDL to submit to a Condor pool. Within this work they are developing on extensions to JSDL to deal with extra functionality core to Condor. One of the elements they are proposing is a <condor:rank> element which is added into a <Resource> element. This is perfectly consistent with the JSDL standard 1.0 and once we get JSDL 1.0 out of the door we will propose an extension to JSDL 1.0 to deal with Condor systems. This I hope you could take an active role in to make this extension specification usable both with Condor and CREAM. Similarly for the idea of parameter sweep operations - they are looking into extensions to provide this. Interestingly I was contacted the other day by someone at the University of Edinburgh who wants to write an extension to our locally developed JSDL consumer so that the JSDL parametric sweep extensions can be used on a Globus cluster by generating multiple Globus submissions to represent the single JSDL job submission. In terms of the Workflow: From my understanding of Condor DAGMAN which is the core of the EGEE job submission system (it's a few years since I've checked this one) a DAGMAN workflow is a file containing a number of Classads along with some control flow instructions. We also envisage a similar process for JSDL. However, the GGF only allows a group to be chartered to complete one task - in our case to develop a Job submission language. With this said I know that there are several people within the group (including some of the chairs) who want to extend JSDL with workflow. To achieve this we will need to develop a new group and then perform these tasks. Hopefully this would be something you would be keen to take part in. I've just realized how much I've written! So I shall summaries as follows: - View JSDL 1.0 as the "core" elements of the system. With the "you may choose to support these features" coming in later versions of JSDL. This by no means prevents you from using these higher level functionalities at this stage, nor from taking an active role in standardizing them. - We are keen for your input. Hopefully you can now see how we see the JSDL work progressing. We are by no means intending to wrap things up with the release of JSDL 1.0 and indeed seek to provide the higher level functionality once this is complete. - Many features that both you and other groups seek can't be put into the core of JSDL 1.0. Though hopefully with your help we can work with these with the goal of obtaining standardization for them in the near future. I'd also like to invite you to talk to us, preferably in person, about this as it is far too easy to misinterpret things within an email. We'd be more than happy to talk with you at GGF14, or if you would prefer we could join you for a teleconference/VRVS/AccessGrid discussion sometime before then. With best regards, steve.. Massimo Sgaravatto - INFN Padova wrote:
Dear all
In the context of the Compute Resource Management Interfaces (CRM) initiative (the same mentioned by Oxana), the Job Description Language (JDL) used by the CREAM (Computing Resource Execution AND Management) service has been compared with JSDL. The CREAM JDL is a subset of the EGEE WMS JDL, that is the language used to specify characteristics and requirements for jobs submitted to the EGEE Workload Management System. This JDL is a high-level, user-oriented language based con Condor Classads for describing jobs and complex requests (aggregates of jobs). It is a fully extensible language, hence the user is allowed to use whatever attribute for the description of a request. However, only a certain set of attributes, referred as "supported attributes", is taken into account.
The result of this evaluation is available at:
http://www.pd.infn.it/grid/jra1/CREAMJDL-vs-JSDL.pdf
A presentation summarizing these results is also available at:
http://www.pd.infn.it/~sgaravat/Grid/CREAMJDL-vs-JSDL-EGEE3.pdf http://www.pd.infn.it/~sgaravat/Grid/CREAMJDL-vs-JSDL-EGEE3.ppt
In short we found that for many functionality for which specific CREAM JDL attributes exist there aren't "default" JSDL directives that could be used. For most of them this is not surprising since they refer to features/functionality very specific to our architecture. In this case I guess JSDL extensions could be used.
On the other hand (and here I agree with Oxana) there are some issues that in my opinion could/should probably be addressed by the default JSDL directives. In particular (and, again, I agree with Oxana):
- the specification of workflows (in our case DAGs, job collections, parametric jobs) - the support for matchmaking, i.e. allowing specifying requirements (and preferences) on the target resource where the job has to be executed.
Thanks for your attention
Cheers, Massimo
\\\|/// \\ ~ ~ // (/ @ @ /) -------oOOo-(_)-oOOo---------------------------------- Massimo Sgaravatto INFN Sezione di Padova Via Marzolo, 8 35131 Padova - Italy Tel: ++39 0498277047 Fax: ++39 0498277102 oooO E-mail: massimo.sgaravatto@pd.infn.it ( ) Oooo Home page: http://www.pd.infn.it/~sgaravat --------\ (----( )---------------------------------- \_) ) / (_/
-- -- ------------------------------------------------------------------------ Dr A. Stephen McGough http://www.doc.ic.ac.uk/~asm ------------------------------------------------------------------------ Technical Coordinator, London e-Science Centre, Imperial College London, Department of Computing, 180 Queen's Gate, London SW7 2BZ, UK tel: +44 (0)207-594-8409 fax: +44 (0)207-581-8024 ------------------------------------------------------------------------ Assistant Warden, West Wing, Beit Hall, Imperial College, Prince Consort Road, London, SW7 2BB tel: +44 (0)207-594-9910 ------------------------------------------------------------------------