application profile proliferation
I've been reviewing JSDL, the BES HPC Application Profile, and JSDL Parallel/SPMD profile to see how they can be used with Globus GRAM execution service. One concern I have is the way these various application profiles are redefining the same basic functionality (executable, arguments, environment, etc), either in their own namespaces or importing from the jsdl-posix namespace, sometimes with different behavior/features. It's not clear to me how (if at all) these are to be/can be composed. The feature set of GRAM seems to span these profiles: some (but not all) of the limits from POSIXApplication are relevant, as is the SPMD functionality of specifying the various process counts and something like SPMDVariation. There is also a small amount of scheduler- specific and site-specific extensions that GRAM might have to deal with. Do other groups have plans for implementing more than one of these profiles, and what approaches are you considering for doing so? joe
Hi, Joseph Bester wrote:
I've been reviewing JSDL, the BES HPC Application Profile, and JSDL Parallel/SPMD profile to see how they can be used with Globus GRAM execution service. One concern I have is the way these various application profiles are redefining the same basic functionality (executable, arguments, environment, etc), either in their own namespaces or importing from the jsdl-posix namespace, sometimes with different behavior/features. It's not clear to me how (if at all) these are to be/can be composed.
Just as sort of an overview of the motivation behind these application extensions: - The HPC Profile Application extension removes elements from POSIXApplication extension that do not apply to non-POSIX systems. Windows was the specific motivating example. Also it narrows down some of the definitions of other elements defined by POSIXApplication for the specific interoperability needs expressed by the HPC Profile. I believe it doesn't add any new features. We did discuss whether it was better to try and modify the definitions of POSIXApplication extension to accommodate explicitly non-POSIX systems and it was felt at the time that for clarity having these similar-but-different definitions in a new namespace was the preferred solution. - The SPMD Application extension main motivation was to add MPI support---though it is likely to do a little bit more than that. A subset of POSIXApplication sub-elements relevant to such submissions are re-used as is and I think this is reasonable usage. We did discuss alternatives (encapsulating one or more POSIXApplication elements inside an SPMDApplication element for example) but we felt that importing the relevant set of elements from the jsdl-posix namespace was the simplest way forward.
The feature set of GRAM seems to span these profiles: some (but not all) of the limits from POSIXApplication are relevant, as is the SPMD functionality of specifying the various process counts and something like SPMDVariation. There is also a small amount of scheduler- specific and site-specific extensions that GRAM might have to deal with.
Do other groups have plans for implementing more than one of these profiles, and what approaches are you considering for doing so?
I can't comment on which project is planning to implement the SPMD extension. But we did receive initial input and interest from GridSAM, NAREGI, BSC (Barcelona Supercomputing Center) and also contributions from Stephen Pickles and Chris Smith. -- Andreas Savva Fujitsu Laboratories Ltd
On Feb 18, 2007, at 10:51 PM, Andreas Savva wrote:
Just as sort of an overview of the motivation behind these application extensions: - The HPC Profile Application extension removes elements from POSIXApplication extension that do not apply to non-POSIX systems. Windows was the specific motivating example. Also it narrows down some of the definitions of other elements defined by POSIXApplication for the specific interoperability needs expressed by the HPC Profile. I believe it doesn't add any new features. We did discuss whether it was better to try and modify the definitions of POSIXApplication extension to accommodate explicitly non-POSIX systems and it was felt at the time that for clarity having these similar-but-different definitions in a new namespace was the preferred solution.
I think there are some semantic differences between the various element definitions. For example (if I'm reading the specifications correctly) things like the executable path are defined by default as relative to the WorkingDirectory in the HPCApplication profile, but not in the POSIXApplication profile.
- The SPMD Application extension main motivation was to add MPI support---though it is likely to do a little bit more than that. A subset of POSIXApplication sub-elements relevant to such submissions are re-used as is and I think this is reasonable usage. We did discuss alternatives (encapsulating one or more POSIXApplication elements inside an SPMDApplication element for example) but we felt that importing the relevant set of elements from the jsdl-posix namespace was the simplest way forward.
Did the group consider having this as a separate extension child of the jsdl:Application element (independent of the POSIXApplication or HPCProfileApplication)? joe
Joseph Bester wrote:
I think there are some semantic differences between the various element definitions.
Yes. That's the point. Isn't it a good thing they're in different namespaces? :-)
For example (if I'm reading the specifications correctly) things like the executable path are defined by default as relative to the WorkingDirectory in the HPCApplication profile, but not in the POSIXApplication profile.
In effect, the HPC app Executable requires (without saying this) that you don't use the PATH to select the binary to run. That's fine for their specific goal (demonstrating interop across a lot of systems) but I suspect that it is too restrictive for many interesting uses. To my mind, that just means that there's room for an "HPC Level 2 Profile" sometime. :-)
Did the group consider having this as a separate extension child of the jsdl:Application element (independent of the POSIXApplication or HPCProfileApplication)?
No, because (IIRC) that's not (technically) type-compatible. This is because we specified that there can only be one xsd:any##other child of the JSDL Application element. Donal.
On Feb 19, 2007, at 5:48 PM, Donal K. Fellows wrote:
Joseph Bester wrote:
I think there are some semantic differences between the various element definitions.
Yes. That's the point. Isn't it a good thing they're in different namespaces? :-)
Sort of, but it makes implementation of JSDL-consuming services more complicated.
Did the group consider having this as a separate extension child of the jsdl:Application element (independent of the POSIXApplication or HPCProfileApplication)?
No, because (IIRC) that's not (technically) type-compatible. This is because we specified that there can only be one xsd:any##other child of the JSDL Application element.
Donal.
Perhaps my version is an old one, but the schema I've seen has <xsd:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded"/> in the definition of the Application_Type. joe
[I'll reply in reverse order if you don't mind.] Joseph Bester wrote:
On Feb 19, 2007, at 5:48 PM, Donal K. Fellows wrote: :
Did the group consider having this as a separate extension child of the jsdl:Application element (independent of the POSIXApplication or HPCProfileApplication)?
We considered a number of different ways, including: - encapsulating (possibly more than one) POSIXApplication element(s) inside SPMDApplication - adding an additional element at the same level as POSIXApplication (as you mention above) - putting it outside JobDescription and inside JobDefinition as a separate general 'parallelism' element - re-using only the elements from POSIXApplication that we agreed were relevant to this extension and defining a new Application extension We decided on the current approach because we thought it was clearer since it uses the mechanism we put in place for new Application extensions---it is a new Application extension so it should replace and be at the same level as POSIXApplication. It also makes the point that not all elements from POSIXApplication would actually be useful when, for example, executing an MPI job.
I think there are some semantic differences between the various element definitions. Yes. That's the point. Isn't it a good thing they're in different namespaces? :-)
Sort of, but it makes implementation of JSDL-consuming services more complicated.
Well, we specifically allow extensibility in application definitions and people can create their own Application extensions. Element names like 'executable' are generic enough that one should expect to see used in other extensions with possibly different definitions/explanations. It may be complicated having jsdl-posix:executable and jsdl-hpcpa:executable but not more so than having, say, a posixecutable and an hpcexecutable. I do understand you point about added complexity, but I'm not sure there is anything that we can do about it except make clear these are different definitions. We can take this discussion offline or we can have a chat on a jsdl teleconference if you like. -- Andreas Savva Fujitsu Laboratories Ltd
participants (3)
-
Andreas Savva -
Donal K. Fellows -
Joseph Bester