Hi;
My responses are in-line below.
Marvin.
From: A S McGough [mailto:asm@doc.ic.ac.uk]
Sent: Friday, June 09, 2006 3:03
AM
To: Marvin Theimer
Cc: JSDL Working Group;
ogsa-bes-wg@ggf.org;
Subject: Re: [jsdl-wg] Questions
and potential changes to JSDL, as seen from HPC Profile point-of-view
Hi Marvin,
Thanks for the (fairly long) email, you've raised quite a few interesting
points - which I'll address inline below. First off I'd just like to say that
the JSDL document is meant to be a language specification document thus a large
number of the issues about how JSDL should be used and what they have to
support is not really in scope for that document. However, I do agree with you
that such a document needs to exist - but for all uses of JSDL not just HPC. I
would like to take your straw man and use it as the starting point for this
document for the section on "using JSDL for HPC". Let me know what
you think.
[Marvin] As long as
the HPC profile specification has some
document/specification that it can employ to normatively define behaviors, I’m
happy. Presumably “compliance” with JSDL will defined to mean
compliance with this second document that you propose to create?
More comments below:
Marvin Theimer wrote:
Hi;
Coming from the point-of-view of the HPC Profile
working group, I have several questions about JSDL, as well as some straw man
thoughts about how JSDL should/could relate to the HPC Profile specification
that I’m involved with. Some of my questions lead me to
restrictions on JSDL that an HPC profile specification might make. Other
questions lead to potential changes that might be made as part of creating
future versions of JSDL. (I’m well aware that JSDL 1.0 was meant as
a starting point rather than the final word on job submission descriptions and
so please interpret my questions as being an attempt at constructive
suggestions rather than a criticism of a very fine first step by the JSDL
working group.)
Will do.
1.
Can JSDL documents describe jobs
other than Linux/Unix/Posix jobs? For example, things like mount points
and mount sources do not map in a completely straight-forward manner to how
file systems are provided in the Windows world.
The idea is that JSDL (possibly through extensions)
will be able to describe all kinds of jobs that can be submitted. This may
include database queries, control of instruments or web invocations. We did
Posix job type fist as that was the one the majority of people in the group
wanted first. We're currently working on a ParallelApplication extension for
JSDL. We'd be more than happy to see if an extension for Windows (or any other
system) can be done through tweaks to the existing setup or by adding a new extension.
Could you say how file systems don't map to the Windows world? My naive
assumption was that you could do it.
[Marvin] With suitable – and hopefully
relatively minor – restrictions the concepts of mount-point and
mount-source might be made to map to the Windows world. It’s the
details that may not map precisely. In general, there are concepts in Unix
and Windows file systems that don’t map to each other. For example,
there are no symbolic links in NTFS and there is no notion of fine-grained
op-locks in the Unix file system.
NO. I doubt JSDL is expressive enough in its current
state to describe the needs of all jobs. We're working with the Information
model people in the OGSA group at the moment on this please help! I liked some
of your ideas for this below by the way.
Agreed. We don't yet have a way to add to the
normative enumerations. I think you suggest below to move these into a separate
document so that they can be updated more easily - this would seem a good idea.
As for OS versioning I have my ideas though JSDL doesn't have a central plan
yet. Again input here would be appreciated.
Yes. The intention with JSDL has always been to
produce more normative extensions post JSDL 1.0.
Personally I'd agree with you that file staging should
be in an extension. Though the view of the group was that most current DRM
systems which would consume JSDL had file staging as a core element. I also
agree on the idea of "optimization hints".
As said above we really need to define this in a
separate "profile" document.
The writing of a resource description language was
something we were told we couldn't do in the JSDL group. I do agree that it's
now important that we have one. I think we'd need to go back to GGF (or
whatever there name is this week) and ask to set up a group to do this. Perhaps
we could take all the stuff out of JSDL which is appropriate as a starting
point?
[Marvin] Whichever group the work is done
in, the HPC profile working group will need to deal with the matter sooner
rather than later (during this summer, to be precise). It may be the case
that the HPC profile working group will end up defining a “Basic Resource
Description” specification in the same spirit as BES is a “basic”
version of what’s being pursued in the
Yes - this is true - though as the current document is
a description of the JSDL "language" this is correct. These
issues should all be clarified in the profile document.
10. Although
the notions of working directory and environment variables are defined in the
posix extension, they are implicitly assuming in the data staging section of
the core specification. This implies to me that either (a) data staging
is made an extension or (b) these concepts are made a normative, required part
of the core specification.
Hmm - well spotted. Personally as I've said I'd like
to see it made into an extension. This probably need s some discussion on the
list.
This is a major problem as many of the systems that
are currently available out there do not support recursive directory copying.
Again we could clarify the use of this through a HPC profile.
12. The
current definitions of the well-known file systems seem imprecise to me.
In particular:
13. What
are the navigation rules associated with each? Can you cd out of the
subtree that each represents? ROOT almost certainly does not allow
that. Is there an assumption that one can cd out of HOME or TMP or
SCRATCH? Hopefully not, since that would make these file systems even
more Unix/Linux-centric, plus one would now need to specify what clients can
expect to see when they do so.
Again not defined here. Though I'd assume we can
easily say in the profile that you can't cd out of it.
Personally I don't have a use for it. Anyone else?
Again profile issue.
Profile.
Profile.
This gets very difficult and political! Though we
should be able to come up with a core set for the profile.
19.
Restructure the JSDL specification
as a small core specification that must be universally implemented – i.e.
not just parsable, but also suppliable by all compliant job submission services
– and a number of optional extension profiles.
Hopefully the language as it stands at the moment
(with a few exceptions) is a good core set. With profiles for different use
cases we could mandate the implemented side too.
Why do these need to be in the core? We had problems
before in a pre-release version when they were in the core as people who wanted
to do database submissions (and other things) were trying to map these into
such elements.
[Marvin] A Windows HPC job is not completely
posix-compliant, yet has overlap on the above-listed set of concepts (and actually
many more). So I would argue that we need something that abstracts out the core concepts of a
traditional HPC job. Given the presence of file data staging elements in
the core specification – which I would argue are meaningless for database
submissions – it seems like the above-listed elements are at least as
generic as the data staging elements.
22. Create
precise definitions of the various concepts introduced in the data staging
extension, including normative requirements about whether or not one can change
directory up and out of a file system’s root directory, etc.
23. Define
which transports are expected to be implemented by all compliant services.
Quite possibly - and the use of a profile.
Sounds good. Even better if we can get someone else to
update these for us.
This should come from the work we are doing with the
Information model people - please join in.
Not entirely sure what you are meaning on this one.
Can you explain further.
[Marvin] I’m basically advocating two
things: (a) tackle the problem of how to describe available resources since it’s
so closely allied to the topic of describing required resources, and (b) start
with a simple approach and a means of allowing evolution/extension to support
richer approaches later on.
Now, as presented above, my straw man proposal looks
like suggestions for changes that might go into a JSDL-1.1 or JSDL-2.0
specification. In the near-term, the HPC profile working group will be
exploring what can be done with just JSDL-1.0 and restrictions to that
specification. The restrictions would correspond to disallowing those
parts of the JSDL-1.0 specification that the above proposal advocates moving to
extension profiles. It will also explore whether a restricted version of
the posix extension could be used to cover most common Windows cases.
Marvin.
OK for those who have made it this far - possibly not
many. I'm going to propose a JSDL call on this in a new email so all can see
it.
[Marvin] Great idea. I will try hard
to be on that call. (If you send me a direct email then that will
increase the likelihood since all my GGF email now goes into one folder and I
sometimes miss important ones in the deluge of all emails.)
steve..
--
------------------------------------------------------------------------
Dr A. Stephen McGough http://www.doc.ic.ac.uk/~asm
------------------------------------------------------------------------
Technical Coordinator, London e-Science Centre, Imperial College London ,
Department of Computing, 180 Queen's Gate, London SW7 2BZ , UK
tel: +44 (0)207-594-8409 fax: +44 (0)207-581-8024
------------------------------------------------------------------------