Hi;
Coming from the point-of-view of the HPC Profile working group, I have
several questions about JSDL, as well as some straw man thoughts about
how JSDL should/could relate to the HPC Profile specification that I'm
involved with. Some of my questions lead me to restrictions on JSDL
that an HPC profile specification might make. Other questions lead to
potential changes that might be made as part of creating future versions
of JSDL. (I'm well aware that JSDL 1.0 was meant as a starting point
rather than the final word on job submission descriptions and so please
interpret my questions as being an attempt at constructive suggestions
rather than a criticism of a very fine first step by the JSDL working
group.)
At a high level, there are several general questions that came up when
reading the JSDL 1.0 specification:
* Can JSDL documents describe jobs other than Linux/Unix/Posix
jobs? For example, things like mount points and mount sources do not
map in a completely straight-forward manner to how file systems are
provided in the Windows world.
* Is JSDL expressive enough to describe all the needs of a job?
For example, it is unclear how one would specify a requirement for
something like a particular instruction set variation of the IA86
architecture (e.g. the SSE3 version of the Pentium) or how one would
specify that AMD processors are required rather than Intel ones (because
the optimized libraries and the optimizations generated by the compiler
used will differ for each). For another example, it is unclear how one
would specify that all the compute nodes used for something like an MPI
job should have the same hardware.
* How will JSDL's normative set of enumeration values for things
like processor architecture and operating system be kept up-to-date and
relevant? Also, how should things like operating system version get
specified in a normative manner that will enable interoperability among
multiple clients and job scheduling services? For example, things like
Linux and Windows versions are constantly being introduced, each with
potentially significant differences in capabilities that a job might
depend on. Without a normative way of specifying these constantly
evolving version sets it will be difficult, if not impossible, to create
interoperable job submission clients and job scheduling services
(including meta-scheduling services where multiple schedulers must
interoperate with each other).
* Although JSDL specifies a means of including additional
non-normative elements and attributes in a document, non-normative
extensions make interoperability difficult. This implies the need for
normative extensions to JSDL beyond the Posix extension currently
described in the 1.0 specification. Are there plans to define
additional extension profiles to address the above questions surrounding
expressive power and normative descriptions of things like current OS
types and versions?
* If one accepts the need for a variety of extension profiles
then this raises the question of what should be in the base case. For
example, it could be argued that data staging - with its attendant
aspects such as mount points and mount sources - should be defined in an
extension rather than in the core specification that will need to cover
a variety of systems beyond just Linux/Unix/Posix. Similarly, one might
argue that the base case should focus on what's functionally necessary
to execute a job correctly and should leave things that are
"optimization hints", such as CPU speed and network bandwidth
specifications, to extension profiles.
* How are concepts such as IndividualCPUSpeed and
IndividualNetworkBandwidth intended to be defined and used in practice?
I understand the concept of specifying things like the amount of
physical memory or disk space that a job will require in order to be
able to run. However, CPU speed and network bandwidth don't represent
functional requirements for a job - meaning that a job will correctly
run and produce the same results irrespective of the CPU speed and
network bandwidth available to it. Also, the current definitions seem
fuzzy: the megahertz number for a CPU does not tell you how fast a given
compute node will be able to execute various kinds of jobs, given all
the various hardware factors that can affect the performance of a
processor (consider the presence/absence of floating point support, the
memory caching architecture, etc.). Similarly, is network bandwidth
meant to represent the theoretical maximum of a compute node's network
interface card? Is it expected to take into account the performance of
the switch that the compute node is attached to? Since switch
performance is partially a function of the pattern of (aggregate)
traffic going through it, the network bandwidth that a job such as an
MPI application can expect to receive will depend on the type of
communications patterns employed by the application. How should this
aspect of network bandwidth be reflected - if at all - in the network
bandwidth values that a job requests and that compute nodes advertise?
* JSDL is intended for describing the requirements of a job being
submitted for execution. To enable matchmaking between submitted jobs
and available computational resources there must also be a way of
describing existing/available resources. While much of JSDL can be used
for this purpose, it is also clear that various extensions are
necessary. For example, to describe a compute cluster requires that one
be able to specify the resources for each compute node in the cluster
(which may be a heterogeneous lot). Similarly, to describe a compute
node with multiple network interfaces would require an extension to the
current model, which assumes that only a single instance of such things
can exist. This raises the question of whether something other than
JSDL is intended to be used for describing available computational
resources or whether there are intensions to extend JSDL to enable it to
describe such resources.
* The current specification stipulates that conformant
implementations must be able to parse all the elements and attributes
defined in the spec, but doesn't require that any of them be supplied.
Thus, a scheduling service that does nothing could claim to be compliant
as long as it can correctly parse JSDL documents. For interoperability
purposes, I would argue that the spec should define a minimum set of
elements that any compliant service must be able to supply. Otherwise
clients will not be able to make any assumptions about what they can
specify in a JSDL document and, in particular, client applications that
programmatically submit job submission requests will not be possible
since they can't assume that any valid JSDL document will actually be
acceptable by any given job submission service.
* I have a number of questions about data staging:
* Although the notions of working directory and environment
variables are defined in the posix extension, they are implicitly
assuming in the data staging section of the core specification. This
implies to me that either (a) data staging is made an extension or (b)
these concepts are made a normative, required part of the core
specification.
* Recursive directory copying can be specified, but is not
required to be supplied by any job submission service. This makes it
difficult to write applications that programmatically define their data
staging needs since they cannot in the current design determine whether
any given job submission service implements recursive directory copying.
In practice this may mean that programmatically generated job
submissions will only ever use lists of individual files to stage.
* The current definitions of the well-known file systems seem
imprecise to me. In particular:
* What are the navigation rules associated with each? Can you cd
out of the subtree that each represents? ROOT almost certainly does not
allow that. Is there an assumption that one can cd out of HOME or TMP
or SCRATCH? Hopefully not, since that would make these file systems
even more Unix/Linux-centric, plus one would now need to specify what
clients can expect to see when they do so.
* What is ROOT intended to be used for? Are there assumptions
about what resides under root? Are there assumptions about what an
application can read/write under the ROOT subtree? (ROOT also seems
like the most Unix-specific of the 4 file system types defined.)
* What are the sharing/consistency semantics of each file system
in situations where a job is a multi-node application running on
something like a cluster? Is HOME visible to all compute nodes in a
data-consistent manner? I'm guessing that TMP would be assumed to be
strictly local to each compute node, so that things like MPI
applications would need to be cognizant that they are writing multiple
files to multiple separate storage systems when they write to a file in
TMP - and furthermore that data staging of such files after a job has
run will result in multiple files that all map to the same target file.
* Can other users write over or delete your data in TMP and/or
SCRATCH? Is data in these file systems visible to other users or does
each job get its own private TMP and SCRATCH?
* How long does data in SCRATCH stay around? Without some
normative definition - or at least a normative lower bound - on data
lifetime clients will have to assume that the data can vanish
arbitrarily and things like multi-job workflows will be very difficult
to write if they try to take advantage of SCRATCH space to avoid
unnecessary data staging actions to/from a computing facility.
* From an interoperability and programmatic submission
point-of-view, it is important to know which transports any given job
submission service can be expected to support. This seems like another
area where a normative minimal set that all job submission services must
implement needs to be defined.
Given these questions, as well as the mandate for the HPC profile to
define a simple base interface (that can cover the HPC use case of
submitting jobs to a compute cluster), I would like to present the
following straw man proposal for feedback from this community:
* Restructure the JSDL specification as a small core
specification that must be universally implemented - i.e. not just
parsable, but also suppliable by all compliant job submission services -
and a number of optional extension profiles.
* Declare concepts such as executable path, command-line
arguments, environment variables, and working directory to be generic
and include them in the core JSDL specification rather than the posix
extension. This may enable the core specification to support things
like Windows-based jobs (TBD). The goal here is to define a core JSDL
specification that in-and-of-itself could enable job submission to a
fairly wide range of execution subsystems, including both the
Unix/Linux/Posix world and the Windows world.
* Move data staging to an extension.
* Create precise definitions of the various concepts introduced
in the data staging extension, including normative requirements about
whether or not one can change directory up and out of a file system's
root directory, etc.
* Define which transports are expected to be implemented by all
compliant services.
* Move the various enumeration types - e.g. for CPU architecture
and OS - to separate specification documents so that they can evolve
without requiring corresponding and constant revision of the core JSDL
specification.
* Define extension profiles (eventually, not right away) that
enable richer description of hardware and software requirements, such as
details of the CPU architecture or OS capabilities. As part of this,
move optimization hints, such as CPU speed and network bandwidth
elements out of the JSDL core and into a separate extension profile.
* Embrace the issue of how to specify available resources at an
execution subsystem. Start by defining a base case that allows the
description of compute clusters by creating a compound JSDL document
that consists of an outer element that ties together a sequence of
individual JSDL elements, each of which describes a single compute node
of a compute cluster. Define an explicit notion of extension profiles
that could define other ways of describing computational resources
beyond just an array of simple JSDL descriptions.
Now, as presented above, my straw man proposal looks like suggestions
for changes that might go into a JSDL-1.1 or JSDL-2.0 specification. In
the near-term, the HPC profile working group will be exploring what can
be done with just JSDL-1.0 and restrictions to that specification. The
restrictions would correspond to disallowing those parts of the JSDL-1.0
specification that the above proposal advocates moving to extension
profiles. It will also explore whether a restricted version of the
posix extension could be used to cover most common Windows cases.
Marvin.
