POSIXApplication Limits
I'm confused about many of the limits in POSIXApplication. Some seem redundant with respect to the Resources requirement elements. Other's simply don't seem clear in their spec descriptions. In general, it seems unclear whether "job" refers to the entire job or a single process. For example, MemoryLimit says "...the maximum amount of physical memory that the job should use when executing", whereas CPUTimeLimit says "...the number of CPU time seconds a job is allowed to consume before a SIGXCPU signal is sent to the job". In the first case I would interpret "job" to mean the one or more processes a job spawns in the process of achieving completion. In the second case it seems that "job" means an individual process. Is it then impossible to specify a total job CPU limit for all processes, or is that what TotalCPUTime in Resources is meant for? But then it's not so clear why MemoryLimit is not redundant with TotalPhysicalMemory. Here are some more specific questions: WallTimeLimit -- Why isn't there a hard wall time limit if this is a soft limit? DataSegmentLimit -- What is meant by a data segment? VirtualMemoryLimit -- Similar to MemoryLimit, what's the difference between this and an UpperBoundedRange value for Resources' TotalVirtualMemory? Is there some reason someone would want to request nodes with a total virtual memory capacity greater than they are limiting their job to use? Thanks! Peter
Peter G. Lane wrote:
I'm confused about many of the limits in POSIXApplication. Some seem redundant with respect to the Resources requirement elements. Other's simply don't seem clear in their spec descriptions.
It depends on the job manager in question whether the limits are implied by the resources (and with some apps, I understand that the limits have to be specified separately. The person that told me this was not at all happy with the situation BTW!) Yes, it's a mess, but that's the truth on the ground anyway. All limits were discovered by examining the output of ulimit across a wide variety of systems. There's a core set of them that are probably universal on Unix, but some of the others seem more specialized. But don't worry. The _correct_ way of handling the limits is pretty clear: if you don't have a system that supports a particular limit in a natural way, throw out documents that ask to set that limit. It's strongly suggested that JSDL authors do not set limits unless they really need them. FYI, jobs compliant with the HPC Profile will not set limits at all.
Is it then impossible to specify a total job CPU limit for all processes, or is that what TotalCPUTime in Resources is meant for? But then it's not so clear why MemoryLimit is not redundant with TotalPhysicalMemory.
The limits are purely things to splat into ulimit, with all that implies.
Here are some more specific questions:
WallTimeLimit -- Why isn't there a hard wall time limit if this is a soft limit?
We thought about this, but decided that we'd only ever set soft limits.
DataSegmentLimit -- What is meant by a data segment?
Some unix systems let you limit this. I've never needed it myself. If you can't map it (and your question implies that you can't) throw it out. We won't complain!
VirtualMemoryLimit -- Similar to MemoryLimit, what's the difference between this and an UpperBoundedRange value for Resources' TotalVirtualMemory? Is there some reason someone would want to request nodes with a total virtual memory capacity greater than they are limiting their job to use?
If I give an upper bounded range for the VM, the resource manager might choose to allocate an amount of memory less than upper bound described in UBR. Also, the RM might have ways other than limits of imposing contraints on the resources open to me. On the other hand, setting the VM limit just means that the OS will send me a signal when (or sometime after) I exceed it (or is VM one of the ones where it makes some syscalls fail?) so it's at least potentially recoverable. By contrast, the resource allocation may be implemented in a way (e.g. a virtualized OS or, depending on underlying architecture, processor affinities) that cannot be defeated. The take-home message is that POSIXApp limits are specifically to do with ulimit settings, but resource management may be done in other ways, and probably would have to be on clustered systems anyway. Donal.
Donal K. Fellows wrote:
The take-home message is that POSIXApp limits are specifically to do with ulimit settings, but resource management may be done in other ways, and probably would have to be on clustered systems anyway.
Just to make sure I'm understanding your point completely, neither JSDL nor any of it's normative extensions provide a way, not including custom extensions, to specify resource limits for jobs submitted to resource managers (i.e. PBS, LSF, Condor, ...). The POSIXApplication limits should be used only by applications that are able to directly set the ulimit values before executing a process. Is this is a valid statement? Peter
Donal.
Peter G. Lane wrote:
Just to make sure I'm understanding your point completely, neither JSDL nor any of it's normative extensions provide a way, not including custom extensions, to specify resource limits for jobs submitted to resource managers (i.e. PBS, LSF, Condor, ...). The POSIXApplication limits should be used only by applications that are able to directly set the ulimit values before executing a process. Is this is a valid statement?
To be exact, we expect people to specify (in the jsdl:Resources section) what resources their job needs to execute successfully (if they specify too little, that's too bad) and for the resource manager to use that information to compute how many resources to allocate to the job. It is also up to the resource manager to choose how to implement said allocation; JSDL does not mandate any particular mechanism. If the mechanism chosen includes the use of POSIX limits, it is obviously incumbent upon the implementation to set the limits such that the requirements within the POSIXApplication section are also met; I would not expect any implementation to allow the Limits to be used to gain any extra allocation of resources above and beyond what was asked for in the Resources section. One way of doing this might be for the resource manager to first set up hard limits and then to use the various PA/*Limits to refine those with extra soft limits. Donal.
Sorry for the late delay - I've been on holiday and only now starting to catch up. To add to Donal's fine response. If you do come up with a good set of custom extensions for some resource managers please let us know and we can see about making them into normative extensions. steve.. Donal K. Fellows wrote:
Peter G. Lane wrote:
Just to make sure I'm understanding your point completely, neither JSDL nor any of it's normative extensions provide a way, not including custom extensions, to specify resource limits for jobs submitted to resource managers (i.e. PBS, LSF, Condor, ...). The POSIXApplication limits should be used only by applications that are able to directly set the ulimit values before executing a process. Is this is a valid statement?
To be exact, we expect people to specify (in the jsdl:Resources section) what resources their job needs to execute successfully (if they specify too little, that's too bad) and for the resource manager to use that information to compute how many resources to allocate to the job. It is also up to the resource manager to choose how to implement said allocation; JSDL does not mandate any particular mechanism. If the mechanism chosen includes the use of POSIX limits, it is obviously incumbent upon the implementation to set the limits such that the requirements within the POSIXApplication section are also met; I would not expect any implementation to allow the Limits to be used to gain any extra allocation of resources above and beyond what was asked for in the Resources section.
One way of doing this might be for the resource manager to first set up hard limits and then to use the various PA/*Limits to refine those with extra soft limits.
Donal.
participants (3)
-
A.S.McGough -
Donal K. Fellows -
Peter G. Lane