Questions and potential changes to BES, as seen from HPC Profile point-of-view
Hi; Coming from the point-of-view of the HPC Profile working group, I have several questions about BES (including recent discussions on the mailing list), as well as some straw man thoughts about how BES should relate to the HPC profile spec. Based on the BES-1.3 spec that Andrew Grimshaw recently sent out, at an abstract level, there seem to be the following aspects to BES: * A core set of operations around activities: * CreateActivityFromJSDL * GetActivityStatus * RequestActivityStateChange * GetActivityJSDLDocuments * A set of BES factory-specific system management operations and resource properties (RPs): * StartAcceptingNewActivities * StopAcceptingNewActivities * IsAcceptingNewActivities RP * Support for notifications. * Support for various resource properties (or their equivalent in a non-WSRF version) having to do with an information model for describing various things about a BES factory, the associated container it represents, and any activities it is currently running. * An extensible activity state model. Things explicitly NOT in the BES specification are: * Generic system management interface. * Security design. * Interface for directly controlling/manipulating an activity once it has been created. Things that used to be in the BES spec but now seem to be extensions (please correct me if I'm wrong here!): * Data staging * Suspension I have the following questions about BES and the various discussions that have recently occurred (including the ESI integration): * Extensibility: * Given that BES has bought into the notion of an extensible activity state diagram, it needs to also normatively define how clients can learn of the extensions that a given BES service supports. Is that something that will be added to the BES specification? Or will the specification point to some other place where notions of extensibility are defined more generically? (Personally, I'd vote for the former approach.) * Is the "base case" for BES now fig.2, which shows states of {new, pending, running, canceled, failed, finished}? * Previously included states, such as Execution-Pending, will presumably be defined in suitable extension profiles? * Assuming that data staging and suspension are now extensions to the base BES spec, will they be defined as such in an appendix of the spec, or as a separate extension profile? * The original BES spec describes a fairly sophisticated data staging design that supports parallelism. Is there any interest in defining a second, simpler data staging extension that avoids the complexity of the parallelism support? * Will the suspension extension be the simple one that is currently presented in sec. 4 as an example? Or do people feel that a more complicated version, such as the ESI one is necessary/important? Can/should we define both? * Given that suspension is no longer in the base design, presumably the createInSuspendedState parameter to CreateActivityFromJSDL should disappear? * RequestActivityStateChange: I believe this operation will pose challenges in an extensible design. The current design is imperative by nature: it specifies an explicit state to move an activity to. However, a client who does not know of all the extensions that a BES service implements may not know how to pick the appropriate state to transition to. It seems better to introduce a more declarative approach in which clients specify "actions" they wish to occur, such as 'CancelActivity'. This approach would allow the BES service to make the appropriate state transition in response to a desired action requested by a client. * Information model: * JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I'm wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification. * The BES information model now includes various posix-specific elements of JSDL. How would other systems - such as a Windows system - be described? * The spec requires that all BES services "support" all the various attributes listed in sec. 5, but they don't have to implement them. What exactly does that mean? For example, if a JSDL doc specifies a CPU-Speed requirement and a particular BES service doesn't implement it (meaning it doesn't keep track of it), then does the associated CreateActivityFromJSDL request have to fail? If so, then do clients have to figure out what the minimal set of implemented attributes are in a system and then only use those in job descriptions? Is there is a notion of "optional" attributes that can be ignored, that specify desired attribute values rather than required ones? * Is there any notion of specifying that all compute nodes should have the same value for some attribute (e.g. CPU architecture, CPU speed, NIC card)? This seems to be missing from the JSDL specification, but seems very important for BES if it is to support things like compute clusters. * Some of the elements seem either incompletely specified, have definitions that are open to multiple interpretations, or have definitions that would be very difficult to implement in practice. In particular: * CPU architecture seems like it can't describe all the variations - let alone all the peripherals such as GPUs - that a computing resource might have (let alone a cluster). * CPU speed seems like the tip of an iceberg having to do with characterizing the performance of a system, which will depend on all manner of things like details of the processor chip used, cache sizes, bus used, etc. * Network bandwidth: is this the theoretical maximum of the NIC on a compute node or is it the current bandwidth actually available in a (shared) system? Note that the latter is difficult to measure in a practically useful way. Note also that network bandwidth only describes one aspect of communications performance and that several others are arguably equally important (e.g. latency). All this leads to the question of whether BES will have a notion of extending the information model that is supplied. If so, then that leads to the question of what the base case should be and whether it should include a smaller set of things than is currently listed in the spec. Are there any plans to tighten the definitions of some of the more vague information elements? (I guess this really is an issue more for the JSDL WG than for BES.) * GetActivityJSDLDocuments returns a JSDL document for each specified activity. Is this sufficient to capture the entire "provenance" for what has happened to the activity? In particular, would it be sufficient to allow someone to (a) run the same activity on another BES service (assuming same hardware and software) and get the same results and (b) debug what has happened to an errant activity? I would argue that both capabilities have proven to be important in actual systems. * System management operations: * Currently BES supports 2 specific system management operations: Start and stop activities commands. Most schedulers support a variety of scheduling-specific system management operations and I'm wondering why these two operations were singled out in particular to be part of the base case? * These operations seem to require a different set of authorization credentials than the other interface operations since they should be invoked by system administrators rather than random users. How will that work, given that these operations are in the same WSDL as the other operations? Wouldn't this argue for moving these operations to a separate system management interface? * Array operations: * Currently one can create a single activity, but all other operations accept an array of AEDs as input. Was there some reason why an array creation operation wasn't included so that, for example, parameter sweep applications can be created with a single request instead of N requests (where N can be in the thousands)? * Given that BES seems to have bought into the notion of extensibility, should the base case be a "non-array" one? For example, currently if you want to handle a fault for a RequestActivityStateChange operation on a single activity you need to look inside the returned array of results to see if a fault infoset was returned. All the exception handling machinery that modern tooling provides can't get used because RequestActivityStateChange never returns an actual fault message (as compared to a fault infoset for the appropriate array elements that are returned. * Other questions: * An entire (small) section is devoted to talking about the optional use of WS-Names. However, since the specification doesn't require them, it's unclear to me whether BES needs to say anything about WS-Names. As far as I understand things, whether an EPR is a WS-Name or not can be determined by inspecting it. Hence the only reason to have a special property on a BES service that indicates what kind of AEDs it returns is to alert potential clients ahead of time about this feature of the service. But it's not clear to me what a client would do with that information, as compared to deciding opportunistically to exploit a WS-Name AED for, e.g. resolution, at the time that that would be necessary. Is there a use case that describes how clients would exploit the AED-type resource property? * Since JSDL documents are self-describing, a BES service can figure out by inspection whether the job description infoset parameter to CreateActivityFromJSDL is JSDL or something else. This would seem to imply that naming the operation CreateActivity would lose no information and would allow for transparent extension to other job description infoset simply by using them (assuming they are self-describing). * Container attributes that I have questions about: * LocalResourceManagerType: where do these get defined normatively? * Job Credential Service and File Credential Service: these imply a specific security model. Given that security is undefined in the BES spec, is this appropriate - especially given the rather vague definition of both? Given these questions, as well as the mandate for the HPC profile to define a simple base interface, I would like to present the following straw man proposal for a modified BES specification for feedback from this community: * Operations: * CreateActivity(jsdlDoc) --> EPR * GetActivity(EPR) --> activityState * GetActivityProvenance(EPR) --> either JSDL doc (if that can describe all the necessary provenance info) or JSDL+ * CancelActivity(EPR) * For non-WSRF versions: QueryResources() --> schedulerResourcesInfoset * 'schedulerResourcesInfoset' is essentially the union of the RPs that would be exported in a WSRF-based version for describing the resources that are available for use at this BES service. Note that a BES service might also want to expose other kinds of information that would not be returned from this operation - this operation is there so that clients can determine whether or not a BES service could potentially meet their needs and is necessary for meta-scheduling scenarios. * One might argue that one could use WS-Transfer for this operation. However, since a BES service might want to export other kinds of information, this would require an extra level of indirection so that the BES service could expose which EPRs to use for retrieving which kinds of information. * Additional topics/summary: * Simple state diagram and no notion of array operations, data staging, suspension, or notifications in base BES case. * Extensions defined as separate profiles for array operations, data staging, suspension, and notifications. * RequestActivityStateChange replaced by operations specifying desired actions rather than states. Base case supports activity cancellation; extensions can define additional operations (e.g. SuspendActivity). * Information model: small base set plus extensions model (which ones to include in the base set TBD) * All system management functions moved out to a separate interface. Thanks for any and all feedback on these questions and this straw man proposal, Marvin.
Hi; One point of clarification on the straw man proposal: For operations such as * GetActivity(EPR) --> activityState * GetActivityProvenance(EPR) --> either JSDL doc (if that can describe all the necessary provenance info) or JSDL+ * CancelActivity(EPR) The EPR is a parameter supplied to the BES service, it is NOT the endpoint to which the request is being sent. (After all, that EPR is a different WSDL altogether.) In general, this raises the question of whether it would be useful to have a more compact "activityID" (akin to the abstract name in a WS-Name) that can be used instead of a fairly bulky EPR that the BES service then needs to parse in order to obtain the activityID that is somewhere embedded in the EPR. Marvin. ________________________________ From: Marvin Theimer Sent: Monday, June 05, 2006 4:19 PM To: ogsa-bes-wg@ggf.org Cc: Marvin Theimer; Ed Lassettre Subject: Questions and potential changes to BES, as seen from HPC Profile point-of-view Hi; Coming from the point-of-view of the HPC Profile working group, I have several questions about BES (including recent discussions on the mailing list), as well as some straw man thoughts about how BES should relate to the HPC profile spec. Based on the BES-1.3 spec that Andrew Grimshaw recently sent out, at an abstract level, there seem to be the following aspects to BES: * A core set of operations around activities: * CreateActivityFromJSDL * GetActivityStatus * RequestActivityStateChange * GetActivityJSDLDocuments * A set of BES factory-specific system management operations and resource properties (RPs): * StartAcceptingNewActivities * StopAcceptingNewActivities * IsAcceptingNewActivities RP * Support for notifications. * Support for various resource properties (or their equivalent in a non-WSRF version) having to do with an information model for describing various things about a BES factory, the associated container it represents, and any activities it is currently running. * An extensible activity state model. Things explicitly NOT in the BES specification are: * Generic system management interface. * Security design. * Interface for directly controlling/manipulating an activity once it has been created. Things that used to be in the BES spec but now seem to be extensions (please correct me if I'm wrong here!): * Data staging * Suspension I have the following questions about BES and the various discussions that have recently occurred (including the ESI integration): * Extensibility: * Given that BES has bought into the notion of an extensible activity state diagram, it needs to also normatively define how clients can learn of the extensions that a given BES service supports. Is that something that will be added to the BES specification? Or will the specification point to some other place where notions of extensibility are defined more generically? (Personally, I'd vote for the former approach.) * Is the "base case" for BES now fig.2, which shows states of {new, pending, running, canceled, failed, finished}? * Previously included states, such as Execution-Pending, will presumably be defined in suitable extension profiles? * Assuming that data staging and suspension are now extensions to the base BES spec, will they be defined as such in an appendix of the spec, or as a separate extension profile? * The original BES spec describes a fairly sophisticated data staging design that supports parallelism. Is there any interest in defining a second, simpler data staging extension that avoids the complexity of the parallelism support? * Will the suspension extension be the simple one that is currently presented in sec. 4 as an example? Or do people feel that a more complicated version, such as the ESI one is necessary/important? Can/should we define both? * Given that suspension is no longer in the base design, presumably the createInSuspendedState parameter to CreateActivityFromJSDL should disappear? * RequestActivityStateChange: I believe this operation will pose challenges in an extensible design. The current design is imperative by nature: it specifies an explicit state to move an activity to. However, a client who does not know of all the extensions that a BES service implements may not know how to pick the appropriate state to transition to. It seems better to introduce a more declarative approach in which clients specify "actions" they wish to occur, such as 'CancelActivity'. This approach would allow the BES service to make the appropriate state transition in response to a desired action requested by a client. * Information model: * JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I'm wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification. * The BES information model now includes various posix-specific elements of JSDL. How would other systems - such as a Windows system - be described? * The spec requires that all BES services "support" all the various attributes listed in sec. 5, but they don't have to implement them. What exactly does that mean? For example, if a JSDL doc specifies a CPU-Speed requirement and a particular BES service doesn't implement it (meaning it doesn't keep track of it), then does the associated CreateActivityFromJSDL request have to fail? If so, then do clients have to figure out what the minimal set of implemented attributes are in a system and then only use those in job descriptions? Is there is a notion of "optional" attributes that can be ignored, that specify desired attribute values rather than required ones? * Is there any notion of specifying that all compute nodes should have the same value for some attribute (e.g. CPU architecture, CPU speed, NIC card)? This seems to be missing from the JSDL specification, but seems very important for BES if it is to support things like compute clusters. * Some of the elements seem either incompletely specified, have definitions that are open to multiple interpretations, or have definitions that would be very difficult to implement in practice. In particular: * CPU architecture seems like it can't describe all the variations - let alone all the peripherals such as GPUs - that a computing resource might have (let alone a cluster). * CPU speed seems like the tip of an iceberg having to do with characterizing the performance of a system, which will depend on all manner of things like details of the processor chip used, cache sizes, bus used, etc. * Network bandwidth: is this the theoretical maximum of the NIC on a compute node or is it the current bandwidth actually available in a (shared) system? Note that the latter is difficult to measure in a practically useful way. Note also that network bandwidth only describes one aspect of communications performance and that several others are arguably equally important (e.g. latency). All this leads to the question of whether BES will have a notion of extending the information model that is supplied. If so, then that leads to the question of what the base case should be and whether it should include a smaller set of things than is currently listed in the spec. Are there any plans to tighten the definitions of some of the more vague information elements? (I guess this really is an issue more for the JSDL WG than for BES.) * GetActivityJSDLDocuments returns a JSDL document for each specified activity. Is this sufficient to capture the entire "provenance" for what has happened to the activity? In particular, would it be sufficient to allow someone to (a) run the same activity on another BES service (assuming same hardware and software) and get the same results and (b) debug what has happened to an errant activity? I would argue that both capabilities have proven to be important in actual systems. * System management operations: * Currently BES supports 2 specific system management operations: Start and stop activities commands. Most schedulers support a variety of scheduling-specific system management operations and I'm wondering why these two operations were singled out in particular to be part of the base case? * These operations seem to require a different set of authorization credentials than the other interface operations since they should be invoked by system administrators rather than random users. How will that work, given that these operations are in the same WSDL as the other operations? Wouldn't this argue for moving these operations to a separate system management interface? * Array operations: * Currently one can create a single activity, but all other operations accept an array of AEDs as input. Was there some reason why an array creation operation wasn't included so that, for example, parameter sweep applications can be created with a single request instead of N requests (where N can be in the thousands)? * Given that BES seems to have bought into the notion of extensibility, should the base case be a "non-array" one? For example, currently if you want to handle a fault for a RequestActivityStateChange operation on a single activity you need to look inside the returned array of results to see if a fault infoset was returned. All the exception handling machinery that modern tooling provides can't get used because RequestActivityStateChange never returns an actual fault message (as compared to a fault infoset for the appropriate array elements that are returned. * Other questions: * An entire (small) section is devoted to talking about the optional use of WS-Names. However, since the specification doesn't require them, it's unclear to me whether BES needs to say anything about WS-Names. As far as I understand things, whether an EPR is a WS-Name or not can be determined by inspecting it. Hence the only reason to have a special property on a BES service that indicates what kind of AEDs it returns is to alert potential clients ahead of time about this feature of the service. But it's not clear to me what a client would do with that information, as compared to deciding opportunistically to exploit a WS-Name AED for, e.g. resolution, at the time that that would be necessary. Is there a use case that describes how clients would exploit the AED-type resource property? * Since JSDL documents are self-describing, a BES service can figure out by inspection whether the job description infoset parameter to CreateActivityFromJSDL is JSDL or something else. This would seem to imply that naming the operation CreateActivity would lose no information and would allow for transparent extension to other job description infoset simply by using them (assuming they are self-describing). * Container attributes that I have questions about: * LocalResourceManagerType: where do these get defined normatively? * Job Credential Service and File Credential Service: these imply a specific security model. Given that security is undefined in the BES spec, is this appropriate - especially given the rather vague definition of both? Given these questions, as well as the mandate for the HPC profile to define a simple base interface, I would like to present the following straw man proposal for a modified BES specification for feedback from this community: * Operations: * CreateActivity(jsdlDoc) --> EPR * GetActivity(EPR) --> activityState * GetActivityProvenance(EPR) --> either JSDL doc (if that can describe all the necessary provenance info) or JSDL+ * CancelActivity(EPR) * For non-WSRF versions: QueryResources() --> schedulerResourcesInfoset * 'schedulerResourcesInfoset' is essentially the union of the RPs that would be exported in a WSRF-based version for describing the resources that are available for use at this BES service. Note that a BES service might also want to expose other kinds of information that would not be returned from this operation - this operation is there so that clients can determine whether or not a BES service could potentially meet their needs and is necessary for meta-scheduling scenarios. * One might argue that one could use WS-Transfer for this operation. However, since a BES service might want to export other kinds of information, this would require an extra level of indirection so that the BES service could expose which EPRs to use for retrieving which kinds of information. * Additional topics/summary: * Simple state diagram and no notion of array operations, data staging, suspension, or notifications in base BES case. * Extensions defined as separate profiles for array operations, data staging, suspension, and notifications. * RequestActivityStateChange replaced by operations specifying desired actions rather than states. Base case supports activity cancellation; extensions can define additional operations (e.g. SuspendActivity). * Information model: small base set plus extensions model (which ones to include in the base set TBD) * All system management functions moved out to a separate interface. Thanks for any and all feedback on these questions and this straw man proposal, Marvin.
Hi; A couple more questions: * Although sec. 8 says that the interface for controlling/managing an existent activity is out-of-scope, it clearly overlaps with the BES interface (in particular, the query and cancel operations). Given this overlap I'm wondering how much of the activity's WSDL interface will effectively be redundant with the BES interface. Moreover, services wanting to support array operations on activities will effectively need to support the full activity WSDL in any case since otherwise it will be impossible to achieve the desired batching for those operations. * This leads me to wonder whether a separate WSDL for activity interaction is really appropriate since it will require that the two specifications be kept continuously synchronized and one will effectively be a strict subset of the other. This issue seems like one that is more general than just HPC. I'm curious to hear what other people have to say about it. Marvin. ________________________________ From: Marvin Theimer Sent: Monday, June 05, 2006 4:40 PM To: 'ogsa-bes-wg@ggf.org' Cc: Ed Lassettre; Marvin Theimer Subject: RE: Questions and potential changes to BES, as seen from HPC Profile point-of-view Hi; One point of clarification on the straw man proposal: For operations such as * GetActivity(EPR) --> activityState * GetActivityProvenance(EPR) --> either JSDL doc (if that can describe all the necessary provenance info) or JSDL+ * CancelActivity(EPR) The EPR is a parameter supplied to the BES service, it is NOT the endpoint to which the request is being sent. (After all, that EPR is a different WSDL altogether.) In general, this raises the question of whether it would be useful to have a more compact "activityID" (akin to the abstract name in a WS-Name) that can be used instead of a fairly bulky EPR that the BES service then needs to parse in order to obtain the activityID that is somewhere embedded in the EPR. Marvin. ________________________________ From: Marvin Theimer Sent: Monday, June 05, 2006 4:19 PM To: ogsa-bes-wg@ggf.org Cc: Marvin Theimer; Ed Lassettre Subject: Questions and potential changes to BES, as seen from HPC Profile point-of-view Hi; Coming from the point-of-view of the HPC Profile working group, I have several questions about BES (including recent discussions on the mailing list), as well as some straw man thoughts about how BES should relate to the HPC profile spec. Based on the BES-1.3 spec that Andrew Grimshaw recently sent out, at an abstract level, there seem to be the following aspects to BES: * A core set of operations around activities: * CreateActivityFromJSDL * GetActivityStatus * RequestActivityStateChange * GetActivityJSDLDocuments * A set of BES factory-specific system management operations and resource properties (RPs): * StartAcceptingNewActivities * StopAcceptingNewActivities * IsAcceptingNewActivities RP * Support for notifications. * Support for various resource properties (or their equivalent in a non-WSRF version) having to do with an information model for describing various things about a BES factory, the associated container it represents, and any activities it is currently running. * An extensible activity state model. Things explicitly NOT in the BES specification are: * Generic system management interface. * Security design. * Interface for directly controlling/manipulating an activity once it has been created. Things that used to be in the BES spec but now seem to be extensions (please correct me if I'm wrong here!): * Data staging * Suspension I have the following questions about BES and the various discussions that have recently occurred (including the ESI integration): * Extensibility: * Given that BES has bought into the notion of an extensible activity state diagram, it needs to also normatively define how clients can learn of the extensions that a given BES service supports. Is that something that will be added to the BES specification? Or will the specification point to some other place where notions of extensibility are defined more generically? (Personally, I'd vote for the former approach.) * Is the "base case" for BES now fig.2, which shows states of {new, pending, running, canceled, failed, finished}? * Previously included states, such as Execution-Pending, will presumably be defined in suitable extension profiles? * Assuming that data staging and suspension are now extensions to the base BES spec, will they be defined as such in an appendix of the spec, or as a separate extension profile? * The original BES spec describes a fairly sophisticated data staging design that supports parallelism. Is there any interest in defining a second, simpler data staging extension that avoids the complexity of the parallelism support? * Will the suspension extension be the simple one that is currently presented in sec. 4 as an example? Or do people feel that a more complicated version, such as the ESI one is necessary/important? Can/should we define both? * Given that suspension is no longer in the base design, presumably the createInSuspendedState parameter to CreateActivityFromJSDL should disappear? * RequestActivityStateChange: I believe this operation will pose challenges in an extensible design. The current design is imperative by nature: it specifies an explicit state to move an activity to. However, a client who does not know of all the extensions that a BES service implements may not know how to pick the appropriate state to transition to. It seems better to introduce a more declarative approach in which clients specify "actions" they wish to occur, such as 'CancelActivity'. This approach would allow the BES service to make the appropriate state transition in response to a desired action requested by a client. * Information model: * JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I'm wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification. * The BES information model now includes various posix-specific elements of JSDL. How would other systems - such as a Windows system - be described? * The spec requires that all BES services "support" all the various attributes listed in sec. 5, but they don't have to implement them. What exactly does that mean? For example, if a JSDL doc specifies a CPU-Speed requirement and a particular BES service doesn't implement it (meaning it doesn't keep track of it), then does the associated CreateActivityFromJSDL request have to fail? If so, then do clients have to figure out what the minimal set of implemented attributes are in a system and then only use those in job descriptions? Is there is a notion of "optional" attributes that can be ignored, that specify desired attribute values rather than required ones? * Is there any notion of specifying that all compute nodes should have the same value for some attribute (e.g. CPU architecture, CPU speed, NIC card)? This seems to be missing from the JSDL specification, but seems very important for BES if it is to support things like compute clusters. * Some of the elements seem either incompletely specified, have definitions that are open to multiple interpretations, or have definitions that would be very difficult to implement in practice. In particular: * CPU architecture seems like it can't describe all the variations - let alone all the peripherals such as GPUs - that a computing resource might have (let alone a cluster). * CPU speed seems like the tip of an iceberg having to do with characterizing the performance of a system, which will depend on all manner of things like details of the processor chip used, cache sizes, bus used, etc. * Network bandwidth: is this the theoretical maximum of the NIC on a compute node or is it the current bandwidth actually available in a (shared) system? Note that the latter is difficult to measure in a practically useful way. Note also that network bandwidth only describes one aspect of communications performance and that several others are arguably equally important (e.g. latency). All this leads to the question of whether BES will have a notion of extending the information model that is supplied. If so, then that leads to the question of what the base case should be and whether it should include a smaller set of things than is currently listed in the spec. Are there any plans to tighten the definitions of some of the more vague information elements? (I guess this really is an issue more for the JSDL WG than for BES.) * GetActivityJSDLDocuments returns a JSDL document for each specified activity. Is this sufficient to capture the entire "provenance" for what has happened to the activity? In particular, would it be sufficient to allow someone to (a) run the same activity on another BES service (assuming same hardware and software) and get the same results and (b) debug what has happened to an errant activity? I would argue that both capabilities have proven to be important in actual systems. * System management operations: * Currently BES supports 2 specific system management operations: Start and stop activities commands. Most schedulers support a variety of scheduling-specific system management operations and I'm wondering why these two operations were singled out in particular to be part of the base case? * These operations seem to require a different set of authorization credentials than the other interface operations since they should be invoked by system administrators rather than random users. How will that work, given that these operations are in the same WSDL as the other operations? Wouldn't this argue for moving these operations to a separate system management interface? * Array operations: * Currently one can create a single activity, but all other operations accept an array of AEDs as input. Was there some reason why an array creation operation wasn't included so that, for example, parameter sweep applications can be created with a single request instead of N requests (where N can be in the thousands)? * Given that BES seems to have bought into the notion of extensibility, should the base case be a "non-array" one? For example, currently if you want to handle a fault for a RequestActivityStateChange operation on a single activity you need to look inside the returned array of results to see if a fault infoset was returned. All the exception handling machinery that modern tooling provides can't get used because RequestActivityStateChange never returns an actual fault message (as compared to a fault infoset for the appropriate array elements that are returned. * Other questions: * An entire (small) section is devoted to talking about the optional use of WS-Names. However, since the specification doesn't require them, it's unclear to me whether BES needs to say anything about WS-Names. As far as I understand things, whether an EPR is a WS-Name or not can be determined by inspecting it. Hence the only reason to have a special property on a BES service that indicates what kind of AEDs it returns is to alert potential clients ahead of time about this feature of the service. But it's not clear to me what a client would do with that information, as compared to deciding opportunistically to exploit a WS-Name AED for, e.g. resolution, at the time that that would be necessary. Is there a use case that describes how clients would exploit the AED-type resource property? * Since JSDL documents are self-describing, a BES service can figure out by inspection whether the job description infoset parameter to CreateActivityFromJSDL is JSDL or something else. This would seem to imply that naming the operation CreateActivity would lose no information and would allow for transparent extension to other job description infoset simply by using them (assuming they are self-describing). * Container attributes that I have questions about: * LocalResourceManagerType: where do these get defined normatively? * Job Credential Service and File Credential Service: these imply a specific security model. Given that security is undefined in the BES spec, is this appropriate - especially given the rather vague definition of both? Given these questions, as well as the mandate for the HPC profile to define a simple base interface, I would like to present the following straw man proposal for a modified BES specification for feedback from this community: * Operations: * CreateActivity(jsdlDoc) --> EPR * GetActivity(EPR) --> activityState * GetActivityProvenance(EPR) --> either JSDL doc (if that can describe all the necessary provenance info) or JSDL+ * CancelActivity(EPR) * For non-WSRF versions: QueryResources() --> schedulerResourcesInfoset * 'schedulerResourcesInfoset' is essentially the union of the RPs that would be exported in a WSRF-based version for describing the resources that are available for use at this BES service. Note that a BES service might also want to expose other kinds of information that would not be returned from this operation - this operation is there so that clients can determine whether or not a BES service could potentially meet their needs and is necessary for meta-scheduling scenarios. * One might argue that one could use WS-Transfer for this operation. However, since a BES service might want to export other kinds of information, this would require an extra level of indirection so that the BES service could expose which EPRs to use for retrieving which kinds of information. * Additional topics/summary: * Simple state diagram and no notion of array operations, data staging, suspension, or notifications in base BES case. * Extensions defined as separate profiles for array operations, data staging, suspension, and notifications. * RequestActivityStateChange replaced by operations specifying desired actions rather than states. Base case supports activity cancellation; extensions can define additional operations (e.g. SuspendActivity). * Information model: small base set plus extensions model (which ones to include in the base set TBD) * All system management functions moved out to a separate interface. Thanks for any and all feedback on these questions and this straw man proposal, Marvin.
Hi;
A couple more questions:
· Although sec. 8 says that the interface for controlling/ managing an existent activity is out-of-scope, it clearly overlaps with the BES interface (in particular, the query and cancel operations). Given this overlap I’m wondering how much of the activity’s WSDL interface will effectively be redundant with the BES interface. Moreover, services wanting to support array operations on activities will effectively need to support the full activity WSDL in any case since otherwise it will be impossible to achieve the desired batching for those operations. I strongly object to the idea of having generic array operations in an activity interface in addition to the BES interface.They're not needed if the BES interface already has them. I think there should be a strong separation between the "factory" or "activity manager" interface and the activities themselves which will have resources for each individually-owned activity. · This leads me to wonder whether a separate WSDL for activity interaction is really appropriate since it will require that the two specifications be kept continuously synchronized and one will effectively be a strict subset of the other. I think there are enough differences between the two interfaces that a separate WSDL is warranted if not required. If the activity interface is a subset of the BES interface, then I think the design is flawed. Operations and properties that operate on/expose data for one and only one activity should only be in the activity interface. Array operations and compute resource data properties should only be in the BES interface. An implementation of the BES interface will
On Jun 5, 2006, at 5:52 PM, Marvin Theimer wrote: potentially support multiple resources corresponding to the various LocalResourceManagerType values (not necessarily a one-to-one relationship). Peter
Peter Lane wrote:
Marvin Theimer wrote:
· This leads me to wonder whether a separate WSDL for activity interaction is really appropriate since it will require that the two specifications be kept continuously synchronized and one will effectively be a strict subset of the other.
I think there are enough differences between the two interfaces that a separate WSDL is warranted if not required. If the activity interface is a subset of the BES interface, then I think the design is flawed. Operations and properties that operate on/expose data for one and only one activity should only be in the activity interface. Array operations and compute resource data properties should only be in the BES interface. An implementation of the BES interface will potentially support multiple resources corresponding to the various LocalResourceManagerType values (not necessarily a one-to-one relationship).
I've been tending to think of the operations on a BES interface (apart from CreateActivityFromJSDL) as being for managerial/admin use, and that the API that is exposed to users should be on the activities themselves. Given this (possibly erroneous) world-view, the API on the activities becomes substantially more interesting. For example, it is perfectly reasonable for activities to have an operation (or property) that allows the discovery of how to find the stdout and stderr of the activity, and yet that's not generally interesting from a managerial PoV, and I think there are use-cases where that information shouldn't be exposed that way (e.g., where it is potentially highly confidential, such as in the medical or industrial design fields). Oh, that's still assuming that the container is generally inspectable to discover its resource configuration. By saying "BES ops are for admins" I'm not intending to exclude resource discovery. I'm just thinking about the state-changing non-submission ops. :-) Donal.
This (somewhat repetitive) discussion makes me think that we still do not have a precise enough nomenclature for the various aspects of jobs that matter in describing the architecture. I think we (the HPC and Grid Job management community) spend an inordinate amount of time talking in circles to reestablish common ground because of this... The terms are important not only to have unambiguous debates, but also to make the semantics in specifications clear, concise, and relevant. At a high level, this thread has revisited (and these are my best efforts at describing them for discussion, and not an assertion of what to call things in specifications): 1. Service management interfaces a. Possible generic control (start, stop, deploy, undeploy, etc.) b. Existing BES-specific controls (accept activities, etc.) [Has strong overlap with WSDM, CIM, et. al] 2. Generic Activity management interfaces (the main point of BES) a. Activity creation and destruction b. BES service "resource capabilities" (what can be run) c. Single and Array activity monitoring and control [Has overlap with provisioning and resource management] 3. Extended Activity management interfaces a. HPC Suspend/resume b. POSIX or Windows-specific controls c. Job File I/O d. ... (huge range of domain-specific stuff in future) [Has overlap with existing "legacy" grid services] 4. Extensibility Discovery/Negotiation mechanisms And I will add: 5. Applications/Services By this last one, I mean that BES is envisioned to support management of complex services through extensions. I think there is a distinction to be made between the "business" end of such a service (5) and its management interface as a special kind of activity (3). I often get an uncomfortable feeling that some people are expecting the "activity EPR" to expose the business interface, while I and some others expect it to only expose the extended management interface. This extended interface might well include some discovery/bootstrapping capabilities to help find the business endpoint(s) of the activity. I will also point out that, in some sense, BES as a generic activity manager (2) ought to be capable enough to recursively act as the manager of another BES service (1a). As an analogy, a well-known BES service might allow creation of VO's and VO services. Some of these VO services might be new BES instances. It seems to me that the BES-service management interfaces (1b) of the second-tier BES should be part of the extended activity interface (3) of those BES-types activities when they exist within the first tier's environment. On the other hand, the ability to create activities in the second tier is part of the "business" interface of those BES services. In my opinion, it does not belong in the activity management view of the service within the first tier BES service! [Note, I mention this to try to cut deeply into the distinctions between interfaces. I realize recursive use of BES is not on the top of everyone's must-have scenario list.] Also, as Marvin and Donal are pointing out, we need to consider various roles and scenarios. On the other hand, Donal's comments lead towards that slippery slope of assuming too much about implementation, e.g. that authorization is scoped at endpoints. Do we wish to assume this? I know it is easy in Globus land, but we can also handle per-operation authorization. We are, as implementers, a little hog-tied as far as providing finer authorization scoping, e.g. access controls on specific message contents. It is a tricky area, fraught with usability problems and non-interoperability. karl -- Karl Czajkowski karlcz@univa.com
Hi; The cluster job scheduler and meta-scheduler scenarios you mention are key ones that the HPC profile work is intended to cover (as base case and as an extension of the base case). Hence I, at least, can't afford to ignore it and have to therefore ask questions around how one would support it using BES and JSDL as the foundations of the HPC profile work. In my opinion BES has to be able to support the compute cluster scenario well, or it risks becoming a somewhat inappropriate basis for the HPC profile design. One thing to keep in mind on the security/authorization issue is the following: it seems a bit strange (to me) to have an interface that consists of operations that only some clients can invoke. It seems like a better design to have an interface of which users can invoke any operation and a separate interface that is available to administrators with additional privileges. I suspect this will also be simpler and more feasible from the point-of-view of what existing tooling can provide/support. Marvin. -----Original Message----- From: owner-ogsa-bes-wg@ggf.org [mailto:owner-ogsa-bes-wg@ggf.org] On Behalf Of Karl Czajkowski Sent: Tuesday, June 06, 2006 2:40 AM To: Donal K. Fellows Cc: Peter Lane; Marvin Theimer; ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] RE: Questions and potential changes to BES, as seen from HPC Profile point-of-view This (somewhat repetitive) discussion makes me think that we still do not have a precise enough nomenclature for the various aspects of jobs that matter in describing the architecture. I think we (the HPC and Grid Job management community) spend an inordinate amount of time talking in circles to reestablish common ground because of this... The terms are important not only to have unambiguous debates, but also to make the semantics in specifications clear, concise, and relevant. At a high level, this thread has revisited (and these are my best efforts at describing them for discussion, and not an assertion of what to call things in specifications): 1. Service management interfaces a. Possible generic control (start, stop, deploy, undeploy, etc.) b. Existing BES-specific controls (accept activities, etc.) [Has strong overlap with WSDM, CIM, et. al] 2. Generic Activity management interfaces (the main point of BES) a. Activity creation and destruction b. BES service "resource capabilities" (what can be run) c. Single and Array activity monitoring and control [Has overlap with provisioning and resource management] 3. Extended Activity management interfaces a. HPC Suspend/resume b. POSIX or Windows-specific controls c. Job File I/O d. ... (huge range of domain-specific stuff in future) [Has overlap with existing "legacy" grid services] 4. Extensibility Discovery/Negotiation mechanisms And I will add: 5. Applications/Services By this last one, I mean that BES is envisioned to support management of complex services through extensions. I think there is a distinction to be made between the "business" end of such a service (5) and its management interface as a special kind of activity (3). I often get an uncomfortable feeling that some people are expecting the "activity EPR" to expose the business interface, while I and some others expect it to only expose the extended management interface. This extended interface might well include some discovery/bootstrapping capabilities to help find the business endpoint(s) of the activity. I will also point out that, in some sense, BES as a generic activity manager (2) ought to be capable enough to recursively act as the manager of another BES service (1a). As an analogy, a well-known BES service might allow creation of VO's and VO services. Some of these VO services might be new BES instances. It seems to me that the BES-service management interfaces (1b) of the second-tier BES should be part of the extended activity interface (3) of those BES-types activities when they exist within the first tier's environment. On the other hand, the ability to create activities in the second tier is part of the "business" interface of those BES services. In my opinion, it does not belong in the activity management view of the service within the first tier BES service! [Note, I mention this to try to cut deeply into the distinctions between interfaces. I realize recursive use of BES is not on the top of everyone's must-have scenario list.] Also, as Marvin and Donal are pointing out, we need to consider various roles and scenarios. On the other hand, Donal's comments lead towards that slippery slope of assuming too much about implementation, e.g. that authorization is scoped at endpoints. Do we wish to assume this? I know it is easy in Globus land, but we can also handle per-operation authorization. We are, as implementers, a little hog-tied as far as providing finer authorization scoping, e.g. access controls on specific message contents. It is a tricky area, fraught with usability problems and non-interoperability. karl -- Karl Czajkowski karlcz@univa.com
Hi; I'm not advocating an array interface on individual activities. Rather I'm asking whether there is some way of avoiding having two interfaces be defined that need to be kept in sync. One possibility is to have a single interface that takes either no activityID as a parameter or an array of them. The former case represents interaction with a particular activity through its AED while the latter represents interaction with a BES services about multiple activities (i.e. one or more). I not so sure that there will be substantive differences between the interface provided by BES and the one provided by each activity directly. In a high throughput compute cluster scenario -- which is a VERY common case -- users -- and not just sys admins -- will routinely want to query and manipulate multiple activities at the same time. For example, modern UIs show multiple activities and hence must query multiple ones. Notifications can substantially reduce this traffic, but there are still plenty of occasions where a user's client software will want to query multiple activities. As another example, users will routinely do things like cancel all the activities related to some larger workflow, such as a parameter sweep. So I would argue that the canonical interface to an execution subsystem should be sets of activities rather than individual activities. This point-of-view would imply that the interface on a single activity is really just a special case of the interface for dealing with sets of activities. Of course, the special case of interacting/querying a single activity is also a very common one. But I would argue that both scenarios need to be equally well supported. Marvin. ________________________________ From: Peter Lane [mailto:lane@mcs.anl.gov] Sent: Monday, June 05, 2006 5:20 PM To: Marvin Theimer Cc: ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] RE: Questions and potential changes to BES, as seen from HPC Profile point-of-view On Jun 5, 2006, at 5:52 PM, Marvin Theimer wrote: Hi; A couple more questions: * This leads me to wonder whether a separate WSDL for activity interaction is really appropriate since it will require that the two specifications be kept continuously synchronized and one will effectively be a strict subset of the other. I think there are enough differences between the two interfaces that a separate WSDL is warranted if not required. If the activity interface is a subset of the BES interface, then I think the design is flawed. Operations and properties that operate on/expose data for one and only one activity should only be in the activity interface. Array operations and compute resource data properties should only be in the BES interface. An implementation of the BES interface will potentially support multiple resources corresponding to the various LocalResourceManagerType values (not necessarily a one-to-one relationship). Peter
On Jun 7, 2006, at 4:35 PM, Marvin Theimer wrote:
Hi;
I'm not advocating an array interface on individual activities. Rather I'm asking whether there is some way of avoiding having two interfaces be defined that need to be kept in sync. One possibility is to have a single interface that takes either no activityID as a parameter or an array of them. The former case represents interaction with a particular activity through its AED while the latter represents interaction with a BES services about multiple activities (i.e. one or more). I just don't see what you're gaining from this. This seems like a hack so that we can avoid the responsibility of defining and keeping an activity interface in sync with the BES spec. If we're worried about keeping two specs in sync, then perhaps we need to simply define the activity interface in the same spec. I don't think this will be a huge problem if the OGSA-BES WG does both specs, though. I not so sure that there will be substantive differences between the interface provided by BES and the one provided by each activity directly. In a high throughput compute cluster scenario -- which is a VERY common case -- users -- and not just sys admins -- will routinely want to query and manipulate multiple activities at the same time. That's fine, but don't munge the factory and activity interfaces together just because this is a common operating scenario. I'll concede that there will basically be a one-to-one mapping between activity operations and similar array operations on the factory, but I don't see anything wrong with this. For example, modern UIs show multiple activities and hence must query multiple ones. Notifications can substantially reduce this traffic, but there are still plenty of occasions where a user's client software will want to query multiple activities. As another example, users will routinely do things like cancel all the activities related to some larger workflow, such as a parameter sweep. So I would argue that the canonical interface to an execution subsystem should be sets of activities rather than individual activities. This point-of-view would imply that the interface on a single activity is really just a special case of the interface for dealing with sets of activities. Of course, the special case of interacting/querying a single activity is also a very common one. But I would argue that both scenarios need to be equally well supported. I'm in total agreement with the idea of supporting both. I just strongly disagree that we should merge the two just to gain some slight convenience of only having to deal with one interface.
Marvin.
From: Peter Lane [mailto:lane@mcs.anl.gov] Sent: Monday, June 05, 2006 5:20 PM To: Marvin Theimer Cc: ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] RE: Questions and potential changes to BES, as seen from HPC Profile point-of-view
On Jun 5, 2006, at 5:52 PM, Marvin Theimer wrote:
Hi;
A couple more questions:
· This leads me to wonder whether a separate WSDL for activity interaction is really appropriate since it will require that the two specifications be kept continuously synchronized and one will effectively be a strict subset of the other.
I think there are enough differences between the two interfaces that a separate WSDL is warranted if not required. If the activity interface is a subset of the BES interface, then I think the design is flawed. Operations and properties that operate on/expose data for one and only one activity should only be in the activity interface. Array operations and compute resource data properties should only be in the BES interface. An implementation of the BES interface will potentially support multiple resources corresponding to the various LocalResourceManagerType values (not necessarily a one-to-one relationship).
Peter
· Security design. I would argue that this is a WS binding issue and is out of scope of
· Interface for directly controlling/manipulating an activity once it has been created. Yes, I was concerned about this as well but never said anything. Generic operations for manipulating or introspecting activities (i.e. factory operation that takes an activity ID) can be useful, but I definitely agree that there should be an activity service interface with operations that imply the activity via the EPR. · Extensibility:
· Given that BES has bought into the notion of an extensible activity state diagram, it needs to also normatively define how clients can learn of the extensions that a given BES service supports. Is that something that will be added to the BES specification? Or will the specification point to some other place where notions of extensibility are defined more generically? (Personally, I’d vote for the former approach.) Good point. I'd also prefer adding something explicitly to the spec. · Is the “base case” for BES now fig.2, which shows states of {new, pending, running, canceled, failed, finished}?
· Previously included states, such as Execution-Pending, will presumably be defined in suitable extension profiles? My understanding is that this kind of thing is out of scope of the spec. I would assume that if there were any defacto standards being used in the community that this would at most be a separate OGSA-BES WG spec. · Given that suspension is no longer in the base design, presumably the createInSuspendedState parameter to CreateActivityFromJSDL should disappear? Yes, this is something I picked on as well. It should be a JSDL extension anyway, but certainly this becomes mandatory if suspension is an interface extension. · RequestActivityStateChange: I believe this operation will pose challenges in an extensible design. The current design is imperative by nature: it specifies an explicit state to move an activity to. However, a client who does not know of all the extensions that a BES service implements may not know how to pick the appropriate state to transition to. It seems better to introduce a more declarative approach in which clients specify “actions” they wish to occur, such as ‘CancelActivity’. This approach would allow the BES service to make the appropriate state transition in response to a desired action requested by a client. Yes, this is what I argued as well albeit from a slightly different angle. · Information model:
· JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I’m wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification. I think I complained about this here as well. Certainly I've complained to the JSDL people and the ESI people about this. The JDSL
· Is there any notion of specifying that all compute nodes should have the same value for some attribute (e.g. CPU architecture, CPU speed, NIC card)? This seems to be missing from the JSDL specification, but seems very important for BES if it is to support things like compute clusters. This relates to my last comment. All these related specs have assumed
All this leads to the question of whether BES will have a notion of extending the information model that is supplied. If so, then that leads to the question of what the base case should be and whether it should include a smaller set of things than is currently listed in the spec. Good point. · These operations seem to require a different set of authorization credentials than the other interface operations since they should be invoked by system administrators rather than random users. How will that work, given that these operations are in the same WSDL as the other operations? Wouldn’t this argue for moving these operations to a separate system management interface? This is why I suggested that security was out of scope. The Globus Toolkit, for example, has no problems specifying different security
On Jun 5, 2006, at 5:19 PM, Marvin Theimer wrote: the spec. I think the best that could be said would be that some sort of security is implied since the implementation has to be able to differentiate between users. people answered something about keeping it simple for now. My guess is that people see it for the daunting task it is an prefer not to address it. I'm not criticizing anyone for this. It's just my take on the problem. that the node composition is homogenous. parameters for different operations. It's all proprietary configuration, though, and probably should be. That said, I believe I said something similar to this in my original comments on the spec. This sort of thing feels to me as well like it shouldn't be in a basic execution service and should be an implementation detail as to whether the management interface is exposed as well as the BES interface.
· Given that BES seems to have bought into the notion of extensibility, should the base case be a “non-array” one? For example, currently if you want to handle a fault for a RequestActivityStateChange operation on a single activity you need to look inside the returned array of results to see if a fault infoset was returned. All the exception handling machinery that modern tooling provides can’t get used because RequestActivityStateChange never returns an actual fault message (as compared to a fault infoset for the appropriate array elements that are returned. Good point, and more reason to have a separate activity interface. · Since JSDL documents are self-describing, a BES service can figure out by inspection whether the job description infoset parameter to CreateActivityFromJSDL is JSDL or something else. This would seem to imply that naming the operation CreateActivity would lose no information and would allow for transparent extension to other job description infoset simply by using them (assuming they are self-describing). Yes! I argued to chop the suffix as well, but this makes the point better. · Container attributes that I have questions about:
· LocalResourceManagerType: where do these get defined normatively? I would argue against explicitly defining values for this. It eliminates much of the flexibility the implementation has in terms of what local resource managers it can support. · Job Credential Service and File Credential Service: these imply a specific security model. Given that security is undefined in the BES spec, is this appropriate – especially given the rather vague definition of both? I think the simple statement that some sort of security is required should be sufficient to justify generic EPR properties.
Peter
Peter Lane wrote:
Marvin Theimer wrote:
· JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I’m wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification.
I think I complained about this here as well. Certainly I've complained to the JSDL people and the ESI people about this. The JDSL people answered something about keeping it simple for now. My guess is that people see it for the daunting task it is an prefer not to address it. I'm not criticizing anyone for this. It's just my take on the problem.
JSDL describes the resources that are to be allocated to the job (err, activity) and it is not designed to handle the description of the resources allocated to the container. This matters for a whole bunch of reasons (some unrelated to this discussion) but what is crucial here is that the case you're describing only matters if you're constructing an atomic activity that is itself inherently heterogenous. Do users do that? Or are they happier with describing their task as a set of coordinated (and presumably isochronous) activities with a different JSDL-style resource requirement for each? If, as I believe, they really want the latter, we can punt the majority of the heterogenous resource mess and focus on easier things. On the other hand, there is still a need to describe heterogenous clusters; they're not going away soon. The easiest way is probably to allow the BES to publish something like multiple JSDL Resources elements, one for each kind of node in the cluster, and for the composition rule for these separate elements to be "I have *all* of these". The only complication with that approach is that it doesn't easily support the case where you want to just use resources at the lowest common denominator level, making resource matching more complex. Oh well. Donal.
Donal: The main concern, I think, is that JSDL is "so close" to having the right terminology for both levels. It would be a shame to lose focus and have to repeat the effort again later. I do not think the heterogeneous solution is a simple "bag of JSDL documents"... Consider this: within my heterogeneous activity set, I do not just want to compose two independent constraint systems. Rather, I want to express some global (or even coupled!) goals about resources allocated to more than one homogeneous array. As I tried to propose to JSDL-WG long ago, I think there is a simple hierarchical generalization of the jsdl:Resources model, wherein the "leaves" are homogeneous array constraint models, and the whole tree is an aggregate constraint model. What I mean by the above: -- A "global" goal is a generalization of the current JSDL "total" constraint, i.e. a sum over a larger set of resources. -- A "coupled" goal is a restriction of the current JSDL model, as Marvin suggested. For example: here is my general range of per-resource values, but I need symmetry. Or, this coupled application is malleable in time/space, but the two coupled codes need to be balanced according to some linear/quadratic/ whatever relation. We have examined this problem a bit for some other work, and we think a modest refactoring of JSDL would make it very valuable here. Namely, I think the "ontological" stuff regarding to sub-resources has been nicely organized. However, there is some redundancy in the "single/total" constraint model. I think a more general composition of the core concept, e.g. "memory", and a scoping construct, e.g. "per resource," "sum over resource array," or "sum over heterogeneous aggregate" would be better. I haven't thought about the coupled cases much, so I do not embarrass myself with short English phrases for them :-) We can reuse the core JSDL concepts for naming and applying units to sub-resources. The set of scoping constructs would probably be expanded over time. The hierarchy of the complex resource description would be used to group the resource constraints and provide hygienic scoping of these more elaborate goal constraints. The idea I am advocating is to have one general constraint model that is hopefully neutral, such that it can be mapped to the local constraint model of various schedulers or meta-schedulers. No single service would necessarily implement the fully general model, but would rather support profiled subsets of this model. The benefit is that the different profiles are all related by a common model, and it is feasible to consider more general solvers/profiles being introduced over time. karl -- Karl Czajkowski karlcz@univa.com
Karl Czajkowski wrote:
The main concern, I think, is that JSDL is "so close" to having the right terminology for both levels. It would be a shame to lose focus and have to repeat the effort again later. I do not think the heterogeneous solution is a simple "bag of JSDL documents"...
I think this is getting rapidly off-topic for BES. ;-) We want to have at most a simple model for basic execution. Or even no resource model at all. There's no particular reason for BES to publish any information about the resources that it is in front of; that is the responsibility of the information service infrastructure and (on the consumption side) resource selection services. Requiring the use of a published information model (let alone one as complex as you're proposing) precludes many possible implementations for what are effectively non-functional reasons. And most (probably way over 99%) of the activity on the grid is, and will continue to be, simple jobs/activities. Too much effort, not enough return.
I think a more general composition of the core concept, e.g. "memory", and a scoping construct, e.g. "per resource," "sum over resource array," or "sum over heterogeneous aggregate" would be better. I haven't thought about the coupled cases much, so I do not embarrass myself with short English phrases for them :-)
What about clusters that provide a shared memory (OpenMP) model; do we have to introduce another modifier to deal with that? What will the term mean when applied to the other resources? And the next extra thing that wasn't originally thought of? What happens if a resource may be a member of multiple heterogenous aggregates simultaneously, with some defined by the user's VO? Can we get away without opening this high-pressure can of worms? :-)
The idea I am advocating is to have one general constraint model that is hopefully neutral, such that it can be mapped to the local constraint model of various schedulers or meta-schedulers. No single service would necessarily implement the fully general model, but would rather support profiled subsets of this model. The benefit is that the different profiles are all related by a common model, and it is feasible to consider more general solvers/profiles being introduced over time.
I don't think it is a practical proposal for interoperability. Either there's a common profile with no significant extensions to it or there's a standard semantics for reconciling terms from one part of the term-space with terms from another part. The first is pretty much equivalent to ignoring your proposal (except with more matching complexity! Yay!) and the second is extremely difficult. The other alternative (that I do not want) is for there to be systems out there that claim to support the same standards, but which profile them in such different ways that interoperation is nigh-on impossible. That's where we are right now, in effect. [I was writing a much longer response to your message, but decided that a total point-by-point rebuttal was foolish. And this is the wrong place to make a fool of myself.] Donal.
On Jun 06, Donal K. Fellows modulated:
Karl Czajkowski wrote:
The main concern, I think, is that JSDL is "so close" to having the right terminology for both levels. It would be a shame to lose focus and have to repeat the effort again later. I do not think the heterogeneous solution is a simple "bag of JSDL documents"...
I think this is getting rapidly off-topic for BES. ;-)
Sorry, I didn't notice this thread was only on the BES list... for some strange reason, I thought it was copied to JSDL. (Is there any point in my forwarding the previous message to JSDL-WG?) I am not following some of your points, because I thought the topic was the resource constraint model used in JSDL and thus BES jobs as they stand today. It was not about publishing resource models to support discovery, though I do appreciate that there ought to be a complementary discovery system for _any_ rich resource management capability. I thought the submission constraint model was the question raised by Marvin too, with respect to HPC scenarios. BES cannot be basic and serve any purpose behind exotic selection and planning services, if the meaning of a job cannot be formulated precisely. Surely, the idea isn't to do fancy planning and then strip away all the important constraints, throw the job over the fence to BES, and hope for the best! So, either BES needs to capture the application-level constraints, or it needs to capture some intermediate "placement directives" generated by the scheduler intervening between user and BES.
... I don't think it is a practical proposal for interoperability. Either there's a common profile with no significant extensions to it or there's a standard semantics for reconciling terms from one part of the term-space with terms from another part. The first is pretty much equivalent to ignoring your proposal (except with more matching complexity! Yay!) and the second is extremely difficult. The other alternative (that I do not want) is for there to be systems out there that claim to support the same standards, but which profile them in such different ways that interoperation is nigh-on impossible. That's where we are right now, in effect.
I'm not sure I buy the specific three categories you've described. But, is it fair to summarize as follows: you doubt there is a general model that covers the more complex constraints out there in a way that is interoperable, because either no intermediate profiles exist, or implementations will not actually support the profiles? You believe that the only feasible point for interoperability is the current resource array construct in JSDL 1.0, which supports symmetry only with exact (single-valued) resource constraints and which supports non-deterministic heterogeneity with variable (range-valued) resource constraints? Personally, I think that if there is no hope for some interoperable "extension profiles", there is not much point in BES nor in the HPC standard Marvin is championing. There are already a number of good-enough systems which capture the easy case. If we want to be this conservative in covering only the low-hanging fruit, why not just formally document an existing de-facto standard interface, perhaps filing off the rough edges where too many implementation techniques are assumed? karl -- Karl Czajkowski karlcz@univa.com
Karl Czajkowski wrote:
Sorry, I didn't notice this thread was only on the BES list... for some strange reason, I thought it was copied to JSDL. (Is there any point in my forwarding the previous message to JSDL-WG?)
I think it's more talking about general information models of grid systems (way out of scope of JSDL too) so probably ought to go to the main OGSA list. That's where the info-modellers hang out, I think. Donal.
Hi; Please remember the goal of my original email: what to do for the HPC profile, which is starting out with defining a very simple base case and then working its way up from there. I don't want to get into discussions about general information models. That's why this is a BES discussion (and shortly I will take it to the jsdl mailing list as well). Marvin. -----Original Message----- From: owner-ogsa-bes-wg@ggf.org [mailto:owner-ogsa-bes-wg@ggf.org] On Behalf Of Donal K. Fellows Sent: Tuesday, June 06, 2006 8:11 AM To: Karl Czajkowski Cc: ogsa-bes-wg@ggf.org Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view Karl Czajkowski wrote:
Sorry, I didn't notice this thread was only on the BES list... for some strange reason, I thought it was copied to JSDL. (Is there any point in my forwarding the previous message to JSDL-WG?)
I think it's more talking about general information models of grid systems (way out of scope of JSDL too) so probably ought to go to the main OGSA list. That's where the info-modellers hang out, I think. Donal.
Hi; I don't agree with the "there is no hope" argument. A set of JSDL documents can describe things like clusters from the point-of-view of available resource descriptions. A few extensions to jsdl will allow the most common forms of cross-cutting job requirements, such as "all cpus must be the same". You can probably hit 80-90% of the common cases with this incremental approach. If a well-defined way is included to allow evolution and extension then the simple design doesn't prevent future forward progress, yet still enables current progress. Marvin. -----Original Message----- From: Karl Czajkowski [mailto:karlcz@univa.com] Sent: Tuesday, June 06, 2006 7:48 AM To: Donal K. Fellows Cc: Peter Lane; Marvin Theimer; ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view On Jun 06, Donal K. Fellows modulated:
Karl Czajkowski wrote:
The main concern, I think, is that JSDL is "so close" to having the right terminology for both levels. It would be a shame to lose focus and have to repeat the effort again later. I do not think the heterogeneous solution is a simple "bag of JSDL documents"...
I think this is getting rapidly off-topic for BES. ;-)
... I don't think it is a practical proposal for interoperability. Either there's a common profile with no significant extensions to it or
Sorry, I didn't notice this thread was only on the BES list... for some strange reason, I thought it was copied to JSDL. (Is there any point in my forwarding the previous message to JSDL-WG?) I am not following some of your points, because I thought the topic was the resource constraint model used in JSDL and thus BES jobs as they stand today. It was not about publishing resource models to support discovery, though I do appreciate that there ought to be a complementary discovery system for _any_ rich resource management capability. I thought the submission constraint model was the question raised by Marvin too, with respect to HPC scenarios. BES cannot be basic and serve any purpose behind exotic selection and planning services, if the meaning of a job cannot be formulated precisely. Surely, the idea isn't to do fancy planning and then strip away all the important constraints, throw the job over the fence to BES, and hope for the best! So, either BES needs to capture the application-level constraints, or it needs to capture some intermediate "placement directives" generated by the scheduler intervening between user and BES. there's
a standard semantics for reconciling terms from one part of the term-space with terms from another part. The first is pretty much equivalent to ignoring your proposal (except with more matching complexity! Yay!) and the second is extremely difficult. The other alternative (that I do not want) is for there to be systems out there that claim to support the same standards, but which profile them in such different ways that interoperation is nigh-on impossible. That's where we are right now, in effect.
I'm not sure I buy the specific three categories you've described. But, is it fair to summarize as follows: you doubt there is a general model that covers the more complex constraints out there in a way that is interoperable, because either no intermediate profiles exist, or implementations will not actually support the profiles? You believe that the only feasible point for interoperability is the current resource array construct in JSDL 1.0, which supports symmetry only with exact (single-valued) resource constraints and which supports non-deterministic heterogeneity with variable (range-valued) resource constraints? Personally, I think that if there is no hope for some interoperable "extension profiles", there is not much point in BES nor in the HPC standard Marvin is championing. There are already a number of good-enough systems which capture the easy case. If we want to be this conservative in covering only the low-hanging fruit, why not just formally document an existing de-facto standard interface, perhaps filing off the rough edges where too many implementation techniques are assumed? karl -- Karl Czajkowski karlcz@univa.com
Marvin Theimer wrote:
I don't agree with the "there is no hope" argument. A set of JSDL documents can describe things like clusters from the point-of-view of available resource descriptions. A few extensions to jsdl will allow the most common forms of cross-cutting job requirements, such as "all cpus must be the same". You can probably hit 80-90% of the common cases with this incremental approach. If a well-defined way is included to allow evolution and extension then the simple design doesn't prevent future forward progress, yet still enables current progress.
I agree with the above paragraph, but I'd like to point out that the 80-90% of cases above probably should refer to distinct cases, and that it will in fact cover a much larger fraction of job throughput. There is going to be a lot of cases out there where people run the same job, time and time again (presumably with different data, though I've seen cases where that wasn't true, which was stupid). I hope that we end up with solutions that are both capable of properly solving these basic cases (I suppose you could call them the interop use-case set) and yet which do not preclude extension to taking on the bigger picture. Both are important. OK, taking on the bigger picture may need further extension profiles and new services, but that's not a big deal in my view. Donal.
Hi; I think it's accurate to say that we're in complete agreement with each other. :-) I will be sending an email to the JSDL mailing list later today that will be similar to the original one I sent out a couple of days ago to the BES mailing list. That is, it will present some questions that I have about JSDL (from the HPC profile point-of-view) as well as a straw man proposal for a set of restrictions that the HPC profile will probably want to impose on JSDL documents it will accept as well as a set of changes/extensions that it will propose for future versions of JSDL. I will try to incorporate what I've learned from the BES-oriented discussions of the past few days. Marvin. -----Original Message----- From: Donal K. Fellows [mailto:donal.k.fellows@manchester.ac.uk] Sent: Thursday, June 08, 2006 5:39 AM To: Marvin Theimer Cc: ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view Marvin Theimer wrote:
I don't agree with the "there is no hope" argument. A set of JSDL documents can describe things like clusters from the point-of-view of available resource descriptions. A few extensions to jsdl will allow the most common forms of cross-cutting job requirements, such as "all cpus must be the same". You can probably hit 80-90% of the common cases with this incremental approach. If a well-defined way is included to allow evolution and extension then the simple design doesn't prevent future forward progress, yet still enables current progress.
I agree with the above paragraph, but I'd like to point out that the 80-90% of cases above probably should refer to distinct cases, and that it will in fact cover a much larger fraction of job throughput. There is going to be a lot of cases out there where people run the same job, time and time again (presumably with different data, though I've seen cases where that wasn't true, which was stupid). I hope that we end up with solutions that are both capable of properly solving these basic cases (I suppose you could call them the interop use-case set) and yet which do not preclude extension to taking on the bigger picture. Both are important. OK, taking on the bigger picture may need further extension profiles and new services, but that's not a big deal in my view. Donal.
Hi; If we go the way you're suggesting then the HPC profile will need to include BES, JSDL, and as-yet ill-defined specs for an information service infrastructure and resource selection services. That implies that the HPC profile WG will end up defining a simple version of such as a specification that is separate from everything else currently going on in GGF. That's OK by me, and in fact, my original email starting this thread proposed the creation of a QueryResources operation that would supply the requisite information. I included it in my straw man proposal for changes to BES, but would be happy to push it to a separate interface (including the associated resource information model). I agree that we should stay away from general constraint models (at least for now) and certainly should stay away from having different systems employ non-standard profile interpretations. My inclination would be to start with your suggest for a set of JSDL descriptions as the way to expose execution systems such as clusters, either as a replacement for the current BES information model or as a separate QueryResources interface that BES services would export. Marvin. -----Original Message----- From: Donal K. Fellows [mailto:donal.k.fellows@manchester.ac.uk] Sent: Tuesday, June 06, 2006 7:15 AM To: Karl Czajkowski Cc: Peter Lane; Marvin Theimer; ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view Karl Czajkowski wrote:
The main concern, I think, is that JSDL is "so close" to having the right terminology for both levels. It would be a shame to lose focus and have to repeat the effort again later. I do not think the heterogeneous solution is a simple "bag of JSDL documents"...
I think this is getting rapidly off-topic for BES. ;-) We want to have at most a simple model for basic execution. Or even no resource model at all. There's no particular reason for BES to publish any information about the resources that it is in front of; that is the responsibility of the information service infrastructure and (on the consumption side) resource selection services. Requiring the use of a published information model (let alone one as complex as you're proposing) precludes many possible implementations for what are effectively non-functional reasons. And most (probably way over 99%) of the activity on the grid is, and will continue to be, simple jobs/activities. Too much effort, not enough return.
I think a more general composition of the core concept, e.g. "memory", and a scoping construct, e.g. "per resource," "sum over resource array," or "sum over heterogeneous aggregate" would be better. I haven't thought about the coupled cases much, so I do not embarrass myself with short English phrases for them :-)
What about clusters that provide a shared memory (OpenMP) model; do we have to introduce another modifier to deal with that? What will the term mean when applied to the other resources? And the next extra thing that wasn't originally thought of? What happens if a resource may be a member of multiple heterogenous aggregates simultaneously, with some defined by the user's VO? Can we get away without opening this high-pressure can of worms? :-)
The idea I am advocating is to have one general constraint model that is hopefully neutral, such that it can be mapped to the local constraint model of various schedulers or meta-schedulers. No single service would necessarily implement the fully general model, but would rather support profiled subsets of this model. The benefit is that the different profiles are all related by a common model, and it is feasible to consider more general solvers/profiles being introduced over time.
I don't think it is a practical proposal for interoperability. Either there's a common profile with no significant extensions to it or there's a standard semantics for reconciling terms from one part of the term-space with terms from another part. The first is pretty much equivalent to ignoring your proposal (except with more matching complexity! Yay!) and the second is extremely difficult. The other alternative (that I do not want) is for there to be systems out there that claim to support the same standards, but which profile them in such different ways that interoperation is nigh-on impossible. That's where we are right now, in effect. [I was writing a much longer response to your message, but decided that a total point-by-point rebuttal was foolish. And this is the wrong place to make a fool of myself.] Donal.
On Jun 6, 2006, at 3:06 AM, Donal K. Fellows wrote:
Peter Lane wrote:
Marvin Theimer wrote:
· JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I’m wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification.
I think I complained about this here as well. Certainly I've complained to the JSDL people and the ESI people about this. The JDSL people answered something about keeping it simple for now. My guess is that people see it for the daunting task it is an prefer not to address it. I'm not criticizing anyone for this. It's just my take on the problem.
JSDL describes the resources that are to be allocated to the job (err, activity) and it is not designed to handle the description of the resources allocated to the container.
Right, I meant something a little different when I mentioned JSDL. JSDL doesn't allow for describing complex requirements. For example, one might need a complex set of resources to run a distributed application on a cluster. The best JSDL can do, IIRC, is to allocate N homogenous resources. There's no way you can say, for example, "give me two IA64 machines, two x86_64 machines, and two i386 machines". This is a very real requirement by users of GRAM. Peter
Peter Lane wrote:
Right, I meant something a little different when I mentioned JSDL. JSDL doesn't allow for describing complex requirements. For example, one might need a complex set of resources to run a distributed application on a cluster. The best JSDL can do, IIRC, is to allocate N homogenous resources. There's no way you can say, for example, "give me two IA64 machines, two x86_64 machines, and two i386 machines". This is a very real requirement by users of GRAM.
Hmm, that's not something I've ever seen in our job flows. On the other hand, we've found that our users aren't interested in specifying what sort of processor they use at all. They focus on the application instead and if that's part of the set described as supported by the container (ignoring questions of how this is discovered for the moment) then it hardly matters what the underlying CPU or OS is. (OK, both of those can certainly make a difference to the relevant performance metrics, but we'd rather state that we support a certain level of performance instead of providing some information that people can use to try to infer what they're really interested in.) The only real use for matching on CPU and OS is if you're staging in the binary executable to run as part of the job. Physicists seem to like to do that; seems to be a peculiarity of that community. We don't see the same thing to anything like as great an extent among other scientists and engineers (and we have very little CS in our usage profile, though if we did I'd expect them to be similar to the physicists). On the other hand, the physicists seem to prefer to buy their own clusters too. Oh, you're supporting physicists? :-) Donal.
Yes, GRAM users often (almost always) provide their own executable, either because they are CS-inclined, or because the "user" for GRAM is actually another piece of middleware that provides the more abstract interface you describe. BTW, this point of Peter's is what I thought everyone was talking about regarding heterogeneous resource descriptions, and that is what I was trying to address in this thread w.r.t. JSDL constraint models. It seems one could read Marvin's original question as being about resource discovery, but since he mentioned JSDL, I just assumed we were all talking about constraints in the job description. karl -- Karl Czajkowski karlcz@univa.com
Hi; My email was focused on both job description and resource discovery. I need both to implement the HPC profile specification. Marvin. -----Original Message----- From: owner-ogsa-bes-wg@ggf.org [mailto:owner-ogsa-bes-wg@ggf.org] On Behalf Of Karl Czajkowski Sent: Wednesday, June 07, 2006 2:04 AM To: Donal K. Fellows Cc: Peter Lane; ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view Yes, GRAM users often (almost always) provide their own executable, either because they are CS-inclined, or because the "user" for GRAM is actually another piece of middleware that provides the more abstract interface you describe. BTW, this point of Peter's is what I thought everyone was talking about regarding heterogeneous resource descriptions, and that is what I was trying to address in this thread w.r.t. JSDL constraint models. It seems one could read Marvin's original question as being about resource discovery, but since he mentioned JSDL, I just assumed we were all talking about constraints in the job description. karl -- Karl Czajkowski karlcz@univa.com
"Staging in one's own executable" is also a requirement when deploying services, another important use case for GRAM. Ian. At 09:23 AM 6/7/2006 +0100, Donal K. Fellows wrote:
The only real use for matching on CPU and OS is if you're staging in the binary executable to run as part of the job. Physicists seem to like to do that; seems to be a peculiarity of that community. We don't see the same thing to anything like as great an extent among other scientists and engineers (and we have very little CS in our usage profile, though if we did I'd expect them to be similar to the physicists). On the other hand, the physicists seem to prefer to buy their own clusters too.
_______________________________________________________________ Ian Foster, Director, Computation Institute Argonne National Laboratory & University of Chicago Argonne: MCS/221, 9700 S. Cass Ave, Argonne, IL 60439 Chicago: Rm 405, 5640 S. Ellis Ave, Chicago, IL 60637 Tel: +1 630 252 4619. Web: www.ci.uchicago.edu. Globus Alliance: www.globus.org.
Much like Donal, I don't see this requirement overmuch in our user base (squarely in the "HPC Profile" space). Mostly users are submitting jobs that can either utilize any architecture in their environment (the admins help "make this so"), or they work within a homogeneous set (x86_64 running Linux), so the JSDL document is good enough for a lot of these cases. They'll also use queues and "host groups" to makes these kinds of selections for them. Even the parallel jobs are almost always constrained such that all tasks are running on a single architecture. I realize that JSDL has some limitations in describing resources at this time. I was also under the impression that some work was being done to define some kind of information model based on generic name/value pairs (where some names could be standardized by convention), but so that you could represent any type of attribute. This would go along with some kind of grammar to put stuff together into expressions that could be evaluated for the purposes of resource requirement evaluation. I'm willing to live with the JSDL v1.0 limitations, given that there will be some changes in the near future if we work at it. -- Chris On 07/6/06 01:23, "Donal K. Fellows" <donal.k.fellows@manchester.ac.uk> wrote:
Peter Lane wrote:
Right, I meant something a little different when I mentioned JSDL. JSDL doesn't allow for describing complex requirements. For example, one might need a complex set of resources to run a distributed application on a cluster. The best JSDL can do, IIRC, is to allocate N homogenous resources. There's no way you can say, for example, "give me two IA64 machines, two x86_64 machines, and two i386 machines". This is a very real requirement by users of GRAM.
Hmm, that's not something I've ever seen in our job flows. On the other hand, we've found that our users aren't interested in specifying what sort of processor they use at all. They focus on the application instead and if that's part of the set described as supported by the container (ignoring questions of how this is discovered for the moment) then it hardly matters what the underlying CPU or OS is. (OK, both of those can certainly make a difference to the relevant performance metrics, but we'd rather state that we support a certain level of performance instead of providing some information that people can use to try to infer what they're really interested in.)
The only real use for matching on CPU and OS is if you're staging in the binary executable to run as part of the job. Physicists seem to like to do that; seems to be a peculiarity of that community. We don't see the same thing to anything like as great an extent among other scientists and engineers (and we have very little CS in our usage profile, though if we did I'd expect them to be similar to the physicists). On the other hand, the physicists seem to prefer to buy their own clusters too.
Oh, you're supporting physicists? :-)
Donal.
Peter Lane wrote:
Right, I meant something a little different when I mentioned JSDL. JSDL doesn't allow for describing complex requirements. For example, one might need a complex set of resources to run a distributed application on a cluster. The best JSDL can do, IIRC, is to allocate N homogenous resources. There's no way you can say, for example, "give me two IA64 machines, two x86_64 machines, and two i386 machines". This is a very real requirement by users of GRAM.
Hmm, that's not something I've ever seen in our job flows. On the other hand, we've found that our users aren't interested in specifying what sort of processor they use at all. They focus on the application instead and if that's part of the set described as supported by the container (ignoring questions of how this is discovered for the moment) then it hardly matters what the underlying CPU or OS is. (OK, both of those can certainly make a difference to the relevant performance metrics, but we'd rather state that we support a certain level of performance instead of providing some information that people can use to try to infer what they're really interested in.)
The only real use for matching on CPU and OS is if you're staging in
Hi; I agree with Chris' characterization of the most common case. I think the JSDL community will eventually want to support additional elements that allow specification of the sorts of things that have come up in this email thread -- especially since there is no standard, interoperable definition of what queues should be supported in a larger system that spans multiple job schedulers and multiple compute clusters. Luckily, it seems (to me) that most of these extensions can be added in a very localized manner that is transparent to most of the grid infrastructure. I do believe that the HPC profile work will require a richer resource description capability for describing available resources than the current one in BES. I think Donal's suggestion of an array of jsdl documents seems like a good starting point for discussion of such a topic. Marvin. -----Original Message----- From: owner-ogsa-bes-wg@ggf.org [mailto:owner-ogsa-bes-wg@ggf.org] On Behalf Of Christopher Smith Sent: Wednesday, June 07, 2006 10:44 AM To: Donal K. Fellows; Peter Lane Cc: ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view Much like Donal, I don't see this requirement overmuch in our user base (squarely in the "HPC Profile" space). Mostly users are submitting jobs that can either utilize any architecture in their environment (the admins help "make this so"), or they work within a homogeneous set (x86_64 running Linux), so the JSDL document is good enough for a lot of these cases. They'll also use queues and "host groups" to makes these kinds of selections for them. Even the parallel jobs are almost always constrained such that all tasks are running on a single architecture. I realize that JSDL has some limitations in describing resources at this time. I was also under the impression that some work was being done to define some kind of information model based on generic name/value pairs (where some names could be standardized by convention), but so that you could represent any type of attribute. This would go along with some kind of grammar to put stuff together into expressions that could be evaluated for the purposes of resource requirement evaluation. I'm willing to live with the JSDL v1.0 limitations, given that there will be some changes in the near future if we work at it. -- Chris On 07/6/06 01:23, "Donal K. Fellows" <donal.k.fellows@manchester.ac.uk> wrote: the
binary executable to run as part of the job. Physicists seem to like to do that; seems to be a peculiarity of that community. We don't see the same thing to anything like as great an extent among other scientists and engineers (and we have very little CS in our usage profile, though if we did I'd expect them to be similar to the physicists). On the other hand, the physicists seem to prefer to buy their own clusters too.
Oh, you're supporting physicists? :-)
Donal.
Hi; I agree that exact matching on cpu and os isn't the most important issue. What IS important is being able to say "all the same": this is really important for various kinds of MPI applications. I suspect you'll see more specification of binary executable as time goes on. As people purchase multiple clusters with somewhat different hardware and as they add new hardware to existing clusters they end up with heterogeneous environments where the exact choice of hardware really does matter. For example, running numerical libraries that have been optimized for Intel or AMD on the opposite chip type can yield substantial performance penalties. Marvin. -----Original Message----- From: owner-ogsa-bes-wg@ggf.org [mailto:owner-ogsa-bes-wg@ggf.org] On Behalf Of Donal K. Fellows Sent: Wednesday, June 07, 2006 1:24 AM To: Peter Lane Cc: ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view Peter Lane wrote:
Right, I meant something a little different when I mentioned JSDL. JSDL doesn't allow for describing complex requirements. For example, one might need a complex set of resources to run a distributed application on a cluster. The best JSDL can do, IIRC, is to allocate N homogenous
resources. There's no way you can say, for example, "give me two IA64
machines, two x86_64 machines, and two i386 machines". This is a very
real requirement by users of GRAM.
Hmm, that's not something I've ever seen in our job flows. On the other hand, we've found that our users aren't interested in specifying what sort of processor they use at all. They focus on the application instead and if that's part of the set described as supported by the container (ignoring questions of how this is discovered for the moment) then it hardly matters what the underlying CPU or OS is. (OK, both of those can certainly make a difference to the relevant performance metrics, but we'd rather state that we support a certain level of performance instead of providing some information that people can use to try to infer what they're really interested in.) The only real use for matching on CPU and OS is if you're staging in the binary executable to run as part of the job. Physicists seem to like to do that; seems to be a peculiarity of that community. We don't see the same thing to anything like as great an extent among other scientists and engineers (and we have very little CS in our usage profile, though if we did I'd expect them to be similar to the physicists). On the other hand, the physicists seem to prefer to buy their own clusters too. Oh, you're supporting physicists? :-) Donal.
Marvin Theimer wrote:
I suspect you'll see more specification of binary executable as time goes on. As people purchase multiple clusters with somewhat different hardware and as they add new hardware to existing clusters they end up with heterogeneous environments where the exact choice of hardware really does matter. For example, running numerical libraries that have been optimized for Intel or AMD on the opposite chip type can yield substantial performance penalties.
The right thing to do is to specify that you need the library using some kind of abstract name for it, and leave it to the BES container to figure out how to do that efficiently on the hardware allocated. Donal.
Hi Donal, Marvin, all, Donal K. Fellows wrote:
Marvin Theimer wrote:
I suspect you'll see more specification of binary executable as time goes on. As people purchase multiple clusters with somewhat different hardware and as they add new hardware to existing clusters they end up with heterogeneous environments where the exact choice of hardware really does matter. For example, running numerical libraries that have been optimized for Intel or AMD on the opposite chip type can yield substantial performance penalties.
The right thing to do is to specify that you need the library using some kind of abstract name for it, and leave it to the BES container to figure out how to do that efficiently on the hardware allocated.
this is already included in the BES specification as a non-formal JSDL 1.0 extension. It has been incorporated from the ESI proposal, as a "esi:Library" element (the namespace probably needs adjustment). See also BES spec v1.3, page 16 (ch. 6.1.5 JSDL 1.0 Extensions) Cheers, Michel -- Michel <dot> Drescher <at> uk <dot> fujitsu <dot> com Fujitsu Laboratories of Europe +44 20 8606 4834
Hi; That works well for pre-installed libraries and is something that should definitely be supported. However, you also have to support the case where someone provides an executable that has been compiled with either an Intel or an AMD-oriented compiler. In that case the library reference won't be applicable. Marvin. -----Original Message----- From: Michel Drescher [mailto:Michel.Drescher@uk.fujitsu.com] Sent: Thursday, June 08, 2006 6:02 AM To: Donal K. Fellows Cc: Marvin Theimer; Peter Lane; ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view Hi Donal, Marvin, all, Donal K. Fellows wrote:
Marvin Theimer wrote:
I suspect you'll see more specification of binary executable as time goes on. As people purchase multiple clusters with somewhat different hardware and as they add new hardware to existing clusters they end up with heterogeneous environments where the exact choice of hardware really does matter. For example, running numerical libraries that have been optimized for Intel or AMD on the opposite chip type can yield substantial performance penalties.
The right thing to do is to specify that you need the library using some kind of abstract name for it, and leave it to the BES container to figure out how to do that efficiently on the hardware allocated.
this is already included in the BES specification as a non-formal JSDL 1.0 extension. It has been incorporated from the ESI proposal, as a "esi:Library" element (the namespace probably needs adjustment). See also BES spec v1.3, page 16 (ch. 6.1.5 JSDL 1.0 Extensions) Cheers, Michel -- Michel <dot> Drescher <at> uk <dot> fujitsu <dot> com Fujitsu Laboratories of Europe +44 20 8606 4834
Hi; My initial email had the intended effect of generating some good discussion. This email is an attempt on my part to try to distill what I took out of those discussions. Feedback, corrections, and extensions are welcomed. The main topics that seemed to generate significant controversy were the following: * The relationship of the BES "array" operations for interacting with multiple activities in a batch manner versus the (system management) interface that should be offered via the EPRs corresponding to individual activities. * How BES should describe the available resources its associated execution container has - especially when that container might be a compute cluster or the BES service might be a meta-scheduler fronting multiple backend BES services. Most of this controversy was really around the details of the JSDL specification and how to use it rather than around BES proper. The rest of my suggestions did not seem to draw much fire (or maybe I'm just blanking it from my mind :-)). Given that, I would like to propose the following second draft of my proposal for how to change BES in order to support the HPC profile work: * Operations: * CreateActivity(jsdlDoc) --> EPR * GetActivity(EPR) --> activityState * GetActivityDescription(EPR) --> JSDL doc * CancelActivity(EPR) * For non-WSRF versions: QueryResources() --> schedulerResourcesInfoset * 'schedulerResourcesInfoset' is essentially the union of the RPs that would be exported in a WSRF-based version for describing the resources that are available for use at this BES service. Note that a BES service might also want to expose other kinds of information that would not be returned from this operation - this operation is there so that clients can determine whether or not a BES service could potentially meet their needs and is necessary for meta-scheduling scenarios. * One might argue that one could use WS-Transfer for this operation. However, since a BES service might want to export other kinds of information, this would require an extra level of indirection so that the BES service could expose which EPRs to use for retrieving which kinds of information. * Information model for describing available resources: * Define an infoset that can describe such resources. The top-level element indicates which description standard is being employed by the infoset. * Define one such description standard to be an array of JSDL descriptions for things like compute clusters. Other, richer, description standards can be defined later. * Additional topics/summary: * Simple state diagram and no notion of array operations, data staging, suspension, or notifications in base BES case. * Extensions defined as separate profiles for array operations, data staging, suspension, and notifications. * Defer the topic of how the interface for (extension) array operations should relate to the interface supplied for querying/canceling individual activities. * RequestActivityStateChange replaced by operations specifying desired actions rather than states. Base case supports activity cancellation; extensions can define additional operations (e.g. SuspendActivity). * Information model: small base set plus extensions model (which ones to include in the base set TBD) * All system management functions moved out to a separate interface. * Provenance log: (eventually) define an extension for a GetActivityProvenance operation.
Hi;
My initial email had the intended effect of generating some good discussion. This email is an attempt on my part to try to distill what I took out of those discussions. Feedback, corrections, and extensions are welcomed.
The main topics that seemed to generate significant controversy were the following:
· The relationship of the BES “array” operations for interacting with multiple activities in a batch manner versus the (system management) interface that should be offered via the EPRs corresponding to individual activities. I still don't understand what you're getting at here. I don't see
· How BES should describe the available resources its associated execution container has – especially when that container might be a compute cluster or the BES service might be a meta- scheduler fronting multiple backend BES services. Most of this controversy was really around the details of the JSDL specification and how to use it rather than around BES proper.
The rest of my suggestions did not seem to draw much fire (or maybe I’m just blanking it from my mind J).
Given that, I would like to propose the following second draft of my proposal for how to change BES in order to support the HPC profile work:
· Operations:
· CreateActivity(jsdlDoc) à EPR
· GetActivity(EPR) à activityState
· GetActivityDescription(EPR) à JSDL doc I would rather see a generic property getter method instead that takes an array of EPRs. This would model better w.r.t. WSRF's getResourceProperty, etc... methods and avoid the precedent of
On Jun 8, 2006, at 6:34 PM, Marvin Theimer wrote: this as the two interfaces are mutually exclusive. I just don't see any point of having identical operations for both the BES/activity factory service and the activity service. If the factory and activity service were the same, then fine. If they are two distinct services then tailor the interfaces. You mentioned "keeping them in sync". I still don't understand your concern here. What is it that frightens you about the BES group being able to keep the two (which are intimately related) in sync? providing individual operations for each property.
· CancelActivity(EPR)
· For non-WSRF versions: QueryResources() à schedulerResourcesInfoset It worries me that this is a non-WSRF-only operation. We should try very hard to avoid branching the spec for two different resource models. If we can't avoid it then there needs to be two separate specs IMHO. What do the WSRF services do with this? · ‘schedulerResourcesInfoset’ is essentially the union of the RPs that would be exported in a WSRF-based version for describing the resources that are available for use at this BES service. Note that a BES service might also want to expose other kinds of information that would not be returned from this operation – this operation is there so that clients can determine whether or not a BES service could potentially meet their needs and is necessary for meta-scheduling scenarios. I don't really understand what you're trying to accomplish here. You say it's a non-WSRF operation and yet this description says it's an aggregation of the WSRF RPs. Then you add that it might return other stuff as well. This seems like a "give me the serialized resource object" operation. My feeling is that if the necessary information were exposed correctly you wouldn't need this operation. · One might argue that one could use WS-Transfer for this operation. However, since a BES service might want to export other kinds of information, this would require an extra level of indirection so that the BES service could expose which EPRs to use for retrieving which kinds of information.
· Information model for describing available resources:
· Define an infoset that can describe such resources. The top-level element indicates which description standard is being employed by the infoset.
· Define one such description standard to be an array of JSDL descriptions for things like compute clusters. Other, richer, description standards can be defined later.
· Additional topics/summary:
· Simple state diagram and no notion of array operations, data staging, suspension, or notifications in base BES case.
· Extensions defined as separate profiles for array operations, data staging, suspension, and notifications.
· Defer the topic of how the interface for (extension) array operations should relate to the interface supplied for querying/ canceling individual activities.
· RequestActivityStateChange replaced by operations specifying desired actions rather than states. Base case supports activity cancellation; extensions can define additional operations (e.g. SuspendActivity).
· Information model: small base set plus extensions model (which ones to include in the base set TBD)
· All system management functions moved out to a separate interface.
· Provenance log: (eventually) define an extension for a GetActivityProvenance operation.
Hi; I still don't understand what you're getting at here. I don't see this as the two interfaces are mutually exclusive. I just don't see any point of having identical operations for both the BES/activity factory service and the activity service. If the factory and activity service were the same, then fine. If they are two distinct services then tailor the interfaces. You mentioned "keeping them in sync". I still don't understand your concern here. What is it that frightens you about the BES group being able to keep the two (which are intimately related) in sync? I'm just trying to advocate good software engineering principles here: I would prefer to avoid having to keep two interfaces in sync with each other across two different specifications. It's not a big deal and I'm religious about it. I believe that in a later email you suggest that both interfaces be kept in the same spec. That seems like a good idea unless another compelling reason should pop up that argues for keeping them separate. I would rather see a generic property getter method instead that takes an array of EPRs. This would model better w.r.t. WSRF's getResourceProperty, etc... methods and avoid the precedent of providing individual operations for each property. The challenge here is the schism between the WSRF and the WS-Transfer camps. Although that schism is now slated to be healed by the new reconciliation effort that IBM, Microsoft, et al. announced recently, that reconciliation won't be finalized for another year or two. Recall also that, although I'm proposing that the base case BES interface just deal with individual activities, one of the extensions that I expect will get added very quickly is an array interface for dealing with multiple activities in a single interaction. For the latter interface, it does not make sense to query the factory service's resource properties (in the WSRF-oriented world) or its EPR (in the WS-Transfer-oriented world) since the query is about the state of a bunch of activities, not the factory. Since I'd like to have the interface for dealing with arrays of activities and for dealing with individual activities look as similar as possible - and given that the reconciliation of WSRF and WS-Transfer has not yet been completed - I've introduced named operations for the type of state I wish to see returned. Now, you could argue that what I've done is expose a single "property" - the activityState property - and a means of retrieving all "properties" in something akin to WS-Transfer style. Based on that, you could argue that we should expose query operations for "all" the individual "properties" of an activity. Since that set is not completely defined and since we will also (at least eventually) want to support extended sets of "properties" and since there is not a reconciliation-approved way of exporting which "properties" an activity exports, I advocate that the base case of BES should aim to export a "minimal" query interface. To me, that's a way to get everything (including any extensions) and a way of getting the one common state element that we know everyone always wants to query - namely the activityState element. Please note that I have no problem with adding in additional query operations in extensions, but I would argue that the BES base case should be kept as simple as possible. (Along those lines, one might argue that we should jettison the GetActivity operation that returns just the activity's state diagram state. However, retrieving all the state that describes an activity just to get the activityState seems excessive to me, especially when thinking about how that would behave when dealing with large arrays of activities.) It worries me that this is a non-WSRF-only operation. We should try very hard to avoid branching the spec for two different resource models. If we can't avoid it then there needs to be two separate specs IMHO. What do the WSRF services do with this? We may have to have two separate specs, if people feel they need a WSRF-based version of the spec. Personally, I'd prefer a single spec that uses neither WSRF nor WS-Transfer and thereby circumvented the problem - but that's my personal opinion. The interface I've proposed essentially tries to circumvent the issue by employing named query and modification operations. I don't really understand what you're trying to accomplish here. You say it's a non-WSRF operation and yet this description says it's an aggregation of the WSRF RPs. Then you add that it might return other stuff as well. This seems like a "give me the serialized resource object" operation. My feeling is that if the necessary information were exposed correctly you wouldn't need this operation. This again comes down to the WSRF/WS-Transfer schism. I come from the WS-Transfer side of the world and so think of returning state as a single infoset. I'm trying to couch my descriptions in a manner that allows bridging to how a WSRF-oriented person would think about things. Marvin.
On Wed, 2006-06-14 at 13:25 -0700, Marvin Theimer wrote:
Hi;
I still don't understand what you're getting at here. I don't see this as the two interfaces are mutually exclusive. I just don't see any point of having identical operations for both the BES/activity factory service and the activity service. If the factory and activity service were the same, then fine. If they are two distinct services then tailor the interfaces. You mentioned "keeping them in sync". I still don't understand your concern here. What is it that frightens you about the BES group being able to keep the two (which are intimately related) in sync?
I’m just trying to advocate good software engineering principles here: I would prefer to avoid having to keep two interfaces in sync with each other across two different specifications. It’s not a big deal and I’m religious about it. I believe that in a later email you suggest that both interfaces be kept in the same spec. That seems like a good idea unless another compelling reason should pop up that argues for keeping them separate.
Ok, I'll consider this reconciled, then. ;-)
I would rather see a generic property getter method instead that takes an array of EPRs. This would model better w.r.t. WSRF's getResourceProperty, etc... methods and avoid the precedent of providing individual operations for each property.
The challenge here is the schism between the WSRF and the WS-Transfer camps. Although that schism is now slated to be healed by the new reconciliation effort that IBM, Microsoft, et al. announced recently, that reconciliation won’t be finalized for another year or two. Recall also that, although I’m proposing that the base case BES interface just deal with individual activities, one of the extensions that I expect will get added very quickly is an array interface for dealing with multiple activities in a single interaction. For the latter interface, it does not make sense to query the factory service’s resource properties (in the WSRF-oriented world) or its EPR (in the WS-Transfer-oriented world) since the query is about the state of a bunch of activities, not the factory. Since I’d like to have the interface for dealing with arrays of activities and for dealing with individual activities look as similar as possible – and given that the reconciliation of WSRF and WS-Transfer has not yet been completed – I’ve introduced named operations for the type of state I wish to see returned.
I have to admit that I had to read up on WS-Transfer. It makes more sense where you're coming from now. It's a very different philosophy of how resource state should be managed. Considering this I think we should still make this activity description a resource state property/element and leave it ambiguous as to how it is accessed. Unfortunately if we don't define the Activity interface we can't define the state item.
Now, you could argue that what I’ve done is expose a single “property” – the activityState property – and a means of retrieving all “properties” in something akin to WS-Transfer style. Based on that, you could argue that we should expose query operations for “all” the individual “properties” of an activity. Since that set is not completely defined and since we will also (at least eventually) want to support extended sets of “properties” and since there is not a reconciliation-approved way of exporting which “properties” an activity exports, I advocate that the base case of BES should aim to export a “minimal” query interface. To me, that’s a way to get everything (including any extensions) and a way of getting the one common state element that we know everyone always wants to query – namely the activityState element. Please note that I have no problem with adding in additional query operations in extensions, but I would argue that the BES base case should be kept as simple as possible. (Along those lines, one might argue that we should jettison the GetActivity operation that returns just the activity’s state diagram state. However, retrieving all the state that describes an activity just to get the activityState seems excessive to me, especially when thinking about how that would behave when dealing with large arrays of activities.)
And here is where I have the biggest problem with WS-Transfer. I think an activity's progress state should be a property that is modeled as desired (i.e. either WS-Transfer or WSRF) as long as there are multiple, popular ways to do so; but as you say, getting the entire state for one item is excessive. That's my personal opinion in general after reading about WS-Transfer. I don't think we should be sacrificing good design just because some implementation detail (I consider this implementation because it's left ambiguous in the spec) doesn't play nice. If you don't like it, don't use WS-Transfer in your implementation. Now then, there is a potentially serious problem here in that if the public state model is not standardized in the spec then how do you know what WS interface to use to manipulate it? In that case I would say we need an operation that specifically targets interop data (kind of like the FTP "FEAT" command). This is yet another reason in support of defining the activity interface. Each activity ought to be exposing via such a method what kind of public state model it uses.
It worries me that this is a non-WSRF-only operation. We should try very hard to avoid branching the spec for two different resource models. If we can't avoid it then there needs to be two separate specs IMHO. What do the WSRF services do with this?
We may have to have two separate specs, if people feel they need a WSRF-based version of the spec. Personally, I’d prefer a single spec that uses neither WSRF nor WS-Transfer and thereby circumvented the problem – but that’s my personal opinion. The interface I’ve proposed essentially tries to circumvent the issue by employing named query and modification operations.
To summarize what is essentially a counter proposal, I think we need to... 1) Expose public state as it was intended to be exposed (i.e. in state properties/elements within the state document) 2) Leave the public state model ambiguous 3) Define the Activity interface 4) Include an interop data fetch operation, say "GetInteropData", on both the BES interface and the Activity interface. The returned data should be well defined but extensible, and include what type of public state model is being used.
I don't really understand what you're trying to accomplish here. You say it's a non-WSRF operation and yet this description says it's an aggregation of the WSRF RPs. Then you add that it might return other stuff as well. This seems like a "give me the serialized resource object" operation. My feeling is that if the necessary information were exposed correctly you wouldn't need this operation.
This again comes down to the WSRF/WS-Transfer schism. I come from the WS-Transfer side of the world and so think of returning state as a single infoset. I’m trying to couch my descriptions in a manner that allows bridging to how a WSRF-oriented person would think about things.
And again I think that if it's burdensome to fetch the entire infoset just for one datum then you shouldn't be using WS-Transfer. It's not the BES-WG's fault that WS-Transfer isn't flexible enough. We shouldn't be making concessions to broken implementations. I apologize if that's a bit biased/harsh, but I'm not sure how else to put it. I want to make it clear that this is essentially a workaround to avoid a limitation of a particular implementation choice. Occasionally that's unavoidable, but I don't think that is the case here. Peter
Hi; I have to admit that I had to read up on WS-Transfer. It makes more sense where you're coming from now. It's a very different philosophy of how resource state should be managed. Considering this I think we should still make this activity description a resource state property/element and leave it ambiguous as to how it is accessed. Unfortunately if we don't define the Activity interface we can't define the state item. ... And here is where I have the biggest problem with WS-Transfer. I think an activity's progress state should be a property that is modeled as desired (i.e. either WS-Transfer or WSRF) as long as there are multiple, popular ways to do so; but as you say, getting the entire state for one item is excessive. That's my personal opinion in general after reading about WS-Transfer. I don't think we should be sacrificing good design just because some implementation detail (I consider this implementation because it's left ambiguous in the spec) doesn't play nice. If you don't like it, don't use WS-Transfer in your implementation. Now then, there is a potentially serious problem here in that if the public state model is not standardized in the spec then how do you know what WS interface to use to manipulate it? In that case I would say we need an operation that specifically targets interop data (kind of like the FTP "FEAT" command). This is yet another reason in support of defining the activity interface. Each activity ought to be exposing via such a method what kind of public state model it uses. I agree that we should define the activity interface since we're implicitly doing so at the BES level now. Leaving the access method ambiguous unfortunately doesn't yield an interface that actual implementations can interoperate through. So the HPC profile work will need to define one or more concrete interfaces one way or the other. One approach is to reflect the schism between WSRF and WS-Transfer and define two separate (but equal :-)) interfaces that each employ one or the other approach. Unfortunately that will make interoperability in practice much harder to achieve. To interoperate with BES services in general would require that either services export both interfaces or clients implement both interfaces. We've effectively doubled the cost of creating BES interfaces, including all the interop testing that has to go with it. Consider also that currently the main development tool options support either WSRF or WS-Transfer or neither. So developers would need to implement compliant implementations of either WSRF or WS-Transfer or both on their own. If you've followed the interop workshop process that Microsoft and IBM have been using for achieving interoperable tooling and implementations of even basic Web Service concepts and protocols, you'll know that that's a seriously expensive commitment to impose on developers. When you consider that in a couple years' time the world will have converged on a merged alternative to both of these protocol suites (and associated tooling), it seems even more questionable to pursue this avenue at this point in time. The alternative that I would advocate is to define a placeholder interface for version 1 that skirts the issue altogether. That was the intent behind my straw man proposal of GetActivity and GetActivityState methods. Using these would allow BES and the HPC profile work to get off the ground quickly without requiring that all parties first create compliant implementations of both WSRF and WS-Transfer. Interoperability testing would be confined strictly to these few methods in the BES/HPC-profile context and hence would be much simpler. Whether these methods should be replaced by the convergent replacement for WSRF/WS-Transfer in a later version of BES is something that could be examined at a later time when that convergence has actually occurred and things like tooling are broadly available for it. To summarize what is essentially a counter proposal, I think we need to... 1) Expose public state as it was intended to be exposed (i.e. in state properties/elements within the state document) 2) Leave the public state model ambiguous 3) Define the Activity interface 4) Include an interop data fetch operation, say "GetInteropData", on both the BES interface and the Activity interface. The returned data should be well defined but extensible, and include what type of public state model is being used. I'm not sure I understand what you mean by "leave the public state model ambiguous". Are you referring to the fact that BES has decided to move to an extensible state model and you'd like to use your proposed GetInteropData to return - among other things - a description of which state model extensions the service implements (including, of course, the base state model)? If that's the case then I totally agree with the need for that information. So I suspect that we're in agreement that, one-way-or-the-other, an activity should expose at least the following state elements for querying by clients: * Description of which state model extensions the service implements. * Activity state. * Full (extensible) description of the activity's state. See my previous comments about not wanting to expose more than one way to retrieve the public state of an activity. Please also note that, whereas one can argue about the pros and cons of the WSRF and WS-Transfer approaches, the reality I have to deal with is that there are two camps and there is no good tooling that supports both, so the choice the HPC profile WG has to make is arguably one of the following: 1. Go with both WSRF and WS-Transfer-based versions of BES and force everyone to implement compliant versions of both. That's too expensive in my opinion. 2. Wait for the convergent follow-on specs and define BES in terms of those. That doesn't meet the time-line requirements that the HPC profile WG faces. 3. Define a single interface that employs neither WSRF nor WS-Transfer in order to get a version 1 HPC profile done now rather than in a couple of years time; then consider for version 2 whether it makes sense to move to the convergent WSRF/WS-Transfer follow-on specs. To me, option 3 seems by-far the most preferable one. Marvin.
On Tue, 2006-06-20 at 11:48 -0700, Marvin Theimer wrote:
Hi;
I have to admit that I had to read up on WS-Transfer. It makes more
sense where you're coming from now. It's a very different philosophy of
how resource state should be managed. Considering this I think we should
still make this activity description a resource state property/element
and leave it ambiguous as to how it is accessed. Unfortunately if we
don't define the Activity interface we can't define the state item.
...
And here is where I have the biggest problem with WS-Transfer. I think
an activity's progress state should be a property that is modeled as
desired (i.e. either WS-Transfer or WSRF) as long as there are multiple,
popular ways to do so; but as you say, getting the entire state for one
item is excessive. That's my personal opinion in general after reading
about WS-Transfer. I don't think we should be sacrificing good design
just because some implementation detail (I consider this implementation
because it's left ambiguous in the spec) doesn't play nice. If you don't
like it, don't use WS-Transfer in your implementation.
Now then, there is a potentially serious problem here in that if the
public state model is not standardized in the spec then how do you know
what WS interface to use to manipulate it? In that case I would say we
need an operation that specifically targets interop data (kind of like
the FTP "FEAT" command). This is yet another reason in support of
defining the activity interface. Each activity ought to be exposing via
such a method what kind of public state model it uses.
I agree that we should define the activity interface since we’re implicitly doing so at the BES level now. Leaving the access method ambiguous unfortunately doesn’t yield an interface that actual implementations can interoperate through. So the HPC profile work will need to define one or more concrete interfaces one way or the other.
I fully agree with this. The only use of the GetInteropData operation in this case would be to tell you whether you *can* interoperate with it.
Consider also that currently the main development tool options support either WSRF or WS-Transfer or neither.
The "neither" option is what frustrates me reading this. We want to expose public state, and yet apparently we have to keep implementors in mind that have no standard way to expose that public state.
So developers would need to implement compliant implementations of either WSRF or WS-Transfer or both on their own. If you’ve followed the interop workshop process that Microsoft and IBM have been using for achieving interoperable tooling and implementations of even basic Web Service concepts and protocols, you’ll know that that’s a seriously expensive commitment to impose on developers. When you consider that in a couple years’ time the world will have converged on a merged alternative to both of these protocol suites (and associated tooling), it seems even more questionable to pursue this avenue at this point in time.
I reluctantly agree. I was operating under the assumption that at very least either WSRF or WS-Transfer would be an acceptable requirement. Not having any public state model requirement at all makes this a slightly different problem in my opinion.
The alternative that I would advocate is to define a placeholder interface for version 1 that skirts the issue altogether. That was the intent behind my straw man proposal of GetActivity and GetActivityState methods. Using these would allow BES and the HPC profile work to get off the ground quickly without requiring that all parties first create compliant implementations of both WSRF and WS-Transfer. Interoperability testing would be confined strictly to these few methods in the BES/HPC-profile context and hence would be much simpler. Whether these methods should be replaced by the convergent replacement for WSRF/WS-Transfer in a later version of BES is something that could be examined at a later time when that convergence has actually occurred and things like tooling are broadly available for it.
I think I'm with you now for the most part. I just have a problem with singling out one property over another in terms of which ones are "good enough" to have their own accessor operation. Especially if we allow for notifications, the activity status might be the least used and yet you are proposing that we make it the only property that gets it's own accessor operation. Perhaps it would be acceptable to create a hybrid operation that either takes an array of property QNames and returns those properties (ala WSRF's "GetMultipleResourceProperties"), or if the array is empty/not specified returns the entire public state document (ala WS-Transfer's "Get"). If we need a multiple-activity operation for the BES services, then just add another argument that is an array of activity EPRs and name the operation to distinguish it from the singular operation.
I’m not sure I understand what you mean by “leave the public state model ambiguous”.
I meant define public state properties in the spec like we do, but leave it up to the implementation as to whether to use WSRF or WS-Transfer to access those properties. The GetInteropData operation would indicate which public state model is implemented.
Are you referring to the fact that BES has decided to move to an extensible state model and you’d like to use your proposed GetInteropData to return – among other things – a description of which state model extensions the service implements (including, of course, the base state model)? If that’s the case then I totally agree with the need for that information.
The term "state" is used both for the description of what vertex the job/Activity is at in the directed progress graph as well as the set of data elements/properties that are associated with a statefull service entity. When I say "public state" I mean the latter. I'll try to use "activity status" for the former from now on.
So I suspect that we’re in agreement that, one-way-or-the-other, an activity should expose at least the following state elements for querying by clients:
· Description of which state model extensions the service implements.
I presume you're referring to what I'm calling the "public state model", yes?
· Activity state.
Not sure what you're referring to here.
· Full (extensible) description of the activity’s state.
And by this you mean the extensions to what I'm now calling the "activity status"?
See my previous comments about not wanting to expose more than one way to retrieve the public state of an activity. Please also note that, whereas one can argue about the pros and cons of the WSRF and WS-Transfer approaches, the reality I have to deal with is that there are two camps and there is no good tooling that supports both, so the choice the HPC profile WG has to make is arguably one of the following:
1. Go with both WSRF and WS-Transfer-based versions of BES and force everyone to implement compliant versions of both. That’s too expensive in my opinion.
2. Wait for the convergent follow-on specs and define BES in terms of those. That doesn’t meet the time-line requirements that the HPC profile WG faces.
3. Define a single interface that employs neither WSRF nor WS-Transfer in order to get a version 1 HPC profile done now rather than in a couple of years time; then consider for version 2 whether it makes sense to move to the convergent WSRF/WS-Transfer follow-on specs.
To me, option 3 seems by-far the most preferable one.
This is essentially what I was pushing with my ambiguous public state model concept. What I wasn't considering was this new (at least to me) requirement for no public state model at all. If we can't assume a public state model, then we have to basically agree on some minimal reconciliation between WSRF and WS-Transfer since the BES spec implies some sort of public state model as a result of it's definition of statefull service properties. The two candidates seem to be the following: 1) Have named accessor operations for each property and one operation for obtaining the entire public state document, or 2) Have a single, generic operation that fetches either an array of properties or the entire set of public state properties. Is that a fair assessment? Peter
Hi; The "neither" option is what frustrates me reading this. We want to expose public state, and yet apparently we have to keep implementors in mind that have no standard way to expose that public state. ... I reluctantly agree. I was operating under the assumption that at very least either WSRF or WS-Transfer would be an acceptable requirement. Not having any public state model requirement at all makes this a slightly different problem in my opinion. I believe that today's reality is that you can't assume universal WSRF or WS-Transfer support anytime soon. I think I'm with you now for the most part. I just have a problem with singling out one property over another in terms of which ones are "good enough" to have their own accessor operation. Especially if we allow for notifications, the activity status might be the least used and yet you are proposing that we make it the only property that gets it's own accessor operation. Perhaps it would be acceptable to create a hybrid operation that either takes an array of property QNames and returns those properties (ala WSRF's "GetMultipleResourceProperties"), or if the array is empty/not specified returns the entire public state document (ala WS-Transfer's "Get"). If we need a multiple-activity operation for the BES services, then just add another argument that is an array of activity EPRs and name the operation to distinguish it from the singular operation. I didn't mean to advocate that there be an explicit accessor function for only the activity status property. Rather I was advocating that there be accessor functions for each of the properties and that the interop standard mandate that there be at least the accessor function for returning an activity's status (in addition to the accessor function for returning all the state information for an activity). All other accessor functions would be part of extensions. In my response further down I explain why I prefer explicity accessor functions over a WSRF-style GetMultipleResourceProperties approach. This is essentially what I was pushing with my ambiguous public state model concept. What I wasn't considering was this new (at least to me) requirement for no public state model at all. If we can't assume a public state model, then we have to basically agree on some minimal reconciliation between WSRF and WS-Transfer since the BES spec implies some sort of public state model as a result of it's definition of statefull service properties. The two candidates seem to be the following: 1) Have named accessor operations for each property and one operation for obtaining the entire public state document, or 2) Have a single, generic operation that fetches either an array of properties or the entire set of public state properties. I would advocate taking approach 1 for the following reasons, derived from an explanation of a colleague of mine recently provided me: The advantage of the explicit accessor function approach, given the current tooling and suite of technologies that most of the world has, is that you can describe it completely using metadata (i.e. WSDL). The latter approach cannot be described in WSDL, hence the need to introduce the additional message exchange (GetListOfProperties). The disadvantage of the explicit accessor function approach is that you can't request more than one property at a time. You might ask why being able to describe the interface completely in WSDL/XSD matters. The reply to a 'GetState' request will have to be a message with open content... <properties> <xsd:any> </properties> Of course you can define in your specification that if property A is included in there, then the structure would look like <properties> <A> <foo /> <bar /> </A> </properties> However, now this description is in a human language specification and not in a machine-processable vocabulary. You have to manually make sure that you implement support for the various infosets before you can deal with the results. If you make an error in your implementation then the tooling won't help you - nor can the tooling help you with any kind of static analysis of your interface for correctness or security purposes. If you define explicit messages, then you can describe the structure using WSDL and XSD and you can point your favorite WSDL->Java or WSDL->C# tool at the WSDL document and everything will be generated for you. There would be no surprises because the two sides have agreed on a contract. Note also that such tools make the creation of client code for using a multitude of explicit accessor functions fairly painless. Marvin.
Marvin Theimer wrote:
I believe that today’s reality is that you can’t assume universal WSRF or WS-Transfer support anytime soon.
Fair enough, I've come to terms with this. ;-)
I didn’t mean to advocate that there be an explicit accessor function for only the activity status property. Rather I was advocating that there be accessor functions for each of the properties and that the interop standard mandate that there be at least the accessor function for returning an activity’s status (in addition to the accessor function for returning all the state information for an activity). All other accessor functions would be part of extensions.
I think I see what you're implying now. The absolute minimum interop would have to include a way to get the status so the user knows at very least whether the activity has finished. If the user can't get the values of any other properties, the job can still be monitored. I buy that. We should probably define the accessors explicitly in an extension spec.
1) Have named accessor operations for each property and one operation
for obtaining the entire public state document, or
2) Have a single, generic operation that fetches either an array of
properties or the entire set of public state properties.
I would advocate taking approach 1 for the following reasons, derived from an explanation of a colleague of mine recently provided me:
This is fine with me. So to summarize... ---------------- There are four implementation options that should be supported by the spec for obtaining public state data: 1) WS-Transfer, 2) WSRF, 3) public state accessor operations, 4) basic interop. An accessor operation for obtaining the activity status MUST be included in each of the above implementation options. Official QNames SHALL be defined for options 1-3. The GetInteropData operation MUST return a list of QNames that represents which of the above options is supported. An empty list implies that ONLY option #4 is available. ---------------- We should also define what port types option #2 implies or break it up into three different options for each of the three port types used for resource property queries in WSRF (wsrp:GetResourceProperty, wsrp:GetMultipleResourceProperties, and wsrp:QueryResourceProperties). Peter
So to summarize... ---------------- There are four implementation options that should be supported by the spec for obtaining public state data: 1) WS-Transfer, 2) WSRF, 3) public state accessor operations, 4) basic interop. An accessor operation for obtaining the activity status MUST be included in each of the above implementation options. Official QNames SHALL be defined for options 1-3. The GetInteropData operation MUST return a list of QNames that represents which of the above options is supported. An empty list implies that ONLY option #4 is available. ---------------- We should also define what port types option #2 implies or break it up into three different options for each of the three port types used for resource property queries in WSRF (wsrp:GetResourceProperty, wsrp:GetMultipleResourceProperties, and wsrp:QueryResourceProperties). This sounds good to me. I presume that "basic interop" means just the two accessor operations for activity status and complete activity infoset? Marvin.
Marvin Theimer wrote:
So to summarize...
----------------
There are four implementation options that should be supported by the
spec for obtaining public state data:
1) WS-Transfer,
2) WSRF,
3) public state accessor operations,
4) basic interop.
An accessor operation for obtaining the activity status MUST be included
in each of the above implementation options.
Official QNames SHALL be defined for options 1-3.
The GetInteropData operation MUST return a list of QNames that
represents which of the above options is supported. An empty list
implies that ONLY option #4 is available.
----------------
We should also define what port types option #2 implies or break it up
into three different options for each of the three port types used for
resource property queries in WSRF (wsrp:GetResourceProperty,
wsrp:GetMultipleResourceProperties, and wsrp:QueryResourceProperties).
This sounds good to me. I presume that “basic interop” means just the two accessor operations for activity status and complete activity infoset?
I was only including the status getter:
An accessor operation for obtaining the activity status MUST be included
in each of the above implementation options.
In terms of absolute bare minimum interop, I don't think a public resource data getter is necessary. The implementation can still provide a getter operation for the public resource data, but the assumption under basic interop is that it is not a standard the spec knows about or chooses to support. Basic interactions can still occur. It's just that a client not built specifically for the implementation won't be able to inspect the more detailed information about the services. Peter
Marvin.
Hi; I think jsdl rightly stays away from workflows. However, it seems like it would be important for it to be able to describe the common single-application case of things like MPI programs -- which need the ability to say things like "all cpus must be the same" but don't need the more complex heterogeneous descriptions you present here. Marvin. -----Original Message----- From: owner-ogsa-bes-wg@ggf.org [mailto:owner-ogsa-bes-wg@ggf.org] On Behalf Of Peter Lane Sent: Tuesday, June 06, 2006 9:08 AM To: Donal K. Fellows Cc: Marvin Theimer; ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view On Jun 6, 2006, at 3:06 AM, Donal K. Fellows wrote:
Peter Lane wrote:
Marvin Theimer wrote:
* JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I'm wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification.
I think I complained about this here as well. Certainly I've complained to the JSDL people and the ESI people about this. The JDSL people answered something about keeping it simple for now. My guess is that people see it for the daunting task it is an prefer not to address it. I'm not criticizing anyone for this. It's just my take on the problem.
JSDL describes the resources that are to be allocated to the job (err, activity) and it is not designed to handle the description of the resources allocated to the container.
Right, I meant something a little different when I mentioned JSDL. JSDL doesn't allow for describing complex requirements. For example, one might need a complex set of resources to run a distributed application on a cluster. The best JSDL can do, IIRC, is to allocate N homogenous resources. There's no way you can say, for example, "give me two IA64 machines, two x86_64 machines, and two i386 machines". This is a very real requirement by users of GRAM. Peter
Marvin Theimer wrote:
I think jsdl rightly stays away from workflows. However, it seems like it would be important for it to be able to describe the common single-application case of things like MPI programs -- which need the ability to say things like "all cpus must be the same" but don't need the more complex heterogeneous descriptions you present here.
Heh. FYI, this is currently ongoing work and was the main JSDL thing discussed at the last GGF. :-) Donal.
Hi; I agree with Karl's later email pointing out that it would be really unfortunate if we end up with two separate specifications to describe the two closely intertwined topics of job requirements and (available) resource descriptions. From a resource description point-of-view, I agree that a simple approach based on a set of jsdl documents would represent an interesting simple approach. If we structure things so that alternative approaches can be transparently substituted -- e.g. by having resource description infosets have a top-level element that describes which description approach they're taking -- then we can get started without having to either solve all the problems of a fully general solution or foreclose more general solutions as they get defined later on. Marvin. -----Original Message----- From: Donal K. Fellows [mailto:donal.k.fellows@manchester.ac.uk] Sent: Tuesday, June 06, 2006 2:07 AM To: Peter Lane Cc: Marvin Theimer; ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view Peter Lane wrote:
Marvin Theimer wrote:
* JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I'm wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification.
I think I complained about this here as well. Certainly I've complained to the JSDL people and the ESI people about this. The JDSL people answered something about keeping it simple for now. My guess is that people see it for the daunting task it is an prefer not to address it.
I'm not criticizing anyone for this. It's just my take on the problem.
JSDL describes the resources that are to be allocated to the job (err, activity) and it is not designed to handle the description of the resources allocated to the container. This matters for a whole bunch of reasons (some unrelated to this discussion) but what is crucial here is that the case you're describing only matters if you're constructing an atomic activity that is itself inherently heterogenous. Do users do that? Or are they happier with describing their task as a set of coordinated (and presumably isochronous) activities with a different JSDL-style resource requirement for each? If, as I believe, they really want the latter, we can punt the majority of the heterogenous resource mess and focus on easier things. On the other hand, there is still a need to describe heterogenous clusters; they're not going away soon. The easiest way is probably to allow the BES to publish something like multiple JSDL Resources elements, one for each kind of node in the cluster, and for the composition rule for these separate elements to be "I have *all* of these". The only complication with that approach is that it doesn't easily support the case where you want to just use resources at the lowest common denominator level, making resource matching more complex. Oh well. Donal.
Hi; Outlook seems unable to do a proper job of incorporating a copy of your email into my reply, so I will reply to your points separately from the in-lined original email: * I would agree with you that security around messaging itself is out-of-scope. I do believe we need to think about how to deal with the credentials used for running activities. Without that, it will be impossible to define an HPC profile that actually enables a working, interoperable system. * I'm confused about your statement that extension profiles are out-of-scope. Assuming that BES will become an extensible specification - which is implied by its acceptance of an extensible state diagram model - then we have to define how extensions can get written and allow for the BES WG to define a few for some of things that were in the original BES spec, such as data staging and suspension. * From an HPC profile point-of-view, we have to be able to describe BES containers that represent compute clusters. Otherwise, the HPC profile will not be able to achieve its goals using primarily BES and JSDL. That is, it would need to define another interface for describing a cluster's resources. That, in turn, would make the equivalent aspects in the current BES spec redundant. * If we want interoperability then we have to define normative values for things like LocalResourceManagerType. That doesn't preclude the notion of allowing non-normative ones to be employed by clients and services that have an implicit understanding amongst themselves. * The problem with simple statements about security is that they don't enable interoperability. So by defining an EPR for something like a job credential service, you're implicitly introducing an interop requirement that all parties must figure out how to adhere to. Marvin. ________________________________ From: Peter Lane [mailto:lane@mcs.anl.gov] Sent: Monday, June 05, 2006 5:02 PM To: Marvin Theimer Cc: ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view On Jun 5, 2006, at 5:19 PM, Marvin Theimer wrote: * Interface for directly controlling/manipulating an activity once it has been created. Yes, I was concerned about this as well but never said anything. Generic operations for manipulating or introspecting activities (i.e. factory operation that takes an activity ID) can be useful, but I definitely agree that there should be an activity service interface with operations that imply the activity via the EPR. * Given that BES has bought into the notion of an extensible activity state diagram, it needs to also normatively define how clients can learn of the extensions that a given BES service supports. Is that something that will be added to the BES specification? Or will the specification point to some other place where notions of extensibility are defined more generically? (Personally, I'd vote for the former approach.) Good point. I'd also prefer adding something explicitly to the spec. * Previously included states, such as Execution-Pending, will presumably be defined in suitable extension profiles? My understanding is that this kind of thing is out of scope of the spec. I would assume that if there were any defacto standards being used in the community that this would at most be a separate OGSA-BES WG spec. * RequestActivityStateChange: I believe this operation will pose challenges in an extensible design. The current design is imperative by nature: it specifies an explicit state to move an activity to. However, a client who does not know of all the extensions that a BES service implements may not know how to pick the appropriate state to transition to. It seems better to introduce a more declarative approach in which clients specify "actions" they wish to occur, such as 'CancelActivity'. This approach would allow the BES service to make the appropriate state transition in response to a desired action requested by a client. Yes, this is what I argued as well albeit from a slightly different angle. * JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I'm wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification. I think I complained about this here as well. Certainly I've complained to the JSDL people and the ESI people about this. The JDSL people answered something about keeping it simple for now. My guess is that people see it for the daunting task it is an prefer not to address it. I'm not criticizing anyone for this. It's just my take on the problem. * These operations seem to require a different set of authorization credentials than the other interface operations since they should be invoked by system administrators rather than random users. How will that work, given that these operations are in the same WSDL as the other operations? Wouldn't this argue for moving these operations to a separate system management interface? This is why I suggested that security was out of scope. The Globus Toolkit, for example, has no problems specifying different security parameters for different operations. It's all proprietary configuration, though, and probably should be. That said, I believe I said something similar to this in my original comments on the spec. This sort of thing feels to me as well like it shouldn't be in a basic execution service and should be an implementation detail as to whether the management interface is exposed as well as the BES interface. * Since JSDL documents are self-describing, a BES service can figure out by inspection whether the job description infoset parameter to CreateActivityFromJSDL is JSDL or something else. This would seem to imply that naming the operation CreateActivity would lose no information and would allow for transparent extension to other job description infoset simply by using them (assuming they are self-describing). Yes! I argued to chop the suffix as well, but this makes the point better. * LocalResourceManagerType: where do these get defined normatively? I would argue against explicitly defining values for this. It eliminates much of the flexibility the implementation has in terms of what local resource managers it can support. *
On Jun 7, 2006, at 4:29 PM, Marvin Theimer wrote:
Hi;
Outlook seems unable to do a proper job of incorporating a copy of your email into my reply, so I will reply to your points separately from the in-lined original email:
I would agree with you that security around messaging itself is out- of-scope. I do believe we need to think about how to deal with the credentials used for running activities. Without that, it will be impossible to define an HPC profile that actually enables a working, interoperable system. I guess I'm still confused about what you are asking to be addressed. How actual credentials are used seems like an implementation decision and thus out of scope of an interface spec.
I’m confused about your statement that extension profiles are out- of-scope. Assuming that BES will become an extensible specification – which is implied by its acceptance of an extensible state diagram model – then we have to define how extensions can get written and allow for the BES WG to define a few for some of things that were in the original BES spec, such as data staging and suspension. Sorry, I didn't mean that defining *how* extensions are added was out of scope. I just meant that the actual implementation of those extensions is out of scope. I think any extensions should be defined in a separate spec. If not, then I don't see why it's an extension to the spec. ;-)
From an HPC profile point-of-view, we have to be able to describe BES containers that represent compute clusters. Otherwise, the HPC profile will not be able to achieve its goals using primarily BES and JSDL. That is, it would need to define another interface for describing a cluster's resources. That, in turn, would make the equivalent aspects in the current BES spec redundant. If we want interoperability then we have to define normative values for things like LocalResourceManagerType. That doesn't preclude the notion of allowing non-normative ones to be employed by clients and services that have an implicit understanding amongst themselves. I'd agree with this. Have a normative set but also don't restrict to just that set.
The problem with simple statements about security is that they don't enable interoperability. So by defining an EPR for something like a job credential service, you're implicitly introducing an interop requirement that all parties must figure out how to adhere to. True, and yet that's what the BES spec is basically doing with activities as well. It gives you an EPR to *something*. So if we are consistent about this interop philosophy then we really ought to define the activity interface within the same spec.
Marvin.
From: Peter Lane [mailto:lane@mcs.anl.gov] Sent: Monday, June 05, 2006 5:02 PM To: Marvin Theimer Cc: ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view
On Jun 5, 2006, at 5:19 PM, Marvin Theimer wrote:
· Interface for directly controlling/manipulating an activity once it has been created.
Yes, I was concerned about this as well but never said anything. Generic operations for manipulating or introspecting activities (i.e. factory operation that takes an activity ID) can be useful, but I definitely agree that there should be an activity service interface with operations that imply the activity via the EPR.
· Given that BES has bought into the notion of an extensible activity state diagram, it needs to also normatively define how clients can learn of the extensions that a given BES service supports. Is that something that will be added to the BES specification? Or will the specification point to some other place where notions of extensibility are defined more generically? (Personally, I’d vote for the former approach.)
Good point. I'd also prefer adding something explicitly to the spec.
· Previously included states, such as Execution-Pending, will presumably be defined in suitable extension profiles?
My understanding is that this kind of thing is out of scope of the spec. I would assume that if there were any defacto standards being used in the community that this would at most be a separate OGSA- BES WG spec.
· RequestActivityStateChange: I believe this operation will pose challenges in an extensible design. The current design is imperative by nature: it specifies an explicit state to move an activity to. However, a client who does not know of all the extensions that a BES service implements may not know how to pick the appropriate state to transition to. It seems better to introduce a more declarative approach in which clients specify “actions” they wish to occur, such as ‘CancelActivity’. This approach would allow the BES service to make the appropriate state transition in response to a desired action requested by a client.
Yes, this is what I argued as well albeit from a slightly different angle.
· JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I’m wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification.
I think I complained about this here as well. Certainly I've complained to the JSDL people and the ESI people about this. The JDSL people answered something about keeping it simple for now. My guess is that people see it for the daunting task it is an prefer not to address it. I'm not criticizing anyone for this. It's just my take on the problem.
· These operations seem to require a different set of authorization credentials than the other interface operations since they should be invoked by system administrators rather than random users. How will that work, given that these operations are in the same WSDL as the other operations? Wouldn’t this argue for moving these operations to a separate system management interface?
This is why I suggested that security was out of scope. The Globus Toolkit, for example, has no problems specifying different security parameters for different operations. It's all proprietary configuration, though, and probably should be. That said, I believe I said something similar to this in my original comments on the spec. This sort of thing feels to me as well like it shouldn't be in a basic execution service and should be an implementation detail as to whether the management interface is exposed as well as the BES interface.
· Since JSDL documents are self-describing, a BES service can figure out by inspection whether the job description infoset parameter to CreateActivityFromJSDL is JSDL or something else. This would seem to imply that naming the operation CreateActivity would lose no information and would allow for transparent extension to other job description infoset simply by using them (assuming they are self-describing).
Yes! I argued to chop the suffix as well, but this makes the point better.
· LocalResourceManagerType: where do these get defined normatively?
I would argue against explicitly defining values for this. It eliminates much of the flexibility the implementation has in terms of what local resource managers it can support.
·
Hi; I guess I'm still confused about what you are asking to be addressed. How actual credentials are used seems like an implementation decision and thus out of scope of an interface spec. The main thing that comes to mind is how a BES service would expose which types of credentials it can handle or which it expects. I think you're right that the rest of it - e.g. declaring a set of standard credentials types so that interop can actually be implemented - is part of the HPC profile work. True, and yet that's what the BES spec is basically doing with activities as well. It gives you an EPR to *something*. So if we are consistent about this interop philosophy then we really ought to define the activity interface within the same spec. "Someone" will have to define all the details that are necessary to enable actual interoperability. Much of that work can be done in the HPC profile. As I said in a previous email response, I like your suggestion of defining the activity interface - NOTE: just the generic query/modify part having to do with JSDL and things like cancellation - in the same spec. Marvin.
On Wed, 2006-06-14 at 13:38 -0700, Marvin Theimer wrote:
Hi;
I guess I'm still confused about what you are asking to be addressed. How actual credentials are used seems like an implementation decision and thus out of scope of an interface spec.
The main thing that comes to mind is how a BES service would expose which types of credentials it can handle or which it expects. I think you’re right that the rest of it – e.g. declaring a set of standard credentials types so that interop can actually be implemented – is part of the HPC profile work.
So something akin to "I accept X509 certs and Kerberos tokens"? I'm fine with that. This sounds like another item that should be included in a GetInteropData operation that I proposed in a different email.
True, and yet that's what the BES spec is basically doing with activities as well. It gives you an EPR to *something*. So if we are consistent about this interop philosophy then we really ought to define the activity interface within the same spec.
“Someone” will have to define all the details that are necessary to enable actual interoperability. Much of that work can be done in the HPC profile. As I said in a previous email response, I like your suggestion of defining the activity interface – NOTE: just the generic query/modify part having to do with JSDL and things like cancellation – in the same spec.
Fair enough. Have a way to get interop data but don't restrict it. I think I'm going to have to start paying attention to the HPC profile work. ;-) Peter
Folks, Peter G Lane wrote:
[...]
The main thing that comes to mind is how a BES service would expose which types of credentials it can handle or which it expects. I think you’re right that the rest of it – e.g. declaring a set of standard credentials types so that interop can actually be implemented – is part of the HPC profile work.
So something akin to "I accept X509 certs and Kerberos tokens"? I'm fine with that. This sounds like another item that should be included in a GetInteropData operation that I proposed in a different email.
[...]
“Someone” will have to define all the details that are necessary to enable actual interoperability. Much of that work can be done in the HPC profile. As I said in a previous email response, I like your suggestion of defining the activity interface – NOTE: just the generic query/modify part having to do with JSDL and things like cancellation – in the same spec.
Fair enough. Have a way to get interop data but don't restrict it. I think I'm going to have to start paying attention to the HPC profile work. ;-)
even though I did only partially participate in this discussion, I think you touching exactly the right spot on interoperability. I think this is a discussion whose outcome would be invaluable input to the GIN working group. I guess we should copy these guys in here. Cheers, Michel -- Michel <dot> Drescher <at> uk <dot> fujitsu <dot> com Fujitsu Laboratories of Europe +44 20 8606 4834
Hi, In-lined, Marvin Theimer wrote: <snip>
· Information model:
· JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I'm wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification.
If I could say here that JSDL 1.0 is not meant to be a final answer. The resource model as used within it is meant to be a "rough place holder" as we are expecting that the CIM people will be providing something much more robust and appropriate. We simply wanted something simple to get things going. We hope that when this model is available we can roll it into a later release of JSDL. If you're interested in following this up I'd suggest getting involved with the OGSA information model people (and obviously) the JSDL people.
· The BES information model now includes various posix-specific elements of JSDL. How would other systems -- such as a Windows system -- be described?
The posix elements are only inside the POSIXApplicationType. The best way would be to define a WindowsApplicationType (or something similar). The JSDL group would be very interested in this.
· Is there any notion of specifying that all compute nodes should have the /same/ value for some attribute (e.g. CPU architecture, CPU speed, NIC card)? This seems to be missing from the JSDL specification, but seems very important for BES if it is to support things like compute clusters.
Again this will hopefully come in the future.
· Some of the elements seem either incompletely specified, have definitions that are open to multiple interpretations, or have definitions that would be very difficult to implement in practice. In particular:
· CPU architecture seems like it can't describe all the variations -- let alone all the peripherals such as GPUs -- that a computing resource might have (let alone a cluster).
· CPU speed seems like the tip of an iceberg having to do with characterizing the performance of a system, which will depend on all manner of things like details of the processor chip used, cache sizes, bus used, etc.
· Network bandwidth: is this the theoretical maximum of the NIC on a compute node or is it the current bandwidth actually available in a (shared) system? Note that the latter is difficult to measure in a practically useful way. Note also that network bandwidth only describes one aspect of communications performance and that several others are arguably equally important (e.g. latency).
All this leads to the question of whether BES will have a notion of extending the information model that is supplied. If so, then that leads to the question of what the base case should be and whether it should include a smaller set of things than is currently listed in the spec.
Are there any plans to tighten the definitions of some of the more vague information elements? (I guess this really is an issue more for the JSDL WG than for BES.)
Again - this is where we are now planning to go with JSDL. JSDL 1.0 should be seen as a starting point and not the end. We hope most of these things can be handled through extensions to JSDL 1.0. Those that can't we'll need to add into future versions.
· GetActivityJSDLDocuments returns a JSDL document for each specified activity. Is this sufficient to capture the entire "provenance" for what has happened to the activity? In particular, would it be sufficient to allow someone to (a) run the same activity on another BES service (assuming same hardware and software) and get the same results and (b) debug what has happened to an errant activity? I would argue that both capabilities have proven to be important in actual systems.
A JSDL document is (by the definition of our charter) a Job Submission document. As such things like provenance was ruled out of scope (not by us but by GGF in general - it was felt that this was too much to do all in one goal). However, there is now the scope to go back and re-address these issues. There was some interesting discussions at the last GGF meeting where the ideas of where non submission information could be placed. The suggestions included in a wrapper around the JSDL document or (and from my recollection) the more popular option was to place it in the outer most level of the JSDL document. Hope this helps, steve..
-- ------------------------------------------------------------------------ Dr A. Stephen McGough http://www.doc.ic.ac.uk/~asm ------------------------------------------------------------------------ Technical Coordinator, London e-Science Centre, Imperial College London, Department of Computing, 180 Queen's Gate, London SW7 2BZ, UK tel: +44 (0)207-594-8409 fax: +44 (0)207-581-8024 ------------------------------------------------------------------------
Hi; I totally understand that jsdl 1.0 isn't the "final" answer. I view my email as being an attempt at constructive suggestion for potential change. :-) Currently the posix extension to jsdl includes various things that I would argue are more generic than posix. I think it would be interesting to define a common subset (that includes things execution pathname, command-line args, and working directory) as well as defining a WindowsApplicationType that describes those things that are Windows-specific. It's OK if provenance is considered out-of-scope for jsdl. But we may still want to define a simple provenance notion for BES. Marvin. ________________________________ From: A S McGough [mailto:asm@doc.ic.ac.uk] Sent: Tuesday, June 06, 2006 8:40 AM To: Marvin Theimer Cc: ogsa-bes-wg@ggf.org; Ed Lassettre Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view Hi, In-lined, Marvin Theimer wrote: <snip> * Information model: 1. JSDL seems to inherently be focused on describing a single job or a single computational resource. For example, it has no notion of describing all the differing compute nodes of a (heterogeneous) compute cluster. By incorporating JSDL elements into the BES information model it seems that BES is foreclosing the ability to describe things like compute clusters. This issue also effects what can get returned from GetActivityJSDLDocuments. If I'm wrong about this, then it seems like it would be worth having an explicit explanation about how to achieve this functionality somewhere in the specification. If I could say here that JSDL 1.0 is not meant to be a final answer. The resource model as used within it is meant to be a "rough place holder" as we are expecting that the CIM people will be providing something much more robust and appropriate. We simply wanted something simple to get things going. We hope that when this model is available we can roll it into a later release of JSDL. If you're interested in following this up I'd suggest getting involved with the OGSA information model people (and obviously) the JSDL people. 2. The BES information model now includes various posix-specific elements of JSDL. How would other systems - such as a Windows system - be described? The posix elements are only inside the POSIXApplicationType. The best way would be to define a WindowsApplicationType (or something similar). The JSDL group would be very interested in this. 3. Is there any notion of specifying that all compute nodes should have the same value for some attribute (e.g. CPU architecture, CPU speed, NIC card)? This seems to be missing from the JSDL specification, but seems very important for BES if it is to support things like compute clusters. Again this will hopefully come in the future. 4. Some of the elements seem either incompletely specified, have definitions that are open to multiple interpretations, or have definitions that would be very difficult to implement in practice. In particular: 5. CPU architecture seems like it can't describe all the variations - let alone all the peripherals such as GPUs - that a computing resource might have (let alone a cluster). 6. CPU speed seems like the tip of an iceberg having to do with characterizing the performance of a system, which will depend on all manner of things like details of the processor chip used, cache sizes, bus used, etc. 7. Network bandwidth: is this the theoretical maximum of the NIC on a compute node or is it the current bandwidth actually available in a (shared) system? Note that the latter is difficult to measure in a practically useful way. Note also that network bandwidth only describes one aspect of communications performance and that several others are arguably equally important (e.g. latency). All this leads to the question of whether BES will have a notion of extending the information model that is supplied. If so, then that leads to the question of what the base case should be and whether it should include a smaller set of things than is currently listed in the spec. Are there any plans to tighten the definitions of some of the more vague information elements? (I guess this really is an issue more for the JSDL WG than for BES.) Again - this is where we are now planning to go with JSDL. JSDL 1.0 should be seen as a starting point and not the end. We hope most of these things can be handled through extensions to JSDL 1.0. Those that can't we'll need to add into future versions. 8. GetActivityJSDLDocuments returns a JSDL document for each specified activity. Is this sufficient to capture the entire "provenance" for what has happened to the activity? In particular, would it be sufficient to allow someone to (a) run the same activity on another BES service (assuming same hardware and software) and get the same results and (b) debug what has happened to an errant activity? I would argue that both capabilities have proven to be important in actual systems. A JSDL document is (by the definition of our charter) a Job Submission document. As such things like provenance was ruled out of scope (not by us but by GGF in general - it was felt that this was too much to do all in one goal). However, there is now the scope to go back and re-address these issues. There was some interesting discussions at the last GGF meeting where the ideas of where non submission information could be placed. The suggestions included in a wrapper around the JSDL document or (and from my recollection) the more popular option was to place it in the outer most level of the JSDL document. Hope this helps, steve..
Marvin Theimer wrote:
Currently the posix extension to jsdl includes various things that I would argue are more generic than posix. I think it would be interesting to define a common subset (that includes things execution pathname, command-line args, and working directory) as well as defining a WindowsApplicationType that describes those things that are Windows-specific.
I believe[*] it is possible to define a new element type that describes an abstract ExecutableApplication element with these generic bits in and which is grandfathered into being a supertype of the POSIXApplication element (as well as being the parent of the WindowsApplication you're positing). This would require a small change to the existing schema, though we'd want to make sure that the documents accepted by the schema remained the same, and would enable this suggestion (which I approve of, by the way) to be slotted into the larger scheme of things without fuss. By the way, what would these extra Windows-specific bits be? I can imagine a number of the POSIX things being dropped (e.g. the limits) but I'm not nearly well-versed enough on Win (despite using it a lot) to be able to think what these extra additions might be. Just out of curiosity! :-) Donal. [* This is believe as in "I've done this sort of thing in other schemas" so I know that XML Schema can support it. ]
Hi; I will look into the question of what sorts of Windows-specific things might be appropriate for a WindowsApplicationType. I suspect that if we have a common ApplicationType that includes the things we've talked about then the WindowsApplicationType may be fairly small -- at least that's what I'm hoping for. :-) Marvin. -----Original Message----- From: Donal K. Fellows [mailto:donal.k.fellows@manchester.ac.uk] Sent: Thursday, June 08, 2006 5:49 AM To: Marvin Theimer Cc: A S McGough; ogsa-bes-wg@ggf.org; Ed Lassettre; JSDL Working Group Subject: Re: [ogsa-bes-wg] Questions and potential changes to BES, as seen from HPC Profile point-of-view Marvin Theimer wrote:
Currently the posix extension to jsdl includes various things that I would argue are more generic than posix. I think it would be interesting to define a common subset (that includes things execution pathname, command-line args, and working directory) as well as defining a WindowsApplicationType that describes those things that are Windows-specific.
I believe[*] it is possible to define a new element type that describes an abstract ExecutableApplication element with these generic bits in and which is grandfathered into being a supertype of the POSIXApplication element (as well as being the parent of the WindowsApplication you're positing). This would require a small change to the existing schema, though we'd want to make sure that the documents accepted by the schema remained the same, and would enable this suggestion (which I approve of, by the way) to be slotted into the larger scheme of things without fuss. By the way, what would these extra Windows-specific bits be? I can imagine a number of the POSIX things being dropped (e.g. the limits) but I'm not nearly well-versed enough on Win (despite using it a lot) to be able to think what these extra additions might be. Just out of curiosity! :-) Donal. [* This is believe as in "I've done this sort of thing in other schemas" so I know that XML Schema can support it. ]
participants (10)
-
A S McGough -
Christopher Smith -
Donal K. Fellows -
Ian Foster -
Karl Czajkowski -
Marvin Theimer -
Michel Drescher -
Peter G Lane -
Peter G. Lane
-
Peter Lane