Hi, I'd like to do some more on the BES spec. However, I have a few questions that I want to raise first: * The extensible state model: this seems a great idea. However, I feel concerned that there are difficulties lurking there that we haven't discovered yet. Is there any prior work here that we can refer to in order to make ourselves (and others) feel more confident in the approach? * Information model: I certainly think we need an information model. On the other hand, I wonder if we might want to think about pulling the details out as a "profile." What we have is just what we need for our current HPC use cases; on the other hand, if one is modeling say a Java container, they might be less appropriate. Thoughts? * Management operations: we currently have a couple of operations ("start/stop accepting requests") that are really container management rather than job operations. Could we separate those out in a distinct port-type? * I know there was discussion about how to handle single-job vs. multi-job operations in a consistent manner, but I don't know how that was reconciled. Ian.
* The extensible state model: this seems a great idea. However, I feel concerned that there are difficulties lurking there that we haven't discovered yet. Is there any prior work here that we can refer to in order to make ourselves (and others) feel more confident in the approach?
This was discussed at the telecon and hopefully reflected in the tracker. My initial interpretation was 'adhoc' extensibility but (thankfully IMHO) this could be better described as state model profiles, i.e. one state model will probably not fit all implementation/scenarios therefore several defined models should be profiled. From discussions there should not be that many... but it would allow implementations to have a defined state model if they offered suspension, but they could still produce a BES compliant implementation if they did not and have a defined state model for it.
* Information model: I certainly think we need an information model. On the other hand, I wonder if we might want to think about pulling the details out as a "profile." What we have is just what we need for our current HPC use cases; on the other hand, if one is modeling say a Java container, they might be less appropriate. Thoughts?
The current model in the spec. was felt to enforce/expect too much homogeneity. Altering it to support vectors of description sets was felt essential. Alongside that to have a well defined and required minimum set up then to let communities define additional parameters to be introduced as extensibility elements. For later standardisation
* Management operations: we currently have a couple of operations ("start/stop accepting requests") that are really container management rather than job operations. Could we separate those out in a distinct port-type?
In my view they need to go in the spec, and are different from the generic management activities you might have on a container (i.e. stop the container). If the are rendered into a different port type... I don't see an issue - a tracker beckons.
* I know there was discussion about how to handle single-job vs. multi-job operations in a consistent manner, but I don't know how that was reconciled.
It hasn't. It has not yet been discussed! Steven -- ---------------------------------------------------------------- Dr Steven Newhouse Mob:+44(0)7920489420 Tel:+44(0)23 80598789 Director, Open Middleware Infrastructure Institute-UK (OMII-UK) c/o Suite 6005, Faraday Building (B21), Highfield Campus, Southampton University, Highfield, Southampton, SO17 1BJ, UK
Ian, My two cents. On 22 Jun 2006, at 23:11, Ian Foster wrote:
Hi,
I'd like to do some more on the BES spec. However, I have a few questions that I want to raise first:
* The extensible state model: this seems a great idea. However, I feel concerned that there are difficulties lurking there that we haven't discovered yet. Is there any prior work here that we can refer to in order to make ourselves (and others) feel more confident in the approach?
I too am pretty happy with this. The only thing I might change is to move the three terminal states to a sub state model and only expose "Done" at the top level, but I won't cry if we leave it the way it is. We had very good luck with a nested state model in Unicore. it wasn't extensible, but not all detailed substates were implemented by all platforms, but with the nesting we could always say something about the state. It worked well for both servers that didn't implement all substates and for clients that only understood the higher states. The attached java file describes the Unicore state model, for those interested. There is one thing I am concerned about and that is how do we express these substates? For example we want to be able to combine the concepts of StagingIn and Held without building a complex matrix of all possible combinations. While at the same time, we want StagingIn_Held to be a substate of StagingIn and restricted accordingly, e.g. limiting what the next states can be. What does this look like in XML? Any ideas folks? The simplest would be to use the schema extension model in XML. Define a type State, the restrict it to New, Pending, Running, Cancelled, Failed, and Finished. Then we can restrict Running to get Running_StagingIn, Running_Executing, and Running_StagingOut. But then when we add the notion of held substate on each of these, we get Running_StagingIn_Held, Running_Executing_Held, and Running_StagingOut_Held. I am happy with this, but it does create a large number of state names. Note, this substate only works if the substates really are specializations on their superstate. In particular there is a two part rule that MUST be followed. "All transitions to or from a substate must either be within the super state or reachable through the super states possible transitions." So a transition from Running_Executing_Held to Running_StagingOut would be possible because there is a transition from Running_Executing to Running_StagingOut. However, going from Running_Executing_Held to Pending would not be allowed. Note that although nothing to do with substates, the transition in Figure 3 from Failed to Pending is not legal since in the super state model there is no transition from Failed to Pending. The mistake is repeated in Figure 4.
* Information model: I certainly think we need an information model. On the other hand, I wonder if we might want to think about pulling the details out as a "profile." What we have is just what we need for our current HPC use cases; on the other hand, if one is modeling say a Java container, they might be less appropriate. Thoughts?
I think the information model we have is fairly simple and universal and would like to see it remain as part of the spec. As most elements are optional, I don't think this over burdens non-HPC implementations. The extensibility allows other profiles of course.
* Management operations: we currently have a couple of operations ("start/stop accepting requests") that are really container management rather than job operations. Could we separate those out in a distinct port-type?
There is an issue we just resolved to do just this.
* I know there was discussion about how to handle single-job vs. multi-job operations in a consistent manner, but I don't know how that was reconciled.
A pending issue still. -- Take care: Dr. David Snelling < David . Snelling . UK . Fujitsu . com > Fujitsu Laboratories of Europe Hayes Park Central Hayes End Road Hayes, Middlesex UB4 8FE +44-208-606-4649 (Office) +44-208-606-4539 (Fax) +44-7768-807526 (Mobile)
On Jul 03, David Snelling modulated:
There is one thing I am concerned about and that is how do we express these substates? For example we want to be able to combine the concepts of StagingIn and Held without building a complex matrix of all possible combinations. While at the same time, we want StagingIn_Held to be a substate of StagingIn and restricted accordingly, e.g. limiting what the next states can be. What does this look like in XML?
Maybe it shouldn't be StagingIn_Held, it should be /StagingIn/Held (using XPath syntax): <stagingIn> <held/> </stagingIn> come on, the answer is right there in the phrase "NESTED states". ;-) The advantage here is you can use open extensibility (with the "any other" namepsace restriction, perhaps) to allow anyone to define extensions. As you mentioned, a client can ignore the inner (extended) content if they wish to operate more generically. I agree with your explanation of state transition rules. Note, this is just rephrasing the theory of hypergraphs, so people could look to that for references if they are worried about the ramifications. karl -- Karl Czajkowski karlcz@univa.com
Karl, To be complete your example would ned to be: <Running> <Stagingin> <Held/> </Stagingin> </Running> Right? I need to think more about this, but it may be ok. Could we standardize the meaning of some of the substates, such as <Held/> and <Suspended/>? Then we could get standardized semantics anywhere. E.g. <bes;Running> <ds:DavesSpecialDataMovementState> <bes:Held/> </ds:DavesSpecialDataMovementState> </bes:Running> On 3 Jul 2006, at 08:42, Karl Czajkowski wrote:
On Jul 03, David Snelling modulated:
There is one thing I am concerned about and that is how do we express these substates? For example we want to be able to combine the concepts of StagingIn and Held without building a complex matrix of all possible combinations. While at the same time, we want StagingIn_Held to be a substate of StagingIn and restricted accordingly, e.g. limiting what the next states can be. What does this look like in XML?
Maybe it shouldn't be StagingIn_Held, it should be /StagingIn/Held (using XPath syntax):
<stagingIn> <held/> </stagingIn>
come on, the answer is right there in the phrase "NESTED states". ;-)
The advantage here is you can use open extensibility (with the "any other" namepsace restriction, perhaps) to allow anyone to define extensions. As you mentioned, a client can ignore the inner (extended) content if they wish to operate more generically.
I agree with your explanation of state transition rules. Note, this is just rephrasing the theory of hypergraphs, so people could look to that for references if they are worried about the ramifications.
karl
-- Karl Czajkowski karlcz@univa.com
-- Take care: Dr. David Snelling < David . Snelling . UK . Fujitsu . com > Fujitsu Laboratories of Europe Hayes Park Central Hayes End Road Hayes, Middlesex UB4 8FE +44-208-606-4649 (Office) +44-208-606-4539 (Fax) +44-7768-807526 (Mobile)
Another rendering for this, which Marvin and I explored a little bit when writing our document, is: <ActivityState BaseState="running"> <SubState StateName="profile1:Stagingin> <SubState StateName="profile1:Held"/> </SubState> </ActivityState> It would also be nice to support being in multiple sub states (if it makes sense) because of various profiles and operations, so if (for example) I was in the above state, but the user also suspended the job, I could be in: <ActivityState BaseState="running"> <SubState StateName="profile1:Stagingin> <SubState StateName="profile1:Held"/> </SubState> <SubState StateName="profile2:Suspended"/> </ActivityState> By representing the actual state as the name of these State/SubState elements, then even a client that doesn't understand the sub states can parse the state element (because everything understands "ActivityState" and "SubState". The BaseState values come from an enumeration, since they are well known, and the StateName values are prefixed with the namespace of the profile they are defined in. -- Chris On 03/7/06 03:08, "David Snelling" <David.Snelling@UK.Fujitsu.com> wrote:
Karl,
To be complete your example would ned to be:
<Running> <Stagingin> <Held/> </Stagingin> </Running>
Right?
I need to think more about this, but it may be ok.
Could we standardize the meaning of some of the substates, such as <Held/> and <Suspended/>? Then we could get standardized semantics anywhere. E.g.
<bes;Running> <ds:DavesSpecialDataMovementState> <bes:Held/> </ds:DavesSpecialDataMovementState> </bes:Running>
On 3 Jul 2006, at 08:42, Karl Czajkowski wrote:
On Jul 03, David Snelling modulated:
There is one thing I am concerned about and that is how do we express these substates? For example we want to be able to combine the concepts of StagingIn and Held without building a complex matrix of all possible combinations. While at the same time, we want StagingIn_Held to be a substate of StagingIn and restricted accordingly, e.g. limiting what the next states can be. What does this look like in XML?
Maybe it shouldn't be StagingIn_Held, it should be /StagingIn/Held (using XPath syntax):
<stagingIn> <held/> </stagingIn>
come on, the answer is right there in the phrase "NESTED states". ;-)
The advantage here is you can use open extensibility (with the "any other" namepsace restriction, perhaps) to allow anyone to define extensions. As you mentioned, a client can ignore the inner (extended) content if they wish to operate more generically.
I agree with your explanation of state transition rules. Note, this is just rephrasing the theory of hypergraphs, so people could look to that for references if they are worried about the ramifications.
karl
-- Karl Czajkowski karlcz@univa.com
Chris, On 3 Jul 2006, at 18:49, Christopher Smith wrote:
Another rendering for this, which Marvin and I explored a little bit when writing our document, is:
<ActivityState BaseState="running"> <SubState StateName="profile1:Stagingin> <SubState StateName="profile1:Held"/> </SubState> </ActivityState>
I don't think we need two different tags. ActivityState only should work, as the substate concept is captured structurally. My personal preference is still for the subtyping approach, but I can't see myself winning on that one.
It would also be nice to support being in multiple sub states (if it makes sense) because of various profiles and operations, so if (for example) I was in the above state, but the user also suspended the job, I could be in:
<ActivityState BaseState="running"> <SubState StateName="profile1:Stagingin> <SubState StateName="profile1:Held"/> </SubState> <SubState StateName="profile2:Suspended"/> </ActivityState>
This is both interesting and very scary. On the interesting side this make sense: <ActivityState StateName="running"> < ActivityState StateName="profile1:StagingIn/> < ActivityState StateName="profile1:Held/> </ActivityState> The problems I think will arise when we start to describe the semantics. It is scary enough that I don't want to try thinking about it with out others around and a beer. -- Take care: Dr. David Snelling < David . Snelling . UK . Fujitsu . com > Fujitsu Laboratories of Europe Hayes Park Central Hayes End Road Hayes, Middlesex UB4 8FE +44-208-606-4649 (Office) +44-208-606-4539 (Fax) +44-7768-807526 (Mobile)
Folks, Dave, David Snelling wrote:
Chris,
On 3 Jul 2006, at 18:49, Christopher Smith wrote:
Another rendering for this, which Marvin and I explored a little bit when writing our document, is:
<ActivityState BaseState="running"> <SubState StateName="profile1:Stagingin> <SubState StateName="profile1:Held"/> </SubState> </ActivityState>
I don't think we need two different tags. ActivityState only should work, as the substate concept is captured structurally. My personal preference is still for the subtyping approach, but I can't see myself winning on that one.
+1 on having one element only.
It would also be nice to support being in multiple sub states (if it makes sense) because of various profiles and operations, so if (for example) I was in the above state, but the user also suspended the job, I could be in:
<ActivityState BaseState="running"> <SubState StateName="profile1:Stagingin> <SubState StateName="profile1:Held"/> </SubState> <SubState StateName="profile2:Suspended"/> </ActivityState>
This is both interesting and very scary. On the interesting side this make sense:
<ActivityState StateName="running"> < ActivityState StateName="profile1:StagingIn/> < ActivityState StateName="profile1:Held/> </ActivityState>
The problems I think will arise when we start to describe the semantics.
... let alone the heroic and at the same time cumbersome task to maintain a list of incompatible activity states that come from two (or even more) different profiles.
It is scary enough that I don't want to try thinking about it with out others around and a beer.
As you know, I almost always agree on "spirited" discussions. :-) Cheers, Michel -- Michel <dot> Drescher <at> uk <dot> fujitsu <dot> com Fujitsu Laboratories of Europe +44 20 8606 4834
On 04/7/06 05:15, "David Snelling" <David.Snelling@uk.fujitsu.com> wrote:
Chris,
On 3 Jul 2006, at 18:49, Christopher Smith wrote:
Another rendering for this, which Marvin and I explored a little bit when writing our document, is:
<ActivityState BaseState="running"> <SubState StateName="profile1:Stagingin> <SubState StateName="profile1:Held"/> </SubState> </ActivityState>
I don't think we need two different tags. ActivityState only should work, as the substate concept is captured structurally. My personal preference is still for the subtyping approach, but I can't see myself winning on that one.
The nice thing about having two elements is that you can require the BaseState and StateName attributes, whereas one element would render them both optional. If re-use of StateName for the base states is desired, then we lose the ability to explicitly enumerate the names of the base states. But other than this use of XML, there is no conceptual difference, as far as I can tell. -- Chris
participants (6)
-
Christopher Smith -
David Snelling -
Ian Foster -
Karl Czajkowski -
Michel Drescher -
Steven Newhouse