On Feb 14, Takuya Araki (Biglobe) loaded a tape reading:

Alain, Karl:

Thank you for the excerpts!

So Karl, let me confirm your opinion: Are you thinking of using the method which is implemented in WS-GRAM? If so, I agree with that it increases the reliability of the system, but it doesn't seem to be able to replace asynchronous operations completely.

Yes, that is my suggestion and no I do not completely agree with your assessment... I am attaching a longer description of my position which was drafted as part of a different activity. I think it addresses this topic well enough so I will only paste it and then add a few more comments specifically about WS-Agreement. I apologize for the length. RELIABILITY: One issue that arises in job management is the potentially high stakes of errors in the management protocol. Specifically, while a client may well be expected to tolerate rejection and even execution failure, it is undesireable to have job management state "disappear" or get duplicated due to message-layer, client, or provider failures. We assert that the simple creation pattern is sufficient because there are multiple binding options available to get reliability. One approach is to have transactional bindings which use commit and rollback to make sure a creation occurs exactly once or not at all. However, not all Internet deployments will have transactional messaging, so another approach is to use WS-Addressing MessageID header to get idempotent invocation. This mechanism allows the client to resend an application message in case it missed the response message, and the provider must resend the response without duplicating actions such as actual execution. This "at most once" semantics is sufficient for real world EMS scenarios, as the client will eventually learn whether the execution was accepted or not. GT 4.0 GRAM optionally uses a proprietary message-level concept that is equivalent to the WS-Addressing MessageID in order to work across any binding and because it was designed before the WS-Addressing mechanism was fully clarified by its authors. In this variant, the idempotent ID is sent as an extra field in the input message and interpreted early in the request processing to avoid duplication. The idempotent operation style optimizes the success case by not requiring additional message exchanges unless there is an error condition or timeout. In contrast, a transactional approach requires more exchanges to setup and commit (or cancel) the invocation. ASYNCHRONY: Another concern is how much delay may be encountered in the creation pattern. The WS architecture makes no statements about the relative duration of an "in-out" message exchange, as that is essentially a binding issue. Two camps seem to dislike long message delays for different reasons which are both somewhat inconsistent with the WS architecture model. First, one camp confuses the WSDL message protocol with an API specification, so they believe that a WSDL "in-out" message must mean a blocking procedure call in their client bindings. They are uncomfortable with the implication that a long delay cannot be processed asynchronously by their application. We believe this is a mistaken viewpoint to take when designing protocols, because the asynchrony of the client can be addressed simply by using an appropriate tooling strategy. They should move to better tooling if their existing client stubs are indeed this limiting. For example, the C language WS tooling in GT 4.0 generates synchronous and asynchronous stubs for each WSDL operation, so our GT 4.0 GRAM client tool is able to perform the creation message exchange using asynchronous "post message" and "response callback" programmatic interfaces. The newest JAX RPC revision also is said to have better support for asynchronous invocation. The second dissenting camp is concerned that long response delays will be fragile because some bindings cannot tolerate the delay. For example, a SOAP over HTTP binding may not be able to wait long enough for a response before the TCP connection is lost. Because an "in-out" message pattern addresses the response implicitly via binding-level context, it is not as durable as an explicit peer-to-peer message exchange using "in only" messages sent to explicit endpoints at both peer sites. Unfortunately, this style is also difficult in constrained binding environments because SOAP over HTTP is often valued specifically for being asymmetric and allowing simple NAT/firewall traversal from "anonymous" clients to well-known providers. If we render a peer-to-peer interface model in order to support fragile bindings, we create obstacles for these other common deployment environments. [Please note, WS-Agreement is meant to support this peer-to-peer pattern optionally, but I admit that we may need to make some technical cleanup on the spec before completion... it seems to have lost some details in the time I have been absent from the workgroup discussions. See the optional initiator's EPR field in the create call. What is missing, I think, is clear normative text on how this will be used by the responding party and how/if it should appear in the Agreement context for correlative purposes.] A third solution which happens to address both camps simultaneously is to render explicit "post" and "poll" interfaces to initiate the logical operation and then hold the response at the provider until the client can reconnect and retrieve the result. This supports fragile bindings in NAT/firewall environments and also yields an asynchronous interface with naive tooling that generates synchronous stubs for "in out" message exchanges. However, it complicates the application-level modelling and lifts transport-level message buffering into the application-level service implementation. We argue that a simple "in-out" message exchange in combination with idempotent ID mechanisms can equally well satisfy the fragile bindings and NAT/firewall asymmetry without significant impact on the application protocol. It still retains state at the provider, but rather than adding post/poll operations to the WSDL it simply uses message send for "post" and message resend for "poll". This also means that the application logic can be written using the more natural "in-out" pattern and a simple buffering layer at (or slightly above) the binding code can handle the resends at the provider. This model supports asynchrony because the polling exchange can "block" at the messaging level until the binding times out. In other words, a client who logically iterates with: ID = new_identifier while is_non_response ( result = EPR->create(ID, input content) ) repeat will not "spin" but rather post a new copy of the idempotent create message at the frequency at which the binding signals an error, e.g. a closed connection. While the binding is still functioning, the underlying protocol such as SOAP over HTTP will provide for asynchronous delivery of the response message. (Note of course that the above snippet could be written in a longer psuedo-code format by using an asynchronous post/callback model such as we use in GT4 C bindings. The sychronous create call is nothing more than a post followed by a conditional wait on the callback monitor.) This approach does not permit visibility as to WHY the response is taking so long, but merely visibility as to WHETHER the response has been issued yet. There is no lifecycle model in WS-Agreement for the decision making that the Agreement provider performs while considering an Agreement creation request, nor should we take lightly the burden of trying to develop such a model.

(By the way, it seems that GRAM has "batch mode" as an application level asynchronous operation. That's why the current method is enough for GRAM, I think.)

No, actually our "globusrun" tool's batch mode is not about asynchronous submission. It simply turns off the subscription and state monitoring that the tool normally does after submission. The submission step itself is roughly equivalent to the WS-Agreement createAgreement operation. karl -- Karl Czajkowski karlcz@univa.com

Hi: I've updated the list to reflect lots of new comments.. Best Regards Toshi -- We have moved to a new Office!! Toshiyuki Nakata ????? Internet System Laboratories NEC t-nakata@cw.jp.nec.com 1753, Shimonumabe, Nakahara-Ku, Kawasaki,Kanagawa 211-8666,Japan Tel +81-44-431-7653 (NEC Internal 22-60210) Fax +81-44-431-7681 (NEC Internal 22-60219)

On Feb 17, Takuya Araki (Biglobe) loaded a tape reading:

Karl:

Thank you for your comment.

Here is summary of my understandings:

* I agree with that the asynchronous operations should be implemented at the binding level if there are proper tools. - Here, you think there are such tools - My opinion is currently available tools are not enough for this purpose

* Your suggestion is to implement asynchrony by "polling"; "message ID" of WS-Addressing can be used to know the uniqueness of the message (, which corresponds to "correlation ID" of my proposal) The difference from the polling method of my proposal is: - It is not visible on WSDL level (so there is no impact on the spec) - On the transport level, it uses a normal synchronous operation; when it timeouts, the same message is sent using the same massage ID again.

- Here, your opinion is this suggestion solves the problem. - My opinion is this is useful, but might not be enough to handle time consuming agreement creation.

I understand your opinion that this kind of "implementation issue" should not affect the specification. However, I feel it is in the future that proper tools solves the problem completely. Therefore, I believe that the WSDL level solution is useful to implement practical systems using currently available tools.

My proposal is perfectly optional, and can be safely ignored if you don't like it.

I agree with your summary. I have a question: would the binding-level method I propose be satisfactory for the asymmetric (no initiator-side endpoints) case? What I propose is that we leave the current model more or less untouched and focus on reintroducing/repairing the symmetric peer-to-peer creation mechanism to address asynchrony at the WSDL level. I went through the public draft very carefully on a long flight the other day and realize there are a number of problems relating to the underspecification and inconsistency of the symmetric case. The creation input's optional "initiator EPR" is useless as the spec stands! I think some related state information was removed in the effort to get rid of negotiation, and this part was overlooked. I am not 100% sure how much overlap there is between this and your proposal, so let me phrase it in terms of adjustments to the public draft that I have been reading: 1) Remove the optional "initiator EPR" from the createAgreement() input. The simple createAgreement() is always asymmetric. 2) Introduce a new FOO() operation which has a mandatory "acceptance EPR" field along with the offer and which outputs an empty message (ACK) or fault. A fault indicates that the offer cannot be considered, e.g. the initiator should not expect any further messages regarding this offer. 3) The acceptance EPR must address an endpoint providing two new operations: -- acceptAgreement() which takes as input the EPR to the newly formed Agreement provided by responder, e.g. the same EPR that would have appeared in the simple createAgreement output -- rejectAgreement() operation which takes a fault as input, e.g. the same fault that would have appeared in the simple createAgreement output 4) the responder MUST drive the acceptAgreement() operation on the acceptance EPR to accept, or the rejectAgreement() operation to reject, once he has made his decision. I think this is in the spirit of the original spec drafts and the whole idea of symmetric parties. To make it completely symmetric: 5) the acceptAgreement() operation outputs the EPR of an Agreement provided by the initiator, allowing both parties to know an Agreement EPR hosted by the other party to which further Agreement-related operations can be directed. This becomes an underpinning for future extensions where the parties can signal more interesting Agreement conditions/changes to one another. I called it "Foo()" above because the naming depends on what is decided for this last feature. I would call it createSymmetricAgreement if we included (5). I would call it something like requestAgreement if we do (1)-(4) only. I continue to be uncomfortable using the words "synchronous" or "asynchronous" to name message patters because they are always asynchronous in the WS architecture. I think synchrony only makes sense when talking about APIs where control is passed from one piece of code to another.

Are you talking here about API level asynchrony?

I am stating that some recurring discussions come from people trying to address API-level concerns in the WSDL instead of in the tooling where they belong. I was not sure if you were stating any of these arguments or not.

If so, API level asynchrony is also important, but what we are discussing is transport level asynchrony, I think.

Good, I think we are on the same page then.

(I checked the recent version of JAX RPC spec. It says it supports "a client level asynchrony", but I couldn't figure out if it also supports transport level asynchrony...)

It's an API concern, which is why I brought it up as a response to those trying to address API asynchrony. Of course, transport asynchrony depends on binding protocol rather than a WSDL->API translation scheme.

...

... This approach does not permit visibility as to WHY the response is taking so long, but merely visibility as to WHETHER the response has been issued yet. There is no lifecycle model in WS-Agreement for the decision making that the Agreement provider performs while considering an Agreement creation request, nor should we take lightly the burden of trying to develop such a model.

Sorry, I couldn't catch the point of these sentences. Are you saying that the WSDL level asynchronous operations should manage some kind of lifecycle model in the provider? If so, well, I admit that there is something which should be done in the application level...

No, I was merely trying to state that we would have to consider what we are doing if we introduced some two-level factory scheme, e.g. an initial "lightweight" factory create call that returns the EPR of a new "pending request" resource which the client can poll to find the eventual "accepted agreement" EPR. I don't think this would be that much more powerful than the binding-level solution I have promoted, and it opens the can of worms as to what is the lifecycle of a "pending request". Another way of rendering this two-level scheme would be to just return the Agreement EPR in a new enhanced lifecycle where it has not been accepted yet. An asynchronous state-change can indicate whether it is accepted or rejected once the provider knows. This is pretty much what our original state machine model looked like before we factored out negotiation. :-) It also happens to be what GRAM does today. :-(

Sorry about the misunderstanding. But my point is that GRAM does not use time consuming operations, which seems to be why the current method is enough for GRAM...

I suppose you may be right. We have not done much testing with long creation delays, because our current GRAM submission semantics are not exactly analogous to Agreement.

My concern about using your suggestion for operations which take a long time is as follows: * As I mentioned in the previous mail, a client program can't distinguish timeout caused by message layer problems; it just thinks that the operation is taking a long time, and might retry forever.

* The server should keep the HTTP connection for the long period, (though the connection is disconnected and reconnected periodically). That might be intolerably high overhead for the server, if there are a lot of agreement creation requests. (Usually, there is a limit of number of HTTP connection.)

I had not really considered that holding HTTP connections open would be a scalability problem... I know most browsers do it by default these days and I assume binding-level technology will be improved as needed as the WS methodology matures. But I will accept that it could be a practical limit today. Would a corrected version of the symmetric agreement mechanism satisfy your needs here, or do you also still feel that a two-phase creation mechanism must be available for asymmetric initiators who will not host any WS-Agreement related endpoints? karl -- Karl Czajkowski karlcz@univa.com

Karl, hope it helps,

attachment, so I cannot review the specific changes. (Does anyone know of a public archive where I can easily retrieve such GRAAP messages with attachments?)

http://www-unix.gridforum.org/mail_archive/graap-wg/threads.html And his message is; http://www-unix.gridforum.org/mail_archive/graap-wg/2004/11/msg00004.html ---- Hiro Kishimoto

-----Original Message----- From: owner-graap-wg@ggf.org [mailto:owner-graap-wg@ggf.org] On Behalf Of Karl Czajkowski Sent: Monday, February 21, 2005 12:30 PM To: Takuya Araki (Biglobe) Cc: graap-wg@gridforum.org Subject: Re: [graap-wg] asynchronous binding

On Feb 19, Karl Czajkowski loaded a tape reading: ...

...

Would a corrected version of the symmetric agreement mechanism satisfy your needs here, or do you also still feel that a two-phase creation mechanism must be available for asymmetric initiators who will not host any WS-Agreement related endpoints?

karl

While waiting for a response, I dug back in my email archive and found this comment from Takuya on 1 Dec 2004:

I added asynchronous request operations (e.g. createAgreementAsync), polling operations for getting the result (e.g. createAgreementGetResult), and response operations (e.g. createAgreementResult). Please see the attached document for detail. It also includes some minor comments to the original specification.

but unfortunately my archive does not retain the Word document attachment, so I cannot review the specific changes. (Does anyone know of a public archive where I can easily retrieve such GRAAP messages with attachments?)

Just based on the above summary, I would say that I am proposing we drop the createAgreementGetResult operation in favor of an equivalent binding-level behavior to reliably get (or "re-get" by polling) the result of the existing createAgreement call in case it takes too long for the baseline bindings. I am also proposing that we attempt to incorporate his notion of a "reverse direction" createAgreementResult operation (which I separated as acceptAgreement and rejectAgreement) into a repair for the now underspecified and broken symmetric agreement pattern.

Does anybody else have opinions on these issues? Will there be a GRAAP call today (Monday 20 Feb)?

karl

-- Karl Czajkowski karlcz@univa.com

On Feb 20, Karl Czajkowski loaded a tape reading: ...

Just based on the above summary, I would say that I am proposing we drop the createAgreementGetResult operation in favor of an equivalent binding-level behavior to reliably get (or "re-get" by polling) the result of the existing createAgreement call in case it takes too long for the baseline bindings. I am also proposing that we attempt to incorporate his notion of a "reverse direction" createAgreementResult operation (which I separated as acceptAgreement and rejectAgreement) into a repair for the now underspecified and broken symmetric agreement pattern.

Thanks to all who pointed out that the GGF site does indeed have an archive for the GRAAP-WG mailing list. :-) After reviewing the proposed changes, I think the above comments still stand. The only additional "async" interface proposal was for termination, and I would also suggest dropping that because it is my belief that there is no implication of long delays in this operation. Termination is an inherently asynchronous mechanism by which the client requests termination and finds out if his request is acceptable or not; he does not get some response synchronized to happen "after" termination but he should assume the resource is not to be accessed if the response is successful. This is a point that I think was argued extensively (and convincingly, to me) in the OGSI and WSRF work groups. I actually would question why we have a wsag:Terminate instead of just using the WS-ResourceLifetime mechanism. I cannot see what value it adds or what would be different about the semantics. The text in the Agreement Context section is hardly convincing on this point. :-( Is it supposed to I am afraid I must have been absent during the time when this was decided... karl -- Karl Czajkowski karlcz@univa.com

Karl, on your issue with termination:

I actually would question why we have a wsag:Terminate instead of just using the WS-ResourceLifetime mechanism. I cannot see what value it adds or what would be different about the semantics. The text in the Agreement Context section is hardly convincing on this point. :-( Is it supposed to I am afraid I must have been absent during the time when this was decided...

I think we might be mixing semantics. There is a difference between the lifetime of the agreement instance and the state of the agreement instance to be observed. The terminate message is supposed to move the agreement state to afterObserved while the WSRF lifetime addresses how long the Agreement service is supposed to be around. We might argue that thi sshould be the same but, in fact, I tend to think to keep it separate. Heiko

Karl: Thank you for your comments and new proposal. (And sorry about the late response.) Basically, I agree with your new proposal. * Your requestAgreement, accepctAgreement, rejectAgreement operations are mostly same as my proposal. Its OK for me to change operation names. - Letting acceptAgreement create a new service ( "5)" in your proposal) seems to be a new topic. If you want to include this, it might be better to discuss it as a different topic in the WG. - It is OK to drop async version of the Terminate operation as you mentioned in another mail. I thought that it might be used as "destructor" of the agreement; there might be something like clean-up which takes some time. If there are dependent agreements and they should be also terminated at the time, the whole time might be long. However, your discussion in another mail (terminate operation is inherently asynchronous operation, etc.) seems to be also convincing. (Terminate operation can return immediately if the provider accepts the termination; clean-ups and calling terminate operations of dependent agreement can be executed asynchronously...) Therefore, I agree with dropping async version of Terminate operation. * It's a bit pity to drop WSDL level polling operation :-) (considering scalability, etc.), but if we have request/accept/rejectAgreement operations, they can be used if scalability is important. Some other comments are in line:

... No, I was merely trying to state that we would have to consider what we are doing if we introduced some two-level factory scheme, e.g. an initial "lightweight" factory create call that returns the EPR of a new "pending request" resource which the client can poll to find the eventual "accepted agreement" EPR. I don't think this would be that much more powerful than the binding-level solution I have promoted, and it opens the can of worms as to what is the lifecycle of a "pending request".

Another way of rendering this two-level scheme would be to just return the Agreement EPR in a new enhanced lifecycle where it has not been accepted yet. An asynchronous state-change can indicate whether it is accepted or rejected once the provider knows. This is pretty much what our original state machine model looked like before we factored out negotiation. :-) It also happens to be what GRAM does today. :-(

Actually, "the EPR of a new pending request resource" is not created in my proposal. The polling method is defined at the factory; to identify the request, I used correlation ID. In this case, it seems to have less lifecycle problem than two-level factory scheme.

Would a corrected version of the symmetric agreement mechanism satisfy your needs here, or do you also still feel that a two-phase creation mechanism must be available for asymmetric initiators who will not host any WS-Agreement related endpoints?

If there are request/accept/rejectAgreement operations, most of my needs can be satisfied, I think. If you still feel uncomfortable about the polling operation I proposed(, though it is not two-level factory scheme), I agree with dropping the polling operation and using the binding level solution instead for the asymmetric case. -- Takuya Araki Grid System Technology Group Internet Systems Research Laboratories NEC Corporation