Hello: I had a brief chat with Karl on the issue below. Basically, Takuya and I agreed that the proposal 1) seems attractive. The only worry that I mentioned to Karl was, the re-introduction of the agreement state. I vaguely remember that the state had been dropped because of the following discussion After the agreement phase, it would be difficult to define the agreement state as some of the SDT's might be observed and some others might not be and that it was rather meaningless to have a state of its own. If this was the real reason it might be good to resurrect the agreement state with the understanding that it would show the status of what the agreement process was. Comments? BestRegards Toshi Takuya Araki (Biglobe) wrote:

Karl:

Basically, I agree with your (another) new proposal.

Here is detailed opinion about it. There are pros and cons: * pros - Programs don't have to create/manage correlation ID (This is good!). * cons - Your proposal 1) * This affects other part of the spec (introduces a new agreement state). * Programs might need to be aware of the agreement state; this might make the program complex. + For example, in the original spec, if an agreement service (resource) and a domain-specific service are implemented as one service, the operations of the domain-specific service can be implemented assuming that the agreement exists when they are called. However, in this proposal, the operations of the domain-specific service should check the agreement state everytime, and return fault or something if the state is "pre-agreement". This should be implemented in all operations, which might be burden of programmers. (We might need AspectJ :-) (However, I admit that if we include the Terminate operation, this kind of state management should be done anyway...) - Your proposal 2) * There are no state problems, but it looks a bit awkward compared to 1)... For example, care should be taken about the lifecyle of the two services (as you mentioned in the previous mail). - Using notification makes the spec depend on WS-BaseNotification, etc. I am not against using it, but it increases the complexity of the spec anyway.

My proposal took into account of keeping simplicity and avoiding change to the current spec. But if this level of complexity and change to the spec are allowed, I agree with your proposal.

I also think your proposal 1) seems to be better than 2), if the complexity can be justified by other reasons, like future negotiation mechanism can be implemented on top of that. -- Takuya Araki Grid System Technology Group Internet Systems Research Laboratories NEC Corporation <t-araki@dc.jp.nec.com>

...

-----Original Message----- From: owner-graap-wg@ggf.org [mailto:owner-graap-wg@ggf.org] On Behalf Of Karl Czajkowski Sent: Thursday, February 24, 2005 12:34 PM To: GRAAP-WG Subject: [graap-wg] More async/asymmetry comments

I am sorry I missed the telecon.

I have been giving this asymmetry/asynchrony topic more thought and discussing it with some WSRF/Web Service experts in the Globus Alliance. I still think that binding-level reliability can extend the utility of the basic "synchronous" Agreement creation patters and that we should repair the "initiator EPR" mess to render the peer-to-peer asynchronous pattern proposal (with the accept/reject operations).

Additionally, I am persuaded that as a practical matter, it may be worth rendering the asymmetric (client-server) interface to support asynchronous (post+poll) messaging because of limitations in the binding and tooling world today. However, I am uncomfortable with the standing proposal and how it uses correlation IDs. As was pointed out to me, there is a related distributed systems management problem of deciding how this ID "namespace" is managed and how information is buffered, e.g. for how long.

The low-level binding solution that uses Message-ID has some unaddressed quality of service issues pertaining to how long the IDs are remembered in order to enforce idempotence. It also has security issues pertaining to whether one client can deny service to another by anticipating and "polluting" IDs. I think it would be unfortunate if we replicated all of these flaws in the WS-Agreement WSDL interface.

The view I share w/ my Globus colleagues is that we should use WSRF resources to model the buffering state and use those resource EPRs as the correlation IDs, since we are already using WSRF. This alternative is what I was hinting at in an earlier message as "another layer of resources".

There are really two reasonable variants to doing this, depending on how we wish to think about the lifecycle of Agreement resources.

1) Change the Agreement resource lifecycle to again capture "pre-agreement" and possibly "post-agreement" states.

The pre-agreement state can be used to allow a simple low-latency Agreement creation step and then have the "responder agrees to offer" decision made asynchronously. The initiator can poll or subscribe for notifications to learn what decision is made.

The post-agreement state would capture the notion Heiko is arguing for that it is possible to "expire" or "cancel" an agreement without destroying the service interface. I mention this only to carry through the modelling approach; the pre and post states are really independent concepts.

2) Introduce separate PreAgreement and PostAgreement resources that have lifetimes linked with the existing Agreement resource.

The PreAgreement resource represents the messaging state for determining whether the responder will agree to the offer or not. When he agrees, the initiator can invoke an operation on the PreAgreement to obtain the Agreement resource EPR.

Similarly, the PostAgreement resource would represent the Agreement after it is "expired" or "canceled", obviating the need for an Agreement resource that exists after Heiko's agreement termination?

I suppose you could mix the two, using a PreAgreement resource and having a post-agreement state on the Agreement resource, but I think that is a bit strange. I will try to start a separate thread about the termination stuff again...

Personally, I prefer (1) as I think it sets us up with a better base on which to integrate future negotiation mechanisms. The Agreement provides a generic introspection interface to present different negotiation lifecycles. Optional RPs could provide info on the specific negotiation process, either as extended pre-agreement states or as references to some other service domain where that info can be found. It was also argued by my WSRF colleagues that this would be easier to implement in practice than a flurry of "smaller" resources, although that might depend on implementation technology?

karl

-- Karl Czajkowski karlcz@univa.com

-- We have moved to a new Office!! Toshiyuki Nakata ????? Internet System Laboratories NEC t-nakata@cw.jp.nec.com 1753, Shimonumabe, Nakahara-Ku, Kawasaki,Kanagawa 211-8666,Japan Tel +81-44-431-7653 (NEC Internal 22-60210) Fax +81-44-431-7681 (NEC Internal 22-60219)

This is actually a very messy question. :-) There are two related problems: the lack of transactional control to group the separate operations, and the lack of QoS guarantees for notification. For the transaction problem, you are worried about a race-condition between the creation and subscription where a state change would be missed. This can be addressed in a number of ways that are all unfortunately not standards: 1) require that a subscription always trigger a notification of the "current" state at the time of subscription in addition to future state changes, etc. 2) do one state query after subscription to get approximately the same value as in (1), with the added excitement of interpreting this result that is delivered outside the notification stream. 3) provide a merged createAgreement and subscribe operation; this is what we do in WS-GRAM in GT 4.0, both to avoid the race-condition and reduce the number of message exchanges. However, even with one of these solutions there is another set of risks: notification does not really guarantee reliability or ordered delivery of notifications that the provider "tried" to send. Of course, Web services in general do not provide message delivery guarantees, but the lack of a response to the deliverNotification operation means that the sender cannot even know if the message was received successfully. One solution for WS-Agreement would be use the peer-to-peer interface variant and use the proposed domain-specific accept() operation in favor of generic subscription/notification for this important state transition. In Globus, our long-term architectural solution to the generic notification QoS problem has been to assume a soft-state methodology w/ temporal metadata. We would advocate sending a stream of redundant messages within certain max/min frequencies and letting the consumer sort it duplication or loss. This leads to a system that will converge on steady state values and may diverge under loss/congestion. These concepts were built into OGSI from the start. However, the WSRF specifications do not provide such primitives and we have not introduced extensions for this yet. So, in GT 4.0 it is incumbent on the subscriber to actually poll values as a fall-back to avoid hanging forever in the presence of notification failures! Furthermore, the network could reverse the order of multiple notifications which might be confusing for an overly-trusting consumer, depending on the state model for the topic in question. With the right temporal metadata, a consumer of a notification stream can reorder or filter out-of-order messages to avoid violating state invariants. Even without such metadata, one can often filter out meaningless or "backwards" transitions, e.g. even our "DUROC" co-allocator that consumed GRAM state change messages would guard against non-sensical job state notifications by comparing them to the currently held view of the remote job's state before doing any further processing. karl On Mar 03, Toshiyuki Nakata loaded a tape reading:

If the initiator decided to subscribe, can you tell me the sequence of subscribe/notify?

1)my original assumption. 1.1) Initiator requests CreateAgreement 1.2)Provider creates an empty Agreement (With agreement state Not-Yet-Complete)and returns an EPR to the created Agreement. 1.3)Initiator subscribes for the notification using the returned EPR.

My worry is what if the provider had been very quick and had agreed to the agreement and had changed the agreement status to Agreed between timing 1.2 and 1.3?

Would the Initiator be notified of this at the timing of 1.3? or would the initiator have to do some kind of atomic chack and subscribe?

Best regards Toshi

-- We have moved to a new Office!! Toshiyuki Nakata ????? Internet System Laboratories NEC t-nakata@cw.jp.nec.com 1753, Shimonumabe, Nakahara-Ku, Kawasaki,Kanagawa 211-8666,Japan Tel +81-44-431-7653 (NEC Internal 22-60210) Fax +81-44-431-7681 (NEC Internal 22-60219)

-- Karl Czajkowski karlcz@univa.com