On Jan 19, 2007, at 7:49 AM, Werner Benger wrote:

If we just add a finegranular timestamp in UTC to each msg, would that help here?

It could, but you 1) have to do a system call (expensive context switch) to get the time 2) Worry about timer granularity 3) in the message bus case, there will be time-skew in the hosts attached to the bus, so you'd also have to keep tables of timeskew corrections for each of the hosts involved in the bus 4) We know what the order of the messages are that have arrived, but we only know after-the-fact whether a message has been delayed. So the timestamp might not be sufficient to ensure message ordering. 5) Still have to have some logic to reorder messages on the recv side 6) There is no 6.... at this point it still looks like a tough problem.

Werner

On Thu, 18 Jan 2007 17:36:15 -0600, John Shalf <JShalf@lbl.gov> wrote:

...

On Jan 18, 2007, at 12:18 PM, Andre Merzky wrote:

...

Hi John, Andrei,

you are right: getting some feedback from the transport level folx is certainly a good idea. The API draft won't go into public comment for another month or so (at least), and then it will stay in public comment for another 2 months or longer - that should give us enough time to contact them.

About ordering: the text Andrei cited is in the spec because ordering is, as of now, not an attribute of the connection or endpoint - so the spec tries to nail it down. It says "MUST be ordered, but no global ordering is required" because I thought that this covers the majority of use cases.

I don't think there are use cases which require global ordering - or at least not enough to justify a requirement for global ordering. What is your opinion? Also, thats really difficult to implement in Grids IMHO.

Well as I mentioned before, global ordering is actually a hot topic for debate in folks who are doing the low-level one-sided messaging interfaces (GA/ARMCI vs. UPC/GASNet). The issue with enforcing global ordering is that it limits opportunities for performance optimization and requires a lot more complexity (SW and HW) and software overhead at the endpoints to ensure the ordering is enforced. However, global ordering makes it much easier to send messages that express fences or barriers. As you can imagine, not enforcing ordering (particularly for the message bus case) is a *lot* easier to implement, but makes the concept of fences and simultaneity of events to be more complicted (starts to look like General Relativity brain teasers).

If we want to steer clear of this nasty debate, it seems we should be able to query the ordering enforcement (or request it if available) offered by the underlying protocol.

...

Use cases which do not require ordering should be happy with order preserving connections, too. Question now is: does the benefit of un-ordered implementations (simplier, smaller footprint) justify an attribute on API level? Or are there use cases which require non-ordered delivery for other reasons?

Cheers, Andre.

Quoting [Andrei Hutanu] (Jan 18 2007):

...

Hi,

...

...

2) I see ordering is enforced, could that be an option?

I think ordering is *not* enforced, but I do wonder if it should be an option or a channel property (certainly semireliable will likely result in some reording whereas a TCP channel would enforce ordering of the messages for instance).

This is a controversial topic in the HPC message passing community (whether msg. ordering is a good or bad-thing to enforce in at the hardware level).

I was thinking the same (no strong feelings for either option or property) but the text tells otherwise : In 2.1 introduction : In contrast, this message API extension guarantees that message blocks of arbitrary size are delivered in order, and intact, without the need for additional application level coordination or synchronization. and

then in 2.1.7 reliability corectness and ordering The order of sent messages MUST be preserved by the implementation. Global ordering is, however, not guaranteed to be preserved:

Assume three endpoints A, B and C, all connected to each other. If A sends two messages [a1, a2], in this order, it is guaranteed that both B and C receive the messages in this order [a1, a2]. If, however, A sends a message [a1] and then B sends a message [b1], C may receive the messages in either order, [a1, b1] or [b1, a1].

Andrei

-- "So much time, so little to do..." -- Garfield -- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg

-- ______________________________________________________________________ __ Dr. Werner Benger <benger@zib.de, Werner.Benger@aei.mpg.de> Zuse Institute Berlin ZIB Takustrasse 7 Tel: +49 30 84185-184 D-14195 Berlin-Dahlem, GERMANY Fax: +49 30 84185-107 Max-Planck-Institut fuer Gravitationsphysik Albert-Einstein- Institut Am Muehlenberg 1 Tel: +49 (331) 567-7115 D-14476 Golm bei Potsdam Fax: +49 (331) 567-7298 http://www.photon.at/~werner/