Fwd (hupfeld@zib.de): Re: SAGA Strawman API Version 1.0
Ha, finally someone really critical :-D Thanks Felix, comments are invited. Cheers, Andre. ----- Forwarded message from Felix Hupfeld <hupfeld@zib.de> -----
I think there is a problem in the foundation of the SAGA approach. The underlying assumptions seem to be that: (a) prospective designers of Grid applications are unaware of distr. system issues (b) distr. system issues are not fundamental, but can be solved by technical means
on (a): Designers of Grid applications are presumably already writing parallel applications on clusters or large HPC machines. They are well aware that remote memory is not local memory and that they can't use DSM just like local memory without serious performance issues. To name one of the fundamental problems of writing distr. systems...
on (b): There are fundamental tradeoffs to be considered when building distributed systems (DS), like availability vs. consistency, scalabillity vs. consistency, ... State of the art of CS is that they can't be solved by technical means but are laws of the nature of DS. In practice that means: if you design an API for a DS, where you _must_ state which consistency guarentees you give, you assume certain requirements of the user and force him into your choice of the tradeoff spectrum.
With assumptions (a) and (b), the designers of SAGA seem to persue an approach, where the user has a very-POSIX like API. Although the spec. ignores all DS issues, it seems to imply that the API gives the user a single-copy view of the overall Grid, which makes the strongest choice on consistency, but results in the worst possible scalability and availability.
To illustrate, I will elaborate on the File API, but the same arguments can be made for namespaces and logical files, replicas, ...
The spec. does not make any statements about the consistency guarantees of the read and write operations. I assume that should imply that it gives POSIX like read and write guarantees, resulting in bad scalability and availability. The authors must explicitly state that. If the consistency guarantees are left open and governed by the API implementation, no applications can be written towards that API. You can't write applications if you don't known when you will be able to read the data you have written on local or remote instances.
A solution to this problem would be the specification of required consistency guarantees by the applications at open() time. This, however, would require thorough research on conistency models and application requirements, so I can't propose a few of hand.
If SAGA choses to give single-system like guarantees, this must be explicititely stated. All interfaces that deal with data are unusable without a specification of consistency guarantees.
However, if you chose to be as POSIX like as possible and have single-system semantics, why not directly use parts of the POSIX spec. and strip it down where needed?
----- End forwarded message ----- -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
<snip> If SAGA choses to give single-system like guarantees, this must be explicititely stated. All interfaces that deal with data are unusable without a specification of consistency guarantees.
I don't believe that it is possible, or even desirable, to try to make distributed systems look like they are not distributed. For example, I don't think you should provide POSIX behaviour on a distributed filesystem. If you look at AFS, it doesn't fit the POSIX model. Most people write code that ignores what the filesystem might be, and assume POSIX. How many people check the failure status on a file close? With AFS, you can get "Host not found" when you do a file close. You can wait, and try again. If you quit, your changes are lost. (As a library writer, you can try and "squash" the errors by putting a clever layer of code between the app and the filesystem that know tricks like this. The Condor people do this, I seem to recall.) The point here isn't that developers should never assume a POSIX filesystem, it is that they should know what kind of filesystem they are dealing with, so that they can write appropriate code. When you go distributed, there are a whole new set of error conditions that can occur. I don't think that there is anything to be gained from pretending that remote objects are the same as local objects, so that people's code can stay the same. If the code doesn't know it's dealing with something that is remote, rather than local, then at best (i.e. if there is lots of error checking) it will fail far more often. Probably though, it won't be robust. It might be worth looking at the following paper, which says eloquently what I'm grasping for. "A note on distributed computing" by Jim Waldo et al., available from: http://research.sun.com/techrep/1994/abstract-29.html Cheers, Jon.
Hi, in a chat with John Shalf, he offered following opinion to the topic of consistence, which he agreed to let me post to the list. The paper John refers to is the paper cited by Jon McLaren (see quoting below). SSI here means Single System Image. John: I think SAGA has no business defining a consistency model, but it should be able to accomodate consistency models that exist in the underlying implementations. SAGA is just the API. We should lay out some existing consistency models and make sure the error handling supports it. The problem with POSIX+NFS that was pointed out in that paper was that they could not extend the error codes that were available to POSIX. Felix is right on that account. However, most grid technologies are trying their best to create an SSI consistency model. The existence proof is there (WAN-GPFS provides the same consistency model semantics as a local FS connection). So I reject any claim that a "Grid" filesystem neccessitates some bizzaro consistency model that is not the same as Single System Image (SSI). I think any remote-filesystem strategy should have SSI as its goal ultimately. I think it is somewhat of a religious battle as to whether a remote consistency model must be radically different than the local one. Certainly, we need to deal with different kinds of failures (eg. an open filehandle that suddenly becomes unavailable). It should be discussed at GGF [...]. But just to keep the discussion focused, the core comment is "SAGA has no business defining a consistency model, but it should be able to accomodate consistency models that exist in the underlying implementations." and also to say that "Felix is right about the programmer potentially having to worry about the worst-case." [...] Best Regards, Andre. Quoting [Jon MacLaren] (Jun 15 2005):
<snip> If SAGA choses to give single-system like guarantees, this must be explicititely stated. All interfaces that deal with data are unusable without a specification of consistency guarantees.
I don't believe that it is possible, or even desirable, to try to make distributed systems look like they are not distributed. For example, I don't think you should provide POSIX behaviour on a distributed filesystem. If you look at AFS, it doesn't fit the POSIX model. Most people write code that ignores what the filesystem might be, and assume POSIX. How many people check the failure status on a file close? With AFS, you can get "Host not found" when you do a file close. You can wait, and try again. If you quit, your changes are lost. (As a library writer, you can try and "squash" the errors by putting a clever layer of code between the app and the filesystem that know tricks like this. The Condor people do this, I seem to recall.)
The point here isn't that developers should never assume a POSIX filesystem, it is that they should know what kind of filesystem they are dealing with, so that they can write appropriate code. When you go distributed, there are a whole new set of error conditions that can occur. I don't think that there is anything to be gained from pretending that remote objects are the same as local objects, so that people's code can stay the same. If the code doesn't know it's dealing with something that is remote, rather than local, then at best (i.e. if there is lots of error checking) it will fail far more often. Probably though, it won't be robust.
It might be worth looking at the following paper, which says eloquently what I'm grasping for.
"A note on distributed computing" by Jim Waldo et al., available from: http://research.sun.com/techrep/1994/abstract-29.html
Cheers,
Jon.
-- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
Hi, the quoted conversation its just one side of a conversation in AIM. As such, I think many of these statements are likely to get misinterpreted because it does not include Andre's side of the conversation (every other paragraph was an exchange between us). So, in order to avert any flame wars on this list, the focus here is 1) Its probably inappropriate for SAGA to definine a consistency model, given it is only an API. However, it should be able to accomodate consistency models that exist in the underlying implementations. This is not to say consistency models are not important, but that the API can't be used to impose a consistency model. Its the other way around... the consistency model is going to impose conditions on the API. 2) At GGF it would be worthwhile to walk through some existing consistency models and find out how they map into the API (are there error states or features that are not accomodated?) 3) Felix is correct that such a model potentially exposes the programmer to the worst-case scenario in terms of consistency models. This is one of the risks involved in anything that carries the moniker "simple". On Jun 18, 2005, at 4:26 AM, Andre Merzky wrote:
Hi,
in a chat with John Shalf, he offered following opinion to the topic of consistence, which he agreed to let me post to the list. The paper John refers to is the paper cited by Jon McLaren (see quoting below). SSI here means Single System Image.
John:
I think SAGA has no business defining a consistency model, but it should be able to accomodate consistency models that exist in the underlying implementations.
SAGA is just the API. We should lay out some existing consistency models and make sure the error handling supports it. The problem with POSIX+NFS that was pointed out in that paper was that they could not extend the error codes that were available to POSIX.
Felix is right on that account.
However, most grid technologies are trying their best to create an SSI consistency model. The existence proof is there (WAN-GPFS provides the same consistency model semantics as a local FS connection).
So I reject any claim that a "Grid" filesystem neccessitates some bizzaro consistency model that is not the same as Single System Image (SSI).
I think any remote-filesystem strategy should have SSI as its goal ultimately. I think it is somewhat of a religious battle as to whether a remote consistency model must be radically different than the local one. Certainly, we need to deal with different kinds of failures (eg. an open filehandle that suddenly becomes unavailable).
It should be discussed at GGF [...].
But just to keep the discussion focused, the core comment is "SAGA has no business defining a consistency model, but it should be able to accomodate consistency models that exist in the underlying implementations." and also to say that "Felix is right about the programmer potentially having to worry about the worst-case."
[...]
Best Regards,
Andre.
Quoting [Jon MacLaren] (Jun 15 2005):
<snip> If SAGA choses to give single-system like guarantees, this must be explicititely stated. All interfaces that deal with data are unusable without a specification of consistency guarantees.
I don't believe that it is possible, or even desirable, to try to make distributed systems look like they are not distributed. For example, I don't think you should provide POSIX behaviour on a distributed filesystem. If you look at AFS, it doesn't fit the POSIX model. Most people write code that ignores what the filesystem might be, and assume POSIX. How many people check the failure status on a file close? With AFS, you can get "Host not found" when you do a file close. You can wait, and try again. If you quit, your changes are lost. (As a library writer, you can try and "squash" the errors by putting a clever layer of code between the app and the filesystem that know tricks like this. The Condor people do this, I seem to recall.)
The point here isn't that developers should never assume a POSIX filesystem, it is that they should know what kind of filesystem they are dealing with, so that they can write appropriate code. When you go distributed, there are a whole new set of error conditions that can occur. I don't think that there is anything to be gained from pretending that remote objects are the same as local objects, so that people's code can stay the same. If the code doesn't know it's dealing with something that is remote, rather than local, then at best (i.e. if there is lots of error checking) it will fail far more often. Probably though, it won't be robust.
It might be worth looking at the following paper, which says eloquently what I'm grasping for.
"A note on distributed computing" by Jim Waldo et al., available from: http://research.sun.com/techrep/1994/abstract-29.html
Cheers,
Jon.
-- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
Hi, the quoted conversation its just one side of a conversation in AIM. As such, I think many of these statements are likely to get misinterpreted because it does not include Andre's side of the conversation (every other paragraph was an exchange between us).
So, in order to avert any flame wars on this list, the focus here is 1) Its probably inappropriate for SAGA to definine a consistency model, given it is only an API. However, it should be able to accomodate consistency models that exist in the underlying implementations. This is not to say consistency models are not important, but that the API can't be used to impose a consistency model. Its the other way around... the consistency model is going to impose conditions on the API.
2) At GGF it would be worthwhile to walk through some existing consistency models and find out how they map into the API (are there error states or features that are not accomodated?)
3) Felix is correct that such a model potentially exposes the programmer to the worst-case scenario in terms of consistency models. This is one of the risks involved in anything that carries the moniker "simple". I see the point that it is hard guarantee a specific consistency model in SAGA or make it a configuration option to choose one. But if you completely ignore
On Saturday 18 June 2005 17:21, John Shalf wrote: this topic in the specs you will end up with a good spec of the syntax but without semantics. My 0.02€, Thorsten
On Jun 18, 2005, at 4:26 AM, Andre Merzky wrote:
Hi,
in a chat with John Shalf, he offered following opinion to the topic of consistence, which he agreed to let me post to the list. The paper John refers to is the paper cited by Jon McLaren (see quoting below). SSI here means Single System Image.
John:
I think SAGA has no business defining a consistency model, but it should be able to accomodate consistency models that exist in the underlying implementations.
SAGA is just the API. We should lay out some existing consistency models and make sure the error handling supports it. The problem with POSIX+NFS that was pointed out in that paper was that they could not extend the error codes that were available to POSIX.
Felix is right on that account.
However, most grid technologies are trying their best to create an SSI consistency model. The existence proof is there (WAN-GPFS provides the same consistency model semantics as a local FS connection).
So I reject any claim that a "Grid" filesystem neccessitates some bizzaro consistency model that is not the same as Single System Image (SSI).
I think any remote-filesystem strategy should have SSI as its goal ultimately. I think it is somewhat of a religious battle as to whether a remote consistency model must be radically different than the local one. Certainly, we need to deal with different kinds of failures (eg. an open filehandle that suddenly becomes unavailable).
It should be discussed at GGF [...].
But just to keep the discussion focused, the core comment is "SAGA has no business defining a consistency model, but it should be able to accomodate consistency models that exist in the underlying implementations." and also to say that "Felix is right about the programmer potentially having to worry about the worst-case."
[...]
Best Regards,
Andre.
Quoting [Jon MacLaren] (Jun 15 2005):
<snip> If SAGA choses to give single-system like guarantees, this must be explicititely stated. All interfaces that deal with data are unusable without a specification of consistency guarantees.
I don't believe that it is possible, or even desirable, to try to make distributed systems look like they are not distributed. For example, I don't think you should provide POSIX behaviour on a distributed filesystem. If you look at AFS, it doesn't fit the POSIX model. Most people write code that ignores what the filesystem might be, and assume POSIX. How many people check the failure status on a file close? With AFS, you can get "Host not found" when you do a file close. You can wait, and try again. If you quit, your changes are lost. (As a library writer, you can try and "squash" the errors by putting a clever layer of code between the app and the filesystem that know tricks like this. The Condor people do this, I seem to recall.)
The point here isn't that developers should never assume a POSIX filesystem, it is that they should know what kind of filesystem they are dealing with, so that they can write appropriate code. When you go distributed, there are a whole new set of error conditions that can occur. I don't think that there is anything to be gained from pretending that remote objects are the same as local objects, so that people's code can stay the same. If the code doesn't know it's dealing with something that is remote, rather than local, then at best (i.e. if there is lots of error checking) it will fail far more often. Probably though, it won't be robust.
It might be worth looking at the following paper, which says eloquently what I'm grasping for.
"A note on distributed computing" by Jim Waldo et al., available from: http://research.sun.com/techrep/1994/abstract-29.html
Cheers,
Jon.
-- +-----------------------------------------------------------------+
| Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | |
+-----------------------------------------------------------------+
Not w/o semantics, but the programmer has to assume worst case. And, as Jon & John point out, that is not so different from what you have 'locally'. If you open a file for write, you SHOULD assume worst case, that is the file is on NFS/AFS, and POSIX consistency is NOT guaranteed. A. Quoting [Thorsten Schuett] (Jun 20 2005):
I see the point that it is hard guarantee a specific consistency model in SAGA or make it a configuration option to choose one. But if you completely ignore this topic in the specs you will end up with a good spec of the syntax but without semantics.
My 0.02???, Thorsten
-- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
Hi Andre, all, Actually, that wasn't the point that I was making. My point is that the code writer has to be aware of whether they are writing code for a local filesystem, or a remote one. (Most application writers assume that they *are* writing for POSIX filesystems, thinking that their code will only be run on local POSIX filesystems.) If I am writing code for a system that's distributed, then I should try to make the code "loosely coupled". In other words, to have as infrequent interactions with the remote services as possible, and to make these more coarse grained. I will also try to allow for failures in the network, knowing that even if it is not possible to communicate with the remote component now, it might be possible in the future. (As an example, I have some recent code which tries to push logical file mappings into Globus RLS. If the RLS service is not up, the mappings are cached locally, and will be uploaded later.) With filesystems, if you are writing code to work on a local filesystem, then the way you would behave in an error condition is different. If you can't write to your local filesystem, the program should probably try to quit (unless perhaps the disk is full, and there are some temporary files you can clean out - but this is already complex). However, if the filesystem might be AFS, or another remote filesystem, I should think about writing the code so that the operations can be retried later. (With AFS, the writes will "succeed", it's the close file operation that you have to be careful with.) Having (briefly) read the discussion on the mailing list, I don't believe that trying to enforce POSIX semantics on remote filesystems solves the above problem. You might get a filesystem that has POSIX semantics, but it would behave like a very unreliable POSIX filesystem. The point, which is made well in the paper I cited, is that you should not try to make remote things look like local things. In terms of practical advice for SAGA, I don't know how important this point is. The application writer, if using the SAGA interface, will probably know that the API is designed to communicate with remote components, and will hopefully code accordingly. I do believe that deciding upon single-system-type POSIX semantics everywhere would be a mistake. But I agree with the point that SAGA has to say something about (consistency) semantics too. If the GGF schedule permits, I hope to attend a lively discussion on this next week. For the meeting, it might be worth trying to contact the Legion (and AVAKI?) people. They have been doing this stuff for around ten years now. They support multiple underlying filesystem semantics, I believe. Perhaps this is the voice of experience? We should try to get someone from their team to come along to the discussion. Jon. On Jun 20, 2005, at 3:21 AM, Andre Merzky wrote:
Not w/o semantics, but the programmer has to assume worst case.
And, as Jon & John point out, that is not so different from what you have 'locally'. If you open a file for write, you SHOULD assume worst case, that is the file is on NFS/AFS, and POSIX consistency is NOT guaranteed.
A.
Quoting [Thorsten Schuett] (Jun 20 2005):
I see the point that it is hard guarantee a specific consistency model in SAGA or make it a configuration option to choose one. But if you completely ignore this topic in the specs you will end up with a good spec of the syntax but without semantics.
My 0.02???, Thorsten
-- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
On Jun 20, 2005, at 12:10 AM, Thorsten Schuett wrote:
2) At GGF it would be worthwhile to walk through some existing consistency models and find out how they map into the API (are there error states or features that are not accomodated?)
3) Felix is correct that such a model potentially exposes the programmer to the worst-case scenario in terms of consistency models. This is one of the risks involved in anything that carries the moniker "simple". I see the point that it is hard guarantee a specific consistency model in SAGA or make it a configuration option to choose one. But if you completely ignore
On Saturday 18 June 2005 17:21, John Shalf wrote: this topic in the specs you will end up with a good spec of the syntax but without semantics. Thorsten
Hence the item #2. Consistency models should be discussed at GGF. The group should find out a) Are there any error conditions or features that need to be added to SAGA in order to support existing consistency models. b) What are the ramifications of the existing consistency models on the high-level programming interface that is being offered by SAGA? How will it affect the way an app writer must implement error checking & recovery? This is, in effect, an exploration of the semantics that are imposed on SAGA by available/underlying implementations for remote I/O. -john
participants (4)
-
Andre Merzky -
John Shalf -
Jon MacLaren -
Thorsten Schuett