May 2005 - saga-rg - lists.ogf.org

Re: comment on SAGA strawman doc.
by Andre Merzky 21 Jul '05

21 Jul '05

Dear Peter, Kalman, many thanks for your detailed comments! We will try to answer them all, but it will take us some time. After a first glance, they look very useful to me! I forward the mail to the SAGA list, for archiving and comments. Cheers, Andre. Quoting [Peter Kunszt] (May 31 2005): > Subject: comment on SAGA strawman doc. > Date: Tue, 31 May 2005 18:08:50 +0200 > From: "Peter Kunszt" <Peter.Kunszt(a)cern.ch> > To: "EGEE Middleware Engineering Management Team" <project-eu-egee-middleware-emt(a)cern.ch> > Cc: "EGEE Middleware Design Team" <project-eu-egee-middleware-design(a)cern.ch>, > "Kalman Kovari" <Kalman.Kovari(a)cern.ch>, > <andre(a)merzky.net> > > hi all > > following up on our task to be interoperable with > standards and to see how we fit into the large world > of GGF, we have looked at the strawman SAGA API doc . > http://wiki.cct.lsu.edu/saga/space/start/strawman-api.pdf > we focussed on the data management side, to see whether we > could implement these interfaces 'easily' or not. > > some of the comments are a bit sketchy.. glancing > at the SAGA mailinglist, some may have been discussed already. > > so we have only looked at the APIs relating to data management. > > the good news is that it seems that with some minor exceptions it would > be quite straightforward > to map the SAGA API to the existing Data Management APIs of the Fireman, > glite-io and the metadata catalog. it is not a surprise that the SAGA > API is much more sketchy and seems at times inconsistent and incomplete > compared with our > APIs since they don't have a real implementation behind them yet. > > the only 'issue' is the introduction of a stream-based network > API in SAGA. in our architecture / design we decided to go with the > managed async data transfer model (FTS), not with direct synchronous > streaming. the > reason is obvious: we need to manage the network resource efficiently. > > unfortunately the SAGA doc does not motivate the interfaces to an > extent where we would be able to deduce an 'architecture'... > we might also argue that some interfaces (like replica location, storage > space management, > lifetime management, quotas, etc) are missing... > > cheers > peter > > ps. kalman kovari [cc'd] did the actual work, i am just echoing and > commenting > his findings. i also cc'd andre merzky of the SAGA-WG. > > -------------- > > some detailed doc comments: > > - The interfaces in the document have some self-inconsistencies, > however, none of them is critical. > We expect these will be cleaned whenever someone starts to implement > them. > > - The examples are sometimes more misleading than useful... > > - The error handling and error semantics are not defined well enough. > This is essential > in order to assure proper interoperability. > > - There is a lack of structure in terms of interface layering. > High level interfaces (like JobManagement) and low level interfaces > (like > Stream) are not distinguished, motivated, or put into context. > E.g. why is there a need of a networkstream-level interface for a > grid application? > > - Example (P18): OO structured, not like the iface. (eg. i = > dir.getNumEntries() > instead of dir.getNumEntries(a) ? ) > > - Example (P29): an exact copy of Ex. 1. > > - Description - Streams (P62): close or destroyStream? maybe close, but > not consistent. > > - General: sometimes bitwise OR-s are used for multiple options > (Stream/ActivityType, P59), sometimes arrays of values > (NameSpace/CopyFlags, P9). Why no convention? > > - Description: Stream/read: Throws: IncorrectState if stream not in > "NotConnected" state. typo. > > - Description: Stream/write: Same as above. C&P. > > - Description: StreamServer/create,destroy: create/createStreamServer, > destroy/destroyStreamServer? > > - Interface NSDir: what are the semantics of the copy method? of > getName and getURL? > > ----------------- > > some details concerning implementation mappings. > below we mention the classes to be implemented, and their methods: > (an 'ok' is put to methods that are trivial to be implemented using the > existing APIs. probably just > name and parameter translations are required in that cases) > > > iface NSDir - except for copy(), everything can be implemented using the > fireman-catalog. > > changeDir: clientside manipulation+validity check in the catalog. > list: we only implement it with limits -- some default limits are > required. > readlink: ok. > exists: getlfnstat+check. > isDir: getlfnstat+check. > isFile: getlfnstat+check. > isLink: getlfnstat+check. > getNumEntries: we have no efficient implementation, possible with > readDir(), but slow. > copy: What do they mean saying "copy"? Needs clarification. > link: ok. > move: ok. > remove: unlink(). > makeDir: ok. > > getURL, getName: undocd. Needs to be clarified what they mean here, but > probably only clientside manipulation. > > iface Directory extends NSDir - internal scheme to implement directory > services. we already have a working catalog, > so we only need to make an interface for it. > getSize: fireman::getlfnstat. > openDir: clientside consturctor+check in the catalog if name is valid. > open: open/create a file. glite_io::open() and for creating some > fireman::create, too. > > iface File - implementable through glite_io. > read, write, seek: ok. > > iface LogicalDirectory extends NSDir - in the implementation scheme we > have no difference > between Directory and LogicalDirectory > > iface LogicalFile > addLocation, removeLocation, listLocations: addReplica, removeReplica, > listReplicas. ok. > replicate: addReplica > > iface Stream, StreamServer - low level, seemingly managing network > streams. > [see comments above] > > iface Attributes - usual key/value pairs, with support for > vectorAttributes. > > not looked at: > iface Task, TaskContainer > iface JobInfo, Job, JobService -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 1

Fwd (s.newhouse@omii.ac.uk): Re: SAGA Strawman API
by Andre Merzky 01 Jun '05

01 Jun '05

Hi Steven, thanks for your comments, I forward them to the list - hope you don't mind. Cheers, Andre. ----- Forwarded message from Steven Newhouse <s.newhouse(a)omii.ac.uk> ----- > Date: Sun, 29 May 2005 14:47:40 +0100 > From: Steven Newhouse <s.newhouse(a)omii.ac.uk> > To: Shantenu Jha <s.jha(a)ucl.ac.uk> > CC: Tom Goodale <goodale(a)cct.lsu.edu>, Andre Merzky <andre(a)merzky.net> > Subject: Re: SAGA Strawman API > > Hi Shantenu, > > >In order to advance the SAGA strawman API, we need urgently > >some feedback from outside the authors group. > > I have been carrying around for over month and have only just (late I'm > afraid!) got round to looking at it. A few general comments... > > 1. OO vs. functional > I can understand specifying in one language - no problem. But you need > to show read across to different languages and systems. > > 2. Task > Didn't understand this at all! Are you proposing to support 5 different > systems or expecting one to be selected from these 5? > > 3. Security > Inconsistent use of security - it appears in the stream section but > nowhere else. I think this is very wrong. Not specifying a security > system makes sense, but I think all of thes emethods should have a > security token being passed into them or it should be an argument in the > relevant object constructor. > > 4. URL - P10 > I assume the URL can support a number of protocols - > http/https/file/gridftp. I see no way to register plugins in the > interface (may be this is an 'internal' interface as opposed to a user > funtion. Maybe there needs to be aet of interfaces to help the > developers, e.g. register protocol plugins. > > 5. Languages > These examples (with the OO background) obviously look very Java. I > think you need a section (but not for each area) showing how the > interfaces get rendered in different areas. This will help adoption & > buyin from Perl, Python, F77, C, & other communities. How are the > exceptions handled in different programming languages? I'd almost be > tempted to return all error codes from the function as the return value. > > 6. Focus > The large number of APIs makes the document appear un-focuses. You > mention a focus on tier 1 interfaces (which is good) but is this the > whole document or a subset of the current draft? If its the whole > document I think you need to drop more! Have this document as a > framework but take forward a smaller document for standardisation. Samll > is better! > > 7. Partial Implementations > How are you going to handle partial functionality implementations? > Should _all_ operations support a NotImplementedException ? Should there > be a stndard static method in each section to discover what is > implemented? e.g. supported protocols & supported methods. > > 8. SAGA File Transfer. > You have an 'LSF' schema for >>, >, < & <<. But in the file/directory > area you use a series of attribute flags. I think you should carry these > attribute flags forward into the job definition area - either way I > think you should have one model not two! > > General Thoughts > The document needs to be broken up IMHO - either into sections or > individual documents. > I. General Model. Language mapping. Standard attribute model. Security. > Tasks > II. Tier 1 Interfaces. > III Tier 2 and above Interfaces. > > This would make it clearer what you are doing now & the constructs & > concepts that you are using to build the model. > > Hope that helps! Happy to clarify any of the above... > > Regards, > > Steven -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

3 4

Last chance to comment on use case document
by Tom Goodale 26 May '05

26 May '05

Hi, it's been over a month now since we posted the use case document on the wiki, and there has been little comment; is there a desire to discuss it further at next GGF or on the mailing list, or should we just bump the revision to 1.0 and submit it to the editors ? Please shout by the middle of next week if you want further discussion. Cheers, Tom

1 0

Fwd (schuett@zib.de): Re: SAGA Strawman API
by Andre Merzky 21 May '05

21 May '05

Hi group, Thorsten sent me some feedback to the strawman API. Since it is in german, I'll translate it to english for the list. Some parts I leave out: those which point out small obvious errors like spelling etc, which I just corrected in the doc. Thanks for the comments, Thorsten! On Friday 20 May 2005 18:47:11, Thorsten Sch"uett wrote: > > Mal abgesehen davon, dass ich noch nie von SIDL gehört habe ... > > - Warum brauche ich im Package NameSpace ein interface NSDir? Sollte das Ding > nicht Directory heissen? Q: Why does the package NameSpace need a interface NSDir? Shouldn't that be named Directory? A: It WAS named directore before, but we thought Dir to be, well, shorter, w/o loosing meaning. Hmm, I prefer Dir for beeing shorter, but have no strong opniion about it. Others? > - Ist isFile und isLink nicht wieder arg File spezifisch? In einem LDAP-Baum > hasst du bestimmt andere isXXX-Funktionen. isDir kann bleiben. Gibt es Links > in jedem Namespace? Q: Isn't isFile and isLink too FILE specific? In an LDAP tree you might find completely different isXXX methods. isDir is ok. Are there inks in all namespaces? A: isFile _sounds_ file specific, but that is a naming problem here. I have to elaborate on this somewhat, sorry. Once we had a NSDirectory and an NSDirectoryEntry interface. The latter one was basically empty, but in any package implementing NS, that would get filles. E.g. a physical file namespace would have a physical file as NSDirectoryEntry. Since that interface was empty, we got rid of it for now (just to be minimalistic). We figured: if we find good reasons (say: methods) for it to exist, we just include it again. Now that left is no good choice for naming the isEntry w/o creating confusion, so we named it isFile. Basically, that checks if the entry is, well, an entry. So, maybe the name _should_ return after all. For links: most name spaces have link. If not, then the is does not hurt. However, that method seems more related to the namespace concept than to its physical representation - thats why it is in the NS interface. Does that make sense? > - Impliziert der openDir-Call irgendwas für das Current working directory von > der neuen Directory-Instanz? Q: Is the openDir call indicatind anything in respect to the CWD of the new directory? A: Good catch! Yes, the CWD is pointing to the opened directory. I changed the doc. > - Was ist das Current working directory von NSDir("/")? Q: what is cwd of NSDir("/"); A: "/". > - Irgendwie sind die Flags merkwürdig. Ich könnte ein Flags-Array mit Create > und NoCreate erzeugen. Es würde dann zwar ein Exception geworfen, aber sowas > kann auch der Compiler überprüfen. Lass doch die NoXXXX Flags weg und mach > flags zu einem set<openFlags>. Dann gibt es auch keine doppelten Flags. Und > wenn ihr schon numerische Werte für die Flags einführt, gestalte sie doch so, > dass man auch ein einen Integer und Bit-Arithmetik in einem Binding verwenden > kann. > enum openFlags{ > Create = 1, > Excl = 2, > Truncate = 4, > Append = 8, > Lock = 16 > }; Q: Flags are strange: I could create a flagg array with both 'Create' and 'NoCreate'. Well, thats would give an exception, but thats something which could be catched by the compiler. You could leave out the NoXXX flags, and change the flags to set<openFlags>. the there are no double flags. And if you go for numeric values, you could do the usual bit arithmetik thingie... A: I agree. I can't for sure remember the rationale for the current notation -- but I think we left that to the language binding. So, in C we would do what you suggest. > - Ist irgendwo definiert, wie gross ein long sein muss? Gerade für Grids > sollte getSize mindestens 64-Bit zurückgeben. Q: Is there any definition for sizeof (long)? In particular for Grids, getSize should return at least 64 bit. A: SIDL defines long to be 64bit. I will put the SIDL documentation on the WIKI. > - read: Posix kennt kein EndOfFile-Error. Dann wird einfach 0 zurückgegeben. Q: in posix, there is no EOF error - just returns 0. A: Oops! removed. > - interface Attribute: Komischer Name, beschreibt das Interface nicht eher > eine "Description" o.ä.? removeAttribute wird nicht genauer beschrieben. Mit > eurem Interface kann man zu einem Key nicht mehrere Values haben. Schade. Q: Interface Attributes: strange name, isn't that rather a description or such? removeAttribute is not fleshed out. You can't have multiple values for one key - a pity. A: Hmm, maybe we didn't transport the idea well enough. Think physical files: attributes would be for example: file size, file type, last mod time, owner, ACL etc. You don't have two values for size. If you have more than one item to one key (acl), you have vector attributes for these. I am not sure what you mean with description, or with multiple values in that context. Do you have an example? removeAttribute: Oops, fixed. > - In den Beispielen taucht manchmal "Context" auf. Im Text wird es glaube ich > nicht erläutert. Könnt ihr die Dinger nicht bitte komplett eleminieren? Q: In some examples appears a 'context' - this is not elaborated elsewhere in the text. Cant you remove them completely? A: A context is supposed to be a SAGA session handle. But, you are right, until they are fleshed out in the spec, they should go away in the examples. Thanks again, Andre. > Thorsten ----- End forwarded message ----- -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

2 2

RE: SAGA sessions at GGF14
by Rajic, Hrabri 11 May '05

11 May '05

It might be a good idea to engage as many people as possible, so to make the transition (straw-man to wider group blessed) between GGF 14 and GGF 15. Once this happens; how to judge?, a smaller team could take a speedier approach for the doc production. Having two sessions for this purpose should be enough. There could be lots of buzz around GT4, so having many sessions could be challenge. Groups to avoid overlap: JSDL, GRAAP, OGSA-BES, DRMAA, ... ( 5 is a max ) In addition to tuning the spec via use cases I would suggest also driving it via 2 reference implementations. <DRMAA hat on> Implementations help a lot to weed out tricky issues from a different perspective. </DRMAA hat on> Greetings -Hrabri >-----Original Message----- >From: Andre Merzky [mailto:andre@merzky.net] >Sent: Wednesday, May 11, 2005 2:27 AM >To: Shantenu Jha >Cc: Andre Merzky; Keith Jackson; David E. Konerding; Tom Goodale; Rajic, >Hrabri; Simple API for Grid Applications WG >Subject: Re: SAGA sessions at GGF14 > >As many as posisble I'd say -- I for one don't plan to >attend many other sessions, and culd spend the whole time on >SAGA. > >Agenda items: I think > > - finalize use case doc (advance in doc process into > public comment phase) > - finalize strawman API (we intended an AG meeting before, > didn't we? Did we set a date?) > - if possible push strawman API Doc into doc process > - collectively work on the requirements doc > - either split up in some 2...3 person groups working on > parts of that, > - or just discuss it on a blackboard, and make extensive > notes > - create a timeplane for transition of strawman to real > - after implementations arise? > - after the req. doc is _drafted_? > - after the req. doc is _finished_? > >Hmm, just mu $0.02. > >Cheers, Andre. > > >PS.: CC to the list. > >Quoting [Shantenu Jha] (May 10 2005): >> >> Folks, How many sessions for SAGA at GGF14? Tentative agenda? >> >> S >-- >+-----------------------------------------------------------------+ >| Andre Merzky | phon: +31 - 20 - 598 - 7759 | >| Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | >| Dept. of Computer Science | mail: merzky(a)cs.vu.nl | >| De Boelelaan 1083a | www: http://www.merzky.net | >| 1081 HV Amsterdam, Netherlands | | >+-----------------------------------------------------------------+

2 1

Re: SAGA sessions at GGF14
by Andre Merzky 11 May '05

11 May '05

As many as posisble I'd say -- I for one don't plan to attend many other sessions, and culd spend the whole time on SAGA. Agenda items: I think - finalize use case doc (advance in doc process into public comment phase) - finalize strawman API (we intended an AG meeting before, didn't we? Did we set a date?) - if possible push strawman API Doc into doc process - collectively work on the requirements doc - either split up in some 2...3 person groups working on parts of that, - or just discuss it on a blackboard, and make extensive notes - create a timeplane for transition of strawman to real - after implementations arise? - after the req. doc is _drafted_? - after the req. doc is _finished_? Hmm, just mu $0.02. Cheers, Andre. PS.: CC to the list. Quoting [Shantenu Jha] (May 10 2005): > > Folks, How many sessions for SAGA at GGF14? Tentative agenda? > > S -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0

Final CFP: GGF14 applications workshop
by Thilo Kielmann 03 May '05

03 May '05

Workshop on Grid Applications: from Early Adopters to Mainstream Users In conjunction with GGF14, (June 26, Chicago, USA) http://www.cs.vu.nl/ggf/apps-rg/meetings/ggf14.html Deadline extended to May 16, 2005! Jointly organized by: GGF Application Developers and Users Research Group (APPS-RG) & Production Grid Management Research Group (PGM-RG) Workshop Organizers: Thomas Hinke (NASA AMES, USA), Thilo Kielmann (Vrije Universiteit, Amsterdam, Netherlands), Laura McGinnis (PSC, USA) & David Wallom (Bristol University, UK) ---------------------------------------------------------------------------- Call for Papers: In this workshop, we aim at experience with bridging the gap between early adopters of grids and the more mainstream use of grid technology. Currently, grids are mostly on the early adopter side of the gap, asking for the move to the more mainstream users. For fostering mature production environments, incentives by application users are vital. We are seeking experience from early adoptors who would like to become mainstream users, from mainstream users who would like to use grids, from those who already do, and from middleware developers and system operators in charge of providing working grid environments to user communities. We are seeking contributions on the following topics: - user experience with early-adoptor and/or mainstream grid applications - management approaches for production-quality grid environments - techniques for robust (fault tolerant) grid applications and middleware - software tools for automatic testing and signaling of error conditions Our focus is on hands-on experience with existing grid environments, focusing on bridging the gap between early adoption and production use. For the workshop presentations, we are seeking papers on the subjects described above. Papers should be typeset in a single-column format, in 12pt font, for letter or A4 sized paper. Papers should not exceed 10 pages. Shorter papers are welcome. Submission format is PDF. Please submit your paper to David Wallom, by email to david.wallom(a)bristol.ac.uk, marked as GGF-14 Grid Applications workshop submission. ---------------------------------------------------------------------------- Publication: All accepted submissions will be published as a GGF Informational Document. We will invite the most promising submissions for a journal special issue in a high-quality journal covering Grid computing. ---------------------------------------------------------------------------- Important Dates: Paper submission: May 16, 2005 (extended on common request) Notification of acceptance: May 30, 2005 Workshop: June 26, 2005

1 0