SAGA-WG session at OGF28, monday march 15, 2010, 16:00-17:30 session chaired by Andre Merkzy notes taken by Thilo Kielmann Due to a technical problem with the beamer, the session agenda was rearranged to start with a discussion on the white board. new agenda: 1. relation of SAGA with the Vine toolkit 2. the SAGA Core experience document 1. SAGA and Vine both have similar concepts (API abstracting from middleware and services, implementations with dynamic plugins/adaptors) There clearly is some conceptual overlap between both, so it was discussed how they could possibly be combined, or implementations be integrated. The outcome was that SAGA could be used to give Vine access to several middleware platforms. The Vine group will try out the Java implementation to do so. 2. the SAGA Core experience document Andre was walking the group through the upcoming experience document. The discussion lead to the following action items: clarify the notions of interoperable vs. compliant what do we need/want to show exactly? should the service discovery (SD) extension package be covered in this experience doc? decision: No remove section 2, there seems no real value here section 3: was debated as well solution: merge its content into the current section 4 make more concise points about what we want to test, exactly The session was closed at 17:30. -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/
SAGA-WG session at OGF28, monday march 15, 2010, 11:00-12:30 session chaired by Andre Merkzy Session was a general session, and covered - SAGA landscape introduction (updated landscape pdf attached) - short SAGA intro - discussion of 3rd part adaptor support in both C++ and Java Best, Andre. -- Nothing is ever easy.
Hi again, attached is another revision of the SAGA Core API Experience document, which contains changes as discussed at OGF28. I hope the changes reflect the discussion points. The complex stack of SAGA implementations, their interdependencies, the fact that interop is weakly defined for APIs, and the fact that GFD.90 specifies the SAGA API in a language independent way, al those things make it somewhat difficult to come up with a concise definition of the goals for our interop document. While there is little doubt in the group that SAGA is implementablt, we should strive to make that point convincing in the document. So I'd like to invite you all to review the document in that respect, and help to improve those points if possible. In particular, I added some short description of the language bindings (not implementations) -- it would be great if Hartmut, Ceriel and Mathijs could shortly review those sections, and fix/expand where appropriate. Quoting [Thilo Kielmann] (Mar 15 2010):
2. the SAGA Core experience document
Andre was walking the group through the upcoming experience document. The discussion lead to the following action items:
clarify the notions of interoperable vs. compliant what do we need/want to show exactly?
done
remove section 2, there seems no real value here
done
section 3: was debated as well solution: merge its content into the current section 4
done
make more concise points about what we want to test, exactly
attempted ;-) Quoting [Thilo Kielmann] (Mar 16 2010):
1. presentation by Go Iwai about the SAGA implementations done by the Japanese NAREGI project. They identified several (fortunately small) issues with the spec.
It was decided to incorporate their findings in the errata document and to close the list of errata then. (We are not aware of another mature implementation effort from which we have not received errata feedback yet.)
done So, a couple of additional errata from the Naregi group have been applied to the Core API - hopefully the last ones. However, there remains one item unresolved: appearently we never considered to add a flush() method to the saga::file instance. As is, our API implies that all writes are immediately flushed. While that is certainly valid, the question remains if we should consider an explicit flush() method, which would, amongst others, allow implementations to perform client side caching of write operations. Iff that is considered useful, one could further discuss if that should be introduced on namespace level, so that other namespace derived packages (replica, advert, etc) can also benefit from flush(). FWIW, a close() should always imply a flush() IMHO. So, please voice your opinion! Best, Andre. -- Nothing is ever easy.
Hi all, Quoting [Andre Merzky] (Apr 04 2010):
From: Andre Merzky
To: Thilo Kielmann Cc: SAGA RG Subject: Re: [SAGA-RG] notes from the OGF28 session on 15/03, 16:00-17:30 attached is another revision of the SAGA Core API Experience document, which contains changes as discussed at OGF28. I hope the changes reflect the discussion points.
I just wanted to let you know that both the advert API extension and the Core experience document have been submitted to the OGF editor, and both docs should be entering public comment sometime soon. That means that the Core API (including errata) is now definitely frozen, unless the public comments require additional changes. The submitted documents can be found in https://svn.cct.lsu.edu/repos/saga-ogf/trunk/documents/saga-package-advert/t... https://svn.cct.lsu.edu/repos/saga-ogf/trunk/documents/saga-core-experience/... https://svn.cct.lsu.edu/repos/saga-ogf/trunk/documents/saga-core/tags/v1.1rc...
So, a couple of additional errata from the Naregi group have been applied to the Core API - hopefully the last ones. However, there remains one item unresolved:
appearently we never considered to add a flush() method to the saga::file instance. As is, our API implies that all writes are immediately flushed. While that is certainly valid, the question remains if we should consider an explicit flush() method, which would, amongst others, allow implementations to perform client side caching of write operations. Iff that is considered useful, one could further discuss if that should be introduced on namespace level, so that other namespace derived packages (replica, advert, etc) can also benefit from flush(). FWIW, a close() should always imply a flush() IMHO.
So, please voice your opinion!
There was not much feedback on this item, so I added it to the list of open items for SAGA 2.0. As of now, caching behaviour on write remains undefined, and the safest assumption (for SAGA implementors) is to always flush after write, even if that is costly in terms of performance. That opens the question on when, and if at all, we should start to discuss a next version of the core API. FWIW, I appen the current list of open issues to this mail. We did not have a phone call since OGF28. There are a number of open TODO items however, and I am not sure that any calls are useful at that point, beyond iterating that those items need to be dealt with :-P So, I suggest to suspend the calls until at least some of these items are handled: - CPR package needs to be finalized - message API examples need to be rendered in different versions, to come to a conclusion on the general design approach. - Python bindings need to be shown to be functional on both Java and C++ - the SAGA rendering of GridRPC.v2 needs to be synced with the final version of GridRPC.v2 If anybody has other items to discuss, please let me know, and I'll schedule the calls. Also, the above items are obviously open for input from all of you, so, please feel free to contribute in any form. Finally, the conversion of our CVS repository to SVN is completed. CCT support did not manage to make the CVS repository ReadOnly, but please don't commit there anymore. The new SVN url is, as you probably guessed from above, https://svn.cct.lsu.edu/repos/saga-ogf/trunk That repository should be world-readable. Please let me know if you would like to have write permissions. Best, Andre. Current known open issues for SAGA Core v2.0 -------------------------------------------- - file / stream server / rpc could have state (Unknown, New, Open, Closed). - task: get_task_description just like job desc, would give you information about what the task does, e.g. - "method" = "copy" - "args" = "internet.txt" "internet.bak" (vector attrib (type??)) - "started" = "11:35pm 12/22/2006" - "finished" = "11:35pm 12/22/2007" inspection would be useful to get type and return type of task after getting it from a task_container. - I/O tasks could have a get_buffer() method, to free application from keeping/tracking I/O buffers. That would return a shallow copy of the buffer object which was given as inout parameter. Method would need to be templetized for the different buffer classes we have in the spec (or limited to the buffer base class) - make state transitions less prone to race conditions. E.g., allow suspend() also on jobs in Suspend state, and cancel() on jobs in a final state (state remains the same). Needs some thought... - what error is thrown on incorrectly formatted attributes, and when? - wait() to also report on other state changes, like suspend/resume (see DRMAA-II). - add inspection: job.list_interfaces () - monitorable - attributable - steerable? - checkpointable? to provide seemless integration of extensions, which then can define additional interfaces for core classes (see cpr). - add resource assignment to job description, e.g.: // name: CPUID // desc: CPU id to assign the process thread to // mode: ReadWrite, optional // type: Int // value: '1' // notes: - if supported, the process is guaranteed to // run on the CPU identified by the id. // - id starts at 1 // - not supported by JSDL, DRMAA.v1 // // name: CPUCoreID // desc: CPu core id to assign the process thread to // mode: ReadWrite, optional // type: Int // value: '1' // notes: - if supported, the process is guaranteed to // run on the CPU core identified by the id. // - id starts at 1 // - not supported by JSDL, DRMAA.v1 This could also go into a resource management package, obviously, together with 'queue' attribute btw (see mailing list discussion with Sylvain, and discussion about DRMAA.v2. - session.list_contexts (string type = ""); returns all contexts of that type. Also works on default session! If no type is given, all contexts are returned. - trigger metrics should have a value of 0 or 1, to allow polling for triggers. So, in fact Trigger metrics should be Boolean. - we have mtime for ns entries - there is no reason not to have ctime, or even atime, even if that is not widely supported. So what. Now we have to add ctime to the cpr package... Messy. - properties which are available via get_xyz() and is_abc() should generally also be expressed as attributes (see get_size(), get_mtime(), but also is_file() etc) - attributes and metrics should be unified. Either a metric IS-A attribute, or even better, callbacks can be added to attributes - no metric needed anymore. - file::dir should inherit file::entry *and* ns::dir. Makes in particular sense for advert and cpr ns derivates, which then don't need to duplicate methods anymore. Language bindings may not allow/encourage multiple inheritance, but it would make the spec (IDL) simpler. - we are not sticking to SIDL syntax anyway, so probably should remove references to it, and define our own *blush*. See attributes, metrics, c'tors, multiple inheritance, etc. - reconsider to split the core into \LF and API packages :-/ - reconsider file.get_fd(), for example for checkpoint writing/reading, where apps often have their own native IO routines. But of course, if they get a saga::fs::file, they can just close() it, and reopen the location natively... - file.flush is missing :-( Same for replica etc. Not sure if it makes sense on the ns::entry though. -- Nothing is ever easy.
participants (2)
-
Andre Merzky
-
Thilo Kielmann