Hi Group, as you have seen in the notes from last GGF, there was some discussion about the SAGA task model. This mail summarizes this discussion, and proposes some changes to the task model to accommodate the suggestions made in Boston. Sorry that this mail comes late, I pushed that off far too long... As a reminder, the current task model in action looks like this: --------------------------------------------------------------------- Example 1: saga::directory d ("/tmp/"); saga::directory::task_factory dtf = d.get_task_factory (); saga::task t = dtf.mkdir ("test/"); t.run (); t.wait (); --------------------------------------------------------------------- So, the minimal number of code lines needed for running a task is 4. Note that the asynchroneous mkdir method has the same signature as the synchroneous, but has a different return value. Compare that to the mechanism used in MPI or used in GridRPC: --------------------------------------------------------------------- Example 2: saga::directory d ("/tmp/"); saga::task t = d.mkdir_async ("test/"); t.wait (); --------------------------------------------------------------------- The number of lines is 2, and there is no explicit additional object (the task factory). Please note that the created task is automatically run on creation time. The discussion in Boston showed that people favor the similar mechanism of example 2. Also, SAGA has not a single use case which requires (or even motivates) the task_factories as explicit objects in the SAGA API. But, we do have use cases which motivate that tasks are NOT autimatically run. In particular the following scenario of bulk operations shows that: --------------------------------------------------------------------- Example 3: saga::file f ("/tmp/t.dat"); saga::file::task_factory dtf = d.get_task_factory (); saga::task_container tc; for ( int i = 0; i < 1000; i++ ) { saga::task t = ftf.read (i, 1, buffer[i]); tc.add_task (t); } tc.run (); tc.wait (); --------------------------------------------------------------------- As the tc.run starts all tasks at the same time, the implementation has the chance to optimize that, and to perform one remote operation instead of 1000 independend ones. With the model shown in example 2, this optimization would be impossible, and the bulk related use cases in SAGA would be difficult to implement (efficiently). In order to accommodate the explicit run() method on tasks, there are two possibilities (well, there are more, but two seem particularily obvious): --------------------------------------------------------------------- Example 4a: distinguish task from async call saga::directory d ("/tmp/"); saga::task t_1 = d.mkdir_async ("test/"); saga::task t_2 = d.mkdir_task ("test/"); t_2.run () t_1.wait (); t_2.wait (); Example 4b: flag to start task saga::directory d ("/tmp/"); saga::task t_1 = d.mkdir_async ("test/", TRUE); // TRUE optional saga::task t_2 = d.mkdir_async ("test/", FALSE); t_2.run () t_1.wait (); t_2.wait (); --------------------------------------------------------------------- Although the version 4b seems somewhat simplier, most people (including me) seem to oppose a seemingly artificial flag on all async calls to flag the runnnin of the task. Also, this would change the signature if compared to the synchroneous version of the method. Version 4a OTOH increases the number of methods. Then again, that might not be sooooo bad. The current task model doubles the number of methods as well. But as we describe the model in text, and do not actually add all signatures twice in the spec, the spec stays terse (for a very specific definition of terse, to be sure). So, version 4a would add only one paragraph to the spec :-) Also, 4a is strictly spoken just a marker to the sync methods, which makes them asynchroneous. In different language bindings, this could look very different, and convenient: --------------------------------------------------------------------- Example 4a: more versions in C++ 4a1: d.mkdir ("test/"); d.mkdir_sync ("test/"); // same saga::task t_1 = d.mkdir_async ("test/"); saga::task t_2 = d.mkdir_task ("test/"); 4a2: d.mkdir ("test/"); d.sync ::mkdir ("test/"); // same saga::task t_1 = d.async::mkdir ("test/"); saga::task t_2 = d.task ::mkdir ("test/"); 4a3: d.mkdir ("test/"); d.mkdir <sync> ("test/"); // same saga::task t_1 = d.mkdir <async> ("test/"); saga::task t_2 = d.mkdir <task> ("test/"); --------------------------------------------------------------------- All these versions are equivalent. I personally like 4a2 best, Thilo likes 4a1 (close to MPI), Hartmut favors 4a3 (nice to implement in C++). Well, here we are really: the proposal is following: P1) get rid of the task factory (there seems no need for it) P2) allow tasks which do run() on creation P3) allow tasks which do NOT run() on creation All both 4a and 4b deliver that, with only small changes in the API spec (but obviously larger changes in the language bindings). For 4a, the versions 4a1 to 4a3 show different posssible C++ language bindings. Questions: Q1) any comments to the Boston discussion? I hope I reflected that correctly... Q2) are there any drawbacks by introducing 4a or 4b, apart from the additional number of calls? Q3) Are the advantages (simplicity for most usages, powerfull for bulk operations and others) good enough to introduce 4a or 4b? Q4) You like 4a or 4b better? Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!) Please answer before November 10th: firstly, we would like to finish the task model soon, in order to get the Strawman stabilizwed before GGF16; secondly, we would like to come to a decision during the F2F meeting at SuperComputing in Seattle (November 12-18). Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
On Fri, Oct 28, 2005 at 11:06:01PM -0500, Andre Merzky wrote:
Hi Group,
Hi Andre!
Well, here we are really: the proposal is following:
P1) get rid of the task factory (there seems no need for it) P2) allow tasks which do run() on creation P3) allow tasks which do NOT run() on creation
Agree!
Questions:
Q1) any comments to the Boston discussion? I hope I reflected that correctly... fine
Q2) are there any drawbacks by introducing 4a or 4b, apart from the additional number of calls? I don't think so
Q3) Are the advantages (simplicity for most usages, powerfull for bulk operations and others) good enough to introduce 4a or 4b? Yes! advantages: a) simpler -- no more factories b) more powerful -- matches e.g. GridRPC requirements
Q4) You like 4a or 4b better?
4a!
Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!)
4a2 has inconsistent syntax. I prefer 4a3, and also 4a1
Please answer before November 10th: firstly, we would like to finish the task model soon, in order to get the Strawman stabilizwed before GGF16; secondly, we would like to come to a decision during the F2F meeting at SuperComputing in Seattle (November 12-18).
See you there! Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/
Hi Thilo, thanks for the answer! One short question: Quoting [Thilo Kielmann] (Nov 01 2005):
Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!)
4a2 has inconsistent syntax. I prefer 4a3, and also 4a1
Why do you think the syntax of 4a2 to be inconsistent? Thanks, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
Oops, forget it. It should have been: --------------------------------------------------------------------- 4a2: d.mkdir ("test/"); d.sync .mkdir ("test/"); // same saga::task t_1 = d.async.mkdir ("test/"); saga::task t_2 = d.task .mkdir ("test/"); --------------------------------------------------------------------- Cheers, Andre. Quoting [Andre Merzky] (Nov 01 2005):
Hi Thilo,
thanks for the answer! One short question:
Quoting [Thilo Kielmann] (Nov 01 2005):
Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!)
4a2 has inconsistent syntax. I prefer 4a3, and also 4a1
Why do you think the syntax of 4a2 to be inconsistent?
Thanks, Andre.
-- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
On Tue, Nov 01, 2005 at 08:27:57PM -0600, Andre Merzky wrote:
Oops, forget it. It should have been:
--------------------------------------------------------------------- 4a2: d.mkdir ("test/"); d.sync .mkdir ("test/"); // same saga::task t_1 = d.async.mkdir ("test/"); saga::task t_2 = d.task .mkdir ("test/"); ---------------------------------------------------------------------
Cheers, Andre.
OK, in this case it is no better no worse (at least not much ;-) than 4a1 and 4a3. Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/
I agree with nearly all of Thilo's comments on this subject. On Oct 28, 2005, at 9:06 PM, Andre Merzky wrote:
Questions:
Q1) any comments to the Boston discussion? I hope I reflected that correctly... Q2) are there any drawbacks by introducing 4a or 4b, apart from the additional number of calls?
I think it is good. (go for it)
Q3) Are the advantages (simplicity for most usages, powerfull for bulk operations and others) good enough to introduce 4a or 4b?
We want a more concise way of describing async operations, so this is good.
Q4) You like 4a or 4b better?
I do not like 4b. The flags seem very klunky. I think the naming of the method call is a more effective means of communicating its function (it is certainly more readable than code that uses the flags). Readability is very important.
Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!)
I prefer 4a1 because it is more readable and the implementation would be quite straightforward. It is also a familiar paradigm for any MPI programmers and anyone who has played with various proprietary Async I/O implementations. (its a very familiar and conventional approach) I kind of like 4a2 as well from the standpoint of a C++ programmer (even with Andre's syntax corrections). However, the resulting bindings will not be very consistent with the approach we would take for Fortran or C bindings (eg. those would likely look more like 4a1). It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity. I do a lot of C++ programming, but I find the 4a3 option a bit obscure both in terms of readability and any advantages it might confer in terms of implementation. It would certainly be easier to create multi-language bindings for a single code base if we stick with something more conventional like 4a1. Each approach is equally readable to me (less so for 4a3). I'm certainly open to any additional information on how the 4a2 and 4a3 approaches could simplify implementation. If the other approaches can offer some implementation benefits, then maybe I'd give them extra consideration, but otherwise, I would prefer a more conventional approach like 4a1. The only implementation I'm outright against is the 4b example. -john
On Wed, Nov 02, 2005 at 02:57:22AM -0800, John Shalf wrote:
I prefer 4a1 because it is more readable and the implementation would be quite straightforward. It is also a familiar paradigm for any MPI programmers and anyone who has played with various proprietary Async I/O implementations. (its a very familiar and conventional approach)
I kind of like 4a2 as well from the standpoint of a C++ programmer (even with Andre's syntax corrections). However, the resulting bindings will not be very consistent with the approach we would take for Fortran or C bindings (eg. those would likely look more like 4a1). It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
Taking these arguments together, we should opt for 4a1 ! - more readable - closer to Fortran bindings and other archaic languages ;-) Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/
Hi Thilo, John, I should have made more clear, that the API spec would contain only 4a (not 4b, as nobody seems to like the flag version, fortunately). the 4a1, 4a2 and/or 4a3 would go into the language binding for C++, and are merilly illustrative for how 4a could look like in a OO language binding. Anyay, yes, it seems people are happy with going for 4a, so I will put that into the strawman until SC05, unless there is any negative feedback by then. Thanks for your comments, Andre. Quoting [Thilo Kielmann] (Nov 02 2005):
Date: Wed, 2 Nov 2005 14:56:24 +0100 From: Thilo Kielmann
To: John Shalf Cc: Andre Merzky , Simple API for Grid Applications WG Subject: Re: [saga-rg] Task model... On Wed, Nov 02, 2005 at 02:57:22AM -0800, John Shalf wrote:
I prefer 4a1 because it is more readable and the implementation would be quite straightforward. It is also a familiar paradigm for any MPI programmers and anyone who has played with various proprietary Async I/O implementations. (its a very familiar and conventional approach)
I kind of like 4a2 as well from the standpoint of a C++ programmer (even with Andre's syntax corrections). However, the resulting bindings will not be very consistent with the approach we would take for Fortran or C bindings (eg. those would likely look more like 4a1). It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
Taking these arguments together, we should opt for 4a1 ! - more readable - closer to Fortran bindings and other archaic languages ;-)
Thilo -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
On Nov 2, 2005, at 5:56 AM, Thilo Kielmann wrote:
On Wed, Nov 02, 2005 at 02:57:22AM -0800, John Shalf wrote:
I prefer 4a1 because it is more readable and the implementation would be quite straightforward. It is also a familiar paradigm for any MPI programmers and anyone who has played with various proprietary Async I/O implementations. (its a very familiar and conventional approach)
I kind of like 4a2 as well from the standpoint of a C++ programmer (even with Andre's syntax corrections). However, the resulting bindings will not be very consistent with the approach we would take for Fortran or C bindings (eg. those would likely look more like 4a1). It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
Taking these arguments together, we should opt for 4a1 ! - more readable - closer to Fortran bindings and other archaic languages ;-)
... gotta keep Tom happy. ;-)
Hi John, Thilo, Group, Hartmut and I would like to argue about the C++ bindings. We would like to come to an agreement soon, as our implementation is dealing with the async part right now. As reminder: --------------------------------------------------------------------- Example 4a: more versions in C++ 4a1: d.mkdir ("test/"); d.mkdir_sync ("test/"); // same saga::task t_1 = d.mkdir_async ("test/"); saga::task t_2 = d.mkdir_task ("test/"); 4a2: d.mkdir ("test/"); d.sync .mkdir ("test/"); // same saga::task t_1 = d.async.mkdir ("test/"); saga::task t_2 = d.task .mkdir ("test/"); 4a3: d.mkdir ("test/"); d.mkdir <sync> ("test/"); // same saga::task t_1 = d.mkdir <async> ("test/"); saga::task t_2 = d.mkdir <task> ("test/"); --------------------------------------------------------------------- Quoting [John Shalf] (Nov 02 2005):
Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!)
I prefer 4a1 because it is more readable and the implementation would be quite straightforward.
You favour 4a1. We think that implementation is very straight forward for all three versions. Readability seems not much different to me, and might largely be a matter of taste.
It is also a familiar paradigm for any MPI programmers and anyone who has played with various proprietary Async I/O implementations. (its a very familiar and conventional approach)
Well, these are C-API's. Is there a C++ binding for MPI, and does it look the same?
I kind of like 4a2 as well from the standpoint of a C++ programmer (even with Andre's syntax corrections). However, the resulting bindings will not be very consistent with the approach we would take for Fortran or C bindings (eg. those would likely look more like 4a1).
But, well, that is the idea of the language binding! I agree that C and Fortran would look more like 4a1, of course. But that is no reason that C++ should look like that as well, or Java, Perl etc.
It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
I think that 4a2/4a3 actually allow nicer implementations, as it allows to have the different async parts somewhat separate from the sync parts. We think its nicer :-)
I do a lot of C++ programming, but I find the 4a3 option a bit obscure both in terms of readability and any advantages it might confer in terms of implementation.
Hehe - I thought the same :-) Hartmut likes that version very much. To me it appealed after pondering over it for a couple of days. Now I think it is cute, and quite expressive.
It would certainly be easier to create multi-language bindings for a single code base if we stick with something more conventional like 4a1.
I think that C and Fortran bindings for this part are straight forward anyway, there is no need to reflect that in C++...
Each approach is equally readable to me (less so for 4a3). I'm certainly open to any additional information on how the 4a2 and 4a3 approaches could simplify implementation.
The main point really is that the object itself has only the sync method calls, and the async calls can be put into separate header/sources easily, and build (by default) on top of the sync calls. Of course, you can do that with all three versions, but its not as obvious in 4a1.
If the other approaches can offer some implementation benefits, then maybe I'd give them extra consideration, but otherwise, I would prefer a more conventional approach like 4a1.
I would vote for 4a2 or 4a3 (in that order), but 4a1 would be ok if the majority likes that most, of course. Basically its a matter of taste I think. I am happy that the general point seems accaptable to all so far: having sync, async, and task versions of the calls, w/o explicit task factory.
The only implementation I'm outright against is the 4b example.
Good! :-) Thanks, Andre.
-john
-- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
Hi Andre and all, taking the arguments into account that have been exchanged so far (and giving it another thought), I am voting for 4a2: reason: 4a1 indeed emulates a C of Fortran style syntax, which is NOT the right thing to do (C++ should look like C++) 4a2 groups the three sets of operations in three sub objects (is that right?), at least, there are 3 consistent places where the 3 respective functionality groups belong (one place for sync, one place for async,...) 4a3 is using templates in the wrong way: a template defines some code skeleton into which arbitrary instantiations can be put. (Like the notorious stack from which you can make both a stack of int and a stack molecules) But here, we anticipate exactly 3 possible instantiations, which goes against the intuition of templates. Thilo On Thu, Nov 03, 2005 at 10:56:14AM -0600, Andre Merzky wrote:
X-Original-To: kielmann@localhost Delivered-To: kielmann@localhost.cs.vu.nl Delivered-To: grdfm-saga-rg-outgoing@mailbouncer.mcs.anl.gov X-Original-To: grdfm-saga-rg@mailbouncer.mcs.anl.gov Delivered-To: grdfm-saga-rg@mailbouncer.mcs.anl.gov Date: Thu, 3 Nov 2005 10:56:14 -0600 From: Andre Merzky
To: John Shalf Cc: Andre Merzky , Simple API for Grid Applications WG Subject: Re: [saga-rg] Task model... X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at mailbouncer.mcs.anl.gov X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at mailbouncer.mcs.anl.gov Hi John, Thilo, Group,
Hartmut and I would like to argue about the C++ bindings. We would like to come to an agreement soon, as our implementation is dealing with the async part right now.
As reminder:
--------------------------------------------------------------------- Example 4a: more versions in C++
4a1: d.mkdir ("test/"); d.mkdir_sync ("test/"); // same saga::task t_1 = d.mkdir_async ("test/"); saga::task t_2 = d.mkdir_task ("test/");
4a2: d.mkdir ("test/"); d.sync .mkdir ("test/"); // same saga::task t_1 = d.async.mkdir ("test/"); saga::task t_2 = d.task .mkdir ("test/");
4a3: d.mkdir ("test/"); d.mkdir <sync> ("test/"); // same saga::task t_1 = d.mkdir <async> ("test/"); saga::task t_2 = d.mkdir <task> ("test/"); ---------------------------------------------------------------------
Quoting [John Shalf] (Nov 02 2005):
Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!)
I prefer 4a1 because it is more readable and the implementation would be quite straightforward.
You favour 4a1. We think that implementation is very straight forward for all three versions. Readability seems not much different to me, and might largely be a matter of taste.
It is also a familiar paradigm for any MPI programmers and anyone who has played with various proprietary Async I/O implementations. (its a very familiar and conventional approach)
Well, these are C-API's. Is there a C++ binding for MPI, and does it look the same?
I kind of like 4a2 as well from the standpoint of a C++ programmer (even with Andre's syntax corrections). However, the resulting bindings will not be very consistent with the approach we would take for Fortran or C bindings (eg. those would likely look more like 4a1).
But, well, that is the idea of the language binding! I agree that C and Fortran would look more like 4a1, of course. But that is no reason that C++ should look like that as well, or Java, Perl etc.
It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
I think that 4a2/4a3 actually allow nicer implementations, as it allows to have the different async parts somewhat separate from the sync parts. We think its nicer :-)
I do a lot of C++ programming, but I find the 4a3 option a bit obscure both in terms of readability and any advantages it might confer in terms of implementation.
Hehe - I thought the same :-) Hartmut likes that version very much. To me it appealed after pondering over it for a couple of days. Now I think it is cute, and quite expressive.
It would certainly be easier to create multi-language bindings for a single code base if we stick with something more conventional like 4a1.
I think that C and Fortran bindings for this part are straight forward anyway, there is no need to reflect that in C++...
Each approach is equally readable to me (less so for 4a3). I'm certainly open to any additional information on how the 4a2 and 4a3 approaches could simplify implementation.
The main point really is that the object itself has only the sync method calls, and the async calls can be put into separate header/sources easily, and build (by default) on top of the sync calls. Of course, you can do that with all three versions, but its not as obvious in 4a1.
If the other approaches can offer some implementation benefits, then maybe I'd give them extra consideration, but otherwise, I would prefer a more conventional approach like 4a1.
I would vote for 4a2 or 4a3 (in that order), but 4a1 would be ok if the majority likes that most, of course. Basically its a matter of taste I think. I am happy that the general point seems accaptable to all so far: having sync, async, and task versions of the calls, w/o explicit task factory.
The only implementation I'm outright against is the 4b example.
Good! :-)
Thanks, Andre.
-john
-- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
-- Thilo Kielmann http://www.cs.vu.nl/~kielmann/
Thilo Kielmann wrote:
taking the arguments into account that have been exchanged so far (and giving it another thought), I am voting for 4a2:
reason:
4a1 indeed emulates a C of Fortran style syntax, which is NOT the right thing to do (C++ should look like C++)
4a2 groups the three sets of operations in three sub objects (is that right?), at least, there are 3 consistent places where the 3 respective functionality groups belong (one place for sync, one place for async,...)
4a3 is using templates in the wrong way: a template defines some code skeleton into which arbitrary instantiations can be put. (Like the notorious stack from which you can make both a stack of int and a stack molecules) But here, we anticipate exactly 3 possible instantiations, which goes against the intuition of templates.
I strongly disagree here. The use case you've described is certainly the most visible one for using templates (mainly for historical reasons and because of the STL). But templates are additionally a very powerful tool to enable generic and metaprogramming in C++ (hey templates are turing complete in the end) and therefor allow for certain code generation and optimization methods not available without them. Moreover, if you look at modern C++ languages (as they evolve in Boost or getting incorporated into the next C+ Standard) templates are getting used in a much wider context as only for creating generic containers. BTW: The templated syntax allows to separate the three implementations evenly good as the implementation in 4a2 without exposing them to the interface and without the additional (ok, perhaps marginal) overhead. Regards Hartmut
Thilo
On Thu, Nov 03, 2005 at 10:56:14AM -0600, Andre Merzky wrote:
X-Original-To: kielmann@localhost Delivered-To: kielmann@localhost.cs.vu.nl Delivered-To: grdfm-saga-rg-outgoing@mailbouncer.mcs.anl.gov X-Original-To: grdfm-saga-rg@mailbouncer.mcs.anl.gov Delivered-To: grdfm-saga-rg@mailbouncer.mcs.anl.gov Date: Thu, 3 Nov 2005 10:56:14 -0600 From: Andre Merzky
To: John Shalf Cc: Andre Merzky , Simple API for Grid Applications WG Subject: Re: [saga-rg] Task model... X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at mailbouncer.mcs.anl.gov X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at mailbouncer.mcs.anl.gov Hi John, Thilo, Group,
Hartmut and I would like to argue about the C++ bindings. We would like to come to an agreement soon, as our implementation is dealing with the async part right now.
As reminder:
---------------------------------------------------------------------
Example 4a: more versions in C++
4a1: d.mkdir ("test/"); d.mkdir_sync ("test/"); // same saga::task t_1 = d.mkdir_async ("test/"); saga::task t_2 = d.mkdir_task ("test/");
4a2: d.mkdir ("test/"); d.sync .mkdir ("test/"); // same saga::task t_1 = d.async.mkdir ("test/"); saga::task t_2 = d.task .mkdir ("test/");
4a3: d.mkdir ("test/"); d.mkdir <sync> ("test/"); // same saga::task t_1 = d.mkdir <async> ("test/"); saga::task t_2 = d.mkdir <task> ("test/");
---------------------------------------------------------------------
Quoting [John Shalf] (Nov 02 2005):
Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the
I prefer 4a1 because it is more readable and the implementation would be quite straightforward.
You favour 4a1. We think that implementation is very straight forward for all three versions. Readability seems not much different to me, and might largely be a matter of taste.
It is also a familiar paradigm for any MPI programmers and anyone who has played with various proprietary Async I/O implementations. (its a very familiar and conventional approach)
Well, these are C-API's. Is there a C++ binding for MPI, and does it look the same?
I kind of like 4a2 as well from the standpoint of a C++
Strawman!) programmer
(even with Andre's syntax corrections). However, the resulting bindings will not be very consistent with the approach we would take for Fortran or C bindings (eg. those would likely look more like 4a1).
But, well, that is the idea of the language binding! I agree that C and Fortran would look more like 4a1, of course. But that is no reason that C++ should look like that as well, or Java, Perl etc.
It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
I think that 4a2/4a3 actually allow nicer implementations, as it allows to have the different async parts somewhat separate from the sync parts. We think its nicer :-)
I do a lot of C++ programming, but I find the 4a3 option a bit obscure both in terms of readability and any advantages it might confer in terms of implementation.
Hehe - I thought the same :-) Hartmut likes that version very much. To me it appealed after pondering over it for a couple of days. Now I think it is cute, and quite expressive.
It would certainly be easier to create multi-language bindings for a single code base if we stick with something more conventional like 4a1.
I think that C and Fortran bindings for this part are straight forward anyway, there is no need to reflect that in C++...
Each approach is equally readable to me (less so for 4a3). I'm certainly open to any additional information on how the 4a2 and 4a3 approaches could simplify implementation.
The main point really is that the object itself has only the sync method calls, and the async calls can be put into separate header/sources easily, and build (by default) on top of the sync calls. Of course, you can do that with all three versions, but its not as obvious in 4a1.
If the other approaches can offer some implementation benefits, then maybe I'd give them extra consideration, but otherwise, I would prefer a more conventional approach like 4a1.
I would vote for 4a2 or 4a3 (in that order), but 4a1 would be ok if the majority likes that most, of course. Basically its a matter of taste I think. I am happy that the general point seems accaptable to all so far: having sync, async, and task versions of the calls, w/o explicit task factory.
The only implementation I'm outright against is the 4b example.
Good! :-)
Thanks, Andre.
-john
-- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
-- Thilo Kielmann http://www.cs.vu.nl/~kielmann/
My vote is for 4a1. (Which was option D in our original discussion I believe.) Cheers, Tom On Thu, 3 Nov 2005, Andre Merzky wrote:
Hi John, Thilo, Group,
Hartmut and I would like to argue about the C++ bindings. We would like to come to an agreement soon, as our implementation is dealing with the async part right now.
As reminder:
--------------------------------------------------------------------- Example 4a: more versions in C++
4a1: d.mkdir ("test/"); d.mkdir_sync ("test/"); // same saga::task t_1 = d.mkdir_async ("test/"); saga::task t_2 = d.mkdir_task ("test/");
4a2: d.mkdir ("test/"); d.sync .mkdir ("test/"); // same saga::task t_1 = d.async.mkdir ("test/"); saga::task t_2 = d.task .mkdir ("test/");
4a3: d.mkdir ("test/"); d.mkdir <sync> ("test/"); // same saga::task t_1 = d.mkdir <async> ("test/"); saga::task t_2 = d.mkdir <task> ("test/"); ---------------------------------------------------------------------
Quoting [John Shalf] (Nov 02 2005):
Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!)
I prefer 4a1 because it is more readable and the implementation would be quite straightforward.
You favour 4a1. We think that implementation is very straight forward for all three versions. Readability seems not much different to me, and might largely be a matter of taste.
It is also a familiar paradigm for any MPI programmers and anyone who has played with various proprietary Async I/O implementations. (its a very familiar and conventional approach)
Well, these are C-API's. Is there a C++ binding for MPI, and does it look the same?
I kind of like 4a2 as well from the standpoint of a C++ programmer (even with Andre's syntax corrections). However, the resulting bindings will not be very consistent with the approach we would take for Fortran or C bindings (eg. those would likely look more like 4a1).
But, well, that is the idea of the language binding! I agree that C and Fortran would look more like 4a1, of course. But that is no reason that C++ should look like that as well, or Java, Perl etc.
It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
I think that 4a2/4a3 actually allow nicer implementations, as it allows to have the different async parts somewhat separate from the sync parts. We think its nicer :-)
I do a lot of C++ programming, but I find the 4a3 option a bit obscure both in terms of readability and any advantages it might confer in terms of implementation.
Hehe - I thought the same :-) Hartmut likes that version very much. To me it appealed after pondering over it for a couple of days. Now I think it is cute, and quite expressive.
It would certainly be easier to create multi-language bindings for a single code base if we stick with something more conventional like 4a1.
I think that C and Fortran bindings for this part are straight forward anyway, there is no need to reflect that in C++...
Each approach is equally readable to me (less so for 4a3). I'm certainly open to any additional information on how the 4a2 and 4a3 approaches could simplify implementation.
The main point really is that the object itself has only the sync method calls, and the async calls can be put into separate header/sources easily, and build (by default) on top of the sync calls. Of course, you can do that with all three versions, but its not as obvious in 4a1.
If the other approaches can offer some implementation benefits, then maybe I'd give them extra consideration, but otherwise, I would prefer a more conventional approach like 4a1.
I would vote for 4a2 or 4a3 (in that order), but 4a1 would be ok if the majority likes that most, of course. Basically its a matter of taste I think. I am happy that the general point seems accaptable to all so far: having sync, async, and task versions of the calls, w/o explicit task factory.
The only implementation I'm outright against is the 4b example.
Good! :-)
Thanks, Andre.
-john
Andre Merzky wrote:
Quoting [John Shalf] (Nov 02 2005):
Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!)
I prefer 4a1 because it is more readable and the
implementation would
be quite straightforward.
You favour 4a1. We think that implementation is very straight forward for all three versions. Readability seems not much different to me, and might largely be a matter of taste.
4a1 is a plain match of some flat API onto the C++ language. It bloats the exposed object surface (every function has to be there 3 or 4 times).
It is also a familiar paradigm for any MPI programmers and anyone who has played with various proprietary Async I/O implementations. (its a very familiar and conventional approach)
Well, these are C-API's. Is there a C++ binding for MPI, and does it look the same?
Frankly, I'm not a specialist here. But from what I can tell from the MPI specs the C++ bindings are more or less a complete copy of the underlying C API, which is not object oriented (but please correct me if I'm wrong).
I kind of like 4a2 as well from the standpoint of a C++ programmer (even with Andre's syntax corrections). However, the resulting bindings will not be very consistent with the approach we would take for Fortran or C bindings (eg. those would likely look more like 4a1).
But, well, that is the idea of the language binding! I agree that C and Fortran would look more like 4a1, of course. But that is no reason that C++ should look like that as well, or Java, Perl etc.
4a2 is good from the standpoint of a C++ language binding but infers non-needed objects in the interface (the sync, async and task subobjects) additionally for every API object there have to be constructed, initialised and destructed these three subobjects. I know, that performance is not an issue in the SAGA context, but I think we should avoid premature pessimisation (i.e. building on top of a obviously non-optimal solution).
It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
I think that 4a2/4a3 actually allow nicer implementations, as it allows to have the different async parts somewhat separate from the sync parts. We think its nicer :-)
4a3 the only implementation in a C++ like manner I can think of, which provides: - an interface which conforms to modern C++ concepts pushed by Boost et.al. - it avoids not needed overhead if some of the features are not needed/used (this gets optimised away by any decent C++ compiler) - it allows for using the SAGA API classes in a generic programming context using the execution model as a (template) parameter (I can elaborate on this further if needed) - minimises the exposed interface
I do a lot of C++ programming, but I find the 4a3 option a bit obscure both in terms of readability and any advantages it might confer in terms of implementation.
Hehe - I thought the same :-) Hartmut likes that version very much. To me it appealed after pondering over it for a couple of days. Now I think it is cute, and quite expressive.
It's not a matter of cuteness, I guess. It's simply a matter of efficiency, maintainability and minimization of the exposed interface.
It would certainly be easier to create multi-language bindings for a single code base if we stick with something more conventional like 4a1.
I think that C and Fortran bindings for this part are straight forward anyway, there is no need to reflect that in C++...
Agreed. Fortran and C bindings probably will end up to be very similar. In C++ we should conform to the main C++ language paradigm (i.e. don't pay for anything you're not going to use etc.)?
Each approach is equally readable to me (less so for 4a3). I'm certainly open to any additional information on how the 4a2 and 4a3 approaches could simplify implementation.
The main point really is that the object itself has only the sync method calls, and the async calls can be put into separate header/sources easily, and build (by default) on top of the sync calls. Of course, you can do that with all three versions, but its not as obvious in 4a1.
It's not a matter of simplifying the implementation, it's a matter of simplifying the exposed interface. I even would go for a more complex implementation if the interface gets simpler. And from my experience the templated syntax isn't counterintuitive if you're used to programming in C++.
If the other approaches can offer some implementation benefits, then maybe I'd give them extra consideration, but otherwise, I would prefer a more conventional approach like 4a1.
I would vote for 4a2 or 4a3 (in that order), but 4a1 would be ok if the majority likes that most, of course. Basically its a matter of taste I think. I am happy that the general point seems accaptable to all so far: having sync, async, and task versions of the calls, w/o explicit task factory.
I kind of agree, but would prefer 4a3 because of the benfits it provides. Regards Hartmut
On Nov 3, 2005, at 2:01 PM, Hartmut Kaiser wrote:
Andre Merzky wrote:
Quoting [John Shalf] (Nov 02 2005):
Q5) Any comments to 4a1, 4a2 or 4a3? (not part of the Strawman!)
I prefer 4a1 because it is more readable and the
implementation would
be quite straightforward.
You favour 4a1. We think that implementation is very straight forward for all three versions. Readability seems not much different to me, and might largely be a matter of taste.
I think the readability is nearly equal for all of 4a1-3. The original statement assumed 4b was still in the mix (and 4b flunks the readability test). I consider 4a3 a bit less readable than the former two because the use of the templating syntax there is a little obscure (normally only see the <> in constructors/declarations), but I agree that is more a matter of taste and not of substance.
4a1 is a plain match of some flat API onto the C++ language. It bloats the exposed object surface (every function has to be there 3 or 4 times).
I don't think replication of the API constitutes a serious issue so long as the function naming convention is regular and consistent. So that is to say that 4a1 does not present a programmer with any greater cognitive load than 4a2 for instance.
It is also a familiar paradigm for any MPI programmers and anyone who has played with various proprietary Async I/O implementations. (its a very familiar and conventional approach)
Well, these are C-API's. Is there a C++ binding for MPI, and does it look the same?
Frankly, I'm not a specialist here. But from what I can tell from the MPI specs the C++ bindings are more or less a complete copy of the underlying C API, which is not object oriented (but please correct me if I'm wrong).
Well, the default is to just use the regular C API, but the mpi++ bindings encapsulate the communicator in the class, but otherwise the calls are very similar. Its something like comm.iSend(...stuff....) comm.Send(...stuff...) comm.iRecv(...stuff...) I think the MPI iSend() vs. bSend() and sSend() is an example where the number of different send and recv calls is increased by 4x, but quite frankly it doesn't affect the cognitive load for the programmer because the naming convention is very regular and consistent. (eg. there really isn't much of a difference between 4a1 and 4a2 in that regard).
I kind of like 4a2 as well from the standpoint of a C++ programmer (even with Andre's syntax corrections). However, the resulting bindings will not be very consistent with the approach we would take for Fortran or C bindings (eg. those would likely look more like 4a1).
But, well, that is the idea of the language binding! I agree that C and Fortran would look more like 4a1, of course. But that is no reason that C++ should look like that as well, or Java, Perl etc.
4a2 is good from the standpoint of a C++ language binding but infers non-needed objects in the interface (the sync, async and task subobjects) additionally for every API object there have to be constructed, initialised and destructed these three subobjects. I know, that performance is not an issue in the SAGA context, but I think we should avoid premature pessimisation (i.e. building on top of a obviously non-optimal solution).
I agree... 4a2 doesn't seem to make the implementation of the bindings any easier or more optimal than the flat approach.
It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
I think that 4a2/4a3 actually allow nicer implementations, as it allows to have the different async parts somewhat separate from the sync parts. We think its nicer :-)
4a3 the only implementation in a C++ like manner I can think of, which provides: - an interface which conforms to modern C++ concepts pushed by Boost et.al. - it avoids not needed overhead if some of the features are not needed/used (this gets optimised away by any decent C++ compiler)
Well, if the compiler can actually take advantage of that syntactical convention to apply more efficient optimizations, then that is a clear advantage to that approach (whether or not I find the syntax obscure).
- it allows for using the SAGA API classes in a generic programming context using the execution model as a (template) parameter (I can elaborate on this further if needed) - minimises the exposed interface
Agreed. Fortran and C bindings probably will end up to be very similar. In C++ we should conform to the main C++ language paradigm (i.e. don't pay for anything you're not going to use etc.)?
Each approach is equally readable to me (less so for 4a3). I'm certainly open to any additional information on how the 4a2 and 4a3 approaches could simplify implementation.
The main point really is that the object itself has only the sync method calls, and the async calls can be put into separate header/sources easily, and build (by default) on top of the sync calls. Of course, you can do that with all three versions, but its not as obvious in 4a1.
It's not a matter of simplifying the implementation, it's a matter of simplifying the exposed interface. I even would go for a more complex implementation if the interface gets simpler. And from my experience the templated syntax isn't counterintuitive if you're used to programming in C++.
Its not clear to me that C++ requires use of templates to be C++- like. I typically think of using templates where the algorithm is the same but the types that have to be passed through the algorithm may differ in different implementations. I'm still having a difficult time wrapping my mind around the concept of using templates where the underlying algorithm will be very different.
If the other approaches can offer some implementation benefits, then maybe I'd give them extra consideration, but otherwise, I would prefer a more conventional approach like 4a1.
I would vote for 4a2 or 4a3 (in that order), but 4a1 would be ok if the majority likes that most, of course. Basically its a matter of taste I think. I am happy that the general point seems accaptable to all so far: having sync, async, and task versions of the calls, w/o explicit task factory.
I kind of agree, but would prefer 4a3 because of the benfits it provides.
Regards Hartmut
Quoting [John Shalf] (Nov 03 2005):
It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
I think that 4a2/4a3 actually allow nicer implementations, as it allows to have the different async parts somewhat separate from the sync parts. We think its nicer :-)
4a3 the only implementation in a C++ like manner I can think of, which provides: - an interface which conforms to modern C++ concepts pushed by Boost et.al. - it avoids not needed overhead if some of the features are not needed/used (this gets optimised away by any decent C++ compiler)
Well, if the compiler can actually take advantage of that syntactical convention to apply more efficient optimizations, then that is a clear advantage to that approach (whether or not I find the syntax obscure).
_S_AGA. If it is obscure to you or others, its not a good choice. Performance is a second order issue. However, as I said before, I kind of like the template thing :-) And, Hartmut, forgive me for calling it cute... ;-) Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
On Thu, 3 Nov 2005, 'Andre Merzky' wrote:
Quoting [John Shalf] (Nov 03 2005):
It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
I think that 4a2/4a3 actually allow nicer implementations, as it allows to have the different async parts somewhat separate from the sync parts. We think its nicer :-)
4a3 the only implementation in a C++ like manner I can think of, which provides: - an interface which conforms to modern C++ concepts pushed by Boost et.al. - it avoids not needed overhead if some of the features are not needed/used (this gets optimised away by any decent C++ compiler)
Well, if the compiler can actually take advantage of that syntactical convention to apply more efficient optimizations, then that is a clear advantage to that approach (whether or not I find the syntax obscure).
_S_AGA. If it is obscure to you or others, its not a good choice. Performance is a second order issue. However, as I
Also, we have not used templates anywhere else in the abstract API, so I think it would be a bad idea to move that way now. This does not preclude the C++ language binding using such if there is a clear benefit. On a practical note, templates are one of the least portable features of C++, especially in libraries (except template libraries), and I would strongly urge that we do not use them for SAGA. Tom
Quoting [Tom Goodale] (Nov 03 2005):
Well, if the compiler can actually take advantage of that syntactical convention to apply more efficient optimizations, then that is a clear advantage to that approach (whether or not I find the syntax obscure).
_S_AGA. If it is obscure to you or others, its not a good choice. Performance is a second order issue. However, as I
Also, we have not used templates anywhere else in the abstract API, so I think it would be a bad idea to move that way now. This does not preclude the C++ language binding using such if there is a clear benefit.
Right now, we are only talking about the C++ binding - sorry if we did not make that clear. The abstract spec will not have templates. Cheers, Andre.
On a practical note, templates are one of the least portable features of C++, especially in libraries (except template libraries), and I would strongly urge that we do not use them for SAGA.
Tom
-- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky@cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+
Tom Goodale wrote:
It is not really much more readable than 4a1. Therefore, I'm uncertain if it is worth fragmenting our approach to bindings in different languages when there is not a clear benefit in terms of readability or implementation complexity.
I think that 4a2/4a3 actually allow nicer implementations, as it allows to have the different async parts somewhat separate from the sync parts. We think its nicer :-)
4a3 the only implementation in a C++ like manner I can think of, which provides: - an interface which conforms to modern C++ concepts pushed by Boost et.al. - it avoids not needed overhead if some of the features are not needed/used (this gets optimised away by any decent C++ compiler)
Well, if the compiler can actually take advantage of that syntactical convention to apply more efficient optimizations, then that is a clear advantage to that approach (whether or not I find the syntax obscure).
_S_AGA. If it is obscure to you or others, its not a good choice. Performance is a second order issue. However, as I
Also, we have not used templates anywhere else in the abstract API, so I think it would be a bad idea to move that way now. This does not preclude the C++ language binding using such if there is a clear benefit.
I agree here. There was a bit of misunderstanding from my side. You were talking about the the SAGA spec but I was talking about the C++ language binding. Could we find a way of doing both, describing a more or less general interface in the spec and giving some freedom of implementation to the language bindings?
On a practical note, templates are one of the least portable features of C++, especially in libraries (except template libraries), and I would strongly urge that we do not use them for SAGA.
We will probably end up writing a C language binding for the affected architectures in the end anyway... But let's first solve the problem for the 99% of the architectures where your concern is a no-issue. It's a matter of how much time is required to do the C++ implementation in the end. And using templates will give a speedup to this for sure. Regards Hartmut
John Shalf wrote:
It's not a matter of simplifying the implementation, it's a matter of simplifying the exposed interface. I even would go for a more complex implementation if the interface gets simpler. And from my experience the templated syntax isn't counterintuitive if you're used to programming in C++.
Its not clear to me that C++ requires use of templates to be C++- like.
It was not my intent to say that. What I was trying to say was that if you're going to use C++ then templates will be a powerful and versatile tool.
I typically think of using templates where the algorithm is the same but the types that have to be passed through the algorithm may differ in different implementations. I'm still having a difficult time wrapping my mind around the concept of using templates where the underlying algorithm will be very different.
That's the initial use case which lead to the addition of templates to the language. As I said, templates are turing complete and thus usable (and used!) for generic metaprogramming - a powerful tool for letting the compiler generate optimal code. Regards Hartmut
participants (7)
-
'Andre Merzky'
-
Andre Merzky
-
Hartmut Kaiser
-
John Shalf
-
John Shalf
-
Thilo Kielmann
-
Tom Goodale