Dear all, Andre asked me to provide some more detailed information prior to the meeting in Lyon, so here goes. Attached you will find two files, one of them is the python example from the LSU Python bindings [1]. The other file is the diff that is needed to make the same simple example run with JPySAGA, which implements the API as defined by VU for their Pysaga implementation [2]. I also did a review of the comparison done in the past, and that still seems to be valid. For reference I've also attached it. The code diff and the snippet below gives some intuition of the differences. =============== File ---- VU module:filesystem, class:File LSU module:filesystem._filesystem,class:file Job --- VU module:job, class:Job LSU module:job._job, class:job JobDescription -------------- VU module:job, class:JobDescription LSU module:job._job, class:description JobService ---------- VU module:job, class:JobService LSU module:job._job, class:service Exception --------- VU module:error, class:SagaException LSU module:saga, class:exception ============== On overall we can summarize that the LSU bindings have C++ style names and the VU bindings have Python style names. The VU bindings have create() calls for explicit constructors (not sure why). Both implementations make extensive use of getters and setters which are discouraged in Python. The VU package structure seems to be closer to the spec. The LSU bindings use "fork://" for localhost execution and the VU bindings use "local://". The VU bindings need RMs to be of type URL only, LSU accepts strings too. To conclude, the list of little differences continues for ever. See the file vulsudiff.txt for an extensive overview. Ideally, the bindings should become Python-like: - CamelCase for classes - lowercase_underscores for members - No getters and setters - No explicit create() constructors Given this list, neither of the two implementations apply to these requirements as is. How to go forward? Functionally they don't seem to differ much, so that's not a deal breaker. The LSU bindings have most probably a larger user community, that could be an argument in favor of LSU. Another option would be that we could go out and define a third set of bindings with the best elements of both implementations, not sure if that would just make the situation worse. I hope this gives some structure to the discussion. Regards, Mark 1. http://apidoc.saga.cct.lsu.edu/saga-python/latest/ 2. http://apidoc.saga.cct.lsu.edu/saga-python-alt/
Hi Mark, all, On Thu, Sep 1, 2011 at 10:48 PM, M.A. Santcroos <m.a.santcroos@amc.uva.nl> wrote:
Dear all,
Andre asked me to provide some more detailed information prior to the meeting in Lyon, so here goes.
Many thanks for the effort! Inlined are some comments and questions... Best, Andre.
Attached you will find two files, one of them is the python example from the LSU Python bindings [1]. The other file is the diff that is needed to make the same simple example run with JPySAGA, which implements the API as defined by VU for their Pysaga implementation [2].
I also did a review of the comparison done in the past, and that still seems to be valid. For reference I've also attached it. The code diff and the snippet below gives some intuition of the differences.
[...]
The VU bindings have create() calls for explicit constructors (not sure why).
Could the reason be to allow for asynchronous object construction?
Both implementations make extensive use of getters and setters which are discouraged in Python.
The SAGA spec requires that some objects, such as job description, do not allow free attributes (like 'my_jobclass = super'), and also restricts values several attributes can have. Is that mappable to a python hashtable (dictoinary)? If not, I would think that is the reason for setters and getters?
The VU package structure seems to be closer to the spec. The LSU bindings use "fork://" for localhost execution and the VU bindings use "local://".
The URL semantics should not be part of the language binding - for good or for worse, that is an implementation detail....
The VU bindings need RMs to be of type URL only, LSU accepts strings too.
To conclude, the list of little differences continues for ever. See the file vulsudiff.txt for an extensive overview.
Ideally, the bindings should become Python-like: - CamelCase for classes - lowercase_underscores for members - No getters and setters - No explicit create() constructors
Given this list, neither of the two implementations apply to these requirements as is.
How to go forward? Functionally they don't seem to differ much, so that's not a deal breaker.
My $0.02: - quickly (re-)agree on a common binding, whatever that is - provide a thin wrapper around the C++ bindings, to be able to maintain both for a time - change the VU bindings to adhere to the agreement - provide a common test suite for both (something along the lines of Sylvain's interop tests)
The LSU bindings have most probably a larger user community, that could be an argument in favor of LSU. Another option would be that we could go out and define a third set of bindings with the best elements of both implementations, not sure if that would just make the situation worse.
I hope this gives some structure to the discussion.
Yes, many thanks! Steven, Sylvain, others - any input before we start bashing our heads in Lyon? :) Cheers, Andre.
Regards,
Mark
1. http://apidoc.saga.cct.lsu.edu/saga-python/latest/ 2. http://apidoc.saga.cct.lsu.edu/saga-python-alt/
-- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg
-- Nothing is ever easy...
Hi Andre, On 9/2/11 6:55 , "Andre Merzky" <andre@merzky.net> wrote:
Both implementations make extensive use of getters and setters which are discouraged in Python.
The SAGA spec requires that some objects, such as job description, do not allow free attributes (like 'my_jobclass = super'),
You won't technically prevent this by having getters and setters, but it might have some social/practical effect. There are ways to limit creation of new attributes (__slots__).
and also restricts values several attributes can have. Is that mappable to a python hashtable (dictoinary)? If not, I would think that is the reason for setters and getters?
Yes, you can use properties for that.
The VU package structure seems to be closer to the spec. The LSU bindings use "fork://" for localhost execution and the VU bindings use "local://".
The URL semantics should not be part of the language binding - for good or for worse, that is an implementation detail....
True, but it still broke the compatibility, so I thought it was worth mentioning.
My $0.02:
- quickly (re-)agree on a common binding, whatever that is - provide a thin wrapper around the C++ bindings, to be able to maintain both for a time - change the VU bindings to adhere to the agreement - provide a common test suite for both (something along the lines of Sylvain's interop tests)
Sounds like a plan. If I have some time before the meeting I can come up with a proposal for the common binding. Cheers, Mark
Hi all, Andre, I think you're right about the reason for the create() method ; it can indeed return a Task object that will contain the created object. Mark, FYI you can configure JSAGA's "local://" adaptor to be mapped to URL scheme "fork://"... but anyway, there are many other differences between URL of various SAGA implementations (independently of the implemented binding). The other main difference between SAGA implementations is between the security contexts. So if you want to develop an application that uses several SAGA implementations, creating URLs and security contexts will require implementation-specific code anyway. Best regards, Sylvain Le 02/09/2011 06:55, Andre Merzky a écrit :
Hi Mark, all,
On Thu, Sep 1, 2011 at 10:48 PM, M.A. Santcroos <m.a.santcroos@amc.uva.nl> wrote:
Dear all,
Andre asked me to provide some more detailed information prior to the meeting in Lyon, so here goes. Many thanks for the effort! Inlined are some comments and questions...
Best, Andre.
Attached you will find two files, one of them is the python example from the LSU Python bindings [1]. The other file is the diff that is needed to make the same simple example run with JPySAGA, which implements the API as defined by VU for their Pysaga implementation [2].
I also did a review of the comparison done in the past, and that still seems to be valid. For reference I've also attached it. The code diff and the snippet below gives some intuition of the differences.
[...]
The VU bindings have create() calls for explicit constructors (not sure why). Could the reason be to allow for asynchronous object construction?
Both implementations make extensive use of getters and setters which are discouraged in Python. The SAGA spec requires that some objects, such as job description, do not allow free attributes (like 'my_jobclass = super'), and also restricts values several attributes can have. Is that mappable to a python hashtable (dictoinary)? If not, I would think that is the reason for setters and getters?
The VU package structure seems to be closer to the spec. The LSU bindings use "fork://" for localhost execution and the VU bindings use "local://". The URL semantics should not be part of the language binding - for good or for worse, that is an implementation detail....
The VU bindings need RMs to be of type URL only, LSU accepts strings too.
To conclude, the list of little differences continues for ever. See the file vulsudiff.txt for an extensive overview.
Ideally, the bindings should become Python-like: - CamelCase for classes - lowercase_underscores for members - No getters and setters - No explicit create() constructors
Given this list, neither of the two implementations apply to these requirements as is.
How to go forward? Functionally they don't seem to differ much, so that's not a deal breaker. My $0.02:
- quickly (re-)agree on a common binding, whatever that is - provide a thin wrapper around the C++ bindings, to be able to maintain both for a time - change the VU bindings to adhere to the agreement - provide a common test suite for both (something along the lines of Sylvain's interop tests)
The LSU bindings have most probably a larger user community, that could be an argument in favor of LSU. Another option would be that we could go out and define a third set of bindings with the best elements of both implementations, not sure if that would just make the situation worse. I hope this gives some structure to the discussion. Yes, many thanks!
Steven, Sylvain, others - any input before we start bashing our heads in Lyon? :)
Cheers, Andre.
Regards,
Mark
1. http://apidoc.saga.cct.lsu.edu/saga-python/latest/ 2. http://apidoc.saga.cct.lsu.edu/saga-python-alt/
-- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg
Hi Sylvain, On 9/2/11 10:40 , "Sylvain Reynaud" <Sylvain.Reynaud@in2p3.fr> wrote:
Andre, I think you're right about the reason for the create() method ; it can indeed return a Task object that will contain the created object.
I don't really understand that. So how does that differ from the LSU functionality?
Mark, FYI you can configure JSAGA's "local://" adaptor to be mapped to URL scheme "fork://"... but anyway, there are many other differences between URL of various SAGA implementations (independently of the implemented binding). The other main difference between SAGA implementations is between the security contexts.
How come? Was there too much room for interpretation in the spec?
So if you want to develop an application that uses several SAGA implementations, creating URLs and security contexts will require implementation-specific code anyway.
I don't consider that a feature obviously. It kind of defeats the purpose of having a standard API. Cheers, Mark
Hi Mark, I would call that a "limitation" rather than a "defeat". Indeed, creating URLs and security contexts is generally a small part of application code, so having a standard API is still useful for all the rest of the code! Yes there is some room for interpretation about URL and security contexts, but I am not sure if it is too much... For example, with URLs ; should the SAGA specification enumerate all the possible schemes for all existing technologies, and then be updated each time a new technology is born ? The same question applies to security contexts as well, because their attributes depends on which information is required by the middleware, rather than which functionnality we want to provide to users. I am not saying that we should not try to overcome this "limitation", but I am not sure that the SAGA-Core specification document would be the best place for this. Cheers, Sylvain Le 03/09/2011 12:21, M.A. Santcroos a écrit :
Hi Sylvain,
On 9/2/11 10:40 , "Sylvain Reynaud"<Sylvain.Reynaud@in2p3.fr> wrote:
Andre, I think you're right about the reason for the create() method ; it can indeed return a Task object that will contain the created object. I don't really understand that. So how does that differ from the LSU functionality?
Mark, FYI you can configure JSAGA's "local://" adaptor to be mapped to URL scheme "fork://"... but anyway, there are many other differences between URL of various SAGA implementations (independently of the implemented binding). The other main difference between SAGA implementations is between the security contexts. How come? Was there too much room for interpretation in the spec?
So if you want to develop an application that uses several SAGA implementations, creating URLs and security contexts will require implementation-specific code anyway. I don't consider that a feature obviously. It kind of defeats the purpose of having a standard API.
Cheers,
Mark
On Sat, Sep 3, 2011 at 12:21 PM, M.A. Santcroos <m.a.santcroos@amc.uva.nl> wrote:
Hi Sylvain,
On 9/2/11 10:40 , "Sylvain Reynaud" <Sylvain.Reynaud@in2p3.fr> wrote:
Andre, I think you're right about the reason for the create() method ; it can indeed return a Task object that will contain the created object.
I don't really understand that. So how does that differ from the LSU functionality?
Well, at least in C++ we do the same: 'normal' constructors for the synchronous operations, factory like create methods for the asynchronous ones. There are other ways to render that in an API, for example a two-step initialization, but that seemed to be easiest back then. I don't think the LSU SAGA bindings expose the async factory methods.
Mark, FYI you can configure JSAGA's "local://" adaptor to be mapped to URL scheme "fork://"... but anyway, there are many other differences between URL of various SAGA implementations (independently of the implemented binding). The other main difference between SAGA implementations is between the security contexts.
How come? Was there too much room for interpretation in the spec?
You cannot sensibly standardize the URL name space - that will get quickly out of sync with that 'reality' thing ;-) The same holds for security contexts, which at the end are also just pointers into some (backend defined) namespace.
So if you want to develop an application that uses several SAGA implementations, creating URLs and security contexts will require implementation-specific code anyway.
I don't consider that a feature obviously. It kind of defeats the purpose of having a standard API.
Well, the URLs are pointers to endpoints. A GRAM endpoint, for example, will in general have a well defined https URL, which should be valid for all SAGA (and non-SAGA) implementations which want to use that endpoint (unless the implementation does strange things). The problem only arises where endpoint have no well defined URLs associated (fork:// versus local://). Similar for security contexts: the spec specifies a set of attributes which seem to map to most security tokens known to us - unless the SAGA implementation does strange things, you should always be able to use the 'UserProxy' attribute to point to an X509 certificate, etc. But SAGA cannot sensibly anticipate all possible security token attributes, as some of them are even deployment-dependent. That is why the security context attributes are extensible... Cheers, Andre.
Cheers,
Mark
-- Nothing is ever easy...
Le 05/09/2011 12:27, Andre Merzky a écrit :
On Sat, Sep 3, 2011 at 12:21 PM, M.A. Santcroos <m.a.santcroos@amc.uva.nl> wrote:
Hi Sylvain,
On 9/2/11 10:40 , "Sylvain Reynaud"<Sylvain.Reynaud@in2p3.fr> wrote:
Andre, I think you're right about the reason for the create() method ; it can indeed return a Task object that will contain the created object. I don't really understand that. So how does that differ from the LSU functionality? Well, at least in C++ we do the same: 'normal' constructors for the synchronous operations, factory like create methods for the asynchronous ones. There are other ways to render that in an API, for example a two-step initialization, but that seemed to be easiest back then. I don't think the LSU SAGA bindings expose the async factory methods.
Mark, FYI you can configure JSAGA's "local://" adaptor to be mapped to URL scheme "fork://"... but anyway, there are many other differences between URL of various SAGA implementations (independently of the implemented binding). The other main difference between SAGA implementations is between the security contexts. How come? Was there too much room for interpretation in the spec? You cannot sensibly standardize the URL name space - that will get quickly out of sync with that 'reality' thing ;-) The same holds for security contexts, which at the end are also just pointers into some (backend defined) namespace.
So if you want to develop an application that uses several SAGA implementations, creating URLs and security contexts will require implementation-specific code anyway. I don't consider that a feature obviously. It kind of defeats the purpose of having a standard API. Well, the URLs are pointers to endpoints. A GRAM endpoint, for example, will in general have a well defined https URL, which should be valid for all SAGA (and non-SAGA) implementations which want to use that endpoint (unless the implementation does strange things). The problem only arises where endpoint have no well defined URLs associated (fork:// versus local://). The problems also arises with protocols which need additional informations from WS-Adressing header or from attributes in the legacy API, with protocols which use non-standard URL (for example with a fragment separator before the path), and with protocols that use the same scheme if the SAGA implementation wants to avoid late-binding (this is the case for JSAGA), and probably many other examples that we have not faced yet...
...but I think these examples are one more reason to leave some room for interpretation of URLs in the spec !
Similar for security contexts: the spec specifies a set of attributes which seem to map to most security tokens known to us - unless the SAGA implementation does strange things, you should always be able to use the 'UserProxy' attribute to point to an X509 certificate, etc. I agree ; SAGA implementations can have lot of differences in the attributes needed to generate the credentials (i.e. *-proxy-init for example), but they should not have so much differences in the way to simply use them.
But be careful, even if they are built with the same string value(s), currently the URL/Session objects passed in the SAGA method arguments must be created with the right SAGA implementation (maybe this last point can be improved in the SAGA implementations ?). Cheers, Sylvain
But SAGA cannot sensibly anticipate all possible security token attributes, as some of them are even deployment-dependent. That is why the security context attributes are extensible...
Cheers, Andre.
Cheers,
Mark
On Mon, Sep 5, 2011 at 3:07 PM, Sylvain Reynaud <Sylvain.Reynaud@in2p3.fr> wrote:
I agree ; SAGA implementations can have lot of differences in the attributes needed to generate the credentials (i.e. *-proxy-init for example), but they should not have so much differences in the way to simply use them.
But be careful, even if they are built with the same string value(s), currently the URL/Session objects passed in the SAGA method arguments must be created with the right SAGA implementation (maybe this last point can be improved in the SAGA implementations ?).
Yes, I agree - but I would not expect to obtain object level interoperability based on an API specification :-) I am not even sure if would be a worthy effort, as the lower implementation layers would become complex and more constraint as well... Cheers, Andre. -- Nothing is ever easy...
On Sep 3, 2011, at 6:21 AM, M.A. Santcroos wrote:
Hi Sylvain,
On 9/2/11 10:40 , "Sylvain Reynaud" <Sylvain.Reynaud@in2p3.fr> wrote:
Andre, I think you're right about the reason for the create() method ; it can indeed return a Task object that will contain the created object.
I don't really understand that. So how does that differ from the LSU functionality?
Mark, FYI you can configure JSAGA's "local://" adaptor to be mapped to URL scheme "fork://"... but anyway, there are many other differences between URL of various SAGA implementations (independently of the implemented binding). The other main difference between SAGA implementations is between the security contexts.
How come? Was there too much room for interpretation in the spec?
So if you want to develop an application that uses several SAGA implementations, creating URLs and security contexts will require implementation-specific code anyway.
I don't consider that a feature obviously. It kind of defeats the purpose of having a standard API.
Yes, this bothered me too. Dan
Cheers,
Mark
-- saga-rg mailing list saga-rg@ogf.org http://www.ogf.org/mailman/listinfo/saga-rg
-- Daniel S. Katz University of Chicago (773) 834-7186 (voice) (773) 834-6818 (fax) d.katz@ieee.org or dsk@ci.uchicago.edu http://www.ci.uchicago.edu/~dsk/
participants (4)
-
Andre Merzky -
Daniel S. Katz -
M.A. Santcroos -
Sylvain Reynaud