Dear Steve, Here are my comments as well on the API. Some of them overlap those of Andre, so sorry for the verbatim. 1) The attributes should be handled by the attribute interface. This will allow better SAGA L&F integration. Also this might remove the need for specific get_vo, get_key & get_values methods. These can be treated as attributes of the class. In SAGA, the getters and setters are in general covered by the attributes interface and the implementation allows specific support for them if needed. The SAGA API should, for the time being, go for the minimum common set of functions. 2) I totally agree with Andre that the service should be an interface and not a class. Well in a Java binding it might be an all interface anyway. 3) The use of SQL is what made me read the section twice. I think we had already a long discussion on the other parts of the API on the usage of REGX, wildcards, SQL, etc. We might try to sort this out either by F2F or telecon. 4) The name space of this API should be better then "sd". Why not "discovery" or something more expressive? The API specification document and the language bindings L&F are different things. In the document we have a more C style coding. When the implementation is done we should use language specifics: - list_services() -> in C: list_services() - list_services() -> in C++: ListServices() - list_services() -> in Java: list () So no much worry at the time being. sub-namespaces are not necessary in the SAGA API because the API is just the front end and should remain thin. We expect the implementations that are behind to either use their own namespace or use sub-spaces. Otherwise thanks for your work. If you have a simple demo it is always nice to see real "software" and not just real "vapoware" :P Best regards, Pascal Kleijer ---------------------------------------------------------------- HPC Marketing Promotion Division, NEC Corporation 1-10, Nisshin-cho, Fuchu, Tokyo, 183-8501, Japan. Tel: +81-(0)42/333.6389 Fax: +81-(0)42/333.6382 http://www2.nec.co.jp/online-tv/en/introduction/society/met_h.html Andre Merzky wrote:

Dear Steve,

thanks for sharing the document draft! I do have quite a number of comments, so would like to emphasize that I do actually like the document, very much so, both in scope and approach, and thus hope you won't take the comments as criticism - they aren't :-)

I think Shantenu plans a phone call for next Wednesday, in particular also to discuss the document. Would that work for you?

Comments:

- it would be good to tie the API package closer to the other SAGA packages. In particular, it would be very good if the url returned by service.get_url() would explicitely be formed so that it can be used for the construction of saga::job_service, saga::directory, saga::rpc instance etc.

- service should be an interface, not a class. A language binding (e.g. for Java) might render that as interface, but in the langauge independend specs we agreed to model the functional API packages as classes. Unless I am missing some point here?

- we do have an attribute interface (doh! ;-) in the SAGA core API spec. I think it would be best to use that interface for the service interface/class.

- why is get_attributes on discoverer, but get_values on service? That seems inconsistent?

- is it possible to tighten the definition of possible query strings? For example, the current spec would allow to query with the service filter for

- type = "ftp" - type = "gsiftp" - type = "gridftp" - type = "remote data"

How does the end user know what to use? Does Glue help here (please excuse my ignorance about Glue...)

- I am still worried that SAGA uses regular expressions in one place, wildcards in other places, and now SQL, too. Not that I know a better approach, or even solution, but we should keep that problem in mind. I think your statement "uses SQL 92 syntax as if it were part of a WHERE clause" is good: that is something which has a good chance to be reusable by other SQL related packages, should they emerge.

- I am somewhat confused about the difference between keys and attributes in the service interface/class... What is the difference, really? Do you have an example?

- Is it actually possible to tighten the format of the VO string, e.g. is there some commonly agreed upon format?

- I think service.get_vo_names should be changed into an attribute, too: service.get_attribute ("vo_names"). Does that make sense?

Quoting [Fisher, SM (Steve)] (Apr 05 2007):

...

Hi,

I attach a first draft of the proposed service discovery SAGA API extension following very closely my presentation at the last GGF. Once I have write access to CVS I will check the source files in.

Ouch, I think I forgot to take care of this. My apologies! I will be offline over the next week. Hartmut, Shantenu, could you possibly take care of an CVS account at CCT?

Again, my apologies...

...

We have a prototpye implementation which does not make use of a proper adapter mechanism yet - however we plan to use the SAGA adapter style for the C++. The C++ prototpye is almost complete for the BDII (LDAP) information system and the R-GMA one is not far behind. The performance is good - we can show it at Manchester.

We would appreciate any comments - especially identifying those areas where we are not doing things "the SAGA way".

:-D I think there aare some of these points above. Main one is about the attributes...

You might want to check the saga::job_description, which is an kind of empty class which _only_ implements the attribute interface. It has a lot of the things which you added in a similar way.

Also, it would be good to specify the available attributes as far a possible.

...

Incidentally we were surprised that we are only expected to use one name space for the implementation - why are sub-namespaces not permitted - especially for the API extensions?

That seemed unneccessary for now. Imagine a C language binding: all name spaces would need rendering the the (flat) names of C functions - that can get messy.

Languages like Java, which have a very hierarchical name space usually, may add name spaces - I am not sure about the neccessity though...

...

For the Java implementation are we expected to use lower case class names and underscore separated method names?

Steve

Thanks, best regards,

Andre.

Content-Description: comments.txt

Andre's email =============

- it would be good to tie the API package closer to the other SAGA packages. In particular, it would be very good if the url returned by service.get_url() would explicitely be formed so that it can be used for the construction of saga::job_service, saga::directory, saga::rpc instance etc.

This depends upon what is published - if the URL is published correctly we will return it.

Ah, I see: there is no prescription on what the URL is usable for, as you have, naturally, no control over what URLs are added to the information repository in the first place. Understand. However, that leaves some questions on how to use the returned URLs in the application. Assume an application queries for list <string> urls = d.list_services ("*data*", "*", "*"); (query probably ill formatted...) Can the application use the returned URL to create a saga::directory, or a saga::file? I assume that this question cannot be answered within the SD API, and I am not sure what mechanism could be used to answer it at all. Any thoughts? However, if that use of the returned URLs (to create saga::xyz class instances) is the intended use of the SD API, adding a statement in that respect would clarify the intendet use of the extension.

- service should be an interface, not a class. A language binding (e.g. for Java) might render that as interface, but in the langauge independend specs we agreed to model the functional API packages as classes. Unless I am missing some point here?

That was a mistake in the text - in the SIDL we have the service as an interface but not in the accompanying text. However if the service is to implement an attribute interface then it can't itself be an interface but must become a class - we could of course extend rather than implement.

I have to apologize, my mistake: I meant it should be a class, not an interface. The following still holds: "in the langauge independend specs we agreed to model the functional API packages as classes."

The discoverer should be a class returning services.

Incidentally it may makes sense to rename "service" to "service_description".

Yes, it does.

- we do have an attribute interface (doh! ;-) in the SAGA core API spec. I think it would be best to use that interface for the service interface/class.

Yes - there seems to be no problem with that in principle for the service attributes.

- why is get_attributes on discoverer, but get_values on service? That seems inconsistent?

Discoverer has get_attribute_names() method which will return an array of all possible standard names deliverable by the existing information system adapters. All services have the same set so associating this data with the discoverer rather then the service seems to make sense.

If the attribute set is fixed, then using the saga::attribute interface in the service (or service_description) instances would be easy. The document should specify what attributes are available IMHO. Again, technically this looks more and more like the saga::job_description class, and IMHO it would be good to design the service(_description) class the same way.

- is it possible to tighten the definition of possible query strings? For example, the current spec would allow to query with the service filter for

- type = "ftp" - type = "gsiftp" - type = "gridftp" - type = "remote data"

How does the end user know what to use? Does Glue help here (please excuse my ignorance about Glue...)

In the case of the type it is the responsibility of the service provider, publishing the service to publish a well known service type. I prefer the service name to take the form of a DNS registered name - registered by the service provider. e.g. org.globus.gridftp.

Ok, so thats out of scope. SHould be mentioned in the text I think.

- I am still worried that SAGA uses regular expressions in one place, wildcards in other places, and now SQL, too. Not that I know a better approach, or even solution, but we should keep that problem in mind. I think your statement "uses SQL 92 syntax as if it were part of a WHERE clause" is good: that is something which has a good chance to be reusable by other SQL related packages, should they emerge.

We really need the power of SQL!

ok

- I am somewhat confused about the difference between keys and attributes in the service interface/class... What is the difference, really? Do you have an example?

The attributes are based on the GLUE attributes. The key/value pairs are defined by the service publisher. The service publisher is free to define his own key names.

The key names may clash with the glue attribute names so we cannot use the attribute interface for both. We favout using the attribute interface for the glue attributes and retain our current solution for the key value pairs.

Ah, that clarifies a lot. The first paragraph above would help to understand the document, and should be added to the text I think. Interesting problem, clashing attribute name spaces... I have no good idea at the moment, will think about it...

- Is it actually possible to tighten the format of the VO string, e.g. is there some commonly agreed upon format?

In EGEE/gLite/WLCG we are moving towards DNS names again to avoid clashes. However again the actual names are out of scope. GLUE makes recommendations but insists upon nothing.

ok

- I think service.get_vo_names should be changed into an attribute, too: service.get_attribute ("vo_names"). Does that make sense?

We can do that. We will keep the get_url call however as it is a very common one.

Its not a problem to have the URL as an attrubute, _and_ to have a specific getter method. We have the same for job state for example, which is an attribute (which makes sense semantically), and has a getter method (get_state), because it's used so often.

Pascal's email ==============

1) The attributes should be handled by the attribute interface. This will allow better SAGA L&F integration. Also this might remove the need for specific get_vo, get_key & get_values methods. These can be treated as attributes of the class. In SAGA, the getters and setters are in general covered by the attributes interface and the implementation allows specific support for them if needed. The SAGA API should, for the time being, go for the minimum common set of functions.

Yes - see above

2) I totally agree with Andre that the service should be an interface and not a class. Well in a Java binding it might be an all interface anyway.

Yes - see above

3) The use of SQL is what made me read the section twice. I think we had already a long discussion on the other parts of the API on the usage of REGX, wildcards, SQL, etc. We might try to sort this out either by F2F or telecon.

I don't think it is feasible to get the desired behaviour without SQL or something similar. We do want the code to be useful in gLite.

It would be good to check what other service discovery backends are using, and if the used SQL is translatable to that. Do you have any experiences with other backends?

4) The name space of this API should be better then "sd". Why not "discovery" or something more expressive?

As we are not allowed to make the namespace visible in the implementation I don't really care too much what goes in the SIDL as a name space. What is the justification for not having sub-namespaces within SAGA as it says in section 2.2.1 of the spec?

1. it would lead to clashes between namespace and class names (file.file, stream.stream, rpc.rpc, job.job). 2. languages w/o name space support would need to flatten the namespace, resulting in method names like: saga_logical_file_logical_directory_list ("*"); 3. given the two (possibly minor) disadvantages above, and no percieved need for additional name spaces, we decided against their use.

The API specification document and the language bindings L&F are different things. In the document we have a more C style coding. When the implementation is done we should use language specifics:

- list_services() -> in C: list_services() - list_services() -> in C++: ListServices() - list_services() -> in Java: list ()

Sometime rather soon we should define these styles.

Oh yes. Cheers, Andre.

For both C++ and Java I would favour listServices(). Howebver it may be best to just follow the majority of the existing implementation code.

Steve and Paventhan

-- "XML is like violence: if it does not help, use more."

Quoting [Fisher, SM (Steve)] (May 07 2007):

The point of service discovery is to locate a service. Any SAGA call which needs to connect to a remote service and that makes this explict can make use of SD. For example the RPC of 3.13 requires the URL of the service provider. Also the job_service requires as input the contact string for the resouce manager. Servide discover provides these URLs. I haved added a sentence or two explaining this.

Yes, that is what I meant. Having that statement in the doc helps to understand the relation between the documents. Thanks!

...

...

- we do have an attribute interface (doh! ;-) in the SAGA core API spec. I think it would be best to use that interface for the service interface/class.

Yes - there seems to be no problem with that in principle for the service attributes.

- why is get_attributes on discoverer, but get_values on service? That seems inconsistent?

Discoverer has get_attribute_names() method which will return an array of all possible standard names deliverable by the existing information system adapters. All services have the same set so associating this data with the discoverer rather then the service seems to make sense.

If the attribute set is fixed, then using the saga::attribute interface in the service (or service_description) instances would be easy. The document should specify what attributes are available IMHO. Again, technically this looks more and more like the saga::job_description class, and IMHO it would be good to design the service(_description) class the same way.

If we want to use the attribute interface, and I agree that we should, perhaps the best way is to have two extra classes both of which implement the Attribute interface. Then you can say, where s is a service_description:

s.get_data().get_attribute("some key value in the key value pair")

or

s.get_glue().get_attribute("some predefined 'glue' attribute")

This avoids the name clash problem - though we need a better name than get_glue()

I would like people's agreement that this is the right way to go before doing this. It makes things simepler but adds 2 trivial classes. I think I like it!

Interesting point! Its a tradeoff, as you say. I am not sure if I like it personally, but it looks like a clean solution, and if others are ok with it we should go that route I guess.

...

3. given the two (possibly minor) disadvantages above, and no percieved need for additional name spaces, we decided against their use.

I think it was a bad choice - but I can live with these rules

:-)

...

...

The API specification document and the language bindings L&F are different things. In the document we have a more C style coding. When the implementation is done we should use language specifics:

- list_services() -> in C: list_services() - list_services() -> in C++: ListServices() - list_services() -> in Java: list ()

Sometime rather soon we should define these styles.

So are we free to add namespaces - like all other large packages ;-)

:-P Yes, but should be done in sync with the language bindings for the saga core spec... Thanks! cu tomorrow, Andre.

Steve

-- "XML is like violence: if it does not help, use more."