gridrpc-wg

Download

gridrpc-wg@lists.ogf.org

December 2006

2 participants
2 discussions

Re: [GRIDRPC-WG] Comments pertinent to GridRPC from ETSI
by Craig Lee 20 Dec '06

20 Dec '06

Hidemoto, The comments from ETSI concerning GridRPC and ByteIO do emphasize the need for OGF to have common terminology regarding interoperability, interoperability testing, versus conformance testing. There should also be clear distinction between _what_ is being tested (test purpose) and _how_ it is being tested (test description). The GFSG should take up this discussion if OGF is to collaborate with other organizations, such as ETSI, and also promote grid interoperability in a serious way. In the meantime, you could make these distinctions clear in the GridRPC Interop document with appropriate terminology, which the GFSG and OGF may eventually adopt. -Craig At 09:59 PM 12/18/2006, Hidemoto Nakada wrote: >Craig, > >Thank you for forwarding the comments. >While the comments are quite useful, some of them >look like not only for us but for whole OGF, e.g. >general terminology should be defined by OGF. > >How should we respond the comments? >Any idea? > >-hidemoto > >2006/12/16, Craig Lee <craig(a)rushg.aero.org>: > > > > Hidemoto, Yoshio, et al., > > > > Below are some comments on the Interoperability document from ETSI. > > These comments seem to be quite useful and we should respond since > > ETSI is quite interested in collaborating w/ OGF to establish and promote > > grid interoperability! > > > > --Craig > > > > > > At 07:49 AM 12/15/2006, Stephen M Pickles wrote: > > >Dear Craig, > > > > > >I invited ETSI to comment on the GridRPC and ByteIO > > >interoperability documents currently in public comment. > > >Stephan Shultz of ETSI has now done so - see attached. > > > > > >Some of these comments are pertinent to GridRPC > > >(and some are more general). > > > > > >Could you pass them on to authors of the document > > >"Interoperability Testing for The GridRPC API Specification"? > > > > > >Best regards, > > > > > >Stephen > > > > > >==================== Stephen M. Pickles ==================== > > > > > >Technical Director, National Grid Service > > >Manchester Computing > > >Room G49.1, Kilburn Building > > >The University of Manchester tel: +44 161 275 5974 > > >Oxford Road fax: +44 161 275 6800 > > >Manchester M13 9PL stephen.pickles(a)manchester.ac.uk > > >From: "Stephan Schulz" <Stephan.Schulz(a)etsi.org> > > >To: "GRID(a)LIST.ETSI.ORG" <GRID(a)LIST.ETSI.ORG> > > >CC: "Anthony Wiles" <Anthony.Wiles(a)etsi.org> > > >Subject: PTCC Review of OGF GRID Test Specs > > >Date: Thu, 14 Dec 2006 14:30:43 +0000 > > >X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 > > >MIME-Version: 1.0 > > >Thread-Topic: PTCC Review of OGF GRID Test Specs > > >Thread-Index: AccNVxOMLDrBcAlfTCOF3zsydsdaawSMv7HAAABNsFA= > > >Content-Type: text/plain; charset=us-ascii > > > > > >Hi all, > > > > > >as promised in the last TC GRID Meeting a PTCC review of the OGF test > > >specifications GridRPC and Byte IO. > > >I would appreciate it if someone from OGF, e.g., Steve Pickles, could > > >forward this to the relevant people in OGF. > > >Thanks > > > > > >Firstly I see a different use of terminology. Both documents talk about > > >"test cases" but understand it each differently > > >- and also differently from PTCC point of view. > > >GridRPC is giving high level descriptions essentially of what to test. > > >PTCC > > >calls that test purposes. What GridRPC calls "test code" comes closer to > > >our idea of a > > >test case. Something which is executable .. a test script if you will. > > >The ByteIO specification is much more detailed than a test purpose, > > >e.g., it gives detailed message > > >contents, arrival of the verdict is convert in great detail, however as > > >such the document > > >is not executable. In our methodology we call this form of specification > > >test descriptions > > >(which is an optional intermediate step between test purpose and test > > >case implementation). > > >In ETSI 3GPP test specifications this intermediate step has also be > > >used. > > > >From PTCC point of view it would be nice to have a consistent use of > > >terminology in specification. > > >Of course it would be nice to follow ETSI methodology but at the end > > >that is of course up to you. > > >In the same way we feel it is beneficial to separate WHAT is to be > > >tested (=purpose) and HOW to > > >test (= description). In the Byte IO case it these two issues seem to be > > >mixed and they should be > > >untangled. Clear separation of WHAT and HOW[StS] gives you a better > > >idea, e.g., to get a clear > > >idea on the passing criteria and about your actual test coverage of your > > >tests . Another benefit of > > >test purposes is that they can be understood by a much wider audience as > > >test description since > > >they do not require deep knowledge, e.g., of protocol syntax etc. > > > > > >Both documents position themselves as being a basis for interoperability > > >testing. From PTCC point of > > >view both are really specifying conformance tests. I am not sure why the > > >term conformance is avoided. > > >Or maybe there is just a mix-up of words here which is (achievement of) > > >interoperability versus > > >interoperability testing. > > >GridRPC states for example that "As such, the requirement is only to > > >test client interoperability: that > > >multiple implementations of a specification provide consistent results > > >to a suite of tests". In my opinion > > >this just falls short of saying that these implementations conform to > > >the specification which has been > > >selected to be the basis of these so called interoperability tests, > > >i.e., GFD-R.52. > > >ByteIO test case descriptions specify validation criteria at the level > > >of bytes, i.e., WSDL messages. > > >In both specs it seems to me that the system under test consists of one > > >entity, i.e., the server, being > > >tested in isolation. I could be wrong. > > >In ETSI (formal) interoperability testing is defined as operating > > >different implementations of different entities > > >against each other, e.g., a NIFT client implementation against a > > >GridSolve server. Therefore an interoperability > > >test would describe how to stimulate and observe client as well as > > >server implementations not the interface > > >betweeen them. > > > > > >The two documents are specifying tests at different levels - GridRPC > > >more at what I would consider software > > >module testing and ByteIO more as protocol/Unit testing. From our point > > >of view that is fine. However I wonder > > >in the case of GridRPC why this specification is limited to C > > >implementations only. It seems there is a lot of > > >Grid middle ware based on Java out there. Should this test specification > > >not be more implementation agnostic? > > >On the ByteIO side I wonder why test purposes have to be bound so > > >specifically to one specific scenario or > > >initial configuration? Again it should be thought about to adopt true > > >test purposes which are more abstract and > > >leave the details to the test description and implementation. One > > >approach could be to change from checking > > >rigidly every single message parameter in a single test to focusing on > > >the information which are needed to fulfil > > >some function. SO for example to say instead of "check that size > > >parameter is 2 and ...". To state that "ensure > > >that the request for size returns a valid value. This test purpose could > > >then have multiple test implementations > > >testing the value 2 or boundary conditions, etc. > > > > > >Possibilities for further improvement: > > >As said before a clearer separation of WHAT is to be test versus HOW > > >specifically in the ByteIO spec - the > > >looks fine. > > >Both documents are lacking references in the test purpose/description to > > >the requirement(s) they pertain to > > >in the specification, e.g., document clause or quotation of > > >specification text. Such reference are very important > > > to trace back test results to the specification or to ensure the test > > >itself is a valid one. > > >Another point is clearer identification and separation of mandatory > > >versus optional features of the specification > > > / implementation under test. That means entire tests or test purposes > > >should be optional not only some of > > >multiple validation criteria within the same test purpose. This > > >categorization is also helpful for identifying the > > >minimum set of tests that a implementation must pass before being > > >compliant or shall we say "considered > > > interoperable". GridRPC has some discussion of optional responses based > > >on about observed differences > > >with various applications but it does not clarify what the response > > >should be, i.e., what the specification actually > > >states. > > > > > >Let me know if you have any questions. > > >Thanks, > > >stephan > > > > > >o--------------------------------o > > > Stephan Schulz, Ph.D. > > > ETSI PTCC Technical Officer > > > 650 Rue des Lucioles > > > F-06921 Sophia Antipolis Cedex > > > Tel: +33 49294 4964 > > > Mobile: +33 63334 8269 > > > Fax: +33 49365 4716 > > >o--------------------------------o > > > > > > >-- > HIDEMOTO NAKADA, GTRC, AIST >-- > gridrpc-wg mailing list > gridrpc-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/gridrpc-wg

2 1

Re: [GRIDRPC-WG] Comments pertinent to GridRPC from ETSI
by Craig Lee 19 Dec '06

19 Dec '06

Hidemoto, Yoshio, et al., Below are some comments on the Interoperability document from ETSI. These comments seem to be quite useful and we should respond since ETSI is quite interested in collaborating w/ OGF to establish and promote grid interoperability! --Craig At 07:49 AM 12/15/2006, Stephen M Pickles wrote: >Dear Craig, > >I invited ETSI to comment on the GridRPC and ByteIO >interoperability documents currently in public comment. >Stephan Shultz of ETSI has now done so - see attached. > >Some of these comments are pertinent to GridRPC >(and some are more general). > >Could you pass them on to authors of the document >"Interoperability Testing for The GridRPC API Specification"? > >Best regards, > >Stephen > >==================== Stephen M. Pickles ==================== > >Technical Director, National Grid Service >Manchester Computing >Room G49.1, Kilburn Building >The University of Manchester tel: +44 161 275 5974 >Oxford Road fax: +44 161 275 6800 >Manchester M13 9PL stephen.pickles(a)manchester.ac.uk >From: "Stephan Schulz" <Stephan.Schulz(a)etsi.org> >To: "GRID(a)LIST.ETSI.ORG" <GRID(a)LIST.ETSI.ORG> >CC: "Anthony Wiles" <Anthony.Wiles(a)etsi.org> >Subject: PTCC Review of OGF GRID Test Specs >Date: Thu, 14 Dec 2006 14:30:43 +0000 >X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 >MIME-Version: 1.0 >Thread-Topic: PTCC Review of OGF GRID Test Specs >Thread-Index: AccNVxOMLDrBcAlfTCOF3zsydsdaawSMv7HAAABNsFA= >Content-Type: text/plain; charset=us-ascii > >Hi all, > >as promised in the last TC GRID Meeting a PTCC review of the OGF test >specifications GridRPC and Byte IO. >I would appreciate it if someone from OGF, e.g., Steve Pickles, could >forward this to the relevant people in OGF. >Thanks > >Firstly I see a different use of terminology. Both documents talk about >"test cases" but understand it each differently >- and also differently from PTCC point of view. >GridRPC is giving high level descriptions essentially of what to test. >PTCC >calls that test purposes. What GridRPC calls "test code" comes closer to >our idea of a >test case. Something which is executable .. a test script if you will. >The ByteIO specification is much more detailed than a test purpose, >e.g., it gives detailed message >contents, arrival of the verdict is convert in great detail, however as >such the document >is not executable. In our methodology we call this form of specification >test descriptions >(which is an optional intermediate step between test purpose and test >case implementation). >In ETSI 3GPP test specifications this intermediate step has also be >used. > >From PTCC point of view it would be nice to have a consistent use of >terminology in specification. >Of course it would be nice to follow ETSI methodology but at the end >that is of course up to you. >In the same way we feel it is beneficial to separate WHAT is to be >tested (=purpose) and HOW to >test (= description). In the Byte IO case it these two issues seem to be >mixed and they should be >untangled. Clear separation of WHAT and HOW[StS] gives you a better >idea, e.g., to get a clear >idea on the passing criteria and about your actual test coverage of your >tests . Another benefit of >test purposes is that they can be understood by a much wider audience as >test description since >they do not require deep knowledge, e.g., of protocol syntax etc. > >Both documents position themselves as being a basis for interoperability >testing. From PTCC point of >view both are really specifying conformance tests. I am not sure why the >term conformance is avoided. >Or maybe there is just a mix-up of words here which is (achievement of) >interoperability versus >interoperability testing. >GridRPC states for example that "As such, the requirement is only to >test client interoperability: that >multiple implementations of a specification provide consistent results >to a suite of tests". In my opinion >this just falls short of saying that these implementations conform to >the specification which has been >selected to be the basis of these so called interoperability tests, >i.e., GFD-R.52. >ByteIO test case descriptions specify validation criteria at the level >of bytes, i.e., WSDL messages. >In both specs it seems to me that the system under test consists of one >entity, i.e., the server, being >tested in isolation. I could be wrong. >In ETSI (formal) interoperability testing is defined as operating >different implementations of different entities >against each other, e.g., a NIFT client implementation against a >GridSolve server. Therefore an interoperability >test would describe how to stimulate and observe client as well as >server implementations not the interface >betweeen them. > >The two documents are specifying tests at different levels - GridRPC >more at what I would consider software >module testing and ByteIO more as protocol/Unit testing. From our point >of view that is fine. However I wonder >in the case of GridRPC why this specification is limited to C >implementations only. It seems there is a lot of >Grid middle ware based on Java out there. Should this test specification >not be more implementation agnostic? >On the ByteIO side I wonder why test purposes have to be bound so >specifically to one specific scenario or >initial configuration? Again it should be thought about to adopt true >test purposes which are more abstract and >leave the details to the test description and implementation. One >approach could be to change from checking >rigidly every single message parameter in a single test to focusing on >the information which are needed to fulfil >some function. SO for example to say instead of "check that size >parameter is 2 and ...". To state that "ensure >that the request for size returns a valid value. This test purpose could >then have multiple test implementations >testing the value 2 or boundary conditions, etc. > >Possibilities for further improvement: >As said before a clearer separation of WHAT is to be test versus HOW >specifically in the ByteIO spec - the >looks fine. >Both documents are lacking references in the test purpose/description to >the requirement(s) they pertain to >in the specification, e.g., document clause or quotation of >specification text. Such reference are very important > to trace back test results to the specification or to ensure the test >itself is a valid one. >Another point is clearer identification and separation of mandatory >versus optional features of the specification > / implementation under test. That means entire tests or test purposes >should be optional not only some of >multiple validation criteria within the same test purpose. This >categorization is also helpful for identifying the >minimum set of tests that a implementation must pass before being >compliant or shall we say "considered > interoperable". GridRPC has some discussion of optional responses based >on about observed differences >with various applications but it does not clarify what the response >should be, i.e., what the specification actually >states. > >Let me know if you have any questions. >Thanks, >stephan > >o--------------------------------o > Stephan Schulz, Ph.D. > ETSI PTCC Technical Officer > 650 Rue des Lucioles > F-06921 Sophia Antipolis Cedex > Tel: +33 49294 4964 > Mobile: +33 63334 8269 > Fax: +33 49365 4716 >o--------------------------------o

2 1