Re: [GRIDRPC-WG] Comments pertinent to GridRPC from ETSI
Hidemoto, The comments from ETSI concerning GridRPC and ByteIO do emphasize the need for OGF to have common terminology regarding interoperability, interoperability testing, versus conformance testing. There should also be clear distinction between _what_ is being tested (test purpose) and _how_ it is being tested (test description). The GFSG should take up this discussion if OGF is to collaborate with other organizations, such as ETSI, and also promote grid interoperability in a serious way. In the meantime, you could make these distinctions clear in the GridRPC Interop document with appropriate terminology, which the GFSG and OGF may eventually adopt. -Craig At 09:59 PM 12/18/2006, Hidemoto Nakada wrote:
Craig,
Thank you for forwarding the comments. While the comments are quite useful, some of them look like not only for us but for whole OGF, e.g. general terminology should be defined by OGF.
How should we respond the comments? Any idea?
-hidemoto
2006/12/16, Craig Lee <craig@rushg.aero.org>:
Hidemoto, Yoshio, et al.,
Below are some comments on the Interoperability document from ETSI. These comments seem to be quite useful and we should respond since ETSI is quite interested in collaborating w/ OGF to establish and promote grid interoperability!
--Craig
At 07:49 AM 12/15/2006, Stephen M Pickles wrote:
Dear Craig,
I invited ETSI to comment on the GridRPC and ByteIO interoperability documents currently in public comment. Stephan Shultz of ETSI has now done so - see attached.
Some of these comments are pertinent to GridRPC (and some are more general).
Could you pass them on to authors of the document "Interoperability Testing for The GridRPC API Specification"?
Best regards,
Stephen
==================== Stephen M. Pickles ====================
Technical Director, National Grid Service Manchester Computing Room G49.1, Kilburn Building The University of Manchester tel: +44 161 275 5974 Oxford Road fax: +44 161 275 6800 Manchester M13 9PL stephen.pickles@manchester.ac.uk From: "Stephan Schulz" <Stephan.Schulz@etsi.org> To: "GRID@LIST.ETSI.ORG" <GRID@LIST.ETSI.ORG> CC: "Anthony Wiles" <Anthony.Wiles@etsi.org> Subject: PTCC Review of OGF GRID Test Specs Date: Thu, 14 Dec 2006 14:30:43 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 MIME-Version: 1.0 Thread-Topic: PTCC Review of OGF GRID Test Specs Thread-Index: AccNVxOMLDrBcAlfTCOF3zsydsdaawSMv7HAAABNsFA= Content-Type: text/plain; charset=us-ascii
Hi all,
as promised in the last TC GRID Meeting a PTCC review of the OGF test specifications GridRPC and Byte IO. I would appreciate it if someone from OGF, e.g., Steve Pickles, could forward this to the relevant people in OGF. Thanks
From PTCC point of view it would be nice to have a consistent use of terminology in specification. Of course it would be nice to follow ETSI methodology but at the end
Firstly I see a different use of terminology. Both documents talk about "test cases" but understand it each differently - and also differently from PTCC point of view. GridRPC is giving high level descriptions essentially of what to test. PTCC calls that test purposes. What GridRPC calls "test code" comes closer to our idea of a test case. Something which is executable .. a test script if you will. The ByteIO specification is much more detailed than a test purpose, e.g., it gives detailed message contents, arrival of the verdict is convert in great detail, however as such the document is not executable. In our methodology we call this form of specification test descriptions (which is an optional intermediate step between test purpose and test case implementation). In ETSI 3GPP test specifications this intermediate step has also be used. that is of course up to you. In the same way we feel it is beneficial to separate WHAT is to be tested (=purpose) and HOW to test (= description). In the Byte IO case it these two issues seem to be mixed and they should be untangled. Clear separation of WHAT and HOW[StS] gives you a better idea, e.g., to get a clear idea on the passing criteria and about your actual test coverage of your tests . Another benefit of test purposes is that they can be understood by a much wider audience as test description since they do not require deep knowledge, e.g., of protocol syntax etc.
Both documents position themselves as being a basis for interoperability testing. From PTCC point of view both are really specifying conformance tests. I am not sure why the term conformance is avoided. Or maybe there is just a mix-up of words here which is (achievement of) interoperability versus interoperability testing. GridRPC states for example that "As such, the requirement is only to test client interoperability: that multiple implementations of a specification provide consistent results to a suite of tests". In my opinion this just falls short of saying that these implementations conform to the specification which has been selected to be the basis of these so called interoperability tests, i.e., GFD-R.52. ByteIO test case descriptions specify validation criteria at the level of bytes, i.e., WSDL messages. In both specs it seems to me that the system under test consists of one entity, i.e., the server, being tested in isolation. I could be wrong. In ETSI (formal) interoperability testing is defined as operating different implementations of different entities against each other, e.g., a NIFT client implementation against a GridSolve server. Therefore an interoperability test would describe how to stimulate and observe client as well as server implementations not the interface betweeen them.
The two documents are specifying tests at different levels - GridRPC more at what I would consider software module testing and ByteIO more as protocol/Unit testing. From our point of view that is fine. However I wonder in the case of GridRPC why this specification is limited to C implementations only. It seems there is a lot of Grid middle ware based on Java out there. Should this test specification not be more implementation agnostic? On the ByteIO side I wonder why test purposes have to be bound so specifically to one specific scenario or initial configuration? Again it should be thought about to adopt true test purposes which are more abstract and leave the details to the test description and implementation. One approach could be to change from checking rigidly every single message parameter in a single test to focusing on the information which are needed to fulfil some function. SO for example to say instead of "check that size parameter is 2 and ...". To state that "ensure that the request for size returns a valid value. This test purpose could then have multiple test implementations testing the value 2 or boundary conditions, etc.
Possibilities for further improvement: As said before a clearer separation of WHAT is to be test versus HOW specifically in the ByteIO spec - the looks fine. Both documents are lacking references in the test purpose/description to the requirement(s) they pertain to in the specification, e.g., document clause or quotation of specification text. Such reference are very important to trace back test results to the specification or to ensure the test itself is a valid one. Another point is clearer identification and separation of mandatory versus optional features of the specification / implementation under test. That means entire tests or test purposes should be optional not only some of multiple validation criteria within the same test purpose. This categorization is also helpful for identifying the minimum set of tests that a implementation must pass before being compliant or shall we say "considered interoperable". GridRPC has some discussion of optional responses based on about observed differences with various applications but it does not clarify what the response should be, i.e., what the specification actually states.
Let me know if you have any questions. Thanks, stephan
o--------------------------------o Stephan Schulz, Ph.D. ETSI PTCC Technical Officer 650 Rue des Lucioles F-06921 Sophia Antipolis Cedex Tel: +33 49294 4964 Mobile: +33 63334 8269 Fax: +33 49365 4716 o--------------------------------o
-- HIDEMOTO NAKADA, GTRC, AIST -- gridrpc-wg mailing list gridrpc-wg@ogf.org http://www.ogf.org/mailman/listinfo/gridrpc-wg
Craig, Thank you for the reply. We will fix our document so that it will use 'test purpose' and 'test case'. 2006/12/20, Craig Lee <craig@rushg.aero.org>:
Hidemoto,
The comments from ETSI concerning GridRPC and ByteIO do emphasize the need for OGF to have common terminology regarding interoperability, interoperability testing, versus conformance testing. There should also be clear distinction between _what_ is being tested (test purpose) and _how_ it is being tested (test description). The GFSG should take up this discussion if OGF is to collaborate with other organizations, such as ETSI, and also promote grid interoperability in a serious way.
In the meantime, you could make these distinctions clear in the GridRPC Interop document with appropriate terminology, which the GFSG and OGF may eventually adopt.
-Craig
At 09:59 PM 12/18/2006, Hidemoto Nakada wrote:
Craig,
Thank you for forwarding the comments. While the comments are quite useful, some of them look like not only for us but for whole OGF, e.g. general terminology should be defined by OGF.
How should we respond the comments? Any idea?
-hidemoto
2006/12/16, Craig Lee <craig@rushg.aero.org>:
Hidemoto, Yoshio, et al.,
Below are some comments on the Interoperability document from ETSI. These comments seem to be quite useful and we should respond since ETSI is quite interested in collaborating w/ OGF to establish and promote grid interoperability!
--Craig
At 07:49 AM 12/15/2006, Stephen M Pickles wrote:
Dear Craig,
I invited ETSI to comment on the GridRPC and ByteIO interoperability documents currently in public comment. Stephan Shultz of ETSI has now done so - see attached.
Some of these comments are pertinent to GridRPC (and some are more general).
Could you pass them on to authors of the document "Interoperability Testing for The GridRPC API Specification"?
Best regards,
Stephen
==================== Stephen M. Pickles ====================
Technical Director, National Grid Service Manchester Computing Room G49.1, Kilburn Building The University of Manchester tel: +44 161 275 5974 Oxford Road fax: +44 161 275 6800 Manchester M13 9PL stephen.pickles@manchester.ac.uk From: "Stephan Schulz" <Stephan.Schulz@etsi.org> To: "GRID@LIST.ETSI.ORG" <GRID@LIST.ETSI.ORG> CC: "Anthony Wiles" <Anthony.Wiles@etsi.org> Subject: PTCC Review of OGF GRID Test Specs Date: Thu, 14 Dec 2006 14:30:43 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 MIME-Version: 1.0 Thread-Topic: PTCC Review of OGF GRID Test Specs Thread-Index: AccNVxOMLDrBcAlfTCOF3zsydsdaawSMv7HAAABNsFA= Content-Type: text/plain; charset=us-ascii
Hi all,
as promised in the last TC GRID Meeting a PTCC review of the OGF test specifications GridRPC and Byte IO. I would appreciate it if someone from OGF, e.g., Steve Pickles, could forward this to the relevant people in OGF. Thanks
From PTCC point of view it would be nice to have a consistent use of terminology in specification. Of course it would be nice to follow ETSI methodology but at the end
Firstly I see a different use of terminology. Both documents talk about "test cases" but understand it each differently - and also differently from PTCC point of view. GridRPC is giving high level descriptions essentially of what to test. PTCC calls that test purposes. What GridRPC calls "test code" comes closer to our idea of a test case. Something which is executable .. a test script if you will. The ByteIO specification is much more detailed than a test purpose, e.g., it gives detailed message contents, arrival of the verdict is convert in great detail, however as such the document is not executable. In our methodology we call this form of specification test descriptions (which is an optional intermediate step between test purpose and test case implementation). In ETSI 3GPP test specifications this intermediate step has also be used. that is of course up to you. In the same way we feel it is beneficial to separate WHAT is to be tested (=purpose) and HOW to test (= description). In the Byte IO case it these two issues seem to be mixed and they should be untangled. Clear separation of WHAT and HOW[StS] gives you a better idea, e.g., to get a clear idea on the passing criteria and about your actual test coverage of your tests . Another benefit of test purposes is that they can be understood by a much wider audience as test description since they do not require deep knowledge, e.g., of protocol syntax etc.
Both documents position themselves as being a basis for interoperability testing. From PTCC point of view both are really specifying conformance tests. I am not sure why the term conformance is avoided. Or maybe there is just a mix-up of words here which is (achievement of) interoperability versus interoperability testing. GridRPC states for example that "As such, the requirement is only to test client interoperability: that multiple implementations of a specification provide consistent results to a suite of tests". In my opinion this just falls short of saying that these implementations conform to the specification which has been selected to be the basis of these so called interoperability tests, i.e., GFD-R.52. ByteIO test case descriptions specify validation criteria at the level of bytes, i.e., WSDL messages. In both specs it seems to me that the system under test consists of one entity, i.e., the server, being tested in isolation. I could be wrong. In ETSI (formal) interoperability testing is defined as operating different implementations of different entities against each other, e.g., a NIFT client implementation against a GridSolve server. Therefore an interoperability test would describe how to stimulate and observe client as well as server implementations not the interface betweeen them.
The two documents are specifying tests at different levels - GridRPC more at what I would consider software module testing and ByteIO more as protocol/Unit testing. From our point of view that is fine. However I wonder in the case of GridRPC why this specification is limited to C implementations only. It seems there is a lot of Grid middle ware based on Java out there. Should this test specification not be more implementation agnostic? On the ByteIO side I wonder why test purposes have to be bound so specifically to one specific scenario or initial configuration? Again it should be thought about to adopt true test purposes which are more abstract and leave the details to the test description and implementation. One approach could be to change from checking rigidly every single message parameter in a single test to focusing on the information which are needed to fulfil some function. SO for example to say instead of "check that size parameter is 2 and ...". To state that "ensure that the request for size returns a valid value. This test purpose could then have multiple test implementations testing the value 2 or boundary conditions, etc.
Possibilities for further improvement: As said before a clearer separation of WHAT is to be test versus HOW specifically in the ByteIO spec - the looks fine. Both documents are lacking references in the test purpose/description to the requirement(s) they pertain to in the specification, e.g., document clause or quotation of specification text. Such reference are very important to trace back test results to the specification or to ensure the test itself is a valid one. Another point is clearer identification and separation of mandatory versus optional features of the specification / implementation under test. That means entire tests or test purposes should be optional not only some of multiple validation criteria within the same test purpose. This categorization is also helpful for identifying the minimum set of tests that a implementation must pass before being compliant or shall we say "considered interoperable". GridRPC has some discussion of optional responses based on about observed differences with various applications but it does not clarify what the response should be, i.e., what the specification actually states.
Let me know if you have any questions. Thanks, stephan
o--------------------------------o Stephan Schulz, Ph.D. ETSI PTCC Technical Officer 650 Rue des Lucioles F-06921 Sophia Antipolis Cedex Tel: +33 49294 4964 Mobile: +33 63334 8269 Fax: +33 49365 4716 o--------------------------------o
-- HIDEMOTO NAKADA, GTRC, AIST -- gridrpc-wg mailing list gridrpc-wg@ogf.org http://www.ogf.org/mailman/listinfo/gridrpc-wg
-- HIDEMOTO NAKADA, GTRC, AIST
participants (2)
-
Craig Lee -
Hidemoto Nakada