Hidemoto, Yoshio, et al.,
Below are some comments on the Interoperability document from ETSI.
These comments seem to be quite useful and we should respond since
ETSI is quite interested in collaborating w/ OGF to establish and promote
grid interoperability!
--Craig
At 07:49 AM 12/15/2006, Stephen M Pickles wrote:
>Dear Craig,
>
>I invited ETSI to comment on the GridRPC and ByteIO
>interoperability documents currently in public comment.
>Stephan Shultz of ETSI has now done so - see attached.
>
>Some of these comments are pertinent to GridRPC
>(and some are more general).
>
>Could you pass them on to authors of the document
>"Interoperability Testing for The GridRPC API Specification"?
>
>Best regards,
>
>Stephen
>
>==================== Stephen M. Pickles ====================
>
>Technical Director, National Grid Service
>Manchester Computing
>Room G49.1, Kilburn Building
>The University of Manchester tel: +44 161 275 5974
>Oxford Road fax: +44 161 275 6800
>Manchester M13 9PL stephen.pickles(a)manchester.ac.uk
>From: "Stephan Schulz" <Stephan.Schulz(a)etsi.org>
>To: "GRID(a)LIST.ETSI.ORG" <GRID(a)LIST.ETSI.ORG>
>CC: "Anthony Wiles" <Anthony.Wiles(a)etsi.org>
>Subject: PTCC Review of OGF GRID Test Specs
>Date: Thu, 14 Dec 2006 14:30:43 +0000
>X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0
>MIME-Version: 1.0
>Thread-Topic: PTCC Review of OGF GRID Test Specs
>Thread-Index: AccNVxOMLDrBcAlfTCOF3zsydsdaawSMv7HAAABNsFA=
>Content-Type: text/plain; charset=us-ascii
>
>Hi all,
>
>as promised in the last TC GRID Meeting a PTCC review of the OGF test
>specifications GridRPC and Byte IO.
>I would appreciate it if someone from OGF, e.g., Steve Pickles, could
>forward this to the relevant people in OGF.
>Thanks
>
>Firstly I see a different use of terminology. Both documents talk about
>"test cases" but understand it each differently
>- and also differently from PTCC point of view.
>GridRPC is giving high level descriptions essentially of what to test.
>PTCC
>calls that test purposes. What GridRPC calls "test code" comes closer to
>our idea of a
>test case. Something which is executable .. a test script if you will.
>The ByteIO specification is much more detailed than a test purpose,
>e.g., it gives detailed message
>contents, arrival of the verdict is convert in great detail, however as
>such the document
>is not executable. In our methodology we call this form of specification
>test descriptions
>(which is an optional intermediate step between test purpose and test
>case implementation).
>In ETSI 3GPP test specifications this intermediate step has also be
>used.
> >From PTCC point of view it would be nice to have a consistent use of
>terminology in specification.
>Of course it would be nice to follow ETSI methodology but at the end
>that is of course up to you.
>In the same way we feel it is beneficial to separate WHAT is to be
>tested (=purpose) and HOW to
>test (= description). In the Byte IO case it these two issues seem to be
>mixed and they should be
>untangled. Clear separation of WHAT and HOW[StS] gives you a better
>idea, e.g., to get a clear
>idea on the passing criteria and about your actual test coverage of your
>tests . Another benefit of
>test purposes is that they can be understood by a much wider audience as
>test description since
>they do not require deep knowledge, e.g., of protocol syntax etc.
>
>Both documents position themselves as being a basis for interoperability
>testing. From PTCC point of
>view both are really specifying conformance tests. I am not sure why the
>term conformance is avoided.
>Or maybe there is just a mix-up of words here which is (achievement of)
>interoperability versus
>interoperability testing.
>GridRPC states for example that "As such, the requirement is only to
>test client interoperability: that
>multiple implementations of a specification provide consistent results
>to a suite of tests". In my opinion
>this just falls short of saying that these implementations conform to
>the specification which has been
>selected to be the basis of these so called interoperability tests,
>i.e., GFD-R.52.
>ByteIO test case descriptions specify validation criteria at the level
>of bytes, i.e., WSDL messages.
>In both specs it seems to me that the system under test consists of one
>entity, i.e., the server, being
>tested in isolation. I could be wrong.
>In ETSI (formal) interoperability testing is defined as operating
>different implementations of different entities
>against each other, e.g., a NIFT client implementation against a
>GridSolve server. Therefore an interoperability
>test would describe how to stimulate and observe client as well as
>server implementations not the interface
>betweeen them.
>
>The two documents are specifying tests at different levels - GridRPC
>more at what I would consider software
>module testing and ByteIO more as protocol/Unit testing. From our point
>of view that is fine. However I wonder
>in the case of GridRPC why this specification is limited to C
>implementations only. It seems there is a lot of
>Grid middle ware based on Java out there. Should this test specification
>not be more implementation agnostic?
>On the ByteIO side I wonder why test purposes have to be bound so
>specifically to one specific scenario or
>initial configuration? Again it should be thought about to adopt true
>test purposes which are more abstract and
>leave the details to the test description and implementation. One
>approach could be to change from checking
>rigidly every single message parameter in a single test to focusing on
>the information which are needed to fulfil
>some function. SO for example to say instead of "check that size
>parameter is 2 and ...". To state that "ensure
>that the request for size returns a valid value. This test purpose could
>then have multiple test implementations
>testing the value 2 or boundary conditions, etc.
>
>Possibilities for further improvement:
>As said before a clearer separation of WHAT is to be test versus HOW
>specifically in the ByteIO spec - the
>looks fine.
>Both documents are lacking references in the test purpose/description to
>the requirement(s) they pertain to
>in the specification, e.g., document clause or quotation of
>specification text. Such reference are very important
> to trace back test results to the specification or to ensure the test
>itself is a valid one.
>Another point is clearer identification and separation of mandatory
>versus optional features of the specification
> / implementation under test. That means entire tests or test purposes
>should be optional not only some of
>multiple validation criteria within the same test purpose. This
>categorization is also helpful for identifying the
>minimum set of tests that a implementation must pass before being
>compliant or shall we say "considered
> interoperable". GridRPC has some discussion of optional responses based
>on about observed differences
>with various applications but it does not clarify what the response
>should be, i.e., what the specification actually
>states.
>
>Let me know if you have any questions.
>Thanks,
>stephan
>
>o--------------------------------o
> Stephan Schulz, Ph.D.
> ETSI PTCC Technical Officer
> 650 Rue des Lucioles
> F-06921 Sophia Antipolis Cedex
> Tel: +33 49294 4964
> Mobile: +33 63334 8269
> Fax: +33 49365 4716
>o--------------------------------o
