[GRIDRPC-WG] Comments pertinent to GridRPC from ETSI

[Hidemoto Nakada]

Craig,

Thank you for the reply.
We will fix our document so that it will
use 'test purpose' and 'test case'.

2006/12/20, Craig Lee <craig at rushg.aero.org>:
>
> Hidemoto,
>
> The comments from ETSI concerning GridRPC and ByteIO do emphasize
> the need for OGF to have common terminology regarding interoperability,
> interoperability testing, versus conformance testing.  There should also
> be clear distinction between _what_ is being tested (test purpose) and
> _how_ it is being tested (test description).  The GFSG should take up
> this discussion if OGF is to collaborate with other organizations, such as
> ETSI, and also promote grid interoperability in a serious way.
>
> In the meantime, you could make these distinctions clear in the GridRPC Interop
> document with appropriate terminology, which the GFSG and OGF may
> eventually adopt.
>
> -Craig
>
> At 09:59 PM 12/18/2006, Hidemoto Nakada wrote:
> >Craig,
> >
> >Thank you for forwarding the comments.
> >While the comments are quite useful, some of them
> >look like not only for us  but for whole OGF, e.g.
> >general terminology should be defined by OGF.
> >
> >How should we respond the comments?
> >Any idea?
> >
> >-hidemoto
> >
> >2006/12/16, Craig Lee <craig at rushg.aero.org>:
> > >
> > > Hidemoto, Yoshio, et al.,
> > >
> > > Below are some comments on the Interoperability document from ETSI.
> > > These comments seem to be quite useful and we should respond since
> > > ETSI is quite interested in collaborating w/ OGF to establish and promote
> > > grid interoperability!
> > >
> > > --Craig
> > >
> > >
> > > At 07:49 AM 12/15/2006, Stephen M Pickles wrote:
> > > >Dear Craig,
> > > >
> > > >I invited ETSI to comment on the GridRPC and ByteIO
> > > >interoperability documents currently in public comment.
> > > >Stephan Shultz of ETSI has now done so - see attached.
> > > >
> > > >Some of these comments are pertinent to GridRPC
> > > >(and some are more general).
> > > >
> > > >Could you pass them on to authors of the document
> > > >"Interoperability Testing for The GridRPC API Specification"?
> > > >
> > > >Best regards,
> > > >
> > > >Stephen
> > > >
> > > >==================== Stephen M. Pickles ====================
> > > >
> > > >Technical Director, National Grid Service
> > > >Manchester Computing
> > > >Room G49.1, Kilburn Building
> > > >The University of Manchester           tel: +44 161 275 5974
> > > >Oxford Road                            fax: +44 161 275 6800
> > > >Manchester M13 9PL          stephen.pickles at manchester.ac.uk
> > > >From: "Stephan Schulz" <Stephan.Schulz at etsi.org>
> > > >To: "GRID at LIST.ETSI.ORG" <GRID at LIST.ETSI.ORG>
> > > >CC: "Anthony Wiles" <Anthony.Wiles at etsi.org>
> > > >Subject: PTCC Review of OGF GRID Test Specs
> > > >Date: Thu, 14 Dec 2006 14:30:43 +0000
> > > >X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0
> > > >MIME-Version: 1.0
> > > >Thread-Topic: PTCC Review of OGF GRID Test Specs
> > > >Thread-Index: AccNVxOMLDrBcAlfTCOF3zsydsdaawSMv7HAAABNsFA=
> > > >Content-Type: text/plain; charset=us-ascii
> > > >
> > > >Hi all,
> > > >
> > > >as promised in the last TC GRID Meeting a PTCC review of the OGF test
> > > >specifications GridRPC and Byte IO.
> > > >I would appreciate it if someone from OGF, e.g., Steve Pickles, could
> > > >forward this to the relevant people in OGF.
> > > >Thanks
> > > >
> > > >Firstly I see a different use of terminology. Both documents talk about
> > > >"test cases" but understand it each differently
> > > >- and also differently from PTCC point of view.
> > > >GridRPC is giving high level descriptions essentially of what to test.
> > > >PTCC
> > > >calls that test purposes. What GridRPC calls "test code" comes closer to
> > > >our idea of a
> > > >test case. Something which is executable .. a test script if you will.
> > > >The ByteIO specification is much more detailed than a test purpose,
> > > >e.g., it gives detailed message
> > > >contents, arrival of the verdict is convert in great detail, however as
> > > >such the document
> > > >is not executable. In our methodology we call this form of specification
> > > >test descriptions
> > > >(which is an optional intermediate step between test purpose and test
> > > >case implementation).
> > > >In ETSI 3GPP test specifications this intermediate step has also be
> > > >used.
> > > > >From PTCC point of view it would be nice to have a consistent use of
> > > >terminology in specification.
> > > >Of course it would be nice to follow ETSI methodology but at the end
> > > >that is of course up to you.
> > > >In the same way we feel it is beneficial to separate WHAT is to be
> > > >tested (=purpose) and HOW to
> > > >test (= description). In the Byte IO case it these two issues seem to be
> > > >mixed and they should be
> > > >untangled. Clear separation of WHAT and HOW[StS]   gives you a better
> > > >idea, e.g., to get a clear
> > > >idea on the passing criteria and about your actual test coverage of your
> > > >tests . Another benefit of
> > > >test purposes is that they can be understood by a much wider audience as
> > > >test description since
> > > >they do not require deep knowledge, e.g., of  protocol syntax etc.
> > > >
> > > >Both documents position themselves as being a basis for interoperability
> > > >testing. From PTCC point of
> > > >view both are really specifying conformance tests. I am not sure why the
> > > >term conformance is avoided.
> > > >Or maybe there is just a mix-up of words here which is (achievement of)
> > > >interoperability versus
> > > >interoperability testing.
> > > >GridRPC states for example that "As such, the requirement is only to
> > > >test client interoperability: that
> > > >multiple implementations of a specification provide consistent results
> > > >to a suite of tests". In my opinion
> > > >this just falls short of saying that these implementations conform to
> > > >the specification which has been
> > > >selected to be the basis of these so called interoperability tests,
> > > >i.e., GFD-R.52.
> > > >ByteIO test case descriptions specify validation criteria at the level
> > > >of bytes, i.e., WSDL messages.
> > > >In both specs it seems to me that the system under test consists of one
> > > >entity, i.e., the server, being
> > > >tested in isolation. I could be wrong.
> > > >In ETSI (formal) interoperability testing is defined as operating
> > > >different implementations of different entities
> > > >against each other, e.g., a NIFT client implementation against a
> > > >GridSolve server. Therefore an interoperability
> > > >test would describe how to stimulate and observe client as well as
> > > >server implementations not the interface
> > > >betweeen them.
> > > >
> > > >The two documents are specifying tests at different levels - GridRPC
> > > >more at what I would consider software
> > > >module testing and ByteIO more as protocol/Unit testing. From our point
> > > >of view that is fine. However I wonder
> > > >in the case of GridRPC why this specification is limited to C
> > > >implementations only. It seems there is a lot of
> > > >Grid middle ware based on Java out there. Should this test specification
> > > >not be more implementation agnostic?
> > > >On the ByteIO side I wonder why test purposes have to be bound so
> > > >specifically to one specific scenario or
> > > >initial configuration? Again it should be thought about to adopt true
> > > >test purposes which are more abstract and
> > > >leave the details to the test description and implementation. One
> > > >approach could be to change from checking
> > > >rigidly every single message parameter in a single test to focusing on
> > > >the information which are needed to fulfil
> > > >some function. SO for example to say instead of "check that size
> > > >parameter is 2 and ...". To state that "ensure
> > > >that the request for size returns a valid value. This test purpose could
> > > >then have multiple test implementations
> > > >testing the value 2 or boundary conditions, etc.
> > > >
> > > >Possibilities for further improvement:
> > > >As said before a clearer separation of WHAT is to be test versus HOW
> > > >specifically in the ByteIO spec - the
> > > >looks fine.
> > > >Both documents are lacking references in the test purpose/description to
> > > >the requirement(s) they pertain to
> > > >in the specification, e.g., document clause or quotation of
> > > >specification text. Such reference are very important
> > > >  to trace back test results to the specification or to ensure the test
> > > >itself is a valid one.
> > > >Another point is clearer identification and separation of mandatory
> > > >versus optional features of the specification
> > > >  / implementation under test. That means entire tests or test purposes
> > > >should be optional not only some of
> > > >multiple validation criteria within the same test purpose. This
> > > >categorization is also  helpful for identifying the
> > > >minimum set of tests that a implementation must pass before being
> > > >compliant or shall we say "considered
> > > >  interoperable". GridRPC has some discussion of optional responses based
> > > >on about observed differences
> > > >with various applications but it does not clarify what the response
> > > >should be, i.e., what the specification actually
> > > >states.
> > > >
> > > >Let me know if you have any questions.
> > > >Thanks,
> > > >stephan
> > > >
> > > >o--------------------------------o
> > > >   Stephan Schulz, Ph.D.
> > > >   ETSI PTCC Technical Officer
> > > >   650 Rue des Lucioles
> > > >   F-06921 Sophia Antipolis Cedex
> > > >   Tel:    +33 49294 4964
> > > >   Mobile: +33 63334 8269
> > > >   Fax:    +33 49365 4716
> > > >o--------------------------------o
> > >
> > >
> >
> >
> >--
> >    HIDEMOTO NAKADA, GTRC, AIST
> >--
> >   gridrpc-wg mailing list
> >   gridrpc-wg at ogf.org
> >   http://www.ogf.org/mailman/listinfo/gridrpc-wg
>
>


-- 
   HIDEMOTO NAKADA, GTRC, AIST