[cddlm] Testing Fault Values Thrown on Unsuccessful CDL Language Tests

[Steve Loughran]

As promised, my thoughts on how to declare what kind of failure we 
expect on a test, in a way that does not mandate that implementations 
actually include the same error message or fault type.

The reason for this is that some of the existing tests I have cannot yet 
move to using the new CDL test file format, because I not only test that 
parsing the CDL doc fails, I verify that the exception contains an error 
string that I know about.

     public void testDirectLoop() throws Exception {
         assertInvalidCDL(EXTENDS_DIRECT_LOOP,
                 ExtendsContext.ERROR_RECURSING);
     }

     public void testIndirectLoop() throws Exception {
         assertInvalidCDL(EXTENDS_INDIRECT_LOOP,
                 ExtendsContext.ERROR_RECURSING);
     }

     public void testBadReference() throws Exception {
         assertInvalidCDL(EXTENDS_BAD_REFERENCE,
                 ExtendsResolver.ERROR_UNKNOWN_TEMPLATE);
     }

     public void testUnknownNamespace() throws Exception {
         //This is a Xom error @ parse time.
         assertInvalidCDL(EXTENDS_UNKNOWN_NAMESPACE,
                 "UndeclaredPrefix");
     }

While we can't control what errors get thrown from 3rd party libraries, 
we can hint as to the fault code expected from an exception, leaving 
each implementation free to act on the hint.

comments?


Testing Fault Values Thrown on Unsuccessful CDL Language Tests
===============================================================

When testing to failure, the actual fault type matters.

That is, if a successful test should normally throw a "no prototype for 
extends"
exception, and suddenly it starts failing with a "undefined element 
cdl:cdl", then the
test should fail.

The current iteration of the CDL test language allows one to declare 
that a test
should fail, but does not specify what the failure message is. This is 
an inevitable
consequence of the specifications not defining the error messages. We 
may specify
that things should fail, but not how.

Proposed changes
================

fault code in test documents
============================

-the <ct:failed/> message is extended to support a SOAP fault code; a 
qname of
namespace and string.

  <ct:failed>
    <ct:code 
namespace="http://http://gridforge.org/cddlm/serviceAPI/faults/2004/10/11/"
     value="extends-loop" />
  </ct:failed>

If the namespace attibute is absent, we default to the one defined above;
the CDDLM faults ns.



Defined fault codes for common failure modes
============================================

These can be added to constants.xml, though we may want to pull this into a
separate document.

It has always been the plan to produce an informative documents of
faults that the deployment API endpoints should produce. Adding parse time
fault codes which implementations may implement could benefit users, as
well as the test suite.


Test routine asks implemententations to verify the fault
========================================================


After catching any Throwable (other than ThreadDeath, which must be 
rethrown),
the exception is handed to the CdlProcessor, which has a new interface 
method

boolean isExpectedFault(String test,QName faultCode,Throwable thrown);

-Parameter "test" is the name of the test
-Parameter "faultCode" is the constructed QName of the fault, built from 
the test document
-Parameter "thrown" is the received exception.

It is up to the individual exception to verify if the exception thrown 
matches that
expected for that test or the faultCode.

Implementation options are

1. The current behaviour could be implemented by always returning true.

boolean isExpectedFault(String test,QName faultCode,Throwable thrown) {
   return true;
}


2. having every routine in the CDL parser throw an exception (such as an 
AxisFault) that
  has a fault code.


boolean isExpectedFault(String test,QName faultCode,Throwable thrown) {
   if(thrown instanceof AxisFault) {
     AxisFault fault=(AxisFault )thrown;
     return faultCode.equals(fault.getFaultCode);
   }
   //anything else
   return true;
}


3. Having a lookup table that maps from faultCode to throwable string

boolean isExpectedFault(String test,QName faultCode,Throwable thrown) {
   String search=lookupErrorText(faultCode);
   return thrown.getMessage().indexOf(search)>=0;
}

4. Having a lookup table that knows what exception to get for a 
particular test

boolean isExpectedFault(String test,QName faultCode,Throwable thrown) {
   String search=lookupErrorText(test);
   return thrown.getMessage().indexOf(search)>=0;
}

5. Having every fault implement a FaultCodeComparison interface

interface FaultCodeComparison {
   boolean isFault(QName faultCode);
}


and casting every fault to the check:

boolean isExpectedFault(String test,QName faultCode,Throwable thrown) {
   if(thrown implements FaultCodeComparison) {
     FaultCodeComparison fault=(FaultCodeComparison)thrown;
     return fault.isFault(faultCode);
   }
   //anything else
   return true;
}

This is better than (2) as the check logic could be different for different
fault classes, and allows the implementation to look for nested faults.

6. Remembering the fault codes generated by an implementation, and
    verifying the same fault is thrown.
    This requires manual verification of the thrown information, and some
    persistence mechanism. It is also potentially very brittle.

7. A combination of the above. Such as looking for a specific string for a
    specific test, then checking for expected faults via a 
FaultCodeComparison
    interface, then falling back to a string search from fault code value if
    the interface was missing.



The point is: it is up to the implementor of the CDL runtime to choose how
to process faults; implementing the default behaviour is easy; extending
it to support other checks requires more effort, but is optional.

The only area that requires collaboration across implementations is in 
defining
the fault codes for common errors. Again, this does not require 
implementation
inside the parser implementations, as the CdlProcessor implementations 
can use
one of the many proposed mechanisms to map verify that the caught fault was
appropriate for the given test name and fault code.