As promised, my thoughts on how to declare what kind of failure we expect on a test, in a way that does not mandate that implementations actually include the same error message or fault type. The reason for this is that some of the existing tests I have cannot yet move to using the new CDL test file format, because I not only test that parsing the CDL doc fails, I verify that the exception contains an error string that I know about. public void testDirectLoop() throws Exception { assertInvalidCDL(EXTENDS_DIRECT_LOOP, ExtendsContext.ERROR_RECURSING); } public void testIndirectLoop() throws Exception { assertInvalidCDL(EXTENDS_INDIRECT_LOOP, ExtendsContext.ERROR_RECURSING); } public void testBadReference() throws Exception { assertInvalidCDL(EXTENDS_BAD_REFERENCE, ExtendsResolver.ERROR_UNKNOWN_TEMPLATE); } public void testUnknownNamespace() throws Exception { //This is a Xom error @ parse time. assertInvalidCDL(EXTENDS_UNKNOWN_NAMESPACE, "UndeclaredPrefix"); } While we can't control what errors get thrown from 3rd party libraries, we can hint as to the fault code expected from an exception, leaving each implementation free to act on the hint. comments? Testing Fault Values Thrown on Unsuccessful CDL Language Tests =============================================================== When testing to failure, the actual fault type matters. That is, if a successful test should normally throw a "no prototype for extends" exception, and suddenly it starts failing with a "undefined element cdl:cdl", then the test should fail. The current iteration of the CDL test language allows one to declare that a test should fail, but does not specify what the failure message is. This is an inevitable consequence of the specifications not defining the error messages. We may specify that things should fail, but not how. Proposed changes ================ fault code in test documents ============================ -the <ct:failed/> message is extended to support a SOAP fault code; a qname of namespace and string. <ct:failed> <ct:code namespace="http://http://gridforge.org/cddlm/serviceAPI/faults/2004/10/11/" value="extends-loop" /> </ct:failed> If the namespace attibute is absent, we default to the one defined above; the CDDLM faults ns. Defined fault codes for common failure modes ============================================ These can be added to constants.xml, though we may want to pull this into a separate document. It has always been the plan to produce an informative documents of faults that the deployment API endpoints should produce. Adding parse time fault codes which implementations may implement could benefit users, as well as the test suite. Test routine asks implemententations to verify the fault ======================================================== After catching any Throwable (other than ThreadDeath, which must be rethrown), the exception is handed to the CdlProcessor, which has a new interface method boolean isExpectedFault(String test,QName faultCode,Throwable thrown); -Parameter "test" is the name of the test -Parameter "faultCode" is the constructed QName of the fault, built from the test document -Parameter "thrown" is the received exception. It is up to the individual exception to verify if the exception thrown matches that expected for that test or the faultCode. Implementation options are 1. The current behaviour could be implemented by always returning true. boolean isExpectedFault(String test,QName faultCode,Throwable thrown) { return true; } 2. having every routine in the CDL parser throw an exception (such as an AxisFault) that has a fault code. boolean isExpectedFault(String test,QName faultCode,Throwable thrown) { if(thrown instanceof AxisFault) { AxisFault fault=(AxisFault )thrown; return faultCode.equals(fault.getFaultCode); } //anything else return true; } 3. Having a lookup table that maps from faultCode to throwable string boolean isExpectedFault(String test,QName faultCode,Throwable thrown) { String search=lookupErrorText(faultCode); return thrown.getMessage().indexOf(search)>=0; } 4. Having a lookup table that knows what exception to get for a particular test boolean isExpectedFault(String test,QName faultCode,Throwable thrown) { String search=lookupErrorText(test); return thrown.getMessage().indexOf(search)>=0; } 5. Having every fault implement a FaultCodeComparison interface interface FaultCodeComparison { boolean isFault(QName faultCode); } and casting every fault to the check: boolean isExpectedFault(String test,QName faultCode,Throwable thrown) { if(thrown implements FaultCodeComparison) { FaultCodeComparison fault=(FaultCodeComparison)thrown; return fault.isFault(faultCode); } //anything else return true; } This is better than (2) as the check logic could be different for different fault classes, and allows the implementation to look for nested faults. 6. Remembering the fault codes generated by an implementation, and verifying the same fault is thrown. This requires manual verification of the thrown information, and some persistence mechanism. It is also potentially very brittle. 7. A combination of the above. Such as looking for a specific string for a specific test, then checking for expected faults via a FaultCodeComparison interface, then falling back to a string search from fault code value if the interface was missing. The point is: it is up to the implementor of the CDL runtime to choose how to process faults; implementing the default behaviour is easy; extending it to support other checks requires more effort, but is optional. The only area that requires collaboration across implementations is in defining the fault codes for common errors. Again, this does not require implementation inside the parser implementations, as the CdlProcessor implementations can use one of the many proposed mechanisms to map verify that the caught fault was appropriate for the given test name and fault code.