Notes taken by Spencer Dawkins (spencer@mcsr-labs.org) . I've post edited them slightly to be less critical of WS-A, since now the spec is final, its just my implementation that doesnt work. On the subject of interop, even though we're testing across the Alpine and Axis soap stacks, I'm actually the author of the fault handling logic and servlets of both systems. There could be a common error in both. Alternatively, my misunderstanding of the specs have set the defacto standard for java soap stacks. We covered both CDL and deploy api tests. There was a very nice demo by Satish, and I got to show my localhost deploying tests against the public endpoints and collecting the results. ----------- CDDLM is "Configuration Description, Deployment, and Lifecycle Management". This is the first specification on system administration issues in a Grid. The chairs for this session weren't available, so it was chaired by Steve Loughran of HP Labs in Bristol, UK. "Any applications without adequate system tests doesn't exist" - nice slide. All CDDLM applications "exist", by this definition. "Interoperation" means that we work together for a set of agreed-upon test cases. HP's "high performance datacenter" is an old laptop no one wants, hosted at home because HP's security types became violent when hearing the description of the application. It's nice that they are "behind a firewall". HP has a test set of components which are also deployable. This means we can deploy to other machines and test our machines, etc. Test cases test all mandatory (non-optional) parts of the spec. Different groups had different ideas about what was optional, so had to split up the test cases. Session included 30 tests run against NEC and HP test boxes. WSA is still troublesome. A lot of interoperability problems happen at this layer (four schema versions have been developed, for example). NEC and UFCG both interoperate successfully. Problems we have are at the top layers. HP broke own SOAP stack. some bits work, some bits don't. Publish all SOAP logs on the net as a public service - recommend this because it lets the caller see what the far-end saw. Would be nice if client could pull these logs back in. Tests can put excessive loads on some systems (should have thought more about this in our tests). Don't have consistent error subcodes (would be nice to know if test that was defined to fail, failed for the right reason). "That doesn't work" isn't quite enough. Patchy interoperation isn't just result of incomplete implementations; it's also differences in spec interpretation. This will be true for a while. We need a better X[HT]ML test result format. Really like CDL language test framework that's being used ("best ever"). Have lots of valid and invalid documents defined. Also have manifest files that drive execution. Tests are available on Sourceforge. Experience is that good test cases help specification designers talk about ambiguity, because everyone's test is failing. CDL tests are passing, other tests are failing (at this time). Have "near-100-percent" test case coverage. If everyone gets the same results, we should have interoperability. How do you know you're covering everything that needs to be covered? Really can't. A lot more invalid things than valid things - probably infinitely more. This comes down to the (in)competence of the developers. When I write CDL and check it in, everyone else has to pass my test, too, and that's nice, but we don't know we have 100 percent coverage. When starting WS-Addressing work, there was no test coverage at all. Things are better now. Former GGF was not test-centric, and OASIS is even less so. Need this for agility in standards bodies. Now we're ahead of OASIS and W3C. Not sure we could ever define a complete set of fault codes, or that we should try. We would need all possible fault codes in the specs, and that's problematic. WSR-F does say that everything should throw a specific fault, but the SOAP stack itself throws faults - can't do what the spec says. If I can deploy anything, why not deploy an unreliable proxy server so we can have network failures? But we can't say this in our own specs. This is the difference between specification and implementation. What we've done is probably as state of the art as testing specifications have gone. Test results would include real-time and post-run display, stack traces, logs from different machines, etc. One decision is about machine-readable versus human-readable. Using a Swing GUI that provides more than success/failure - partial success, test in progress, etc. Would like file format that everyone can read, but we don't have this yet. With virtualization, some machines that you ran your tests on may not exist again the next morning - hard to duplicate tests because virtual host may be hosted somewhere different on a re-run. Also looking at NEC implementation - using a script to verify results of tests. Using a Java component implementation, showing an API front end to the implementation. Remaining work on interoperability - only common test deployments. Not particularly hard to resynchronize on different versions of the same specification - file formats, etc. Now that we have CDL and components working, we can focus on component logs. As far as CDL language goes we are complete barring new test cases. For deploy API, think we're complete but may identify more work that must be done as we get experience. As we define more complex components, this gets blurry because it's tied to the component model. We went through most of the NEC tests as well as the HP tests.