September 2007 - dfdl-wg

Fw: Notes from 2007-09-12 call
by Steve Hanson 31 Oct '07

31 Oct '07

More on expressions, <smh>below</smh> Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 19/09/2007 14:14 ----- Mike Beckerle <beckerle(a)us.ibm.com> 19/09/2007 13:43 To Steve Hanson/UK/IBM@IBMGB cc dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Subject Re: [DFDL-WG] Notes from 2007-09-12 call Comments below in BLUE Steve Hanson <smh(a)uk.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 09/19/2007 06:04 AM To dfdl-wg(a)ogf.org cc Subject Re: [DFDL-WG] Notes from 2007-09-12 call Some thoughts since last week's call: 1) Expression language We've not thought much about how expressions will work on output. It's fine to say something like dfdl:length="..\count+1" when parsing, but what happens on output. I think we should not try to reverse engineer expressions, and rely on the user to set output fields correctly. So, taking my example, on output we would assume count had been set by the user, apply the expression to calculate the intended length of data, then apply padding etc rules as needed. Can we generalise that philosophy across all our uses of expressions? If we can't then perhaps that places a bound on the actual uses of expressions that we permit. Inverting will generally not be possible. Just make the example dfdl:length="{ ../count * ../scale + 1 }" How do you split up the length into count and scale? <smh>Agree</smh> In your example, I would expect the count field to have an outputValueCalc="{ ../x.length() - 1 }" (I'm assuming the field with the length calculation formula is named "x".) <smh>Doesn't outputValueCalc mean that we are deriving the count from the length of the data supplied for x? That forces the user to pad x to the correct value, in order to derive count. Which is not how we want things to work. We want count to define the length of x, so the DFDL serialiser can pad x according to other DFDL properties. Maybe I'm missing something about input/outputValueCalc?</smh> In general, when something uses something else in it's calculation (length or just the value - inputValueCalc), then the inverse is outputValueCalc on the contributing parts. 2) dfdl:length for sequences We have three cases here: a) Empty sequence - we agreed to disallow this b) Non-empty normal sequence - what does the length mean here? It means the box is potentially larger than the contents. If it isn't at least as big it's an error. If these lengths are data dependent it could be a processing error. Otherwise a schema definition error. Draft 025 discusses this in the part on sequences with length. We also could disallow this case if we want for now, knowing we could add it back if we want. One can always convert this case into the one below by wrapping the sequence's child elements in an array element, with array occurrences determined by "fillAvailableSpace" policy. If we allow this at all, I think this should be the way we explain the semantics of it. (Though with the inserted array the paths would all change which is undesirable. - so we would say it works like this, but without the paths being changed...) c) Non-empty sequence used as box array - the motivating scenario I think we should also disallow b). If we are disallowing a) on the grounds of not using sequence with a length to model opaque data then we should also disallow b). 3) Handling of comments in data Mentioning this as it was discussed on the call but not minuted. Example: A record is allowed to be followed by multiple free form text lines where the first such line contained //ADDINFOSTART, and the last such line contained //ADDINFOEND. This was put forward as a use case for regular expressions but it was noted that an explicit dfdl:commentScheme based on, or maybe an extension of, the existing dfdl:escapeSchema property would be a more natural solution for users Whooops. Forgot this. We need a concrete proposal here. Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Mike Beckerle <beckerle(a)us.ibm.com> Sent by: dfdl-wg-bounces(a)ogf.org 12/09/2007 21:20 To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] Notes from 2007-09-12 call Mike Beckerle, Alan Powell, Steve Hanson, Suman Kalia attended. Discussed these questions from Alan about expression language. 1. Accessing hidden values - it seems inconsistent to allow access to hidden values when xpath is used within the DFDL domain but not when used outside. 2. Where xpath is allowed in the schema - It is currently allowed in an arbitrary set of properties (initiator, terminator, separator, occurseparator, null, etc ). Why not allow it everywhere? Wr.t. (1) we decided this is correct. path expressions for dfdl properties can see hidden elements, path expressions in other places (e.g., schematron assertions) cannot. Wr.t (2) we decided that expressions should be allowed in principle everywhere for the value of any property; however, there may be exceptions for certain properties. Particularly, it seems some enum-valued properties are unlikely to ever want to be expressions. Example: dfdl:representation. However, it was also pointed out that once we put selectors back into the language you can interleave multiple formats in the same schema, and for any enumerated property you could just have one selector-chosen format for each possible value of the enumerated property. The reason we don't want a blanket statement that you can have expressions anywhere you need a property value is that there is some potential that this makes implementations unnecessarily complex due to the excess flexibility. Digression: (This added by MikeB - was not part of the call today.) Consider dfdl:byteOrder=" if (../../x = 'B') then 'bigEndian' else if (../../x='L') then 'littleEndian' else 'I don't know' }" DFDL implementations must be prepared to cope with recieving "I don't know" as the proposed value for the byteOrder. This is a schema definition error, but it is happening at run time so becomes a processing error. The only way to rule this out is to treat enumerated property values not as strings but as an enum type and force the expressions that compute them to return an enum type, not a string. This is a kind of type inference I had hoped implementations would not need. Selectors have the advantage of being statically verifiable. i.e., each selected format is known to use a value of the enum that is valid or a diagnostic could be issued by the DFDL processor. If we allow an arbitrary expression to return the value of an enumerated property then it presumably could also return a nonsense value: We discussed proposals circulated by MikeB: Here's an update to the first one. We decided sequences shouldn't be another way to carry opaque data. Easy and conservative way to fix this is to require the length of an empty sequence to be zero. Second proposal to eliminate hexBinary and base64Binary was discussed lightly. It was suggested that one could have both, and that would make it easy to explain what the hexBinary type is, because it is a shorthand for a string with encoding="hex", and similarly for base64Binary. We did not resolve this issue on the call. Finally, we discussed regular expression features for DFDL. There does appear to be need for regexp features to support parsing data which is delimited by changing data content. E.g. consider "12345Mike Beckerle". and a two-element sequence. One is a number which continues until the first non-digit character. The other is a string which begins with a non-digit character. Regexp length appears to be a good way to handle this kind of thing. Alan Powell has the action item to talk with the IBM internal TX product group. They have a speculative parser and so have fewer regular-expression features in their language. We want to understand how they deal with the header, body[], trailer use case. This case is where the data is lines of text, the header is the first line, the trailer is the last line, the body records are everything in between and there's no content that can be used to distinguish the record types. This is handled in some format-description systems with regexp features. In TX this is handled by speculative parsing and we want to understand how this comes out and if it is preferable to adding regexp features. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

4 4

Fw: OGSA Data Architecture Now in Public Comment
by Mike Beckerle 27 Sep '07

27 Sep '07

I believe it would be valuable for someone to review this and present to the group. This is a 30 day comment period. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046 ----- Forwarded by Mike Beckerle/Worcester/IBM on 09/27/2007 11:09 AM ----- "Dave Berry" <daveb(a)nesc.ac.uk> 09/27/2007 07:08 AM To "Erwin Laure" <Erwin.Laure(a)cern.ch>, "Mark Morgan" <mmm2a(a)virginia.edu>, "Neil Chue Hong" <neilc(a)epcc.ed.ac.uk>, "Pearson Dave" <dave.pearson(a)oracle.com>, <jimmyers(a)ncsa.uiuc.edu>, <M.Westhead(a)epcc.ed.ac.uk>, Mike Beckerle/Worcester/IBM@IBMUS, "Arun swaran Jagatheesan" <arun(a)sdsc.edu>, <o.tatebe(a)aist.go.jp>, "Igor Mandrichenko" <ivm(a)fnal.gov>, <allcock(a)mcs.anl.gov>, <shoshani(a)lbl.gov>, <dieter.gawlick(a)oracle.com>, "Steve Fisher" <s.m.fisher(a)rl.ac.uk>, "Michel Drescher" <Michel.Drescher(a)uk.fujitsu.com>, <madduri(a)mcs.anl.gov>, "Jens Gunner Jensen" <j.jensen(a)rl.ac.uk>, <grimshaw(a)virginia.edu>, "Birger Koblitz" <Birger.Koblitz(a)cern.ch>, <mwan(a)sdsc.edu>, <ASim(a)lbl.gov> cc Allen Luniewski/Almaden/IBM@IBMUS, "Dave Berry" <daveb(a)nesc.ac.uk> Subject OGSA Data Architecture Now in Public Comment Dear Data Area chairs, The OGSA Data Architecture architecture document has been placed into its 30 day public comment period. It refers to several other OGF data standards, most strongly WS-DAI, ByteIO and SRM but also OGSA-DMI, DFDL and RNS. Please take a look at it and submit your comments, even if it is little more than "I read it". You can make public comments at http://forge.gridforum.org/sf/discussion/do/listTopics/projects.ggf-editor/… . You have to be logged in to be able to leave comments. Thanks. Dave Berry Deputy Director, Research & E-infrastructure Development National e-Science Centre 15 South College Street, Edinburgh, EH8 9AA +44 131 651 4039 www.nesc.ac.uk

1 0

DFDL working group call 26th Sept 2007
by Steve Hanson 26 Sep '07

26 Sep '07

Mike can't make the call today, and it is also awkward for me. Suggest we cancel unless anyone has something to discuss? Regards, Steve Steve Hanson WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

2 1

DFDL nulls/optionals/defaults - next draft
by Mike Beckerle 21 Sep '07

21 Sep '07

Change tracking is on for those who saw v3, but for those of you who didn't you probably want to view this without the change highlighting to make reading easier. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

DFDL Proposal for how XSD facets, min/maxOccurs, default, and fixed are used.
by Mike Beckerle 21 Sep '07

21 Sep '07

This came up as a subtopic of the defaults/optionals/nulls, etc. work. I have mistakenly been referring to xs:fixed as a facet. it is not, nor are xs:default, xs:minOccurs, xs:maxOccurs. Of particular interest is the semantics of the 'fixed' attribute, which is last in the doc. I picked one of 3 possible meanings, but described the other two also. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

Notes from 2007-09-12 call
by Mike Beckerle 19 Sep '07

19 Sep '07

Mike Beckerle, Alan Powell, Steve Hanson, Suman Kalia attended. Discussed these questions from Alan about expression language. 1. Accessing hidden values - it seems inconsistent to allow access to hidden values when xpath is used within the DFDL domain but not when used outside. 2. Where xpath is allowed in the schema - It is currently allowed in an arbitrary set of properties (initiator, terminator, separator, occurseparator, null, etc ). Why not allow it everywhere? Wr.t. (1) we decided this is correct. path expressions for dfdl properties can see hidden elements, path expressions in other places (e.g., schematron assertions) cannot. Wr.t (2) we decided that expressions should be allowed in principle everywhere for the value of any property; however, there may be exceptions for certain properties. Particularly, it seems some enum-valued properties are unlikely to ever want to be expressions. Example: dfdl:representation. However, it was also pointed out that once we put selectors back into the language you can interleave multiple formats in the same schema, and for any enumerated property you could just have one selector-chosen format for each possible value of the enumerated property. The reason we don't want a blanket statement that you can have expressions anywhere you need a property value is that there is some potential that this makes implementations unnecessarily complex due to the excess flexibility. Digression: (This added by MikeB - was not part of the call today.) Consider dfdl:byteOrder=" if (../../x = 'B') then 'bigEndian' else if (../../x='L') then 'littleEndian' else 'I don't know' }" DFDL implementations must be prepared to cope with recieving "I don't know" as the proposed value for the byteOrder. This is a schema definition error, but it is happening at run time so becomes a processing error. The only way to rule this out is to treat enumerated property values not as strings but as an enum type and force the expressions that compute them to return an enum type, not a string. This is a kind of type inference I had hoped implementations would not need. Selectors have the advantage of being statically verifiable. i.e., each selected format is known to use a value of the enum that is valid or a diagnostic could be issued by the DFDL processor. If we allow an arbitrary expression to return the value of an enumerated property then it presumably could also return a nonsense value: We discussed proposals circulated by MikeB: Here's an update to the first one. We decided sequences shouldn't be another way to carry opaque data. Easy and conservative way to fix this is to require the length of an empty sequence to be zero. Second proposal to eliminate hexBinary and base64Binary was discussed lightly. It was suggested that one could have both, and that would make it easy to explain what the hexBinary type is, because it is a shorthand for a string with encoding="hex", and similarly for base64Binary. We did not resolve this issue on the call. Finally, we discussed regular expression features for DFDL. There does appear to be need for regexp features to support parsing data which is delimited by changing data content. E.g. consider "12345Mike Beckerle". and a two-element sequence. One is a number which continues until the first non-digit character. The other is a string which begins with a non-digit character. Regexp length appears to be a good way to handle this kind of thing. Alan Powell has the action item to talk with the IBM internal TX product group. They have a speculative parser and so have fewer regular-expression features in their language. We want to understand how they deal with the header, body[], trailer use case. This case is where the data is lines of text, the header is the first line, the trailer is the last line, the body records are everything in between and there's no content that can be used to distinguish the record types. This is handled in some format-description systems with regexp features. In TX this is handled by speculative parsing and we want to understand how this comes out and if it is preferable to adding regexp features. Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 2

DFDL core draft 025 released
by Mike Beckerle 14 Sep '07

14 Sep '07

I've released internal draft 025 which you can find here: https://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/… I didn't get everything into it that I'd hoped. But I promised a number of people an improved draft by COB today, and this one is a big improvement over 019. Here's things I wanted to get in but couldn't: - we've settled the opaque data discussion. Someone should figure out where in 025 this treatment belongs and how the text of the proposal (circulated in email earlier this week) should be redone inside the core spec. - the discussion of variables is now in back matter which is gradually being phased out. (too formal/mathematical suitable for implementors only) But somebody needs to read the discussion of variables there and create a more user-centric discussion of the same. - null/default/optional - this is the focus of my work right now. Should have a proposal out next week. Next week on our wednesday call I want to go over the new organizational structure of this draft. Note: change tracking is turned ON in this draft. Please DO NOT send me comments on this draft in any form except using MS-WORD COMMENTS in the document. I really really don't want to piece through emails that say "on page xxx section yyy I think ....". It is much easier if people use MS-WORD Comments feature and then we can easily just merge stuff. Thanks Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

1 0

Conference
by Simon Parker 12 Sep '07

12 Sep '07

Sorry, an urgent assignment has come up and I can't attend today's conference. See you next week, Simon Simon Parker Software Consultant PolarLake 80 Harcourt Street Dublin 2, Ireland Phone +353 (1) 449-1075 Fax +353 (1) 449-1011 Web http://www.polarlake.com/ The information transmitted in this email is intended for the addressee only and may contain confidential and/or privileged material. Any review, retransmission, dissemination, reliance upon or other use of this information by persons or entities other than the addressee is prohibited. If you think, for any reason, that this message may have been addressed to you in error, we would ask you to notify the sender immediately by return email and delete the material. PolarLake Limited | Registered in Dublin, Ireland | Number 357324 | Registered office as above.

1 0

WG call 2007-09-12
by Mike Beckerle 12 Sep '07

12 Sep '07

I have two things that we can discuss: The above is v3. It differs in that sequences with length were added. There's an issue in comments of the doc we can discuss. Also: a new related proposal below - I can overview this, but we don't really need people to weigh in with opinions/decisions until more of a chance to think it through. ...mikeb Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 1

v4 on length/occurs simplification proposal
by Mike Beckerle 06 Sep '07

06 Sep '07

How's lengthKind="maximum" sound for the "endOfData" fitting into a box thing? Other possibilities in this doc: Mike Beckerle STSM, Architect, Scalable Computing IBM Software Group Information Platform and Solutions Westborough, MA 01581 direct: voice and FAX 508-599-7148 assistant: Pam Riordan priordan(a)us.ibm.com 508-599-7046

2 2