February 2009 - dfdl-wg - lists.ogf.org

Reducing the number of DFDL properties.
by Alan Powell 03 Mar '09

03 Mar '09

Mike A number of people at IBM have become concerned at the number of properties in DFDL and have identified a number of 'usability' properties that could be dropped. They feel that we should be simplifying the properties wherever possible and not introducing multiple ways of doing the same function without very good reason. The following are offered for consideration. lengthKind='nullterminated' This is just shorthand for lengthKind=delimited and terminator='%Null'. It was felt this this is not even the most common terminator so why have a special case? trimKind It is felt that there aren't any cases when you would want to pad but not trim and vice versa so make padKind control both. typeSubstitution. Is this needed in DFDL v1? Can you consider these before the call next week Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

6 12

Fw: Reducing the number of DFDL properties.
by Suman Kalia 27 Feb '09

27 Feb '09

nullTerminated is a very familiar paradigm which the users understand. We need to keep usability in mind; keep simple tasks simple. Yes it is redundant and you can do the same with 2 properties ( lengthKind=delimited and terminator=NULL) but it is well worth to keep it.. Same argument can be made for trimKind.. I am ok if we drop typeSubstitution although it is a convenient way to specify whole bunch properties grouped together for a type and reuse it. Suman Kalia IBM Toronto Lab WMB Toolkit Architect and Development Lead WebSphere Business Integration Application Connectivity Tools http://www.ibm.com/developerworks/websphere/zones/businessintegration/wmb.h… Tel : 905-413-3923 T/L 969-3923 Fax : 905-413-4850 T/L 969-4850 Internet ID : kalia(a)ca.ibm.com ----- Forwarded by Suman Kalia/Toronto/IBM on 02/26/2009 01:52 PM ----- From: Alan Powell <alan_powell(a)uk.ibm.com> To: mbeckerle.dfdl(a)gmail.com Cc: dfdl-wg(a)ogf.org Date: 02/26/2009 12:27 PM Subject: [DFDL-WG] Reducing the number of DFDL properties. Mike A number of people at IBM have become concerned at the number of properties in DFDL and have identified a number of 'usability' properties that could be dropped. They feel that we should be simplifying the properties wherever possible and not introducing multiple ways of doing the same function without very good reason. The following are offered for consideration. 1. lengthKind='nullterminated' This is just shorthand for lengthKind=delimited and terminator='%Null'. It was felt this this is not even the most common terminator so why have a special case? 2. trimKind It is felt that there aren't any cases when you would want to pad but not trim and vice versa so make padKind control both. 3. typeSubstitution. Is this needed in DFDL v1? Can you consider these before the call next week Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg

2 1

DFDL: Minutes from OGF WG call, February-25-2009
by Alan Powell 26 Feb '09

26 Feb '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, February-25-2009, Attendees Alan Powell (IBM) Steve Hanson (IBM) Dave Glick (drac) Tim Kimber (IBM) Apologies Mike Beckerle(Oco) 1. Discussion of D Glick Spec 033 review Dave has done a thorough review of DFDL Spec v033 and circulated an email with the major point which was discussed on this call. Major areas discussed - Balance of formal spec vs Primer. The spec should be make a clearer distiction between normative and non-normative information. - Parts of the document sound like implementation details which shouldn't be included - DG suggested clearer document structure but wasn't sure on the best oreder to list representation properties. Possible suggestion is alphabetic list of properties with a number of indexes giving different views. - Should be better with RFC2119 terminology SHOULD, MUST, etc - Discussed whether dfdl infoset should be should be subset/compatible with XML infoset or XDM. General agreement that it should be as compatible as possible with XDM. DG to investigate hwo close they can be made. - Types of DFDL properties should be more clearly specified. - DG wondered whether case sensitivity would work for all locales. Suggested investigate using collations. 5. AOB IBM want to have the spec technically complete within the next 6-8 weeks. Next call 1 March 14:00 Meeting closed, 15:00 GMT Actions raised at this meeting No Action 032 DG: Investigate compatibility between DFDL infoset and XDM Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 25/02: no progress 023 MB: Review Schema 1.1 29/1: AP and SH to talk to Sandy Gao 04/02 Call arranged for Friday 11/02: Call took place. Identified useful changes. Consolidate with previous list. 024 String XML type 025 Escape schemes 21/1: discussed requirements 04/02: AP/SH to describe behaviour for known length text fields. Need to discuss if comment escapes should be supported. 11/02 new draft distributed: 18/02: SH up document concerns 25/02: SH and AP have refined proposal ready for approval. 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 031 DG: Review dfdl v033 11/02: Initial comments received 18/02: Will include work items 5 and 12. 032 DG: Investigate compatibility between DFDL infoset and XDM Closed actions: 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work 026 Remove duration complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL Revision 033 Comments
by Dave Glick 25 Feb '09

25 Feb '09

All, I have completed my review of draft 33 of the standard. I've read through the document as a whole at several times and spent a considerable amount of time digesting each of the sections. I've included nearly all my comments in the document (attached) but because there are so many, and reconciling them with ongoing revisions of the document may be difficult, I've included what I feel are the important points in this email (I realize there are a lot of points here, but I guess that's what happens when someone takes a totally fresh look at things). I would like to note that these are all just suggestions - I commented wherever I had a question or concern for completeness sake, but certainly don't expect all of my feedback to be incorporated - especially since many of the concerns may have already been discussed and addressed in previous iterations of the standard. My only goal and motivation is to and help make the best standard possible, and hopefully some of these suggestions will be food for thought. General - The document feels overly verbose and explanatory to me. There are many whole sections and blocks of text that, while very valuable, don't really seem appropriate in a normative standards document. The document should explain "what is," not necessarily "why it is." I understand that it was previously discussed as to whether portions of the document should be extracted and instead included in a separate non-normative "DFDL Primer" similar to the way W3C structured the XML Schema standard. My reaction is that doing so would help clean up the document. Using technical books as a metaphor, my own feeling is that a normative standard should be more like "The Definitive UNIX Reference" and less like "Introduction to UNIX" or even "Expert-Level UNIX". I think the standard falls a little too far into the latter category right now. - Related to the previous comment, the section that seemed the most out of place to me was the discussion on the parsing and unparsing processes and their relationship to grammar and general parsing concepts ("DFDL Properties Introduction"). Though the discussion was extremely valuable from the standpoint of a potential implementer and may actually be the only way to implement the standard, I think it may fall too far into the "how to implement" category and might be more appropriate in an appendix (marked as non-normative) or a primer if one is created. - There are certain sections that seemed a little misplaced to me. In every case, it became clear over time why the document was organized the way it was, but some revision may make it easier to digest. Generally, it seemed that some concepts were "spread out" and not organized under encompassing umbrella sections. Though the way it's currently structured may make the document more componentized, it makes it harder to understand from a "where do I look for all the information on X" standpoint. The most obvious example is all the sections dealing with representation properties such as the list of representation property precedence and the sections on sequences, choices, etc. My thought is that since all those sections are really discussing different representation properties and aspects thereof that it seems reasonable to group them into one overarching section. There are other areas where I thought the organization could be improved including the discussion on element vs. attribute vs. short binding forms (seemed misplaced given that several other non-representation property annotation element attributes such as setVariableName can use the alternate forms, and also broke up the flow of annotation element descriptions) and the glossary (I like the idea of defining specific terms used broadly throughout the document to remove ambiguity, but a general purpose glossary feels more appropriate as an appendix). To codify my thoughts, I worked up a TOC that I think exhibits a more understandable organizational structure. I've attached it not because I want or expect the entire structure of the document to be modified, but as a "jumping off" point for discussion. - The standard references RFC 2119 for defining certain terms such as MUST, SHALL, etc. In most other standards I've seen, emphasis is placed on these terms when their meaning is to be taken from RFC 2119 - I would suggest that DFDL do the same. It also appears as though the terms aren't being used throughout the document as regularly as they could or should be. I would suggest that at some point in the final revision process we scrub the document for requirements concepts and make sure to use the appropriate RFC 2119 terms where possible. This should remove ambiguity about what's expected from implementations. DFDL Information Set - I'm sure this has already been discussed at length, but I wonder if it would be possible to define the DFDL Information Set as an extension to the XPath Data Model (XDM) as XSLT does for its data model. This would have many advantages. The XDM is compatible with both the XML Schema PSVI and the XML Information Set (and the XDM standard explicitly explains the conversion process to and from each). This therefore provides interoperability with the alternate representations and uses of a DFDL Schema as an XML Schema and as plain XML content. Additionally, an XDM (or some reasonable facsimile) will have to be constructed from the DFDL Schema anyway to support the XPath capabilities of DFDL - basing the DFDL Infoset on XDM to begin with would ensure seamless (or at least easier) use with XPath libraries and infrastructure. Also, XDM and DFDL both use the XML Schema type system (after all, DFDL is a subset of XML Schema) and as such XDM already supports the DFDL types. - If the above is infeasible or too big of a change for this late in the process, would it at least be possible to define the DFDL Information Set in terms of the XML Information Set standard? The DFDL Information Set already appears to be loosely based on it, and may actually be compatible (I don't know) but the relationship is not explicit. Without such a statement and the satisfaction of the requirements of extending the XML Information Set as defined in that standard, implementations can't rely on the compatibility. If the relationship was made explicit and we ensured that the DFDL Information Set was indeed compatible with and extended the XML Information Set, then the XDM needed to process DFDL expressions could be generated using the Infoset to XDM process described in the XDM standard. If we went this route, I would also make sure that we maintain compatibility with the PSVI - that is, we don't want to introduce concepts or information set members that conflict with the PSVI. This will make it easier on implementers because they could potentially reuse the same internal infoset representation for both the DFDL Information Set and the PSVI during validation processes. - I imagine it will take some investigation to determine if either of these options is possible and compatible with DFDL concepts - I don't mind taking on the task if there is interest in modifying the DFDL infoset. It just seems a shame to me to forgo an opportunity to establish some synergy with related XML standards. - In any case, the concept of simple element information items and complex element information items seems contrary to established convention. The concept of using character information items (and groupings of them as explicitly allowed in the XML Information Set standard) to represent child simple content has already been established through the XML Information Set standard and other related XML standards. It is especially confusing given that the same terms as the XML Information Set are used. - Should everything in the DFDL Information Set have a corresponding representation in an XML document generated from or used to generate it (not the DFDL Schema, but the result or input to parsing or unparsing)? This question occurred to me based on the discussion in the most recent teleconference about treating and representing comments as separate kinds of content. It was suggested that the infoset would need to handle comments in a special way as to differentiate them from non-commented content. My concern is that there may not be an appropriate XML representation of such an infoset item. Creating a special element in the result document would break the property that the result of DFDL processing can be validated by the DFDL Schema (because the commented element wouldn't have been declared in the original DFDL Schema - it couldn't be a declared element because comments can appear anywhere in the source content). The only other option I can see would be to treat source content identified as comments and indicated as such in the infoset as XML comments in the result document. This brings up the interesting complication during unparsing of differentiating between "real" XML comments (those that should truly be ignored) and "output" XML comments (those that should be output to the result stream as commented content). The solution might be to treat all XML comments in a document used for unparsing as available for output as commented content, but it seems unreasonable to redefine XML comments in that way. This brings us back to the original question: if there is no way to adequately describe commented content in a resultant XML document, does everything in the infoset need a representation in an XML result document? What are the implications to upholding the ability to round-trip (if the resultant XML document doesn't contain everything in the infoset, and everything in the infoset is needed to fully describe the source content, then unparsing the resultant XML document will not result in the original source content)? Annotation Elements and Representation Properties - There seems to be inconsistencies throughout the document, specifically in the descriptions of annotation elements and representation properties. This is to be expected in a document that's been under heavy revision over such a long time span, but an effort will need to be made to scrub out all inconsistencies before the final version. To this end, I've found creating a table of all annotation elements and their properties helpful. I've attached what I have so far. It has all annotation elements and their attributes and a notional start for the representation properties. I intend to complete it as I go and hopefully make sure everything matches up in the process. - I'm not sure I understand the value in having the specialized annotation elements. From the DFDL user/developer perspective it seems more difficult because they need to recognize additional syntax. For example, when they see a dfdl:choice annotation element they need to understand that it's really a dfdl:format with a subset of allowed representation properties appropriate to xs:choice elements. They still must refer to the standard document to find out which representation properties are allowed, and the alternate syntax doesn't necessarily help in validation because a standard dfdl:format annotation element would also have been valid (and the DFDL XML Schema can't determine which representation properties are valid on a dfdl:format based only on usage location). It also makes the document more confusing because representation properties are refered to as being valid for specific dfdl:* annotation elements as opposed to the real meaning which is that they're valid for dfdl:format elements that annotate specific XML elements. To put it another way, a representation property that is valid for dfdl:choice is also only valid for dfdl:format when used as an annotation of xs:choice or as a short form property on xs:choice elements - but this isn't necessarily clear from the property descriptions since they only refer to the dfdl:* special annotation elements. From an implementation perspective, it adds complexity because the extra element names must be accepted. The DFDL parser will still have to validate representation properties and their validity as applied to the parent schema element regardless of whether the annotation element is a dfdl:format or a special annotation element. Not to mention, wouldn't short form be used most frequently anyway, in which case there are no annotation elements? In any case, I see very little value for a disproportionate amount of added complexity and potential confusion and I suggest the concept of special annotation properties that restrict dfdl:format be removed. - The standard isn't totally clear and unambiguous on the behavior with respect to the dfdl:format selector property. It is mentioned that the selector is externally identified, but no additional information is given. Are the selectors implementation specific? If so, does that break compatibility with alternate DFDL parsers if the selector property is used? What if a selector is referred to but doesn't exist in the parser? Is it a schema definition error or a parsing error (when are external selectors resolved)? What if there is no "default" dfdl:format block and they all contain non-matching selectors? Is that a processing error (should be explicit)? - When a defined format is put into use, how/when are the representation properties checked for validity with respect to the schema element (such as xs:choice) that put the defined format into use? To put it another way, is it an error (and what kind) if a defined format specifies representation properties that aren't valid for the schema element that uses it? Can a defined format contain the special format annotation elements (dfdl:sequence, dfdl:choice, etc.)? - The standard says that a dfdl:defineFormat can contain any of the other annotation elements. How are the other annotation elements contained within a dfdl:defineFormat (such as dfdl:assert or dfdl:hidden) applied when a named format definition is referenced by a dfdl:format ref attribute? Can named format definitions be referenced anywhere else other than where a dfdl:format is expected? Do all other annotation elements make sense or be valid wherever a defined format would be referenced? If not, suggest explicitly stating what annotation elements are allowed within a dfdl:defineFormat as opposed to saying any are allowed. - The descriptions for dfdl:assert and dfdl:discriminator read very similarly (and probably for good reason) but it's not clear how they're different. If the failure of a dfdl:discriminator results in a processing error, doesn't that make it equivalent to an assert? In other words, how can it be used for control when one of the two possible outcomes results in an error that (potentially) halts processing? May want to refine the description of dfdl:discriminator. - What about the positioning of hidden elements relative to siblings? Can they appear anywhere within the parent - in which case, is relative position important? May want to address this one way or the other. - The properties for dfdl:textNumberFormat are defined in the representation property section. Granted, they may be representation properties from the conceptual level, but syntactically, they would appear to be different. Can the text number format properties be used in a dfdl:format or dfdl:property element? If not, then suggest treating them more as attributes of the dfdl:textNumberFormat element and defining them there. If so, then I wonder what the purpose of the dfdl:textNumberFormat element is...it would seem to fall into the same category as the other special dfdl:format annotation elements that restrict the set of valid representation properties. Also seems to apply to dfdl:defineEscapeScheme and its representation properties. - The document isn't clear on how the position of a variable declaration impacts its scope. Does it apply to all children of the element to which the definition belong (regardless of position relative to the definition), to all siblings following the definition (but not preceding), or to all elements following the definition (regardless or hierarchy). I assume the first, but more clarification would be helpful in order to make it unambiguous. - The value type for representation properties is listed in conceptual terms, but shouldn't all properties actually accept one or more specific XML Schema (within the DFDL subset) types (usually atomic)? Making this explicit would remove confusion on the part of implementers. For example, several are defined as 'Enum' - though the value may logically be an enumerated type, the actual atomic type is something else like xs:string or xs:token - with additional validation to ensure it's one of the allowed enumerated values. The normative standard is first and foremost a reference for implementations and as such should be totally unambiguous with regard to typing information. - There appear to be cases (such as alignment) where multiple types can be accepted unnecessarily. In the alignment case, a specific xs:string or a positive integer type is valid. Wouldn't this be easier on both the DFDL XML Schema and the implementations if, wherever possible, only one atomic type was accepted? In the alignment case, a xs:nonNegativeInteger could be used where '0' means 'implicit'. - With regard to case sensitivity, how is the case equivalency defined for different character sets? Is this (or should it be) related to XPath collations? Perhaps instead of an ignore case switch the user should be allowed to specify a collation for initiator/terminator comparison and the DFDL standard would require implementations include a case-insensitive collation for common character sets. This would open the door to using more general character/string comparison operations and could be important in certain settings - for example, the XPath standard has an example that 'v' and 'w' are equivalent in Swedish. This may have some other advantages - if collations are needed for this kind of thing, then we could probably support fn:compare and fn:codepoint-equal in the DFDL XPath subset. If you've made it this far, congrats :) Hopefully this list will spur some discussion. Thanks, Dave --- David Glick | dglick(a)dracorp.com <mailto:dglick@dracorp.com> | 703.299.0700 x212 Data Research and Analysis Corp. | www.dracorp.com <http://www.dracorp.com>

2 1

Re: [DFDL-WG] IBM Virtual XML Garden
by Jason Kastner 23 Feb '09

23 Feb '09

I overstated with the "not quite 100%" statement. We are working with Defuddle but have not been updating vocabulary parsing as the spec has evolved. Defuddle is still based on the version of DFDL language when layers were still around. We still believe layers are important to what we are trying to accomplish which is one reason we are branched from the main DFDL spec and not trying to stay up with things. Jason Kastner Research Programmer NCSA at University of Illinois ----- Original Message ----- From: "Steve Hanson" <smh(a)uk.ibm.com> To: "Jason Kastner" <jkastner(a)ncsa.uiuc.edu> Cc: dfdl-wg(a)ogf.org Sent: Monday, February 23, 2009 9:38:41 AM GMT -06:00 US/Canada Central Subject: Re: [DFDL-WG] IBM Virtual XML Garden Hello Jason That's interesting news, thanks for the update. What draft of the spec have you been working to? Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Jason Kastner <jkastner(a)ncsa.uiuc.edu> Sent by: dfdl-wg-bounces(a)ogf.org 23/02/2009 15:25 To dfdl-wg(a)ogf.org cc Subject Re: [DFDL-WG] IBM Virtual XML Garden Although we (NCSA) aren't working 100% up to the spec, we are close to releasing an updated version of Defuddle to sourceforge site that is built on the latest JaxMe libraries. Jason Kastner Research Programmer NCSA at University of Illinois -----Original Message----- From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Pere Estaques Sent: Friday, February 20, 2009 12:20 PM To: dfdl-wg(a)ogf.org Subject: Re: [DFDL-WG] IBM Virtual XML Garden Hi, is there any place with a (more or less) recent dfdl parser? Defuddle in sourceforge seems to be dead, its latest version is Sep 2007. Xml garden is even older. regards, Jordi -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] IBM Virtual XML Garden
by Jason Kastner 23 Feb '09

23 Feb '09

Although we (NCSA) aren't working 100% up to the spec, we are close to releasing an updated version of Defuddle to sourceforge site that is built on the latest JaxMe libraries. Jason Kastner Research Programmer NCSA at University of Illinois -----Original Message----- From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of Pere Estaques Sent: Friday, February 20, 2009 12:20 PM To: dfdl-wg(a)ogf.org Subject: Re: [DFDL-WG] IBM Virtual XML Garden Hi, is there any place with a (more or less) recent dfdl parser? Defuddle in sourceforge seems to be dead, its latest version is Sep 2007. Xml garden is even older. regards, Jordi

2 1

IBM Virtual XML Garden
by Pankaj Agarwal 20 Feb '09

20 Feb '09

Hello, I was writing to find out if the source code for IBM Virtual XML Garden is available anywhere. I could not find it on the IBM site, but maybe I missed it. Thanks, - Pankaj Pankaj Agarwal Duke Bioinformatics Shared Resource Duke Comprehensive Cancer Center Duke University 919-681-6573 p.agarwal(a)duke.edu

4 3

DFDL: Minutes from OGF WG call, February-18-2009
by Alan Powell 19 Feb '09

19 Feb '09

Open Grid Forum: Data Format Description Language Working Group OGF DFDL Working Group Call, February-18-2009, Attendees Alan Powell (IBM) Steve Hanson (IBM) Dave Glick (drac) Apologies Mike Beckerle(Oco) 1. XSD 1.1 Discussion deferred 2. Escape Schemes Further discussion of latest proposal. Noted the need to properly describe the function of escapeSceheme and in particular distinguish from substitution mechanisms such as XML entities which are sometimes called escapes. SH felt there could be confusion with the two use cases of escapeCharacter. Suggested separate properties. Action SH to document issues and improve wording Discussed whether comment type escapes should be supported. Decided they are not needed on V1. AP to confirm that if IBM products provide support. 2. Review Actions Action 20 - There had been a number of email exchanges following on from last weeks call. Decided to keep 'zoned' as it is well known to users who need the support. Discussion on whether dfdl should allow unsigned logical types to have 'signed' phsyical representation. If so what should be the behaviour when there is a mismatch of signs. Action: AP to describe behaviour 3. calendarFormats SH and AP have been discussing the behaviour of strict and lax parsing which needs a more complete description. 4. TypeSubsitution AP raised whether typeSubstitution needs to be supported in DFDL v1. 5. AOB DG to take a look at Work items 5 and 12 while continuing review MB will not be available next week Next call 25 February 14:00 Meeting closed, 15:00 GMT Actions raised at this meeting No Action Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properties 18/2: Need more investigation of ICU strict/lax behaviour. 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 18/02: AP to document unsigned type behaviour 023 MB: Review Schema 1.1 29/1: AP and SH to talk to Sandy Gao 04/02 Call arranged for Friday 11/02: Call took place. Identified useful changes. Consolidate with previous list. 024 String XML type 025 Escape schemes 21/1: discussed requirements 04/02: AP/SH to describe behaviour for known length text fields. Need to discuss if comment escapes should be supported. 11/02 new draft distributed: 18:02: SH up document concerns 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 031 DG: Review dfdl v033 11/02: Initial comments received 18/02: Will include work items 5 and 12. Closed actions: 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work 026 Remove duration complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

DFDL: Minutes from OGF WG call, 11 February 2009
by Alan Powell 12 Feb '09

12 Feb '09

Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 14:00 GMT, 11 February 2009 Attendees Alan Powell (IBM) Steve Hanson (IBM) Dave Glick (drac) Aplogies Mike Beckerle(Oco) 1. XSD 1.1 Went through Schema 1.1 changes presentation from Sandy Gao. Some items need further investigation 2. Review Actions Action 20 - Discussed suggested changes from SH. Observed that non-overpunching zoned was the same as text. Need to decide if unsigned types are allowed to have representation zoned. Other actions updated below 3. Escape Schemes Discussed latest proposal. General agreement that list of references was better than list of pairs of delimiters. Observed that property names within xxxFormats don't need type prefix eg calendarPatternKind -> patternKind Need to discuss if comment escapes should be supported. 4. Calendar/Number formats Didn't discuss 4. AOB Next call 18 February 14:00 Meeting closed, 15:00 GMT Actions raised at this meeting No Action Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 04/02: changed from locale to specific properies 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 11/02: proposal largely agreed. SH to make minor changes 023 MB: Review Schema 1.1 29/1: AP and SH to talk to Sandy Gao 04/02 Call arranged for Friday 11/02: Call took place. Idnetified useful changes. Consolidate with previous list. 024 String XML type 025 Escape schemes 21/1: discussed requirements 04/02: AP/SH to describe behaviour for known length text fields. Need to discuss if comment escapes should be supported. 11/02 new draft distributed: 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 031 DG: Review dfdl v033 11/02: Initial comments received Closed actions: 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work 026 Remove duration complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: DFDL OGF WG call - Action 020
by Steve Hanson 12 Feb '09

12 Feb '09

It was noted on the call this week that there is an alternative to my zoned decimal overpunching proposal i) below. I said: - If it is an unsigned type then DFDL expects the rightmost byte to have a zone nibble when parsing, and outputs a zone nibble when unparsing. - If it is a signed type then DFDL expects it to have a sign nibble when parsing, and outputs a sign nibble when unparsing. But my unsigned type behaviour could be achieved by specifying a rep of text instead of zoned. If that is the case, the alternative is to only allow zoned rep for signed decimal logical types. Thoughts? Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 05/02/2009 10:17 ----- Steve Hanson/UK/IBM 28/01/2009 13:54 To DFDL Working Group cc Subject DFDL OGF WG call - Action 020 Action 020: 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position a) Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy Add new property to section 15.4 Properties Specific to Number with Binary representation. binaryNumberCheckPolicy Enum Values are ?strict? and ?lax?. Indicates how lenient to be when parsing binary numbers. If ?lax? then the parser tolerates all valid alternatives where such alternatives exist. Specifically, for binaryNumberRepresentation = 'packed' the sign nibble for positive, negative, unsigned and zero is allowed to be any of the valid respective values. On unparsing, the specified value is always used. Also suggest changing some of the other property names in 15.4: "decimalVirtualPoint" -> "binaryDecimalVirtualPoint" "packedDecimalSignCodes" -> "binaryPackedSignCodes" And changing binaryNumberRepresentation enumeration: "BCD" -> "bcd" b) Zoned decimals: How to decide to overpunch and sign position Spec assumes that overpunching of the rightmost character always takes place. IBM architecture allows no overpunching (ie, Fx instead of Cx/Dx) - this is supported by IBM MRM & WTX parsers. Additionally IBM MRM parser allows separate sign byte, and sign byte on left. Let's deal with these separately: i) No overpunching. The IBM architecture allows the rightmost byte to have a zone (Fx) or a sign (Cx/Dx) as the left nibble. I don't see why we can't base what to expect when parsing, and output when unparsing, on the logical xsd type. - If it is an unsigned type then DFDL expects the rightmost byte to have a zone nibble when parsing, and outputs a zone nibble when unparsing. - If it is a signed type then DFDL expects it to have a sign nibble when parsing, and outputs a sign nibble when unparsing. For analogy with DFDL packed decimals, it seems at first glance that we should also extend the numberCheckPolicy 'lax' setting to treat a zone nibble as a +ve sign nibble for a signed type. However, IBM iSeries always outputs Fx to mean +ve but accepts both Fx & Cx on input. It is perhaps better therefore that DFDL always tolerates Fx when parsing a signed zoned decimal, otherwise iSeries users would always have to set numberCheckPolicy to 'lax', which might have other implications in the future. ii) Separate sign byte. I don't believe the IBM architecture allows this. I'm not sure where MRM got this from. I don't think DFDL needs to support it. iii) Sign byte on left. I don't believe the IBM architecture allows this. I'm not sure where MRM got this from. I don't think DFDL needs to support it. Conclusion: No new DFDL properties needed, but words need adding to explain zoned parse/unparse behaviour better. Also suggest changing property names: "zonedDecimalSignStyle" -> "numberZonedSignStyle" "zeroNumberRep" -> "numberZeroRep" Should also make clear that any explicit negative pattern in numberPattern will be ignored if the xsd type is unsigned. (We could make this an error but it precludes creation of a textNumberFormat that works with both signed and unsigned types, plus pattern "##0.0" implictly is equivalent to "##0.0;(##0.0)" ). Regards Steve Hanson Programming Model Architect WebSphere Message Brokers Hursley, UK Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 Alan Powell/UK/IBM@IBMGB Sent by: dfdl-wg-bounces(a)ogf.org 23/01/2009 13:36 To dfdl-wg(a)ogf.org cc Subject [DFDL-WG] DFDL: Minutes from OGF WG call, 21 January 2009 Open Grid Forum: Data Format Description Language Working Group Weekly Working Group Conference Call 14:00 GMT, 21 January 2009 Attendees Alan Powell (IBM) Mike Beckerle(Oco) Apologies Steve Hanson (IBM) 1. XSD 1.1 Deferred to next call 2. Calendar formats Discussed updated (v4) supplement emailed by AP Agreed millisec/secSinceEpoc cannot be implied by length of logical data so need seperate enumerations. Observed that these options were really combination of 3 properties binary, length and sec/millisec. Suggested renaming to binarySeconds and binaryMilliseconds Packed calendars: decided that need to be able to specify at least the packedDecimalSignCodes property rather than assuming a default so reference will be added to calendar description Locale needs to be specified for numberformats and calendarFormats (didn't identify any other areas) as it modifies the behaviour of ICU. Decided to add locale to numberFormat and CalendarFormat 3. Escape Schemes Agreed need for multiple escape delimiter pairs but not nested. Need an escape for escape character even though in most cases this will be the same character, eg /n //, There are some formats that have a different escape, eg /n &/. Only need single escape characters and one level of escape characters. Discussed how to deal with comments of the form /* comment */ where the escape delimiters are also the initiator and terminator of the field. Semantic needed is 'only look for field terminator not any parent terminator or any other syntax elements'. May fall out naturally from the speculative parsing rules. Need further discussion. 4. AOB Next call 28 January 14:00 Meeting closed, 15:00 GMT Actions raised at this meeting No Action 031 Current Actions: No Action 012 AP/SH: Update decimalCalendarScheme 10/9: Not allocated yet 17/9: No update 24/9: Add calendar binary formats to actions 22/10: No progress 16/1: proposal distributed and discussed. Will be redistributed 21/1: add locale, 020 SH: Resolve packedDecimalSignCodes behaviour depends on NumberCheckPolicy 22/10: No progress 10/12: added how to decide to overpunch and sign position 023 MB: Review Schema 1.1 024 String XML type 025 Escape schemes 21/1: discussed requirements 026 SH: Envelopes and Payloads 027 Property precedence tables 028 Variable markup 029 valueCalc (output length calculation) 030 AP: confirm with WTX that can drop duration 21/6: WTX confirm that they do not have a duration type so do not need it in dfdl. Will drop from spec. Closed Closed actions: 030 AP: confirm with WTX that can drop duration 21/6: WTX confirm that they do not have a duration type so do not need it in dfdl. Will drop from spec. Closed 034 Work items: No Item 001 String XML type (Ian P) - Apr 30, 2008 002 Escape schemes (Ian P) - Apr 30, 2008 003 Variables - ??, 2008 (Mike) 005 Improvements on property descriptions - ??, 2008 (All - split TBD) 006 Envelopes and Payloads (Steve) - Apr 30, 2008 007 (from draft 32) valueCalc (Mike) - ??, 2008 mostly complete 008 (from draft 32) Property precedence for writing (Steve) - under review 009 (from draft 32) Variable markup (Steve) - Mar 31, 2008 proposal needs writing up 010 (from draft 32) Assertions, discriminators and choice, including discussion of timing option (Suman) - Mar 31, 2008 * in progress * 011 (from draft 32) How speculative parsing works (combining choice and variable-occurence - currently these are separate) ??, 2008 (IBM) in progress 012 (from draft 32) Reordering the properties discussion: move representation earlier, improve flow of topics ??, 2008 (Alan) * not started * 025 Augmented infoset and unparsing (Alan) added but needs work complete - specification updated Alan Powell MP 211, IBM UK Labs, Hursley, Winchester, SO21 2JN, England Notes Id: Alan Powell/UK/IBM email: alan_powell(a)uk.ibm.com Tel: +44 (0)1962 815073 Fax: +44 (0)1962 816898 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 3