August 2014 - dfdl-wg - lists.ogf.org

Fw: Action 261
by Steve Hanson 06 Aug '14

06 Aug '14

In the spec as it stands, occursCountKind 'expression' has behaviour like occursCountKind 'fixed' when parsing (ie, there is a count that is fixed before the occurrences are parsed), but is also stated to have behaviour like occursCountKind 'parsed' when unparsing (ie, there is no count that is fixed before occurrences are unparsed). In terms of expected separators, it behaves like 'never' when parsing, but like 'anyEmpty' when unparsing. This leads to undesirable asymmetric behaviour, such as this example. Data stream is 5,A,B,,D,E. To preserve indexes in the infoset after parsing, I make the array element nillable with ES as nil value. But on unparsing I will get 5,A,B,D,E because the 'anyEmpty' behaviour has suppressed the output of the separator for the zero-length nil value. Not only will modellers find this behaviour odd, it breaks dfdl:outputValueCalc on the count element if fn:count() is used. (Same happens if not nillable but minOccurs > 2 so empty string ends up in the infoset). The proposal is that 'expression' behaves like 'never' on unparsing, the count being taken as the number of items in the augmented infoset. That way the application or outputValueCalc can be certain that no separators will be suppressed, and outputValueCalc will work. It was proposed further down this email thread that the expression is evaluated at the start of unparsing the element and used to obtain the count. There are scenarios where that causes problems though, hence the above is preferred. However, to ensure that the output data stream can be re-parsed, it is proposed that the expression is evaluated at the end of the unparsing of the element (and any elements with expressions dependent on it), and if it fails to match the number that was output, it is a processing error. Similarly for lengthKind 'explicit' where length is an expression. The element continues to be considered variable length on unparsing as the spec says today, but the expression is evaluated at the end of the unparsing of the element, and if it fails to match the length that was output, it is a processing error. It would be great to do the analogous thing for lengthKind 'pattern' but it is not always possible as some patterns look ahead beyond the element content in order to match. So it could be done for some patterns but not all. Implementation-dependent? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 06/08/2014 13:08 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Date: 25/06/2014 11:33 Subject: Fw: [DFDL-WG] Action 261 261 Implied separatorSuppressionPolicy for occursCountKind 'expression' (All) 10/6: Spec says it is 'never' (positional sequence) but you have to parse to identify the position, so isn't that non-positional? 17/6: Some other issues noted around 'expression' as per email thread. IBM have discussed this internally and will submit a proposal. As was noted in the email for Action 260, if it is decided that the meaning of "Each occurrence in the sequence can be identified by its position in the data" is more strictly that an observer of the raw data can identify an occurrence of an element in the sequence solely by counting separators then that would appear to make dfdl:occursCountKind 'expression' more like 'parsed', and not eligible to be in a Positional sequence. But if the meaning is a parser does not have to speculate to identify an occurrence of an element in the sequence then it can be in a Positional sequence. While discussing the nature of 'expression' it was noted that it is very easy for a DFDL user to create a data stream from an infoset and for that data stream to be un-parse-able. If dfdl:outputValueCalc is not used, then the element(s) in the infoset must be correctly set manually to match the number of occurrences. The same observation applies to dfdl:lengthKind 'explicit' where dfdl:length is an expression. To address this, IBM proposes the following changes to the DFDL specification for occursCountKind 'expression' when unparsing: When all occurrences have been obtained from the infoset (and defaulting applied if needed), the occursCount expression is evaluated If any element that is referenced by the expression has dfdl:outputValueCalc, then that expression is (re-)evaluated as part of the above; given that the number of occurrences is now known, the outputValueCalc expression should now evaluate successfully If the result of the occursCount does not match the number of occurrences it is a processing error This ensures the integrity of the data stream (the non-outputValueCalc use case), while still allowing the infoset to dictate the count (the outputValueCalc use case). Similarly, IBM proposes the following changes to the DFDL specification for lengthKind 'explicit', where length is an expression, when unparsing: When an element has been obtained from the infoset (and defaulting applied if needed) and unparsed but before padding is applied, the expression is evaluated to give a length. If any element that is referenced by the expression has dfdl:outputValueCalc, then that expression is (re-)evaluated; given that the unpadded length is now known, the outputValueCalc expression should now evaluate successfully Now that we have a length, the unparser behaviour is the same as if the length was obtained from a fixed dfdl:length value. (That is, truncation, padding, filling or error depending on other property settings). This ensures the integrity of the data stream (the non-outputValueCalc use case), while still allowing the infoset to dictate the length (the outputValueCalc use case). Note that it means the 'awkward' behaviour whereby lengthKind 'explicit' (expression) is a specified length when parsing but variable length when unparsing is avoided - it is now always specified length. IBM believes that none of the above should impose additional burden on implementers, as no brand new behaviour is being added. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 25/06/2014 10:24 ----- From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Alex Wood1/UK/IBM@IBMGB, Andrew Edwards/UK/IBM@IBMGB, Mark Frost/UK/IBM@IBMGB, Date: 16/06/2014 22:08 Subject: Re: [DFDL-WG] Action 261 I think this needs to be discussed before the meeting tomorrow. I wanted to avoid turning this thread into a monster - but I don't think it's possible to discuss without the context so I've continued the thread as before with <tk> tags. I'm clear in my own mind that I understand the issues now. We should give priority to the separatorSuppressionPolicy question because the current rules are close to being unimplementable. The occursCountKind and lengthKind questions are important but are at least implementable as they stand. regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 11/06/2014 16:12 Subject: Re: [DFDL-WG] Action 261 Replies in <smh> tags Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 11/06/2014 13:58 Subject: Re: [DFDL-WG] Action 261 Sent by: dfdl-wg-bounces(a)ogf.org comments in <tk>tags regards, Tim Kimber, IBM Integration Bus Development (Industry Packs) Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 11/06/2014 10:47 Subject: Re: [DFDL-WG] Action 261 Some thoughts on this... I agree that the definition of positional sequence in the spec needs tightening as it is ambiguous as it stands and could be interpreted as a) or b). If we adopted b) then that would appear to allow 'expression' to appear in a positional sequence, but wouldn't it also allow 'stopValue'? <tk>Yes - according to definition b) stopValue would be allowable in a positional sequence. We could still disallow it if we do not believe there is any benefit in allowing it. I don't believe it introduces any particular complexities for an implementer.</tk> occursCountKind 'expression' is analogous to lengthKind 'explicit' with an expression and to lengthKind 'prefixed'. Both these lengthKinds are classified as 'specified length' when parsing but 'variable length' when unparsing. We are observing that occursCountKind 'expression' is like 'fixed' when parsing but not quite so like 'fixed' when unparsing - which is why section 16 groups 'expression' with 'parsed' for unparsing. <tk>Yes - we took a decision that the unparser should ignore the expression in lengthKind/occursCountKind, and just output whatever data happens to be in the info set. I'm not sure that it saves a lot of effort in the implementation and it certainly is not easy to justify as a consistent behaviour. For me, the unparser should treat lengthKind='explicit' the same way whether the value is static or calculated. And the unparser should treat occursCountKind='expression' the same way as occursCountKind='fixed'. </tk> When unparsing occursCountKind 'expression' you don't always have the calculated array length N. If the infoset was derived from XML, there is likely no 'count' element, just a bunch of elements with the same name that make up the 'array'. DFDL gives you the choice whether to a) manually set the count element, or b) have the unparser set it automatically via outputValueCalc. In the former case, you can create a document that can not be parsed; the unparser could check the 'count' element matches the infoset, but that would involve reverse engineering an arbitrarily complex expression and is why the specification does not say that. <tk>It would involve evaluating the expression. In most cases, that will not require any lookahead because the Length/Count field will precede the array or element. Not sure where the reverse engineering comes in?</tk> <smh>I see what you are saying. Just evaluate the expression and see what it gives for N. That handles case a) but not b) where I explicitly want the unparser to set the count via outputValueCalc - which is presumably referring to the number of elements in the array, which is not known. For case b) N has to be the number in the infoset. Given that we have to support case b) the unparser can not treat occursCountKind 'expression' exactly the same as 'fixed' when unparsing.</smh> <smh>Similarly with lengthKind 'explicit' with an expression. For the equivalent to case a) the length is known which makes the length fixed, but for the equivalent of case b) with outputValueCalc the length is not known so it is variable. When this was discussed in the past, it was decided not to bifurcate the expression scenario. Hence the spec is the way it is. </smh> <tk> That helps. So your belief is that case a) is workable but case b) is not because the number of elements in the array is not known. I don't think that holds up under scrutiny. In case b), the outputValueCalc expression cannot be evaluated until all of the array has been received. So if the info set is received as an event stream then the unparser must wait until the array completes before evaluating the expression. Note that the 'count' ( or 'length' ) field cannot be serialized until its value is known. But the array ( or variable-length field ) comes *after* the count/length field. By the time unparsing of the array/field begins, the value *is* known. The implementation of case a) is actually less straightforward. If the field is an array length and the value is less than the number of items in the info set then I think the unparser must issue an error. If greater then the unparser could output default values ( if available ) or delimiters ( if the parent sequence is a positional sequence with a delimiter ) or else an error if neither are possible. More simply, the unparser could simply insist that the value must correctly describe the data, and I think that's a reasonable rule. Similarly for the length. If the length of the *unparsed* value is greater than the value in the info set then the unparser should issue an error. If it is shorter and the field is simple then pad characters could be added. But again, I think real-world usage of length counts suggests that padding is unlikely to be wanted, and the unparser should simply insist on the length field being correct. </tk> Here's a real example of such an expression (albeit with lengthKind 'explicit' but the principle is the same): dfdl:length="{xs:nonNegativeInteger(fn:floor((../Length + 1) div 2))}" Alex brought up the case where the expression evaluates to 0. In a positional sequence, would you still expect a delimiter for this case? <tk>Yes, unless it is in the trailing optional region of the group and SSP='trailingEmpty'. In a positional sequence, every delimiter must be present until suppression begins ( if allowed )</tk> If 'yes' then the resultant zero length string must be treated as the 'absent representation' and ignored. If 'no' then is the sequence still positional? <tk>I don't understand the point. Why would it not be the 'empty representation'? Why must it be 'ignored' if it does happen to be the 'absent representation'? What does 'ignored' mean?</tk> <smh>The point is that the parser has been told there are 0 occurrences. So it would be odd if the infoset ended up containing an occurrence, which can happen if the normal nil/empty rules are followed. (Eg, nilValue=%ES;). <tk> If the occursCount expression evaluates to zero then the parser will not attempt to parse even one occurrence of the array. That's the natural meaning of 'zero occurrences'. So nothing would go into the info set apart from the 'count' field. This is entirely consistent with my definition b) of 'positional' ( the identity of every delimited field is known before parsing of the field begins ). </tk> Hence the 0 occurrence case must treat it as absent which means nothing is added to the infoset. Take the ISO8583 bitmap use case - if the bit is 0 we must not try to parse anything at all for that element - it is totally absent.</smh> <tk>Yes - that's exactly my point. The fact that there is ( or could be ) a delimiter after the 'count' field is irrelevant.</tk> Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 10/06/2014 21:22 Subject: [DFDL-WG] Action 261 Sent by: dfdl-wg-bounces(a)ogf.org Implied separatorSuppressionPolicy for occursCountKind 'expression ' (All) 10/6: Spec says it is 'never' (positional sequence) but you have to parse to identify the position, so isn't that non-positional? I think there are two alternative definitions of 'positional': a) the identity of every delimited field is known before parsing of the sequence group begins b) the identity of every delimited field is known before parsing of the field begins As an implementer, b) is sufficient because it means that the parser never needs to backtrack while parsing the group. a) allows the field identities to be statically known, but that is less important - it does not allow optimised extraction of a particular field as would be the case for a fixed-length group ( the possibility of escaped separators/terminators means that every character will need to be scanned anyway ). It may sound like a small point, but it affects two decisions 1. whether ock='expression' should be allowed within a positional sequence group ( action 261 ) 2. what the behaviour of the unparser should be w.r.t. ock='expression'. My own feeling is that ock='expression' should be treated almost exactly like ock='fixed', except that the calculated array length N is used instead of maxOccurs. - When parsing a positional sequence group it should cause N delimiters to be expected for the array. - When unparsing a positional sequence group it should cause N delimiters to be written. These rules are consistent and straightforward to describe and implement. The current rule ( unparser outputs the occurrences that are in the info set only ) allows the unparser to write a document that cannot be parsed using the same schema. regards, Tim Kimber, ----- Forwarded by Tim Kimber/UK/IBM on 10/06/2014 20:34 ----- From: Steve Hanson/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 10/06/2014 17:57 Subject: [DFDL-WG] OGF DFDL WG Call Minutes 2014-06-10 Sent by: dfdl-wg-bounces(a)ogf.org Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13263?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Fw: Action 260
by Steve Hanson 06 Aug '14

06 Aug '14

So my suggestion below to wrap the array in a sequence does not work, it just moves the problem down into the new sequence. After much deliberation, we think that the definitions of Positional sequence and Non-positional sequence should not be viewed as driving the behaviour of a sequence, but simply as the resultant characteristics of a sequence that has certain properties. That leaves modellers free to mix occursCountKinds, as in Tim's example. No need for any new SDE scenarios. Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. In DFDL, a sequence is considered positional if it contains only required elements and/or optional and array elements that have dfdl:occursCountKind 'implicit', 'fixed' or 'expression', and it has dfdl:separatorSuppressionPolicy 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Often the components of such a sequence have an initiator. Such sequences sometimes allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing might need to be employed by to identify each occurrence. In DFDL, a sequence is non-positional if it contains any optional or array elements that have dfdl:occursCountKind 'parsed' or 'stopValue', and/or it has dfdl:separatorSuppressionPolicy 'anyEmpty'. See parallel email for action 261 that ensures 'expression' behaves itself. One behaviour that is missing from the spec. For a sequence with separators, what is expected in the data stream if occursCount = 'fixed' / 'implicit' and maxOccurs = '0', or occursCountKind = 'expression' and occursCount evaluates to 0 ? We believe that no separator should be expected when parsing and none output when unparsing (same behaviour as inputValueCalc). Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 06/08/2014 12:42 ----- From: Steve Hanson/UK/IBM To: Tim Kimber/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org, dfdl-wg-bounces(a)ogf.org Date: 30/06/2014 10:04 Subject: Re: [DFDL-WG] Action 260 You would wrap the array and it's count in a sequence. Then the 'count+array' is treated as a single entity as far as the parent sequence is concerned. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM@IBMGB To: dfdl-wg(a)ogf.org, Date: 26/06/2014 20:06 Subject: Re: [DFDL-WG] Action 260 Sent by: dfdl-wg-bounces(a)ogf.org Before we settle one way or the other, I would like the following data format to be taken into consideration. chars,5,A,B,C,D,E,integers,1,2,3 chars,3,C,,,integers,2,10,11 I am assuming that the occursCountKind for the arrays is 'expression' and the occursCount refers to the integer field that precedes the array. In order to represent the empty strings on the second line it is essential to specify SSP as 'trailingEmpty' or 'never'. If we disallow the combination of ock='expression' and SSP='trailingEmpty' then how would this format be modelled? regards, Tim Kimber, Technical Lead for IBM Integration Bus Healthcare Pack Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 25/06/2014 16:25 Subject: Re: [DFDL-WG] Action 260 Sent by: dfdl-wg-bounces(a)ogf.org I prefer choice (a) for two reasons * It is more restrictive and therefore more conservative (preserving freedom to change in future if needed) * If a user has a positional data format, you don't want them to even have to understand the concept of speculation in order to model their data. So choice (a) allows a simpler description that doesn't need to introduce the notion that the parser might be speculation. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Jun 25, 2014 at 5:20 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: 260 Positional and non-positional sequences (All) 10/6: Spec defines the above but also allows different occursCountKinds within the same sequence which may have different (implied) separatorSuppressionPolicy, which results in a sequence which is a mixture of both. Should this be allowed? If so what are the rules? Can certain combinations be disallowed? 17/6: IBM have discussed internally and will submit a proposal. In the spec we define Positional Sequence and Non-Positional Sequence: Positional sequence - Each occurrence in the sequence can be identified by its position in the data. Typically the components of such a sequence do not have an initiator. In some such sequences, the separators for optional zero-length occurrences may or must be omitted when at the end of the group. A positional sequence can be modelled by setting dfdl:separatorSuppressionPolicy to 'never', 'trailingEmptyStrict' or 'trailingEmpty'. Non-positional sequence - Occurrences in the sequence cannot be identified by their position in the data alone. Typically the components of such a sequence have an initiator. Such sequences allow the separator to be omitted for optional zero-length occurrences anywhere in the sequence. Speculative parsing is employed by the parser to identify each occurrence. A non-positional sequence can be modelled by setting dfdl:separatorSuppressionPolicy to 'anyEmpty'. The problem is that the setting of dfdl:separatorSuppressionPolicy is only examined for child elements with dfdl:occursCountKind 'implicit'. For other dfdl:occursCountKinds, there is the concept of an 'implied' dfdl:separatorSuppressionPolicy: When dfdl:occursCountKind is 'fixed' then ... the implied behaviour is 'never'. When dfdl:occursCountKind is 'expression' ... the implied behaviour is 'never'. When dfdl:occursCountKind is 'parsed' ... the implied behaviour is 'anyEmpty'. When dfdl:occursCountKind is 'stopValue' ...the implied behaviour is 'anyEmpty'. So if a Positional sequence as defined above contains children with dfdl:occursCountKind 'parsed' or 'stopValue' then surely it is no longer a Positional sequence. A solution to this is to prevent the appearance of certain values of dfdl:occursCountKind within a Positional sequence. However, precisely which values to outlaw is subject to interpretation of the phrase "Each occurrence in the sequence can be identified by its position in the data". Is this intended to mean: a) an observer of the raw data can identify an occurrence of an element in the sequence solely by counting separators => SDE if 'parsed', 'stopValue' or 'expression' ** appeared in a Positional sequence; ** Although 'expression' would appear to be like 'fixed' it actually breaks a) so must be included in the SDE list. or b) a parser does not have to speculate to identify an occurrence of an element in the sequence => SDE only if 'parsed' appeared in a Positional sequence. Note that it is possible to wrap a 'parsed' etc element in a local sequence or another element to avoid an SDE. But this could still be seen as a violation of a) if the separators of both are the same, as the observer can not count the separators. So should the rule be applied recursively, ie, a Positional sequence can not contain a non-Positional sequence unless the separators are different? Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2014-08-04
by Steve Hanson 04 Aug '14

04 Aug '14

Please find minutes from the above call at http://redmine.ogf.org/dmsf_files/13316?download= Please review spec draft r21 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Note next call is 26th Aug. Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: DFDL Working Group Call
by Steve Hanson 04 Aug '14

04 Aug '14

Steve on holiday

1 0

Cancelled: DFDL Working Group Call
by Steve Hanson 04 Aug '14

04 Aug '14

Steve on holiday

1 0

OGF DFDL WG Call Agenda 2014-08-04
by Steve Hanson 04 Aug '14

04 Aug '14

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13316?download= Please review spec draft r18 on Redmine. Email any comments to dfdl-wg(a)ogf.org. Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

bit order documents updated
by Mike Beckerle 01 Aug '14

01 Aug '14

There are now several working documents having to do with bit-order. The prior document has been taken down and replaced by 3 others. These are all up on redmine: http://redmine.ogf.org/dmsf/dfdl-wg?folder_id=5485 1) draft-gwdi-bit-order-features-v2.05.docx - Just describes the new features + TDML extension. The biggest section is organized like an errata specifying where things change in the spec. This should get incorporated into errata, or perhaps just referenced from an errata. Section 3 of this doc is the proposed errata and spec language for review. There is one major open issue here, which is it is unclear whether the mixed-endian byte order that was previously proposed is actually needed or not. The swapping of 16-bit words appears to be not a per-element thing, but something done as a pre-processing of an entire message body before parsing. This isn't something we can handle in DFDL, much like base64-encoded data and so forth. This may be the only place that needed the 16-bit word swapping. 2) Understanding Bit Orderings - draft-gwdi-mil-std-2045-understanding-bit-order-v2.05.docx - Material about bit order - the Wire model vs. Number model material. This is effectively just archiving this material for posterity. If you already read this, you don't need to read it again. 3) draft-gwdi-mil-std-2045-additional-features-v2.05.docx - Material about additional DFDL features that would be helpful in modeling MIL-STD-2045. This can also be reviewed. Not as urgent as (1) above. 4) draft-gwdi-dfdl-standard-encodings-v03.docx - This is material that will be integrated back into the spec as an appendix, but this incorporates feedback on prior versions, and adds a 6-bit ascii encoding that is used by the same binary format standards as the 7-bit one. Works same way, just 6 bits not 7, so there's some codepoint changes. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 7

draft r20
by Mike Beckerle 01 Aug '14

01 Aug '14

I have posted draft r20 to redmine. - Resolved a few comment bubbles raised by steve h. - moved a few things from Appendix on standard charsets to glossary, improved examples there. - Changed header and footer to August 2014. - Added missing "Code Point" glossary entry. - Minor editorial fixes. I believe the only thing left is to get the bit-order materials finalized and in. One open: Steve H suggested appendix tables and figures should be numbered A.1, A.2, B.1, etc. I am uncertain how to do this, so r20 is content changes, if I can figure out how to do the numbering that will be another revision. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Thu, Jul 31, 2014 at 12:31 PM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Mike > > Reviewed draft r18, and created draft r19. A few editorial changes, plus > some editorial comments, plus a comment about the Glossary definition of > Bit Position for when bitOrder is added. I'll let you post to Redmine in > case you want to make any more edits. > > > > Regards > > Steve Hanson > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org>, > Date: 25/07/2014 02:34 > Subject: [DFDL-WG] draft r18 on redmine > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > I posted spec r18 to redmine. > > This incorporates all the errata as of the 1.5 experience doc 1. > > - One exception - the errata about implementation-defined and > implementation-dependent. I went through and updated all the places where > this was not correctly stated per Jonathan's document. > - However, I did not create an appendix also listing all these things. > That turns out to be hard to do, and redundant (you end up copying text > blocks - then they're in two places....) > - I think if we want an index to everyplace that > "implementation-defined" and "implementation-dependent" appear in the > document, that's what an index is for, so we should generate one and put > those in as indexed terms. > > I added the appendix of DFDL Standard Character Set Encodings, which > includes both 7-bit and 6-bit flavors now. > > I believe the last thing to do is bit order, but that should be reviewed > as a separate document first. > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > *www.tresys.com* <http://www.tresys.com/> > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the *OGF Intellectual Property Policy* > <http://www.ogf.org/About/abt_policies.php> > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >

2 1