- dfdl-wg - lists.ogf.org

Apache Daffodil (incubating) 2.6.0 Released
by Beckerle, Mike 24 Apr '20

24 Apr '20

The Apache Daffodil (incubating) community is pleased to announce the release of version 2.6.0. Notable changes in this release include performance and API updates, and the addition of Sonarqube analysis to our workflow. Detailed release notes and downloads are available at: https://daffodil.apache.org/releases/2.6.0/ Daffodil is an open source implementation of the Data Format Description Language (DFDL) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format. For more information about Daffodil visit: https://daffodil.apache.org/ Regards, The Apache Daffodil Team ===== *Disclaimer* Apache Daffodil is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 16 Apr '20

16 Apr '20

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Re: [DFDL-WG] Behavior of nilKind literalValue with respect to binaryNumberRep of packed
by Steve Hanson 15 Apr '20

15 Apr '20

Fill byte is not used for trimming, only padding (to the right). Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Bradd Kadlecik/Poughkeepsie/IBM To: Steve Hanson/UK/IBM Cc: DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org>, Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Date: 15/04/2020 17:53 Subject: Re: [EXTERNAL] Re: [DFDL-WG] Behavior of nilKind literalValue with respect to binaryNumberRep of packed Thanks, that makes sense. It is more a scenario hit here in test based on data from a customer. For background, the way their applications handle signed packed decimal numbers is there can be a value including 0 or unset (meaning no value exists). For a simple fixed length packed decimal of say 2 bytes it could have 0 (00 0C) or unset (00 0F). There is also some cases that these are variable in length and the TPF DFDL parser supports both DFDL expressions for "length" as well as delimited for "lengthKind". The idea of nil is very similar to the idea of unset and the only way to really do this for a packed decimal is to use literalValue since a signed decimal would understand a 0 to end in 0C instead of 0F. It also seemed that fillByte can be used with lteralValue if the literalValue is less than the simpleContent region (and not a string) as shown by: NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill So the question arose how one might indicate this unset state for variable length packed decimals. It does however seem reasonable that in such a case the length should be 0, literalCharacter is used, or the length should be no more than 1. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Date: 04/15/2020 10:51 AM Subject: Re: [EXTERNAL] Re: [DFDL-WG] Behavior of nilKind literalValue with respect to binaryNumberRep of packed There is no explicit control over justification and trimming for binary data. For a specific type of binary data, it is what it is. Packed decimals for example are always right-justified. I don't think interpreting x00x00x0F as a nil value is a good idea. Typically this is unsigned zero, but it is a valid number and not an out-of-type value. I can see that one might want to use xFFFFFFF or x000000 as nil, as these values are often blatted into storage by (eg COBOL) programs and both are out-of-type (although you can handle the latter as zero using dfdl:binaryPackedSignCodes property). The way you handle these as nil is using dfdl:nilLiteralCharacter, set to "%#rFF" or "%#r00" respectively, which handles the variable length. There is no way to provide a nil literal value for a variable length binary element, because no trimming takes place. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Bradd Kadlecik <braddk(a)us.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 14/04/2020 23:57 Subject: [EXTERNAL] Re: [DFDL-WG] Behavior of nilKind literalValue with respect to binaryNumberRep of packed Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Can you give what the bytes look like for typical values of various sizes small and large, how their length is determined, and what a nil value looks like in bytes? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Apr 14, 2020 at 4:54 PM Bradd Kadlecik <braddk(a)us.ibm.com> wrote: Yes that works for fixed length but not variable length which is possible for packed decimal with bigEndian. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States Mike Beckerle ---04/14/2020 03:56:45 PM---Not sure I understand the mixture of the concepts of justification and packed decimal here. From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 04/14/2020 03:56 PM Subject: [EXTERNAL] Re: [DFDL-WG] Behavior of nilKind literalValue with respect to binaryNumberRep of packed Not sure I understand the mixture of the concepts of justification and packed decimal here. I usually think of packed decimal as fixed length and without padding. Let me assume this example: 12345C is value 12345, 00000C is zero, and 00000F is the nil indicator. So, bigEndian byte order, I think dfdl:nilvalue="%#r00;%#r00;%#r0F;" is what I'd expect to see for a literalValue nilValue to match that. I'm guessing some assumption in the above doesn't match your use case, so please correct. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Apr 14, 2020 at 2:59 PM Bradd Kadlecik <braddk(a)us.ibm.com> wrote: I think there is a problem when the literalValue is left-justified for binary data such as packed decimals. This seems problematic because a "0" value might be indicated by having the last byte be 0x0C for a signed numeric while a nil value might be desired to be understood by having the last byte be a 0x0F. In both cases, all preceding bytes are 0x00. In the case that the packed decimal is of variable length, there seems no way to represent this nil value unless it is understood that the fillByte is used for the area preceding the NilElementLiteralContent. Apologies if I might of missed some clarification made regarding this. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 E-mail: braddk(a)us.ibm.com 2455 South Rd Poughkeepsie, NY 12601-5400 United States -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] Behavior of nilKind literalValue with respect to binaryNumberRep of packed
by Bradd Kadlecik 15 Apr '20

15 Apr '20

Thanks, that makes sense. It is more a scenario hit here in test based on data from a customer. For background, the way their applications handle signed packed decimal numbers is there can be a value including 0 or unset (meaning no value exists). For a simple fixed length packed decimal of say 2 bytes it could have 0 (00 0C) or unset (00 0F). There is also some cases that these are variable in length and the TPF DFDL parser supports both DFDL expressions for "length" as well as delimited for "lengthKind". The idea of nil is very similar to the idea of unset and the only way to really do this for a packed decimal is to use literalValue since a signed decimal would understand a 0 to end in 0C instead of 0F. It also seemed that fillByte can be used with lteralValue if the literalValue is less than the simpleContent region (and not a string) as shown by: NilElementLiteralContent = LeftPadding NilLiteralValue RightPadOrFill So the question arose how one might indicate this unset state for variable length packed decimals. It does however seem reasonable that in such a case the length should be 0, literalCharacter is used, or the length should be no more than 1. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org>, "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Date: 04/15/2020 10:51 AM Subject: Re: [EXTERNAL] Re: [DFDL-WG] Behavior of nilKind literalValue with respect to binaryNumberRep of packed There is no explicit control over justification and trimming for binary data. For a specific type of binary data, it is what it is. Packed decimals for example are always right-justified. I don't think interpreting x00x00x0F as a nil value is a good idea. Typically this is unsigned zero, but it is a valid number and not an out-of-type value. I can see that one might want to use xFFFFFFF or x000000 as nil, as these values are often blatted into storage by (eg COBOL) programs and both are out-of-type (although you can handle the latter as zero using dfdl:binaryPackedSignCodes property). The way you handle these as nil is using dfdl:nilLiteralCharacter, set to "%#rFF" or "%#r00" respectively, which handles the variable length. There is no way to provide a nil literal value for a variable length binary element, because no trimming takes place. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 Note: I work Tuesday to Friday From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Bradd Kadlecik <braddk(a)us.ibm.com> Cc: DFDL-WG <dfdl-wg(a)ogf.org> Date: 14/04/2020 23:57 Subject: [EXTERNAL] Re: [DFDL-WG] Behavior of nilKind literalValue with respect to binaryNumberRep of packed Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Can you give what the bytes look like for typical values of various sizes small and large, how their length is determined, and what a nil value looks like in bytes? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Apr 14, 2020 at 4:54 PM Bradd Kadlecik <braddk(a)us.ibm.com> wrote: Yes that works for fixed length but not variable length which is possible for packed decimal with bigEndian. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States Inactive hide details for Mike Beckerle ---04/14/2020 03:56:45 PM---Not sure I understand the mixture of the concepts of justifMike Beckerle ---04/14/2020 03:56:45 PM---Not sure I understand the mixture of the concepts of justification and packed decimal here. From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Bradd Kadlecik <braddk(a)us.ibm.com>, DFDL-WG <dfdl-wg(a)ogf.org> Date: 04/14/2020 03:56 PM Subject: [EXTERNAL] Re: [DFDL-WG] Behavior of nilKind literalValue with respect to binaryNumberRep of packed Not sure I understand the mixture of the concepts of justification and packed decimal here. I usually think of packed decimal as fixed length and without padding. Let me assume this example: 12345C is value 12345, 00000C is zero, and 00000F is the nil indicator. So, bigEndian byte order, I think dfdl:nilvalue="%#r00;%#r00;%#r0F;" is what I'd expect to see for a literalValue nilValue to match that. I'm guessing some assumption in the above doesn't match your use case, so please correct. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Tue, Apr 14, 2020 at 2:59 PM Bradd Kadlecik <braddk(a)us.ibm.com> wrote: I think there is a problem when the literalValue is left-justified for binary data such as packed decimals. This seems problematic because a "0" value might be indicated by having the last byte be 0x0C for a signed numeric while a nil value might be desired to be understood by having the last byte be a 0x0F. In both cases, all preceding bytes are 0x00. In the case that the packed decimal is of variable length, there seems no way to represent this nil value unless it is understood that the fillByte is used for the area preceding the NilElementLiteralContent. Apologies if I might of missed some clarification made regarding this. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Agenda for OGF DFDL WG Call
by Steve Hanson 15 Apr '20

15 Apr '20

Please find agenda for next call on Redmine at https://redmine.ogf.org/dmsf_files/13609?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Clarification: Unparser choice branch selection (section 15.1.3)
by Mike Beckerle 15 Apr '20

15 Apr '20

We have choices like this: <choice> <sequence> <sequence dfdl:hiddenGroupRef="PI_true"/> <sequence> <element name="foo" minOccurs="0" ...../> <element name="bar" minOccurs="0" ..../> </sequence> <sequence dfdl:hiddenGroupRef="PI_false"/> </choice> PI_true and PI_false are presence indicator flags. When unparsing, if you have element "foo" or "bar" in the infoset, then the first branch is selected, and the PI_true flag is unparsed. The problem is when neither "foo" nor "bar" is in the infoset. Note that both are optional above. In that case, we want the PI_false branch to be chosen. So point 1: The DFDL spec (section 15.1.3) is unclear about how a choice branch is chosen if it contains no visible element. Possible fix 1: One reasonable clarification might be that the first branch that admits no elements when unparsing would be chosen. Possible fix 2: Another reasonable clarification would be that a branch with no possible elements is preferred to branches that have possible elements. If there is more than one such branch, the first would be chosen. Maybe there are other possible fixes? Daffodil currently is implementing possible fix 2, but that's not necessarily right. It maintains backward compatibility with older daffodil-oriented DFDL schemas. If we decide Fix 1 is better, then we would have to put in a backward compatibility flag defaulting to true that enables users to continue to use schemas written as above, but we'd have to revise schemas like the above to reverse the order of the branches, and eventually flip the state of the flag to require use of these new reordered schemas. Thoughts? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

3 2

OGF DFDL Working Group Call
by Steve Hanson 15 Apr '20

15 Apr '20

1 0

Behavior of nilKind literalValue with respect to binaryNumberRep of packed
by Bradd Kadlecik 15 Apr '20

15 Apr '20

I think there is a problem when the literalValue is left-justified for binary data such as packed decimals. This seems problematic because a "0" value might be indicated by having the last byte be 0x0C for a signed numeric while a nil value might be desired to be understood by having the last byte be a 0x0F. In both cases, all preceding bytes are 0x00. In the case that the packed decimal is of variable length, there seems no way to represent this nil value unless it is understood that the fillByte is used for the area preceding the NilElementLiteralContent. Apologies if I might of missed some clarification made regarding this. Regards, Bradd Kadlecik z/TPF Development Phone: 1-845-433-1573 2455 South Rd E-mail: braddk(a)us.ibm.com Poughkeepsie, NY 12601-5400 United States

3 4

what data format was dfdl:floating intended to support?
by Mike Beckerle 14 Apr '20

14 Apr '20

I got asked this recently, and I know this property got added to DFDL for some particular cases, but I don't recall what format(s) they are, and if they are well-known, or something that was just encountered in some users' ad-hoc formats. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Owl Cyber Defense | www.owlcyberdefense.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

OGF DFDL WG Call 2020-04-02 cancelled
by Steve Hanson 02 Apr '20

02 Apr '20

Still busy from March. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0