- dfdl-wg - lists.ogf.org

Example format with bitOrder leastSignificantBitFirst, but byteOrder bigEndian
by Mike Beckerle 11 Jan '18

11 Jan '18

This format, which is a envelope format for an envelope-and-payload style of data, describes data as bigEndian, but LSBF. https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?Comman… Thanks to Marshall Miller of Quark Security for this insight. This extracted from the dev(a)daffodil.apache.org mailing list. One can workaround this problem in Daffodil by way of dfdl:inputValueCalc/dfdl:outputValueCalc and hidden groups. But that is, perhaps, unsatisfying. Example: Data is comprised of 3 unsigned integer bitfields: slot1 is 12 bits slot2 is 32 bits slot3 is 20 bits Given an input with the following bytes in hex: 01 23 45 67 89 AB CD EF Parsed results need to be (in hex): slot1: 0567 slot2: DEF01234 slot3: 089ABC Explanation of results: Since the bit fields need to consume data in a leastSignificantBitFirst order, it is easier to represent the input in right-to-left order where bits are consumed starting at the right and moving toward the left. Since words are 4 bytes, and in big endian order, the lowest bit in the 4th byte is the first bit to be consumed. Reordering the input data and consuming data from right to left yields the correct field boundaries. 89 AB CD EF 01 23 45 67 parse fields right to left 05 06 07 08 01 02 03 04 input byte position slot1 consumes the right 3 nibbles, slot2 consumes the next 8 nibbles, and slot3 consumes the left 5 nibbles. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 09 Jan '18

09 Jan '18

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Agenda for next OGF DFDL WG Call
by Steve Hanson 09 Jan '18

09 Jan '18

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13570?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group Call
by Steve Hanson 08 Jan '18

08 Jan '18

Calls for 2018. OGF DFDL WG weekly call dial-in details. Passcode for Participants: 5381214 Canada Toll-Free 888-426-6840 China Toll-Free 10-800-711-1071 CHINA NETCOM GROUP USERS China Toll-Free 10-800-110-0996 CHINA TELECOM SOUTH USERS France Toll-Free 0800-94-0558 Germany Toll-Free 0800-000-1018 India Toll-Free 000-800-100-1176 Ireland Toll-Free 1-800-943-427 Israel Toll-Free 1-809-417-783 Netherlands Toll-Free 0-800-363-6036 United Kingdom Caller Paid 0-20-30596451 United Kingdom Toll-Free 0800-368-0638 USA Caller Paid 215-861-6239 USA Toll-Free 888-426-6840 Other international numbers available - e-mail smh(a)uk.ibm.com. OGF DFDL Home: http://www.ogf.org/dfdl Redmine DFDL: http://redmine.ogf.org/projects/dfdl-wg DFDL Getting Started: http://www.ibm.com/developerworks/library/se-dfdl/ DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf

1 0

Latest OGF DFDL WG Call Minutes
by Steve Hanson 12 Dec '17

12 Dec '17

Please find minutes from the latest call at https://redmine.ogf.org/projects/dfdl-wg/news Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Agenda for next OGF DFDL WG Call
by Steve Hanson 12 Dec '17

12 Dec '17

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13569?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group call
by Steve Hanson 12 Dec '17

12 Dec '17

1 0

Fw: OPC-UA DateTime values
by Steve Hanson 12 Dec '17

12 Dec '17

Requirement to allow DFDL to understand Windows FILETIME type as well as Unix time type. Although DFDL allows the epoch to be specified via dfdl:binaryCalendarEpoch, it can only handle seconds or milliseconds. It would need to add a new binaryCalendarRep 'binaryTick' to handle the '100 nanoseconds' units. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 ----- Forwarded by Steve Hanson/UK/IBM on 12/12/2017 09:48 ----- From: Andreas Martens1/UK/IBM To: Steve Hanson/UK/IBM@IBMGB Date: 08/12/2017 17:48 Subject: OPC-UA DateTime values Hi Steve, I'm working on processing OPC-UA DateTime elements, which are specified by: "A 64-bit signed integer representing the number of 100 nanoseconds intervals since 1601-01-01 00:00:00. This is the same as the WIN32 FILETIME type." (From OPC UA Part 3 - Address Space Model 1.03 Specification, available from opcfoundation.org) Now they might have been a bit conservative on the truth, here's Micosoft's specification: https://msdn.microsoft.com/en-us/library/windows/desktop/ms724284(v=vs.85).… Which doesn't mention whether it's signed or not (in fact, it's split in two parts, potentially to maintain compatibility backwards (have thay always used 1601 as their epoch?)) typedef struct _FILETIME { DWORD dwLowDateTime; DWORD dwHighDateTime; } FILETIME, *PFILETIME; actually, DWORD is unsigned, so it's definitely unsigned, so OPC foundation are liars! cheers, Andreas -- Andreas Martens Emerging Technology IBM Research IBM Hursley Park, Winchester, SO21 2JN Phone: +44 (0) 1962 816833 37246833 www.emerging-technology.co.uk Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] how to trim inside of escape block?
by Steve Hanson 22 Nov '17

22 Nov '17

Mike, your 2nd solution has a problem with the simple case when 3rd field is a single unquoted item - the List element will consume C and D - unless you can put a discriminator on C to stop it being parsed as a List - then it would work. This even simpler solution suffers from the same problem but can also be cured with a discriminator on C. <xs:sequence dfdl:separator=', ,%WSP*;" "%WSP*;,'> <xs:element name="A" type="xs:string"/> <xs:element name="B" type="xs:string"/> <xs:sequence dfdl:separator=","> <xs:element name="List" type="xs:string" maxOccurs="unbounded"/> </xs:sequence> <xs:element name="C" type="xs:string"/> <xs:element name="D" type="xs:string"/> </xs:sequence> An alternative would be to parse the 3rd field as a string with the " as escape block start/end, but make it hidden, then use an inputValueCalc expression to split the string and assign each chunk to an occurrence of List. But that would mean relaxing the restriction that inputValueCalc is not allowed on arrays. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson <smh(a)uk.ibm.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 22/11/2017 17:41 Subject: Re: [DFDL-WG] how to trim inside of escape block? Another related problem: a, b, notList, c, d a, b, "list1, list2, list3",c,d Here the 3rd field is a list, comma separated. Quoted if there is more than one list item. I think to parse this I have to treat the quotation marks as initiator/terminator, and set dfdl:separator="", but since the quotes are optional for the single-list-item case, I'm going to need a choice. I think the best I can do is <ignore:ListOf1__XMLSchemaMakesMeHaveThisForUPA/> <List>notList</List> and <ignore:ListOfN__XMLSchemaMakesMeHaveThisForUPA/> <List>list1</List><List>list2</List><List>list3</List> as the XML representations. Are there any better/cleaner solutions? I did think of this way: (note: I've omitted xs:annotation and xs:appinfo for brevity), but it isn't exactly "clean". This is what I call "modeling syntax as data".... <dfdl:defineVariable name="foundOpenQuote" type="xs:boolean"/> <xs:group name="optionalOpenQuote"> <choice> <xs:sequence dfdl:initiiator='"'> <dfdl:setVariable ref="foundOpenQuote" value="{ fn:true() }"/> </xs:sequence> <xs:sequence dfdl:initiator=""/> </choice> </xs:group> <xs:group name="matchingCloseQuote"> <choice> <xs:sequence dfdl:terminator='"'> <dfdl:discriminator>{ $foundOpenQuote eq fn:true() }</dfdl:assert> </xs:sequence> <xs:sequence /> </choice> </xs:group> // The main sequence for the data would then have this as the list element: <xs:sequence> <dfdl:newVariableInstance ref="foundOpenQuote" defaultValue="false"/> <xs:sequence dfdl:hiddenGroupRef="optionalOpenQuote"/> <xs:sequence dfdl:separator=","> <xs:element name="List" type="xs:string" maxOccurs="unbounded"/> </xs:sequence> <xs:sequence dfdl:hiddenGroupRef="matchingCloseQuote"/> </xs:sequence> I'd try this out, except that we haven't got dfdl:newVariableInstance yet. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy On Wed, Nov 22, 2017 at 4:11 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: I don't think there is a way to achieve what you want. As you say, trimming pad chars takes precedence over applying escape scheme. I wondered if you could define the escapeBlockStart and End as "%WSP*; and %WSP*;" respectively but the white space entities are not allowed as escape character or in escape block start/end. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 22/11/2017 01:28 Subject: [DFDL-WG] how to trim inside of escape block? Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> I have a CSV file Some lines look like this a,b," started with spaces, appearing right after the escape block start ",c,d,e I reviewed the spec, and I see that pad characters appear outside of the quotation marks (escape block start/end). What I'm trying to do is remove the whitespace after the escape block start, and before the escape block end. This is just spurious whitespace, appears because some of these CSV files were edited by people. In my data the quoting characters are not always present. They are only there if a comma appears in the data string. Is there a technique for getting rid of the leading/trailing whitespace inside the escape block start/end that I have forgotten? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] how to trim inside of escape block?
by Steve Hanson 22 Nov '17

22 Nov '17

I don't think there is a way to achieve what you want. As you say, trimming pad chars takes precedence over applying escape scheme. I wondered if you could define the escapeBlockStart and End as "%WSP*; and %WSP*;" respectively but the white space entities are not allowed as escape character or in escape block start/end. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 22/11/2017 01:28 Subject: [DFDL-WG] how to trim inside of escape block? Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> I have a CSV file Some lines look like this a,b," started with spaces, appearing right after the escape block start ",c,d,e I reviewed the spec, and I see that pad characters appear outside of the quotation marks (escape block start/end). What I'm trying to do is remove the whitespace after the escape block start, and before the escape block end. This is just spurious whitespace, appears because some of these CSV files were edited by people. In my data the quoting characters are not always present. They are only there if a comma appears in the data string. Is there a technique for getting rid of the leading/trailing whitespace inside the escape block start/end that I have forgotten? ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ogf.org_mailman_li… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

2 1