- dfdl-wg - lists.ogf.org

Clarification needed: textNumberFormat quoting and prefix/suffix
by Mike Beckerle 18 Jan '23

18 Jan '23

I created an ICU DecimalFormat string where the positive sign is the word POSITIVE, and the negative sign is the word NEGATIVE. Per unit test below, this works in ICU, so long as they are escaped, because "E" is a special character in the ICU pattern language. But... I can find no example of this online for ICU nor DFDL. All examples are of a sign being just a single character, or a pair of single characters before and after for negative values. So my question: Do we intend DFDL to support this full generality, or only single characters for the prefix/suffix regions of the textNumberPattern? Right now I think Daffodil requires the sign to be a single character, which can be one of the special characters only by quoting it individually. E.g., if I wanted the positive sign to be P and negative sign to be N, then I'd use textNumberPattern: 'P'#0.0#;'N'#0.0# I don't have to quote the N, but I did just for consistency. If we want the full generality, I'd appreciate a realistic motivating example of it. /** * This test shows that quoting of characters in ICU text number patterns * and hence DFDL text number patterns can quote strings, not just individual * characters. * * I found no examples of this anywhere. Everything shows patterns where * quotes surround only single characters, or are doubled up for self quoting as in * the o''clock example. */ @Test def testICUDecimalFormatQuoting_01(): Unit = { { // Note that E is a pattern special character val pattern = "'POSITIVE' #.0###;'NEGATIVE' #.0###" val df = new DecimalFormat(pattern) val actual = df.format(-6.847) assertEquals("NEGATIVE 6.847", actual) } { // here we quote only the E, not the other characters. val pattern = "POSITIV'E' #.0###;NEGATIV'E' #.0###" val df = new DecimalFormat(pattern) val actual = df.format(6.847) assertEquals("POSITIVE 6.847", actual) } } Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

Clarify/Simplify - description of textNumberPattern with V, P
by Mike Beckerle 05 Jan '23

05 Jan '23

I am implementing dfdl:textNumberPattern characters V and E for Apache Daffodil. The DFDL Spec says (Section 13.6.1.1) It is a Schema Definition Error if any symbols other than "0", "1" through "9" or # are used in the vpinteger region of the pattern. Note that the vpitneger region of the pattern excludes the leading and/or trailing sign indicator characters which are the prefix and suffix. The prefix and suffix can contain any character except the special characters of the pattern language. Later in the section the spec has a number of detailed statements that are subsumed by the above I believe. - A pattern with a V symbol must not have @ or * symbols. - A pattern with P symbols must not have @ or E or * symbols. So those latter 2 statements could be omitted since @ and * are "special characters" so aren't allowed in the prefix/suffix, and per above stipulation can't be in the vpinteger region either. Can't we simplify and just say a pattern with V or P cannot have any special characters other than 0-9 and # ? Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2

OGF DFDL Working Group Call
by Steve Hanson 05 Jan '23

05 Jan '23

Calls for 2023 OGF DFDL WG call dial-in details. https://ibm.webex.com/meet/smh OGF DFDL Home: http://www.ogf.org/dfdl GitHub DFDL: https://github.com/OpenGridForum/DFDL DFDL Getting Started: https://community.ibm.com/community/user/middleware/viewdocument/get-starte… DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Daffodil Apache incubator: https://daffodil.apache.org/ IBM DFDL: https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.etools… Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

1 0

OGF DFDL Working Group Call
by Steve Hanson 05 Jan '23

05 Jan '23

Calls for 2023 OGF DFDL WG call dial-in details. https://ibm.webex.com/meet/smh OGF DFDL Home: http://www.ogf.org/dfdl GitHub DFDL: https://github.com/OpenGridForum/DFDL DFDL Getting Started: https://community.ibm.com/community/user/middleware/viewdocument/get-starte… DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Daffodil Apache incubator: https://daffodil.apache.org/ IBM DFDL: https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.etools… Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

1 0

OGF DFDL Working Group Call
by Steve Hanson 05 Jan '23

05 Jan '23

Calls for 2023 OGF DFDL WG call dial-in details. https://ibm.webex.com/meet/smh OGF DFDL Home: http://www.ogf.org/dfdl GitHub DFDL: https://github.com/OpenGridForum/DFDL DFDL Getting Started: https://community.ibm.com/community/user/middleware/viewdocument/get-starte… DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Daffodil Apache incubator: https://daffodil.apache.org/ IBM DFDL: https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.etools… Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

1 0

OGF DFDL Working Group Call
by Steve Hanson 05 Jan '23

05 Jan '23

Calls for 2023 OGF DFDL WG call dial-in details. https://ibm.webex.com/meet/smh OGF DFDL Home: http://www.ogf.org/dfdl GitHub DFDL: https://github.com/OpenGridForum/DFDL DFDL Getting Started: https://community.ibm.com/community/user/middleware/viewdocument/get-starte… DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Daffodil Apache incubator: https://daffodil.apache.org/ IBM DFDL: https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.etools… Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

1 0

OGF DFDL Working Group Call
by Steve Hanson 05 Jan '23

05 Jan '23

Calls for 2023 OGF DFDL WG call dial-in details. https://ibm.webex.com/meet/smh OGF DFDL Home: http://www.ogf.org/dfdl GitHub DFDL: https://github.com/OpenGridForum/DFDL DFDL Getting Started: https://community.ibm.com/community/user/middleware/viewdocument/get-starte… DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Daffodil Apache incubator: https://daffodil.apache.org/ IBM DFDL: https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.etools… Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

1 0

OGF DFDL Working Group Call
by Steve Hanson 05 Jan '23

05 Jan '23

Calls for 2023 OGF DFDL WG call dial-in details. https://ibm.webex.com/meet/smh OGF DFDL Home: http://www.ogf.org/dfdl GitHub DFDL: https://github.com/OpenGridForum/DFDL DFDL Getting Started: https://community.ibm.com/community/user/middleware/viewdocument/get-starte… DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Daffodil Apache incubator: https://daffodil.apache.org/ IBM DFDL: https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.etools… Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

1 0

OGF DFDL Working Group Call
by Steve Hanson 04 Jan '23

04 Jan '23

Calls for 2023 OGF DFDL WG call dial-in details. https://ibm.webex.com/meet/smh OGF DFDL Home: http://www.ogf.org/dfdl GitHub DFDL: https://github.com/OpenGridForum/DFDL DFDL Getting Started: https://community.ibm.com/community/user/middleware/viewdocument/get-starte… DFDL 1.0 Spec: http://www.ogf.org/documents/GFD.207.pdf Daffodil Apache incubator: https://daffodil.apache.org/ IBM DFDL: https://www.ibm.com/support/knowledgecenter/en/SSTTDS_11.0.0/com.ibm.etools… Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

1 0

interesting format: RINEX
by Mike Beckerle 04 Jan '23

04 Jan '23

I ran into an interesting format today called RINEX that is problematic for DFDL v1.0: http://acc.igs.org/misc/rinex304.pdf This format has a multi-line header with various kinds of header lines which are 80 characters, but characters 1 to 60 are data, and 61 to 80 are the "label", and you need the label in order to know how to parse the data of each header line. This either requires deep (and slow) speculation, or some sort of look ahead feature. E.g., See page 69 of that spec document for an example. The headers appear not only at the start of the file, but at the start of each data block. We have run into the need for short, fixed-distance look ahead before in other formats. In the case of RINEX the lookahead is exactly 60 characters. In other formats it is also a very short distance, but one which is tricky to figure out as the spec doesn't say exactly how big the region to be looked past is. One would have to figure it out from the data format spec. It's always a constant of course. Mike Beckerle Apache Daffodil PMC | daffodil.apache.org OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl Owl Cyber Defense | www.owlcyberdefense.com

2 2