- dfdl-wg - lists.ogf.org

OGF DFDL WG Call Agenda 2011-08-09 - Cancelled.
by Mike Beckerle 08 Aug '11

08 Aug '11

Due to a large number of WG members being unable to attend, the call is cancelled for Tomorrow August 9th, 2011. Mike Beckerle DFDL Work group Co-Chair Senior Technology Leader/Manager Deloitte Managed Analytics 100 Fifth Avenue, Waltham, MA 02451 Tel/Direct: +1 781 330 0412 | Fax: +1 866 253 3006 www.deloitte.com Please consider the environment before printing.

1 0

OGF DFDL WG Call Minutes 2011-07-26
by Steve Hanson 27 Jul '11

27 Jul '11

Please find minutes of the above meeting on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… No call next week due to holidays, next call 9th August. Regards Steve Hanson Architect, DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Cancelled: OGF DFDL Working Group weekly call
by Steve Hanson 27 Jul '11

27 Jul '11

Canceled as Steve and Mike are on holiday

1 0

[wg-all] Repeating groups in DFDL
by Ryan Farrell 27 Jul '11

27 Jul '11

For anyone that was at the 19/07 phone meeting, this is what Adam Fox and I were trying to find a solution to. If anyone knows of a solution, please contact me or adam.fox(a)nrl.navy.mil. ---------------------------------------------------------------------------------------------------- Please see the attached xml schema while reading this message. The problem we want to solve is how to represent a structure that can have a variable amount of occurences (even zero), and the amount of repeats is controlled by a one bit field, which we will call"repeatBit". The repeating structure in this case is "A1". The element before it, "presentBit", determines if there is at least one occurence of A1. If presentBit is zero, then A1 has zero occurences. If presentBit is one, then A1 has AT LEAST one occurence. The first element in A1 is "repeatBit". What repeatBit does is tells us if there will be another occurence of A1 after the current occurence. When repeatBit is zero, then that means we will read through the rest of A1, then that is the last occurence of A1. As long as repeatBit continues to be one however, we will read through the rest of A1, then start a new occurence. I have included all the known required DFDL notation. Please let me know what is missing. EXAMPLE INPUT (results are in base 10): NOTE: Not all bits are used in these examples. Unused bits should be ignored for the purpose of the examples. 1) 0111 0101 presentBit = 0 aFieldAfterA1 = 3 2) 1011 1010 presentBit = 1 A1 -repeatBit = 0 -someField = 3 aFieldAfterA1 = 2 3) 1101 0110 0000 0000 presentBit = 1 A1 -repeatBit = 1 -someField = 1 A1 -repeatBit = 0 -someField = 3 aFieldAfterA1 = 0

3 5

FW: US NRL's problem format
by Mike Beckerle 26 Jul '11

26 Jul '11

I think the array example from the NRL guys is not an uncommon pattern, and it really does need a general solution. It’s a variation on OCK parsed. I think your idea of a way to tell the parsed kind that you are on the last one is the right notion. occursCountKind=”parse” occursCountParseStop=”…expression...” Problem is for unparsing. We need to say how whatever it is we’re testing gets assigned. This could be done by outputValueCalc expressions. Something like this perhaps: <element name=”nrlArray” maxOccurs=”unbounded” dfdl:occursCountKind=”parse” dfdl:occursCountParseStop=”{ ./isLastFlag = 0 }”> <sequence> <element name=”isLastflag” type=”int” dfdl:outputValueCalc=”{ if (./count() = ./position()) 0 else 1 }“ /> <element name=”field1” type=”int”/> <element name=”field2” type=”int”/> </sequence> </element> From: Steve Hanson [mailto:smh@uk.ibm.com] Sent: Wednesday, July 20, 2011 5:56 AM To: Mike Beckerle Subject: RE: US NRL's problem format Hi Mike Using a separate scalar last element runs into a UPA violation, because the array element and the scalar last element have the same name. If the scalar last element had a different name this would work. I think we have to be careful with occursStopValue. The property today only works for arrays of simple elements because we need a logical type, and it does not put the stop value in the infoset. With NRL's format, not only is the element complex, but the last element is intended to go into the infoset. What is needed is an ability to assert 'I am the last item in an array' and for that to stop occursCountKind 'parsed' from looking any further, I think we should be able to do this with variables. A variable is defined with a default value of (in this case) 1. The array element carries a discriminator that fails if the variable is 0. The trick is to place the setVariable on the child of the array element which contains the last indicator, so the next time round, the discriminator fails. There is one problem with this - it violates variable rules, as (1) a variable can not be set multiple times, and (2) it can't be set after it has been tested. (Even adding a test attribute to setVariable, whereby it is only set when the test expression evaluates to true, still violates (2) ). <element name="data"> <dfd;defineVariable name="last" type="int" defaultValue="1" /> <complexType> <sequence> <element name="array"> <dfd;discriminator test="{$last eq 0 /> <complexType> <sequence> <element name="lastIndicator" type="int"> <dfdl:setVariable ref="last"value="{.}" test="{. eq 0}"/> </element> .... </sequence> </complexType> </element> </sequence> </complexType> </element> A more convenient mechanism is to add a new property occursParsedLast which is an expression (able to look downwards) that returns a boolean and which is tested after each item in the array has been parsed. An aside ... if lengthUnits is 'bits' we could have lengthKind 'pattern' interpret the regexp as if each bit was a character 0 or 1. Then my pattern idea below would work. Could be useful for bit oriented formats? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, <http://www.ogf.org/dfdl/> OGF DFDL Working Group IBM SWG, Hursley, UK <mailto:smh@uk.ibm.com> smh(a)uk.ibm.com tel:+44-1962-815848 From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 19/07/2011 18:32 Subject: RE: US NRL's problem format _____ To clarify: I think we can handle this, but it requires us to model it as an array of the not-last element, followed by a separate scalar last element. Then input value calc could put it all into an array, and you could tease this back apart with output value calc on output. The problem…. This is too simple a concept to require that much work. (It might make a good example of input value calc/output value calc J We’re so close with stopValue already… it seems like a small miss. ----- Forwarded by Steve Hanson/UK/IBM on 20/07/2011 09:34 ----- From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 19/07/2011 18:26 Subject: RE: US NRL's problem format _____ I thought of a choice that disambiguates on the flag, but you’ve identified what doesn’t work on that below. The scheme is a variation on stop value as a concept. I think we need property dfdl:stopValuePath=”…path expression…” which says where (relative to the array element, i.e., inside it) you find the stop value. This would be inverted for output, so can’t be a complex expression, simply must be a path. …mikeb From: Steve Hanson [ <mailto:smh@uk.ibm.com> mailto:smh@uk.ibm.com] Sent: Tuesday, July 19, 2011 1:06 PM To: mbeckerle.dfdl(a)gmail.com Subject: US NRL's problem format Hi Mike For a format like... ......................1xxxxxxxxxxxxxxxxxxxxxxx1xxxxxxxxxxxxxxxxxxxx0xxxxxxxxxxxxxxxxxxx................ where the xxxx are all the same length n, you could use lengthKind="pattern" with a regexp that looked for any number of n+1 length bytes starting with a 1, followed by exactly one n+1 length bytes starting with a 0. You could place an assert on the 1/0 element that said it could either be 0 or 1, but that only works if the next things can't be 0 or 1 in that position. However, I have a feeling that the NRL's data was at the bit-level which would not allow either of these anyway. What was your model? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, <http://www.ogf.org/dfdl/> OGF DFDL Working Group IBM SWG, Hursley, UK <mailto:smh@uk.ibm.com> smh(a)uk.ibm.com <tel:+44-1962-815848> tel:+44-1962-815848 _____ Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _____ Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Variable occurrence of variable/fields
by Michele Zundo 26 Jul '11

26 Jul '11

Dear all, my 2 cents on variable occurrence fields. We have been using XML syntax (in now defunct BinX format) to describe the structure of ESA Earth Observation SMOS Mission Science binary data Product. (see http://www.smos.com.pt/) We also developed some custom C/C++ library to allow Read/Write access to these binary data as we felt this would allow to easily modify the binary structure of the data product without requiring application code changes but only XML "schema". One of the major limitation we have found in BinX (which I'm not sure has been solved in DFDL) is indeed related to describing a binary structure with variable number of records where the number is described (dynamically) within the binary itself. We had to define a pre-processor of the binary data which would read each instance of the file find out how many of the records we had and create at run-time a XML schema with the correct number of replication of the XML code describing each of the records. This is very inconvenient and un-elegant. For example a binary data Product with a variable number of scientific observation would be (symbolically) structured as: Number_of_Records Data_Record_1 (blah, blah, etc etc ) Data_Record_2 ((blah, blah, etc etc) ..... Record_N (with N= Number_of_Records ) It would be desirable to be able to specify such a data product like: Number_of_Records List (count=Number_of_Records) of Data_Record The other feature which we greatly missed is bit level operation since many of our Space data structure are formatted at bit level e.g. 3 bits field + 2 bit field + 11 bit field etc. Regards > > From: Ryan Farrell <ryan.farrell.ctr(a)nrl.navy.mil> > Date: July 19, 2011 20:12:05 GMT+02:00 > To: dfdl-wg(a)ogf.org > Subject: [DFDL-WG] [wg-all] Repeating groups in DFDL > > > For anyone that was at the 19/07 phone meeting, this is what Adam Fox and I were trying to find a solution to. If anyone knows of a solution, please contact me or adam.fox(a)nrl.navy.mil. > > ---------------------------------------------------------------------------------------------------- > > Please see the attached xml schema while reading this message. > > The problem we want to solve is how to represent a structure that can have a variable amount of occurences (even zero), and the amount of repeats is controlled by a one bit field, which we will call"repeatBit". The repeating structure in this case is "A1". The element before it, "presentBit", determines if there is at least one occurence of A1. If presentBit is zero, then A1 has zero occurences. If presentBit is one, then A1 has AT LEAST one occurence. > > The first element in A1 is "repeatBit". What repeatBit does is tells us if there will be another occurence of A1 after the current occurence. When repeatBit is zero, then that means we will read through the rest of A1, then that is the last occurence of A1. As long as repeatBit continues to be one however, we will read through the rest of A1, then start a new occurence. > > I have included all the known required DFDL notation. Please let me know what is missing. > > EXAMPLE INPUT (results are in base 10): > NOTE: Not all bits are used in these examples. Unused bits should be ignored for the purpose of the examples. > > 1) 0111 0101 > presentBit = 0 > aFieldAfterA1 = 3 > > 2) 1011 1010 > presentBit = 1 > A1 > -repeatBit = 0 > -someField = 3 > aFieldAfterA1 = 2 > > 3) 1101 0110 0000 0000 > presentBit = 1 > A1 > -repeatBit = 1 > -someField = 1 > A1 > -repeatBit = 0 > -someField = 3 > aFieldAfterA1 = 0 > <repeat_example.xsd> > > > From: Steve Hanson <smh(a)uk.ibm.com> > Date: July 20, 2011 10:54:27 GMT+02:00 > To: Ryan Farrell <ryan.farrell.ctr(a)nrl.navy.mil> > Cc: dfdl-wg(a)ogf.org > Subject: Re: [DFDL-WG] [wg-all] Repeating groups in DFDL > > > > A couple of clarifications if I may.... > > - In your example, you have someField as being of simple type, can it be complex type in your real format? > > - You said "If presentBit is zero, then A1 has zero occurences". But in your xsd you have A1 as (implied) minOccurs="1". I think you meant minOccurs="0"? > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, OGF DFDL Working Group > IBM SWG, Hursley, UK > smh(a)uk.ibm.com > tel:+44-1962-815848 > > > From: Ryan Farrell <ryan.farrell.ctr(a)nrl.navy.mil> > To: dfdl-wg(a)ogf.org > Date: 19/07/2011 19:12 > Subject: [DFDL-WG] [wg-all] Repeating groups in DFDL > Sent by: dfdl-wg-bounces(a)ogf.org > > > > > For anyone that was at the 19/07 phone meeting, this is what Adam Fox > and I were trying to find a solution to. If anyone knows of a solution, > please contact me or adam.fox(a)nrl.navy.mil. > > ---------------------------------------------------------------------------------------------------- > > Please see the attached xml schema while reading this message. > > The problem we want to solve is how to represent a structure that can > have a variable amount of occurences (even zero), and the amount of > repeats is controlled by a one bit field, which we will call"repeatBit". > The repeating structure in this case is "A1". The element before it, > "presentBit", determines if there is at least one occurence of A1. If > presentBit is zero, then A1 has zero occurences. If presentBit is one, > then A1 has AT LEAST one occurence. > > The first element in A1 is "repeatBit". What repeatBit does is tells us > if there will be another occurence of A1 after the current occurence. > When repeatBit is zero, then that means we will read through the rest of > A1, then that is the last occurence of A1. As long as repeatBit > continues to be one however, we will read through the rest of A1, then > start a new occurence. > > I have included all the known required DFDL notation. Please let me know > what is missing. > > EXAMPLE INPUT (results are in base 10): > NOTE: Not all bits are used in these examples. Unused bits > should be ignored for the purpose of the examples. > > 1) 0111 0101 > presentBit = 0 > aFieldAfterA1 = 3 > > 2) 1011 1010 > presentBit = 1 > A1 > -repeatBit = 0 > -someField = 3 > aFieldAfterA1 = 2 > > 3) 1101 0110 0000 0000 > presentBit = 1 > A1 > -repeatBit = 1 > -someField = 1 > A1 > -repeatBit = 0 > -someField = 3 > aFieldAfterA1 = 0 > [attachment "repeat_example.xsd" deleted by Steve Hanson/UK/IBM] -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > http://www.ogf.org/mailman/listinfo/dfdl-wg > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > > > > > > > > From: Steve Hanson <smh(a)uk.ibm.com> > Date: July 20, 2011 13:28:47 GMT+02:00 > To: dfdl-wg(a)ogf.org > Cc: DFDL-Technical-Core%IBMGB(a)uk.ibm.com > Subject: [DFDL-WG] OGF DFDL WG Call Minutes 2011-07-19 > > > > Please find minutes of the above meeting on GridForge at: > > http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… > > Regards > > Steve Hanson > Architect, DFDL > Co-Chair, OGF DFDL Working Group > IBM SWG, Hursley, UK > smh(a)uk.ibm.com > tel:+44-1962-815848 > > > > > > > From: "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> > Date: July 23, 2011 00:43:49 GMT+02:00 > To: "'Ryan Farrell'" <ryan.farrell.ctr(a)nrl.navy.mil>, <dfdl-wg(a)ogf.org> > Subject: Re: [DFDL-WG] [wg-all] Repeating groups in DFDL > > > Great example Ryan, > > We've spent quite a bit of time over the last several months on textual > formats with these kinds of characteristics, which I would describe as > "occurrances determined by parsing" - that is, you look at the data to > determine "is there another element or not" one by one as they are parsed. > > We need to make sure the properties are there to make this work for the > binary case as well. > > I actually believe we cannot handle this format right now. We don't have a > property which says to look at the current element to determine whether to > expect a subsequent element, as a way of determining occurrence counts. > > Next week we'll add resolving this example to the workgroup actions list. > > ...mikeb > > Mike Beckerle > Senior Technology Leader/Manager > Deloitte Managed Analytics > 100 Fifth Avenue, Waltham, MA 02451 > Tel/Direct: +1 781 330 0412 | Fax: +1 866 253 3006 > www.deloitte.com > > > -----Original Message----- > From: dfdl-wg-bounces(a)ogf.org [mailto:dfdl-wg-bounces@ogf.org] On Behalf Of > Ryan Farrell > Sent: Tuesday, July 19, 2011 2:12 PM > To: dfdl-wg(a)ogf.org > Subject: [DFDL-WG] [wg-all] Repeating groups in DFDL > > For anyone that was at the 19/07 phone meeting, this is what Adam Fox > and I were trying to find a solution to. If anyone knows of a solution, > please contact me or adam.fox(a)nrl.navy.mil. > > ---------------------------------------------------------------------------- > ------------------------ > > Please see the attached xml schema while reading this message. > > The problem we want to solve is how to represent a structure that can > have a variable amount of occurences (even zero), and the amount of > repeats is controlled by a one bit field, which we will call"repeatBit". > The repeating structure in this case is "A1". The element before it, > "presentBit", determines if there is at least one occurence of A1. If > presentBit is zero, then A1 has zero occurences. If presentBit is one, > then A1 has AT LEAST one occurence. > > The first element in A1 is "repeatBit". What repeatBit does is tells us > if there will be another occurence of A1 after the current occurence. > When repeatBit is zero, then that means we will read through the rest of > A1, then that is the last occurence of A1. As long as repeatBit > continues to be one however, we will read through the rest of A1, then > start a new occurence. > > I have included all the known required DFDL notation. Please let me know > what is missing. > > EXAMPLE INPUT (results are in base 10): > NOTE: Not all bits are used in these examples. Unused bits > should be ignored for the purpose of the examples. > > 1) 0111 0101 > presentBit = 0 > aFieldAfterA1 = 3 > > 2) 1011 1010 > presentBit = 1 > A1 > -repeatBit = 0 > -someField = 3 > aFieldAfterA1 = 2 > > 3) 1101 0110 0000 0000 > presentBit = 1 > A1 > -repeatBit = 1 > -someField = 1 > A1 > -repeatBit = 0 > -someField = 3 > aFieldAfterA1 = 0 > > > > > > From: Steve Hanson <smh(a)uk.ibm.com> > Date: July 26, 2011 11:58:59 GMT+02:00 > To: dfdl-wg(a)ogf.org > Subject: [DFDL-WG] OGF DFDL WG Call Agenda 2011-07-26 > > > > Please find agenda for the above call on GridForge at: > > http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… > > As per action 144 an errata to the spec has been created here: http://forge.gridforum.org/sf/go/doc16280?nav=1 > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, OGF DFDL Working Group > IBM SWG, Hursley, UK > smh(a)uk.ibm.com > tel:+44-1962-815848 > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > ----------------------------------------- Michele Zundo EOP-PEP Ground System Definition and Verification Office European Space Agency, ESTEC e-mail: michele.zundo(a)esa.int

2 2

OGF DFDL WG Call Agenda 2011-07-26
by Steve Hanson 26 Jul '11

26 Jul '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… As per action 144 an errata to the spec has been created here: http://forge.gridforum.org/sf/go/doc16280?nav=1 Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Minutes 2011-07-19
by Steve Hanson 20 Jul '11

20 Jul '11

Please find minutes of the above meeting on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… Regards Steve Hanson Architect, DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Fw: Spec issues from Mike Beckerle
by Steve Hanson 19 Jul '11

19 Jul '11

----- Forwarded by Steve Hanson/UK/IBM on 19/07/2011 15:43 ----- From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org Date: 27/06/2011 17:27 Subject: Spec issues from Mike Beckerle For discussion on DFDL WG calls. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 27/06/2011 17:26 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB Date: 15/06/2011 01:59 Subject: Issues to add to work items list Steve, Below is the list I have so far of items in the spec that go beyond just typos - where re-wording is required or advisable. All but the last of these is tied up with the length & delimiters issue, but separate of the "known to exist" vs. "missing" topic. I've not bothered to provide anything about the section on "known to exist" and such. That section is already the discussion of a work item/topic. The last one is just a nit about BOMs. ...mikeb -------------------------------------------------------------------------------------------- 5.2.2 MinLength, MaxLength These facets are used: When dfdl:lengthKind=”implicit”. In that case the length is given by the value of xs:maxLength. In this case minLength if specified is required to be equal to maxLength (schema definition error otherwise). For validation of variable length string elements. It is a processing error when a fixed-length string is found to have a number of characters not equal to the fixed number. For example, if a fixed-length string also has delimiters we might be able to successfully separate it from the surrounding elements depending on the delimiter specifications; however, if the length of the fixed-length string is not equal to the number specified as the fixed length then it is a processing error (not simply a validation error).[MB1] [MB1]Contradicts statement that scanning for delimiters is off.(Discussed where dfdl:lengthKind=’explicit’ is described) What is a fixed length string? Clearly if it has lengthKind=”explicit” it is fixed length. What if it has lengthKind=”implicit” and maxLength=”10”. Is that a fixed length string which shuts off delimiter scanning also? If so then this paragraph is erroneous and misleading. <SMH>Yes, Tim and I have noted that this paragraph needs revising depending on the outcome of action 139 <SMH> -------------------------------------------------------------------------------------- 9.2 DFDL Syntax Grammar Change to introduce concept of EnclosedItem or ChildItem (I used EnclosedItem below): Sequence = LeftFraming SequenceContent RightFraming SequenceContent = [ PrefixSeparator EnclosedItem [ Separator EnclosedItem ]* PostfixSeparator ] FinalUnusedRegion EnclosedItem = Element | Array | ComplexContent[MB1] Choice = LeftFraming ChoiceContent RightFraming ChoiceContent = [ EnclosedItem ] FinalUnusedRegion[MB2] [MB1]Refactored to share the EnclosedItem concept to Choices also. Should perhaps be named ChildItem. This is useful when discussing how parsing, defaulting, etc. work as well. <SMH>Seems fine, it is effectively just a renaming<SMH> --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Table 14 Implicit Alignment in bits Note: Specifying the implicit alignment in bits does not imply that dfdl:lengthUnits 'bits' can be specified for all simple types.[MB1] [MB1]I do not understand this comment. What exactly is the restriction? <SMH>It is really saying that alignmentUnits and lengthUnits are independent and have their own rules for when they are applicable. <SMH> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- lengthKind Enum Controls how the representation length of the component is determined. Valid values are: 'explicit', 'delimited', 'prefixed', 'implicit', 'pattern', 'endOfParent' A full description of each enumeration is given in the later sections. 'explicit' means the length of the item is given by the dfdl:length property 'delimited' means the item is delimited by a terminator or separator[MB1] ‘prefixed’ means the length of the item is given by an immediately preceding prefix field specified using prefixLengthType. ‘implicit means the length is to be determined in terms of the type of the element and its schema-specified properties if any. ‘pattern’ means the length of the item is given by a regular expression specified using the dfdl:lengthPattern property. ‘endOfParent’ means that the item is terminated by the termination of the containing construct. Annotation: dfdl:element, dfdl:simpleType [MB1]To me this is a very strong statement. It means that an outside-in parse is allowed where for a sequence, we can scan and determine its end, and then parse the children. It requires that “scan” is a well-defined concept for the contents of the sequence. It means there can be nothing inside which requires the suspension of scanning. It means contained elements that have length explicit and representation binary, are simply not allowed. <SMH>Not entirely true, we allow some binary types to have delimited lengthKind (eg, BCD and Packed Decimals). There are formats out there that require this. <SMH> To me it also means you cannot change the character set encoding, or have a contained element that itself uses an overlapping set of delimiters with the enclosing group’s delimiters. – Is that going too far? – Delimited is like pattern. It restricts what is in the data stream substantially. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The rules for resolving ambiguity between delimiters are: 1. When two delimiters have a common prefix, the longest delimiter has precedence. 2. When two delimiters have exactly the same value, the innermost (most deeply nested) delimiter has precedence. 3. When the separator and terminator on a group have the same value, the separator has precedence.[MB1] [MB1]By precedence, this must mean it is tried first, but the parser might backtrack and assume it to be a terminator instead. This seems problematic to me. I’d like to either rule this out and say they can’t be the same, or see a use case where this is needed, a backtracking parser can parse it, and there’s no more reasonable way to structure the schema. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 12.3.5.1 Pattern-Based Lengths - Scanability Any element (complex, simple text, simple binary) may have a dfdl:lengthKind 'pattern' as long as the bytes in the content region of the element are legal in the stated encoding of that element. Where a complex element has children with binary representation in practice this means an 8-bit ASCII encoding. [MB1] [MB1]Not necessarily ASCII. An 8-bit encoding such that any byte value is valid is the real requirement. The point is no single byte value is invalid, and no combinations of adjacent byte values are invalid so that any binary data won’t trip up a character conversion and subsequent scan. Hmmm. The 8-bit character set used must have a transformation into Unicode which is bijective and information preserving. I.e., a unique Unicode character for each code point, and no “invalid” chracters which have no corresponding Unicode value. However, Ascii-based sets like 8859-1 are not strictly speaking, required. ----------------------------------------------------------------------------------------------------------------------------------------------- 12.3.7.1.3 Byte Order Mark If a byte-order mark codepoint appears at the start of a UTF-8, [MB1] UTF-16 or UTF-32 encoded string then the byte-order mark will be included as part of the string payload[1]. That is, for the UTF-8, UTF-16 and UTF-32 character encodings, a byte-order-mark codepoint is treated as a character of the string in DFDL and contributes to the length. A way of eliminating the byte-order mark so that it does not end up in the infoset is that the byte-order mark can be modeled as a separate element before the string. This BOM element can be either required or optional depending on whether one is expected or optional at the beginning of the string. [1] Byte-order marks are explicitly stated to be “not characters” in the Unicode standard. [MB1]No such thing as a BOM codepoint in a UTF-8 string. A UTF-8 byte sequence might encode the character code for a BOM, but this would be a meaningless inclusion of a BOM character code in a context where it will never be interpreted. I suggest that we drop the term UTF-8 here, and BOM’s that get encoded when they are interpreted as character codes, and translated by the UTF-8 encoding algorithm into a multi-byte UTF-8 byte sequence, is handled the same way as other non-characters, i.e., what do we do when a high or low surrogate codepoint is present and we’re to encode as UTF-8. I think the answer is we run the UTF-8 encode/decode algorithm, and whatever Unicode character code it creates is what it creates, and if that happens to come out as any of the non-characters (BOMs, surrogates, others perhaps), so be it. The topic is about Unicode non-characters, not specifically BOMs. The general topic is encoding/decoding our infoset Unicode character codes which have no real representation in the specified encoding. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org http://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2011-07-19
by Steve Hanson 19 Jul '11

19 Jul '11

Please find agenda for the above call on GridForge at: http://forge.gridforum.org/sf/docman/do/downloadDocument/projects.dfdl-wg/d… As per action 144 an errata to the spec has been created here: http://forge.gridforum.org/sf/go/doc16280?nav=1 Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0