Open Grid Forum: Data Format Description Language Working Group
OGF DFDL Working Group Call, March 03-2010
Attendees
Suman Kalia (IBM)
Steve Hanson (IBM)
Alan Powell (IBM)
Steve Marting (Progeny)
Peter Lambros (IBM)
Stephanie Fetzer (IBM)
Tim Kimber(IBM)
Apologies
Mike Beckerle (Oco)
1. 16.2 scannablility with lengthKind pattern:
In summary, you can use a data pattern on any element (complex, simple
text, simple binary) as long as the bytes are legal in the stated
encoding, which where binary data is involved in practice means an 8-bit
ASCII encoding.
Binary data can be handled using some of the conveniences of text by way
of treating it as text with encoding="iso-8859-1". In this case literal
text, such as length patterns, is interpreted as in the iso-8859-1
character encoding, and the correspondence of byte values in the data to a
string in the DFDL infoset is one to one. That is, byte with value N,
produces an infoset character with character code N.
The WG agreed to the solution of recommending the use of "iso-8859-1" for
binary fields. The write-up should note that he main use case is for a
complex element that contains binary fields.
2. Current Actions:
Updated below
3 Steve H issues with draft 039
1) Name of property dfdl:textNumberRepresentation is not consistent with
dfdl:binaryNumberRep, dfdl:binaryFloatRep, etc.
Agreed
2) The dfdl:numberPattern etc properties that have been moved from the
defunct dfdl:textNumberFormat annotation to dfdl:element etc should be
called dfdl:textNumberPattern etc. Otherwise users will think they apply
to binary numbers too.
Agreed
3) In section 14.3 on sequences, there are several sub-sections that talk
about parsing according to different ways of specifying length (ie,
lengthKind). But dfdl:sequence no longer carries dfdl:lengthKind so I
think these sub-sections are not in the right place. I think they should
be in section 12, under the correct 12.3.x lengthKind sub-section.
MB has moved these sections
4) Section 19 on built-in specifications. Given that we don't have any for
public comment phase we should reword this section.
Agreed
4 Tim's (major) issues with draft 039
12.2 Delimiters: Text Markup
- The term 'Delimiters' is not accurate. Most readers will not think of
an initiator as a 'delimiter'.
- It's not 'Text' markup any more - especially since v0.39 has allowed
lengthKind="delimited" for elements with binary representation.
Title should be 'Markup' and explanation can then deal with what it really
is, rather than justifying the innaccurate title :-)
Will be changed to Delimiters - Add delimiters to glossary (DFDLlimits
delimiters to terminators and separator when dfdl:lengthKind=delimited)
Syntax for specifying markup:
It's not clear from this description that each item in the space-separated
list is a DFDL string literal.
Section 6.3.1 will be revised to separate string literals and dfdl
expressions etc.
If possible will add the list of property types (listof DFDL string
literals, etc) and update property tables
initiator ( and all other space-separated properties )
It is not clear whether the order of the space-separated properties
matters. Must the parser test them in the order in which they are
specified?
( Q: What if %ES; is the first in the list? )
The parser must look for the longest first. Unparsing will output the
first in the list.
The %ES entity should be not be allowed for initiator, separator and
terminator. Should probably be limited to use in nilValues but will check
whether it is useful in other properties..
occursStopValue should a value in the value space of the base type of the
array.
terminator:
is it OK if the final terminator is missing within the scope of a
known-length parent? Seems like a reasonable extension of the rule ( in
all other scenarios, the end of a known-length parent acts like the end of
the data stream for items with its scope ).
documentFinalTerminatorCanBeMissing:
Let's try to avoid creating another property for the postfix separator
scenario. I think this property provides a way of modelling the data
naturally.
We can recommend use of infix-with-a-terminator rather than 'postfix' if
the final terminator can be missing.
outputNewLine
Should we validate that the 'characterOrCharacters' are all newline
characters from the set described by the %NL; mnemonic? Otherwise the DFDL
serializer will output data which cannot be parsed by the DFDL parser.
Agreed
dfdl:lengthKind endOfParent
'endOfParent' has almost the same meaning as 'delimited' so should have
the same semantics.
· the item?s terminator (if specified)
· an enclosing construct?s separator or terminator
· the end of an enclosing construct designated by its known length
· the end of the data stream
The effect would be the the element could be ended by the nearest known
length parent not just the immediate parent. Also the immediate parent
could have lengthKind 'implicit'
Not discussed but emails indicate Agreed
choiceKind 'Fixed'
When lengthKind='implicit' all alternative branches of the choice are
padded to the fixed length of the largest one so that overall the entire
choice construct is fixed length
There must be a restriction that the length of at least one choice must be
statically defined.
Not discussed but emails indicate Agreed
5 DFDL v1 Specification completion.
Draft 40 will be published early next week. Will be sent to OGF next week
prior to OGF 28
Meeting closed, 14:10
Next call Wednesday 10 March January 2010 13:00 UK
Next action: 084
Actions raised at this meeting
No
Action
Current Actions:
No
Action
066
Investigate format for defining test cases
25/11:IBM to see if it is possible to publish its test case format.
04/12: no update
09/12: no update
16/12: reminded dent to project manager
23/12: SH will send another reminder.
06/01: Another reminder will be sent
13/01: no update
20/01: no update
27/01: no progress
29/01: no progress
03/02: IBM is still investigating
10/02: IBM is still investigating
17/02: IBM is willing in principle to publish the test case format and
some of the test cases. May need some time to build a 'compliance suite'
24/03: No progress
03/03: Discussions have been taking place on the subset of tests that will
be provided.
080
AP:Clarify semantics of fn:poisition and fn:count
17/02: no progress
24/03: No progress
03/03: no progress. There are other functions which return a duration
which need investgating
Closed actions
No
Action
049
20/05 AP Built-in specification description and schemas
03/06: not discussed
24/06: No Progress
24/06: No Progress (hope to get these from test cases)
15/07: No progress. Once available, the examples in the spec should use
the dfdl:defineFormat annotations they provide.
...
14/10: no progress
21/10: Discussed the real need for this being in the specification. It
seemed that the main value is it define a schema location for downloading
'known' defaults from the web.
28/10: no progress
04/11: no progress
11/11: no update
18/11: no update
25/11: Agreed to try to produce for CSV and fixed formats
04/12: no update
09/12: no update
16/12: no update
23/12: no update
06/01: no progress. If there is no resource to complete this action it can
be deferred
13/01:no progress
20/01: no progress
27/01: no progress
29/01: No progress. The predefined formats do not need to be available
when the spec is published.
Suman said that he had been mapping COBOL structures to DFDL and it didn't
look as though the way text numbers are define is very usable. He will
document for next call
03/02: No progress
10/02: No progress
17/03: No progress
24/03: No progress
03/03: The wording of this section will be changed to say that
'implementations may choose to provide predefined formats. The DFDL WG
intends to supply a limited set'
Closed
079
MB:Encoding for binary fields when lenghtkind is pattern
17/02: Discussed but no conclusion
24/03: Mike has found an encoding that matches the first 255 codepoints of
iso 10646. Will document its use for binary fields.
03/03: Wording in minutes agreed.
Closed
083
MB:To correct syntax diagram for FinalUnused and suggest wording for the
Sequence section
03/03: Mike has supplied updates to Sytax, length and Sequence sections.
Closed
Work items:
No
Item
target version
status
005
Improvements on property descriptions
not started
012
Reordering the properties discussion: move representation earlier, improve
flow of topics
not started
036
Update dfdl schema with change properties
ongoing
042
Mapping of the DFDL infoset to XDM
none
not required for V1 specification
070
Write DFDL primer
071
Write test cases.
083
Implement RFC2116
084
MB:Encoding for binary fields when lenghtkind is pattern
085
MB:To correct syntax diagram for FinalUnused and suggest wording for the
Sequence section
086
dfdl:textNumberRepresentation to dfdl:textNumberRep
087
dfdl:NumberPattern etc to dfdl:textNumberPattern (not for calendarformat
but check individual properties)
088
Update built-in specifications section.
089
12.2 Will be changed to Delimiters - Add delimiters to glossary
(DFDLlimits delimiters to terminators and separator when
dfdl:lengthKind=delimited)
090
Section 6.3.1 will be revised to separate string literals and dfdl
expressions etc.
If possible will add the list of property types (listof DFDL string
literals, etc) and update property tables
091
The parser must look for the longest first. Unparsing will output the
first in the list.
The %ES entity should be not be allowed for initiator, separator and
terminator. Should probably be limited to use in nilValues but will check
whether it is useful in other properties..
092
occursStopValue should a value in the value space of the base type of the
array.
093
outputNewLine MUST be one of newline characters from the set described by
the %NL; mnemonic
094
dfdl:lengthKind endOfParent
'endOfParent' has almost the same meaning as 'delimited' so should have
the same semantics.
· the item?s terminator (if specified)
· an enclosing construct?s separator or terminator
· the end of an enclosing construct designated by its known length
· the end of the data stream
The effect would be the the element could be ended by the nearest known
length parent not just the immediate parent. Also the immediate parent
could have lengthKind 'implicit'
095
ChoiceKinf fixed:There must be a restriction that the length of at least
one choice must be statically defined.
096
The WG agreed to the solution of recommending the use of "iso-8859-1" for
binary fields. The write-up should note that he main use case is for a
complex element that contains binary fields.
Regards
Alan Powell
Development - MQSeries, Message Broker, ESB
IBM Software Group, Application and Integration Middleware Software
-------------------------------------------------------------------------------------------------------------------------------------------
IBM
MP211, Hursley Park
Hursley, SO21 2JN
United Kingdom
Phone: +44-1962-815073
e-mail: alan_powell(a)uk.ibm.com
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
