April 2017 - dfdl-wg - lists.ogf.org

clarification needed: unparsing and hidden group choices and choices with dfdl:choiceDispatchKey
by Mike Beckerle 13 Dec '19

13 Dec '19

So interesting issue. Comes up as part of our experience implementing the advanced expression language features: hidden groups, and computed elements. We have hidden groups in DFDL. If a xs:choice is inside a hidden group, then by definition, when unparsing, elements corresponding to some branch of the choice are not in the Infoset. DFDL spec clearly states that dfdl:outputValueCalc is not allowed on the head of a choice branch. But elements inside hidden groups must have dfdl:outputValueCalc, or a default value, otherwise there's no way for them to be filled in as part of the augmented infoset. So the only way a choice can be inside a hidden group is if the head of the choice branches have default values. This is not sufficient. I do not think it was our intention to make hidden groups incapable of holding things complex enough to contain a choice. Rather, the point is to be able to hide things that have excess complexity needed only for parsing/unparsing, so that they need not appear in the final Infoset. So if a choice is allowed inside a hidden group, then a mechanism is needed by which when unparsing a choice branch is selected based on values of other parts of the Infoset, and then the elements of that branch are computed and added to the augmented infoset. Note that without hidden groups, this problem never arises, as the Infoset always has to take a position on choice branches by having an unambiguous element in it. There are 3 cases: 1. unparsing an xs:choice where for parsing, each branch has a discriminator, or is resolved by the simple "first successful parse" rule. 2. unparsing an xs:choice with dfdl:choiceDispatchKey 3. unparsing xs:choice with dfdl:initiatedContent='yes' For all cases, I suggest we must remove the restriction on dfdl:outputValueCalc being disallowed on choice branch head. For (1). First branch where the branch can be constructed successfully is added to the augmented infoset. This is a case where an unparse error is non-fatal, and causes a form of unparser backtracking through alternatives. Branches would be attempted in schema order. For (2) I suggest dfdl:unparseChoiceDispatchKey, evaluated at unparse time only. Produces a choice branch key. That choice branch is then constructed, and default values or dfdl:outputValueCalcs are used to fill in values. (BTW: this is the driving use case we have in Daffodil for this issue.) For (3).... I have no specific solution. I think it may not actually be a separate case and has to be treated as (1) or (2). Comments/Discussion?

2 1

Action 287: Find a way to handle a variable path step in DFDL expression - Re: DFDL4S use of wildcard/regex in length path expression
by Mike Beckerle 13 Nov '17

13 Nov '17

I have analyzed this use of (.*) notation in the Sentinel2X-bandTMISPData.xsd file. Below is my discussion of what this is, why it is used, the alternative to it, and why it is more problematic than it seems at first, as an addition to DFDL. There are two instances of (.*) in the Sentinel schema files. What it expresses is a partial step-name wildcard. This is not a variation on the XPath * notation, as that matches any step, and can't match a partial name. In the usage in the Sentinel schema, this wild card is a regex that matches against a step name, and it matches exactly a single name. It is not used in a way where it can result in a node-set instead of a single node. The question really is why is this (.*) needed - that is what is it trying to achieve, and whether there is an acceptable alternative for what it is doing. I find that this wildcard (.*) is used to achieve a parameterization of the types TypeISPData and TypeISPData_HKTM. These types are polymorphic, in that their exact behavior depends on aspects of their surrounding context. Each of these types incorporates a length of some content that is outside of its own definition. As the types are used now, the length comes from a thing outside of them in the schema that happens to have a particular suffix on its name, which is "Packet_Secondary_Header" or just "_Secondary_Header". So these names, while outside of the type, are in some sense being hard-coded in these types. Even though they are outside of these types, you cannot change the names of these elements without breaking the ability of the type to find them via this (.*) notation and a name suffix. These types, TypeISPData and TypeISPData_HKTM are placed inside other types, and those are then placed in context with various packet-header structures. Those structures have various distinguishing prefixes of sub-elemetnts such as MSI_Packet_Secondary_Header. There are a variety of different things instead of "MSI_", but there's always something with suffix "Packet_Secondary_Header" or "Secondary_Header" in it. The use of this (.*) name wildcard seems convenient, but doesn't offer anything that isn't better captured by a true parameterization mechanism which decouples the names used outside the type from those used inside it. DFDL provides a general mechanism for this sort of parameterization using variables and dfdl:newVariableInstance. Let's look at just one of the two instances, TypeISPData. To use variables, a variable is created which represents this parameter to the TypeISPData. It is declared in the schema file where TypeISPData is defined. <dfdl:defineVariable name="Additional_Content_Length" type="xs:int"/> The expression within TypeISPData that currently contains "contentLength(/Packet_Data_Field/(.*)Packet_Secondary_Header, 'bytes')" That part of the expression instead references the parameter variable $Additional_Content_Length At the point of use, where this packet secondary header element is combined with the element that contains the TypeISPData, at that location, a dfdl:newVariableInstance is created and bound like so: ... <xs:element name="MSI_Packet_Secondary_Header" type="TypePacketData_MSI"/> <xs:sequence> <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl"> <dfdl:newVariableInstance ref="Additional_Content_Length" value="{ dfdl:contentLength(../MSI_Packet_Secondary_Header, 'bytes') }"/> </xs:appinfo></xs:annotation> <xs:element name="MSI_User_Data_Field" type="TypeUserData"/> </xs:sequence> ... This binds the parameter to the needed length around the point of use of the type that needs it (which is down inside the TypeUserData) This seems bulky due to the XML and XSD-annotation notational overheads, but it is just parameter-binding as would occur in an ordinary programming language when passing an argument. Using variables decouples the names of the two parts of the schema entirely. For example, if MSI_Packet_Secondary_Header were to be renamed to MSI_PSH, it would change only the point of use where the variable binding occurs, and would not affect the type definition. This also facilitates testing the types in isolation. It eliiminates the need to have the complete data structure surrounding them. Use of (*.) name matching might seem convenient, but really it depends on an unstated invariant about the way names are chosen in the Sentinel schema. If the names weren't constructed so uniformly, this notational trick would be unable to make the necessary distinctions, and you'd have to fall back on using variables. So, the above explains an alternative, already in DFDL, that can achieve the parameterization of types that is needed. There is an additional issue that makes this (.*) notation problematic. The issue is thes semantics relative to QNames and ordinary XML namespace management for name conflict control, The Sentinel schema is all a "no namespace" schema. In that world a path step is just an identifier with no structure. However, more generally in DFDL, a path step might be a QName, and those have a namespace prefix part, and a name part. Name collisions and conflicts can be managed by use of different target namespaces and namespace prefixes. Using (.*) prefix regex-matching against QNames is problematic, as it subverts the ability to use namespace prefixes to eliminate name conflicts. E.g., does (.*)_Secondary_Header match "foo:My_Secondary_Header" given that it is in the foo namespace? Matching against things without taking namespace prefixes into consideration breaks XSD's ability to use prefixes to manage name collsions. So the answer of does "(.*)_Secondary_Header" match "foo:My_Secondary_Header" would have to be that it does not match, and that if one wanted to match that one would have to write "bar:(.*)_Secondary_Header" where bar prefix is bound to the same namespace as foo in the schema file where My_Secondary_Header is defined. This mixture of some plain-textual regex matching, and some name-prefix-qualified namespace-sensitive matching is not very attractive. There is certainly nothing like this in XPath. Last issue is implementation complexity: So, if one wants to do without the (.*) notation, in terms of making DFDL4S more complex to implement, you would need to add variables. Fortunately variables are really one of the simpler things in DFDL to implement at least for parsing, and is very simple if the dfdl:setVariable annotation is not implemented since it is not needed in this case. Each variable has a stack. the dfdl:newVariableInstance pushes a new entry onto the stack, and in this case, populates it immediately with a value. Reference to the variable from an expression always takes the value of the top-of-stack location. Exiting the scope of the annotation where the dfdl:newVariableInstance appeared pops the stack. In summary, given (a) the availability of a robust parameterization mechanism using parameter variables and newVariableInstance to bind them (b) the ease of implementing variables, and (c) the complexity of working out interactions with XML/XSD namespaces/prefixes and name management,..... given those reasons, I would be disinclined to advocate something like this (.*) notation to DFDL unless it was first added to XPath, where all necessary details were worked out for us. Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Sep 13, 2016 at 10:18 AM, Michele Zundo <michele.zundo(a)esa.int> wrote: > Dear Steve, > > sorry for the delay due to Summer break and other projects. > > Here is the .zip. > > Please note that we realised that one of our previous replies date 25 July > 2016 at 17:45:05 GMT+2 > was a misunderstanding from our part and not applicable. > > Regarding your questions please find answers interleaved below > > Michele > > This message and any attachments are intended for the use of the addressee or addressees only. > The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its > content is not permitted. > If you received this message in error, please notify the sender and delete it from your system. > Emails can be altered and their integrity cannot be guaranteed by the sender. > > Please consider the environment before printing this email. > > > > > Regards > > > > > > > *From:* Steve Hanson [*smh(a)uk.ibm.com* <smh(a)uk.ibm.com>] > *Sent:* Tuesday, July 26, 2016 3:20 AM > *To:* Michele Zundo > *Cc:* Mike Beckerle; *rui.mestre(a)deimos.com.pt* <rui.mestre(a)deimos.com.pt> > *Subject:* Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05 > > > ….snip > > > dfdl:length="{/Packet_Primary_Header/Packet_Data_Length + 1 - > contentLength(/Packet_Data_Fi > eld/(.*)Packet_Secondary_Header, 'bytes') - 2}" > > Firstly, contentLength is a DFDL function so it needs to be in the DFDL > namespace, eg, dfdl:contentLength(). > > > Yes agree with you. We will add the* dfdl:* it in future releases and > modify the applications accordingly. > > > Secondly, the first argument to dfdl:contentLength() is a path, so you are > effectively still using regular expressions in path steps. > > > Yes. For now we are using it and expect this to become part of the > standard. > > > Regards > > Steve Hanson > *IBM Integration Bus* > <http://www-03.ibm.com/software/products/en/ibm-integration-bus>, > Hursley, UK > Architect, *IBM DFDL* > <http://www.ibm.com/developerworks/library/se-dfdl/index.html> > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:*+44-1962-815848* <%2B44-1962-815848> > mob:*+44-7717-378890* <%2B44-7717-378890> > > > > From: Michele Zundo <*michele.zundo(a)esa.int* > <michele.zundo(a)esa.int>> > To: Steve Hanson/UK/IBM@IBMGB > Cc: Mike Beckerle <*mbeckerle(a)tresys.com* <mbeckerle(a)tresys.com>> > Date: 25/07/2016 17:10 > Subject: Fwd: OGF DFDL WG Call Minutes 2016-07-05 > ------------------------------ > > > > > Dear Steve, > > Please find below the answer from our developers and example. > > Note that we have updated our implementation of DFDL to be as compliant as > we > can at this point in time with the exception noted below. > > Michele > > Begin forwarded message: > > *From: *"Rui Mestre (DME)" <*rui.mestre(a)deimos.com.pt* > <rui.mestre(a)deimos.com.pt>> > *Subject: Re: Fwd: OGF DFDL WG Call Minutes 2016-07-05* > *Date: *25 July 2016 at 17:45:05 GMT+2 > > Dear Michele, > > I believe that after our DFDL compliance effort the mentioned "use of a > regex in the path step of a DFDL expression" is no longer in place. > > Currently the only extension implemented in DFDL4S regarding the use of > regular expressions is that implementation of dfdl:contentLength is > extended to support also regular expressions when specifying the node. > > Please find attached a schema file example containing such extension in > the use of dfdl:contentLength. > > Best regards, > Rui > > > Begin forwarded message: > > *From: *Steve Hanson < <smh(a)uk.ibm.com>*smh(a)uk.ibm.com* <smh(a)uk.ibm.com>> > *Subject: OGF DFDL WG Call Minutes 2016-07-05* > *Date: *5 July 2016 at 17:49:13 GMT+2 > *To: * <dfdl-wg(a)ogf.org>*dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org> > *Cc: *"Mike Beckerle" < <mbeckerle(a)tresys.com>*mbeckerle(a)tresys.com* > <mbeckerle(a)tresys.com>>, "Michele Zundo" <*michele.zundo(a)esa.int* > <michele.zundo(a)esa.int>> > > Please find minutes from the above call at > <https://redmine.ogf.org/dmsf_files/13537?download=> > *https://redmine.ogf.org/dmsf_files/13537?download=* > <https://redmine.ogf.org/dmsf_files/13537?download=> > > *@Michele - please can you send to the WG a schema that shows your use of > a regex in the path step of a DFDL expression ?* > > Next call *Aug 2nd* > > Regards > > Steve Hanson > Architect, IBM DFDL, > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK > *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > *tel:+44-1962-815848* <+44-1962-815848> > > ----------------------------------------- > Michele Zundo > > Head of Ground System Definition and Verification Office > EOP-PEP > European Space Agency, ESTEC > e-mail: <michele.zundo(a)esa.int>*michele.zundo(a)esa.int* > <michele.zundo(a)esa.int> > > > This message and any attachments are intended for the use of the addressee > or addressees only. > The unauthorised disclosure, use, dissemination or copying (either in > whole or in part) of its > content is not permitted. > If you received this message in error, please notify the sender and delete > it from your system. > Emails can be altered and their integrity cannot be guaranteed by the > sender. > > Please consider the environment before printing this email. > > ----------------------------------------- > Michele Zundo > > Head of Ground System Definition and Verification Office > EOP-PEP > European Space Agency, ESTEC > e-mail: *michele.zundo(a)esa.int* <michele.zundo(a)esa.int> > > > > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > #### Sentinel2X-bandTMISPData.xsd moved to MyAttachments Repository V3.8 ( > Link > <notes:///802575AF0030E827/5DE5236E5AD1645685256EE0001BBADF/ABC83CA7B3700B0380258018002AE581>) > on 23 August 2016 by Steve Hanson. > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > ----------------------------------------- > Michele Zundo > > Head of Ground System Definition and Verification Office > EOP-PEP > European Space Agency, ESTEC > e-mail: michele.zundo(a)esa.int > > > > > > > > > >

3 5

clarification: element properties like nilValue and nilKind on simpleType - scoping rules
by Mike Beckerle 30 Apr '17

30 Apr '17

So the nil-related properties are not allowed on dfdl:simpleType annotation elements. As well, the corresponding short-form annotations are not allowed on xs:simpleType elements. But is this workaround legal? <dfdl:defineFormat name="nilStuff"> <dfdl:format nilKind="literalValue" nilValue="-"/> </dfdl:defineFormat> ... <xs:simpleType dfdl:ref="tns:nilStuff"> .... </xs:simpleType> You see that I'm not putting the nil properties on a dfdl:simpleType nor in short form. I'm getting them indirectly via a ref. If this workaround is allowed, then I claim the restriction is pretty meaningless and we should probably drop it. A variation on this theme: if nilKind and nilValue are defined in the default format surrounding an xs:simpleType, can the element referencing that simpleType pick up these definitions from the default format surrounding the simpleType, or can it only get them its own default format. Another variation: what about dfdl:occursCountKind. Can I put that in a named format, reference it from an xs:simpleType, and have it seen by an element referencing that simpleType? What about the default-format variation? I think there are only two stable design points (a) the restriction is meaningless, the workaround works, and the default format surrounding a simpleType def can contribute "element" centric properties such as nilKind, nilValue, and occursCountKind to the element referencing the type. (b) the restriction is enforced and only the properties meaningful to a simpleType can be inherited by an element that references that type. The workaround above doesn't work. Nor does having nilValue or nilKind as the default format surrounding a simpleType mean anything. They would always be ignored. I guess there is also this design point: (c) the restriction prevents direct placement of element-centric properties on dfdl:simpleType or in short form on xs:simpleType, but the workaround works, and default format inheritance from the simpleType's default format would also work. This last one just seems odd to me. It's a restriction for which the workaround is trivial, so one gains nothing from the restriction. ...mikeb Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php>

2 1

Re: [DFDL-WG] My understanding of binary encodings in DFDL
by Champagne, Roger 28 Apr '17

28 Apr '17

Mike That's great, many thanks! I was not expecting such elaborate feedback! The schema generator is a great idea! I'll work on that. Most sincerely, Roger -------------------------------------------------------------- Roger Champagne, P.Eng., Ph. D. Professeur agrégé, Département de génie logiciel et des TI Associate Professor, Department of Software and IT Engineering École de technologie supérieure 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada Bureau / Office : A-4499 Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / Fax : +1 514 396 8405 <(514)%20396-8405> Web: http://etsmtl.ca/Professeurs/rchampagne Le 27 avr. 2017 12:21 PM, "Mike Beckerle" <mbeckerle.dfdl(a)gmail.com> a écrit : > Roger, > > This is an interesting format. > > I have found that DFDL newbies frequently have very difficult formats to > deal with. They end up on the cutting edge of DFDL language features. > Basically, one of the first things people have to explore is whether DFDL > can really handle their data problems. So hard cases first. > > But anyway, to your use case.... > > The first way I started to model this is as a vector of bits that are the > physical representation, and a computed value (dfdl:inputValueCalc). > > Something like this: > > ...assume the schema has dfdl:lengthUnits='bits' > > <dfdl:defineVariable name="Range173" type="xs:double">0.4</dfdl:def > ineVariable> > > <sequence> > <element name="bits" minOccurs="12" maxOccurs="12"> > <sequence> > <element name="bit" type="xs:unsignedInt" dfdl:length="1" > dfdl:outputValueCalc="{ .... omit for now. ...}" /> > </sequence> > </element> > <element name="value" type="xs:double" > dfdl:inputValueCalc="{ > ( ../bits[1]/bit * $Range173 / 2.0) + > ( ../bits[2]/bit * $Range173 / 4.0) + > ... > ( ../bits[12]/bit * ... you get the idea...range divided by the > right number here...) > }" /> > </sequence> > > This is doable, but really isn't that great a way to model this. Lots of > overhead. It has the advantage of being able to give the bits any weighting > factors at all. Note that I omitted the dfdl:outputValueCalc needed for > unparsing the format above. That's also doable, but a little painful. > > But the right way is to model this as an unsigned integer of length 12 > (bits), and then convert to a double float and multiply by the right scale > factor. > > For your message example labeled 173 the value of the double is > > ( ( x * sign) / 4096.0) * 0.4 > > where sign is -1 or 1 depending on the sign bit value. > > That formula is easily inverted (for the unparse direction), so you end up > with this schema having raw integer and value floating point elements. > > <sequence> > <element name="raw" type="xs:unsignedInt" dfdl:length="12" > dfdl:outputValueCalc="{ (../value / 0.4) * 4096.0 * (if (../value lt > 0.0) -1.0 else 1.0) }"/> > <element name="value" type="xs:double" > dfdl:inputValueCalc="{ ((../raw * (if (../sign eq 1) -1.0 else 1.0) ) > / 4096.0) * 0.4) }"/> > </sequence> > > You'd also need a dfdl:outputValueCalc on the sign element, which is > elsewhere in the data, so > that it takes on 1 or 0 based on the sign of the value element when > unparsing. > > This is not as generic as you would like. You need one of these for every > range e.g., the 0.4 above, and for every bits length (12 above), but the > scheme is clear, and at runtime this would be very efficient. > > If you have many of these combinations of different lengths and ranges, > then I'd whip up a little DFDL schema generator that just creates all the > combinations from a common template. > > I hope that helps. > > -Mike Beckerle > > > Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | > www.tresys.com > Please note: Contributions to the DFDL Workgroup's email discussions are > subject to the OGF Intellectual Property Policy > <http://www.ogf.org/About/abt_policies.php> > > > On Tue, Apr 25, 2017 at 9:11 PM, Champagne, Roger < > roger.champagne(a)etsmtl.net> wrote: > >> Steve, Mike, >> >> I just shared a spreadsheet illustrating the encoding/decoding of two >> different ARINC429 messages, you should have received the link through >> Google. >> >> The data portion of both messages is 12 bits long, but have different >> maximum values or ranges. Both messages admit positive and negative values. >> >> Encoding of floats in ARINC429 follows a few simple rules: >> >> - a maximum value and number of bits are established for the data; >> - the most significant bit represents one half of the maximum value; >> - the value of each successive bit is one half of the previous; >> - negative values are represented in two's complement. The sign is >> contained elsewhere in the 32 bit word and is not included in the data >> shown. I am showing the binary values for both positive and negative values >> for each of the two messages. I produced the twos complement values by >> hand, having not figured out how to do that automatically in a spreadsheet >> (bitwise binary addition). >> >> In an ideal world, we would like to be able to specify in DFDL the >> maximum value (float) and number of bits for each message, and the encoding >> and decoding would be performed the same way for all messages, based on >> these two parameters. >> >> If anything is unclear, please let me know. I most appreciate your help >> with this. >> >> Sincerely, >> >> Roger >> >> >> -------------------------------------------------------------- >> Roger Champagne, P.Eng., Ph. D. >> Professeur agrégé, Département de génie logiciel et des TI >> Associate Professor, Department of Software and IT Engineering >> >> École de technologie supérieure >> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada >> Bureau / Office : A-4499 >> Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / >> Fax : +1 514 396 8405 <(514)%20396-8405> >> Web: http://etsmtl.ca/Professeurs/rchampagne >> >> On 25 April 2017 at 05:46, Steve Hanson <smh(a)uk.ibm.com> wrote: >> >>> Roger >>> >>> Please can you supply a couple of concrete examples - for each, the >>> ARINC 429 message, what the bits mean, and what you want to appear in the >>> infoset. >>> >>> Regards >>> >>> Steve Hanson >>> >>> IBM Hybrid Integration, Hursley, UK >>> Architect, *IBM DFDL* >>> <http://www.ibm.com/developerworks/library/se-dfdl/index.html> >>> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >>> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >>> tel:+44-1962-815848 <+44%201962%20815848> >>> mob:+44-7717-378890 <+44%207717%20378890> >>> >>> >>> >>> From: "Champagne, Roger" <roger.champagne(a)etsmtl.net> >>> To: Steve Hanson/UK/IBM@IBMGB >>> Cc: smhdfdl(a)gmail.com, mbeckerle.dfdl(a)gmail.com >>> Date: 23/04/2017 15:50 >>> Subject: Re: My understanding of binary encodings in DFDL >>> ------------------------------ >>> >>> >>> >>> Steve, >>> >>> Thanks for the swift reply. >>> >>> I am a DFDL newbie, so not familiar with the concepts you propose. As >>> there are hundreds of ARINC 429 messages, and each has its own range of >>> values and number of bits, I am guessing this is a bit complicated. >>> >>> Ideally, we would be able to specify the number of bits, plus one of >>> either range of values or value of most/least significant bit, and that >>> would be it. If this is not directly supported but can be done through >>> other means such as expression language, that would be great. >>> >>> Otherwise, this is kind of a show stopper for us. We should have caught >>> this sooner. >>> >>> Sincerely, >>> >>> Roger >>> >>> -------------------------------------------------------------- >>> Roger Champagne, P.Eng., Ph. D. >>> >>> Professeur agrégé, Département de génie logiciel et des TI >>> Associate Professor, Department of Software and IT Engineering >>> École de technologie supérieure >>> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada >>> >>> Bureau / Office : A-4499 >>> >>> Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / >>> Fax : +1 514 396 8405 <(514)%20396-8405> >>> >>> Web: *http://etsmtl.ca/Professeurs/rchampagne* >>> <http://etsmtl.ca/Professeurs/rchampagne> >>> >>> >>> Le 23 avr. 2017 3:13 AM, "Steve Hanson" <*smh(a)uk.ibm.com* >>> <smh(a)uk.ibm.com>> a écrit : >>> Hi Michael >>> >>> Thanks for reaching out. Direct contact is fine but better to post to >>> the DFDL mailing list *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>. >>> >>> DFDL supports binary integers of arbitrary length, but not binary >>> floats. >>> >>> Could you treat the mantissa and exponent as integers inside a hidden >>> group, then use inputValueCalc to fabricate a float using the DFDL >>> expression language? >>> >>> Michael has more experience with calculated values and may be able to >>> offer more advice. >>> >>> Regards >>> >>> Steve Hanson >>> IBM Hybrid Integration, Hursley, UK >>> Architect, *IBM DFDL* >>> <http://www.ibm.com/developerworks/library/se-dfdl/index.html> >>> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >>> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >>> tel:*+44-1962-815848* <+44%201962%20815848> >>> mob:*+44-7717-378890* <+44%207717%20378890> >>> >>> >>> >>> From: "Champagne, Roger" <*roger.champagne(a)etsmtl.net* >>> <roger.champagne(a)etsmtl.net>> >>> To: *mbeckerle.dfdl(a)gmail.com* <mbeckerle.dfdl(a)gmail.com>, >>> *smhdfdl(a)gmail.com* <smhdfdl(a)gmail.com> >>> Date: 22/04/2017 22:24 >>> Subject: My understanding of binary encodings in DFDL >>> ------------------------------ >>> >>> >>> >>> Michael, Steve, >>> >>> I am not sure this is the proper way to proceed, but I figured I would >>> give it a try. Please redirect me to the proper channel if relevant. >>> >>> I am working on an avionics project where we are trying to describe >>> ARINC429 messages in DFDL. After a couple of months of a student working on >>> this and requesting my assistance, I realize DFDL is perhaps not a good >>> match for our needs. >>> >>> My understanding, based on section 13.8 of the specification, is that >>> for floating-point values represented in binary, only two encodings are >>> supported in DFDL: IEEE 754 and IBM 390 HEX. In both cases, the data must >>> be exactly 32 bits (4 bytes) long. >>> >>> * Question 1*: Is this assumption exact? >>> >>> The reason I ask is that ARINC429 messages typically consist in a single >>> 32 bit word. The message data is a subset of the 32 bits, and has variable >>> length (up to 18 bits), based on the nature of the message. The way the >>> data is encoded is different for each message, based of the range of values >>> for that message and the number of bits it uses. We are try to represent >>> these messages in DFDL. >>> >>> * Question 2: *If my assumption is correct, are there any plans to >>> support in DFDL binary encodings of arbitrary length, based on the >>> knowledge of number of bits and range of admissible values? >>> >>> I can supply more details if required, I tried to keep this initial >>> message short. >>> >>> Thanks for any light you can shed on this matter. >>> >>> Most sincerely, >>> >>> Roger >>> >>> -------------------------------------------------------------- >>> Roger Champagne, P.Eng., Ph. D. >>> Professeur agrégé, Département de génie logiciel et des TI >>> Associate Professor, Department of Software and IT Engineering >>> >>> École de technologie supérieure >>> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada >>> Bureau / Office : A-4499 >>> Téléphone / Phone : *+1 514 396 8825* <(514)%20396-8825> -- Télécopieur >>> / Fax : *+1 514 396 8405* <(514)%20396-8405> >>> Web: *http://etsmtl.ca/Professeurs/rchampagne* >>> <http://etsmtl.ca/Professeurs/rchampagne> >>> >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 >>> 3AU >>> >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 >>> 3AU >>> >>> >> >

1 0

Re: [DFDL-WG] My understanding of binary encodings in DFDL
by Mike Beckerle 27 Apr '17

27 Apr '17

Roger, This is an interesting format. I have found that DFDL newbies frequently have very difficult formats to deal with. They end up on the cutting edge of DFDL language features. Basically, one of the first things people have to explore is whether DFDL can really handle their data problems. So hard cases first. But anyway, to your use case.... The first way I started to model this is as a vector of bits that are the physical representation, and a computed value (dfdl:inputValueCalc). Something like this: ...assume the schema has dfdl:lengthUnits='bits' <dfdl:defineVariable name="Range173" type="xs:double">0.4</dfdl: defineVariable> <sequence> <element name="bits" minOccurs="12" maxOccurs="12"> <sequence> <element name="bit" type="xs:unsignedInt" dfdl:length="1" dfdl:outputValueCalc="{ .... omit for now. ...}" /> </sequence> </element> <element name="value" type="xs:double" dfdl:inputValueCalc="{ ( ../bits[1]/bit * $Range173 / 2.0) + ( ../bits[2]/bit * $Range173 / 4.0) + ... ( ../bits[12]/bit * ... you get the idea...range divided by the right number here...) }" /> </sequence> This is doable, but really isn't that great a way to model this. Lots of overhead. It has the advantage of being able to give the bits any weighting factors at all. Note that I omitted the dfdl:outputValueCalc needed for unparsing the format above. That's also doable, but a little painful. But the right way is to model this as an unsigned integer of length 12 (bits), and then convert to a double float and multiply by the right scale factor. For your message example labeled 173 the value of the double is ( ( x * sign) / 4096.0) * 0.4 where sign is -1 or 1 depending on the sign bit value. That formula is easily inverted (for the unparse direction), so you end up with this schema having raw integer and value floating point elements. <sequence> <element name="raw" type="xs:unsignedInt" dfdl:length="12" dfdl:outputValueCalc="{ (../value / 0.4) * 4096.0 * (if (../value lt 0.0) -1.0 else 1.0) }"/> <element name="value" type="xs:double" dfdl:inputValueCalc="{ ((../raw * (if (../sign eq 1) -1.0 else 1.0) ) / 4096.0) * 0.4) }"/> </sequence> You'd also need a dfdl:outputValueCalc on the sign element, which is elsewhere in the data, so that it takes on 1 or 0 based on the sign of the value element when unparsing. This is not as generic as you would like. You need one of these for every range e.g., the 0.4 above, and for every bits length (12 above), but the scheme is clear, and at runtime this would be very efficient. If you have many of these combinations of different lengths and ranges, then I'd whip up a little DFDL schema generator that just creates all the combinations from a common template. I hope that helps. -Mike Beckerle Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy <http://www.ogf.org/About/abt_policies.php> On Tue, Apr 25, 2017 at 9:11 PM, Champagne, Roger < roger.champagne(a)etsmtl.net> wrote: > Steve, Mike, > > I just shared a spreadsheet illustrating the encoding/decoding of two > different ARINC429 messages, you should have received the link through > Google. > > The data portion of both messages is 12 bits long, but have different > maximum values or ranges. Both messages admit positive and negative values. > > Encoding of floats in ARINC429 follows a few simple rules: > > - a maximum value and number of bits are established for the data; > - the most significant bit represents one half of the maximum value; > - the value of each successive bit is one half of the previous; > - negative values are represented in two's complement. The sign is > contained elsewhere in the 32 bit word and is not included in the data > shown. I am showing the binary values for both positive and negative values > for each of the two messages. I produced the twos complement values by > hand, having not figured out how to do that automatically in a spreadsheet > (bitwise binary addition). > > In an ideal world, we would like to be able to specify in DFDL the maximum > value (float) and number of bits for each message, and the encoding and > decoding would be performed the same way for all messages, based on these > two parameters. > > If anything is unclear, please let me know. I most appreciate your help > with this. > > Sincerely, > > Roger > > > -------------------------------------------------------------- > Roger Champagne, P.Eng., Ph. D. > Professeur agrégé, Département de génie logiciel et des TI > Associate Professor, Department of Software and IT Engineering > > École de technologie supérieure > 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada > Bureau / Office : A-4499 > Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / > Fax : +1 514 396 8405 <(514)%20396-8405> > Web: http://etsmtl.ca/Professeurs/rchampagne > > On 25 April 2017 at 05:46, Steve Hanson <smh(a)uk.ibm.com> wrote: > >> Roger >> >> Please can you supply a couple of concrete examples - for each, the ARINC >> 429 message, what the bits mean, and what you want to appear in the infoset. >> >> Regards >> >> Steve Hanson >> >> IBM Hybrid Integration, Hursley, UK >> Architect, *IBM DFDL* >> <http://www.ibm.com/developerworks/library/se-dfdl/index.html> >> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >> tel:+44-1962-815848 <+44%201962%20815848> >> mob:+44-7717-378890 <+44%207717%20378890> >> >> >> >> From: "Champagne, Roger" <roger.champagne(a)etsmtl.net> >> To: Steve Hanson/UK/IBM@IBMGB >> Cc: smhdfdl(a)gmail.com, mbeckerle.dfdl(a)gmail.com >> Date: 23/04/2017 15:50 >> Subject: Re: My understanding of binary encodings in DFDL >> ------------------------------ >> >> >> >> Steve, >> >> Thanks for the swift reply. >> >> I am a DFDL newbie, so not familiar with the concepts you propose. As >> there are hundreds of ARINC 429 messages, and each has its own range of >> values and number of bits, I am guessing this is a bit complicated. >> >> Ideally, we would be able to specify the number of bits, plus one of >> either range of values or value of most/least significant bit, and that >> would be it. If this is not directly supported but can be done through >> other means such as expression language, that would be great. >> >> Otherwise, this is kind of a show stopper for us. We should have caught >> this sooner. >> >> Sincerely, >> >> Roger >> >> -------------------------------------------------------------- >> Roger Champagne, P.Eng., Ph. D. >> >> Professeur agrégé, Département de génie logiciel et des TI >> Associate Professor, Department of Software and IT Engineering >> École de technologie supérieure >> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada >> >> Bureau / Office : A-4499 >> >> Téléphone / Phone : +1 514 396 8825 <(514)%20396-8825> -- Télécopieur / >> Fax : +1 514 396 8405 <(514)%20396-8405> >> >> Web: *http://etsmtl.ca/Professeurs/rchampagne* >> <http://etsmtl.ca/Professeurs/rchampagne> >> >> >> Le 23 avr. 2017 3:13 AM, "Steve Hanson" <*smh(a)uk.ibm.com* >> <smh(a)uk.ibm.com>> a écrit : >> Hi Michael >> >> Thanks for reaching out. Direct contact is fine but better to post to >> the DFDL mailing list *dfdl-wg(a)ogf.org* <dfdl-wg(a)ogf.org>. >> >> DFDL supports binary integers of arbitrary length, but not binary floats. >> >> Could you treat the mantissa and exponent as integers inside a hidden >> group, then use inputValueCalc to fabricate a float using the DFDL >> expression language? >> >> Michael has more experience with calculated values and may be able to >> offer more advice. >> >> Regards >> >> Steve Hanson >> IBM Hybrid Integration, Hursley, UK >> Architect, *IBM DFDL* >> <http://www.ibm.com/developerworks/library/se-dfdl/index.html> >> Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> >> *smh(a)uk.ibm.com* <smh(a)uk.ibm.com> >> tel:*+44-1962-815848* <+44%201962%20815848> >> mob:*+44-7717-378890* <+44%207717%20378890> >> >> >> >> From: "Champagne, Roger" <*roger.champagne(a)etsmtl.net* >> <roger.champagne(a)etsmtl.net>> >> To: *mbeckerle.dfdl(a)gmail.com* <mbeckerle.dfdl(a)gmail.com>, >> *smhdfdl(a)gmail.com* <smhdfdl(a)gmail.com> >> Date: 22/04/2017 22:24 >> Subject: My understanding of binary encodings in DFDL >> ------------------------------ >> >> >> >> Michael, Steve, >> >> I am not sure this is the proper way to proceed, but I figured I would >> give it a try. Please redirect me to the proper channel if relevant. >> >> I am working on an avionics project where we are trying to describe >> ARINC429 messages in DFDL. After a couple of months of a student working on >> this and requesting my assistance, I realize DFDL is perhaps not a good >> match for our needs. >> >> My understanding, based on section 13.8 of the specification, is that for >> floating-point values represented in binary, only two encodings are >> supported in DFDL: IEEE 754 and IBM 390 HEX. In both cases, the data must >> be exactly 32 bits (4 bytes) long. >> >> * Question 1*: Is this assumption exact? >> >> The reason I ask is that ARINC429 messages typically consist in a single >> 32 bit word. The message data is a subset of the 32 bits, and has variable >> length (up to 18 bits), based on the nature of the message. The way the >> data is encoded is different for each message, based of the range of values >> for that message and the number of bits it uses. We are try to represent >> these messages in DFDL. >> >> * Question 2: *If my assumption is correct, are there any plans to >> support in DFDL binary encodings of arbitrary length, based on the >> knowledge of number of bits and range of admissible values? >> >> I can supply more details if required, I tried to keep this initial >> message short. >> >> Thanks for any light you can shed on this matter. >> >> Most sincerely, >> >> Roger >> >> -------------------------------------------------------------- >> Roger Champagne, P.Eng., Ph. D. >> Professeur agrégé, Département de génie logiciel et des TI >> Associate Professor, Department of Software and IT Engineering >> >> École de technologie supérieure >> 1100, rue Notre-Dame Ouest, Montréal, Québec, H3C 1K3, Canada >> Bureau / Office : A-4499 >> Téléphone / Phone : *+1 514 396 8825* <(514)%20396-8825> -- Télécopieur >> / Fax : *+1 514 396 8405* <(514)%20396-8405> >> Web: *http://etsmtl.ca/Professeurs/rchampagne* >> <http://etsmtl.ca/Professeurs/rchampagne> >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >

1 0

OGF DFDL WG Call Minutes 2017-04-18
by Steve Hanson 19 Apr '17

19 Apr '17

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13552?download= Next call May 2nd Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL Working Group call
by Steve Hanson 18 Apr '17

18 Apr '17

1 0

OGF DFDL WG Call Agenda 2017-04-18
by Steve Hanson 18 Apr '17

18 Apr '17

Please find agenda for call on Redmine at https://redmine.ogf.org/dmsf_files/13552?download= Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

OGF DFDL WG Call Minutes 2017-04-04
by Steve Hanson 04 Apr '17

04 Apr '17

Please find minutes from the above call at https://redmine.ogf.org/dmsf_files/13551?download= Next call April 18th Regards Steve Hanson IBM Hybrid Integration Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890

1 0

Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:lengthKind='explicit' for elements of complex type, with dfdl:lengthUnits='characters' and a variable-width encoding
by Steve Hanson 04 Apr '17

04 Apr '17

The intent of the last sentence of erratum 3.9 is that the parser simply assumes the content of the complex element is valid text in the stated encoding, and reads the specified number of characters. If a decode error occurs, that's too bad, your model stated otherwise. So, no point in making this optional, it's straightforward to implement. Action 291 closed. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 04/04/2017 11:56 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Some light on action 291 - see the last sentence of this extract from the original errata document (experience doc 1): 3.9. Section 12.3.5, 7.3.1, 7.3.2. The spec originally allows lengthKind ‘pattern’ to be used when the representation of the current element, or of a child element, is binary, but imposes restrictions on the encoding that can be in force. Clarify that the encoding property must be defined for the element (else schema definition error), and that a decoding processing error is possible if the match of the regex encounters data that does not decode in that encoding, dependent on the setting of encodingErrorPolicy. Remove section 12.3.5.1. Same clarifications needed for testKind ”pattern” property for asserts and discriminators. For consistency, the restriction that a complex element of specified length and lengthUnits ‘characters’ must have children that are all text and that have the same encoding as the complex element, is dropped So that explains how IBM DFDL's error message CTDV1524E came about, it was policing a restriction in the original GFD.174 spec, a restriction which no longer exists. IBM DFDL has not yet implemented the erratum. It wasn't an extra IBM restriction. Regards Steve Hanson IBM Hybrid Integration, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 14/09/2016 08:44 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Actions 290 and 291 raised to investigate further - see minutes. Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Steve Hanson/UK/IBM To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> Cc: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 13/09/2016 13:14 Subject: Re: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Mike I am assuming that the processing for utf-16 'fixed' or 'variable' is entirely handled by ICU so there should be no coding overhead. IBM DFDL works ok for dfdl:lengthKind='explicit' for an element of complex type with dfdl:lengthUnits='characters' and dfdl:encoding="utf-8". But there are conditions the content of the complex type must satisfy otherwise an SDE results, such as: CTDV1524E : For a complex element, when 'lengthKind' is 'explicit' or 'prefixed', and 'lengthUnits' is characters, all simple child elements must have text representation, 'lengthUnits' set to 'characters' and the same encoding. So we insist that the properties of the children are consistent with the properties of the parent. If you recall, IBM DFDL does all these kinds of validation checks in a pre-processing phase. That seems a pretty sensible rule but I am not sure if the rule appears in the spec as such - I just had a quick look but didn't spot anything. So I guess I don't see a need for these things to be optional features? Regards Steve Hanson IBM Integration Bus, Hursley, UK Architect, IBM DFDL Co-Chair, OGF DFDL Working Group smh(a)uk.ibm.com tel:+44-1962-815848 mob:+44-7717-378890 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: "dfdl-wg(a)ogf.org" <dfdl-wg(a)ogf.org> Date: 10/08/2016 18:57 Subject: [DFDL-WG] Suggest should be optional feature of DFDL - dfdl:utf16Width='variable' and other corner cases Sent by: "dfdl-wg" <dfdl-wg-bounces(a)ogf.org> Given the limited set of required encodings for a conforming DFDL processor, I believe dfdl:utf16Width='variable' should be an optional feature. That's just consistency with what is optional already. But it is also quite hard to implement. There are other situations that are very hard to implement, probably never used by real users, yet which are non optional in the spec: I would suggest that dfdl:lengthKind='explicit' for elements of complex type, with dfdl:lengthUnits='characters' and a variable-width encoding like utf-8 is very problematic to implement. I am pretty sure IBM DFDL has no implementation of this per email threads, and I know I don't want to implement this in Daffodil even though we're trying to be very comprehensive in the implementation eventually. I think implementations should be free to just not implement this. These sorts of cases often exist just because we're trying to preserve some orthogonality of composition in the language. So it's possible to do quite a few things that probably aren't ever needed by anyone, that reflect ill-defined data formats, etc. I'd rather not document a bunch of "non-conformances" for Daffodil or other implementations for these sorts of things. I'd like to say we don't implement them, but they're optional, and so that's allowed. Comments? Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Please note: Contributions to the DFDL Workgroup's email discussions are subject to the OGF Intellectual Property Policy -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0