- dfdl-wg - lists.ogf.org

question relevant to evaluation timing - short form
by Mike Beckerle 31 Oct '12

31 Oct '12

Can I do this? <sequence> <annotation><appinfo...> <dfdl:setVariable ref="foo" .../> <dfdl:sequence initiator="{...expression involving foo...}"/> <dfdl:newVariableInstance ref="foo" value="{...some expression...}"/> <dfdl:sequence terminator="{...expression involving foo...}"/> </appinfo></annotation> </element> See how I am spreading out the evaluation of the runtime-valued properties with the setVariable and newVariable stuff. This works so long as we evaluate everything in schema definition order. Now let me throw one more wrench (spanner for the UK folks) in the works. I'm going to add a short form annotation: <sequence dfdl:separator="{... expression...}"> <annotation><appinfo...> <dfdl:setVariable ref="foo" .../> <dfdl:sequence initiator="{...expression involving foo...}"/> <dfdl:newVariableInstance ref="foo" value="{...some expression...}"/> <dfdl:sequence terminator="{...expression involving foo...}"/> </appinfo></annotation> </element> Now, a short form annotation is equivalent to a long form conversion thereof, but in this case is it this one??? <sequence> <annotation><appinfo...>  <dfdl:sequence separator="{...expression ...}"/>  <dfdl:setVariable ref="foo" .../> <dfdl:sequence initiator="{...expression involving foo...}"/> <dfdl:newVariableInstance ref="foo" value="{...some expression...}"/> <dfdl:sequence terminator="{...expression involving foo...}"/> </appinfo></annotation> </element> Or... is it this one: <sequence> <annotation><appinfo...> <dfdl:setVariable ref="foo" .../> <dfdl:sequence initiator="{...expression involving foo...}"/> <dfdl:newVariableInstance ref="foo" value="{...some expression...}"/> <dfdl:sequence terminator="{...expression involving foo...}"/>  <dfdl:sequence separator="{...expression ...}"/>  </appinfo></annotation> </element> I would argue I prefer the first one, because makes the lexical order of appearance consistent with evaluation order. ...mikeb -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

3 4

Document Migration GridForge -> Redmine
by Steve Hanson 31 Oct '12

31 Oct '12

To all DFDL WG mailing list members. As per the email below, the DFDL WG is moving its operations to the new OGF Redmine repository. The OGF GridForge repository will not be deleted but should no longer be used. All documents including WG call minutes have been migrated to Redmine. The URL is http://redmine.ogf.org/projects/dfdl-wg. Please note that if you had a user account with GridForge then you must create a new account for Redmine. The URL for this is http://redmine.ogf.org/account/register. Please don't bookmark any DFDL WG documents in Redmine just yet. The migration has placed our existing folder structure into a 'gridforge' folder, and I will be restructuring over the coming week. Please let me know if you have any problems. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 31/10/2012 12:08 ----- From: Andre Merzky <andre(a)merzky.net> To: wg-all(a)ogf.org, Date: 31/10/2012 11:29 Subject: [wg-all] Document Migration GridForge -> Redmine Sent by: wg-all-bounces(a)ogf.org Dear OGF Groups, Most groups have by now migrated their operations from GridForge to Redmine. Over the last days, we have been migrating documents from the various GridForge projects to the respective Redmine projects. We plan a similar migration of tracker items (sorry, we can't give a timeline for that, yet). A couple of notes about the document migration: - The documents (incl. the folder structure) have been moved into a separate folder on your redmine project's document module. That folder is named 'gridforge'. That way, the migrations would not collide with existing redmine documents. Note that you can move folders and documents to other folders, as needed. - For some GridForge projects, we do not have equivalent redmine projects. The respective documents have been moved to a separate archival project: http://redmine.ogf.org/dmsf/migration-archive Please let us know if you want to have access to that project, (it is publicly readable though), and if you want to have parts of that archive migrated to specific redmine projects. - We are aware of some problems, but decided not to delay the migration as several groups are waiting for the documents. Please let us know if you encounter faulty migrations. In particular, you may encounter folders which cannot be opened. Please pop me a note, and I'll make sure that this is getting fixed timely. Those problems should be rare though (so far 2 documents made trouble, out of O(100) checked). It is very likely that for many redmine accounts, permissions are not yet set correctly. In particular, chair and secretary roles may not yet be assigned to your account. Please let me know when you have trouble with permissions, and I'll flip the preciousss bitsesss... Best, Andre. -- Nothing is really difficult... -- The wg-all mailing list is an umbrella list for all OGF mailing lists. If you are subscribed to any OGF mailing list, you will receive mails to wg-all. You can unsubscribe from this list on http://www.ogf.org/cgi-bin/perl/unsubscribe-wg-all.pl wg-all mailing list wg-all(a)ogf.org https://www.ogf.org/mailman/listinfo/wg-all Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

Re: [DFDL-WG] DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance)
by Steve Hanson 30 Oct '12

30 Oct '12

Regarding the self-axis, the rule is simple. If the expression contains any self-axis references it is evaluated after. Otherwise it is evaluated before - even if an absolute or relative path turns out to be to the element in question. This means it is easy to decide when to evaluate it. (I meant to say that originally). Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Tim Kimber/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org Date: 30/10/2012 13:34 Subject: Re: [DFDL-WG] DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance) I agree with most of that. I certainly would want newVariableInstance to be evaluated before a complex element is parsed. The complication with the setVariable rule is that an absolute path may be equivalent to a path which uses the self-axis but it is not possible, in general, to determine this by static analysis of the xsd. Furthermore, one XPath expression can contain zero to many path refs, any or all of which might ( or might not ) start with the self-axis. I can only see two possible solutions to this. - Asserts/discriminators/setVariable are always executed after the component on which they are positioned has been fully parsed. With the exception of asserts/discriminators with test=pattern. - Asserts/discriminators/setVariable have a 'timing' flag that defaults to 'after'. If it is set to 'before' and the expression does not evaluate successfully then it is a schema definition error. This would allow earlier rejection of the wrong branch in the model ( and therefore more efficient parsing ) in cases where the expression only refers to items that have already been parsed. regards, Tim Kimber, DFDL Team, Hursley, UK Internet: kimbert(a)uk.ibm.com Tel. 01962-816742 Internal tel. 37246742 From: Steve Hanson/UK/IBM@IBMGB To: Mike Beckerle <mbeckerle.dfdl(a)gmail.com>, Cc: dfdl-wg(a)ogf.org Date: 30/10/2012 13:12 Subject: Re: [DFDL-WG] DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance) Sent by: dfdl-wg-bounces(a)ogf.org Thanks for writing this up, I have a couple of comments from thinking about this in parallel, and reading the write-up. - In 2), replace element with object as applies to sequence/choice as well - newVariableInstance should be evaluated before the object is parsed (note: self-axis is not allowed) - setVariable should be evaluated before the object is parsed unless it uses self-axis - setVariable should be evaluated after the object is parsed if it uses self-axis - consider only allowing self-axis in setVariable for simple elements/types - add statement about early evaluation of non-pattern asserts/discriminators if that can be done (as per IBM implementation) Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: dfdl-wg(a)ogf.org, Date: 29/10/2012 22:54 Subject: [DFDL-WG] DFDL Statement Evaluation Timing (Assert, Discriminator, SetVariable, NewVariableInstance) Sent by: dfdl-wg-bounces(a)ogf.org I'll write this up like an errata, but this is for discussion of whether we believe this is clear and complete. ------------------------------------- Glossary: DFDL Statements are the annotation elements dfdl:assert, dfdl:discriminator, dfdl:setVariable, and dfdl:newVariableInstance. Errata: Locations where DFDL Statements are allowed to appear are extended to also include Global Element Declarations, and on Simple Type Definitions. Errata: Clarification about discriminators: Discriminators exclude Assertions even when combining across references. Beyond the stipulation that there can be only one dfdl:discriminator at any annotation point of the DFDL schema, there are further constraints. A single dfdl:discriminator annotation may appear on an element reference, or on the global element declaration it refers to, or on the simple type appearing immediately within or referenced from the global element declaration. But only one of those places. In addition, if a discriminator occupies one of those places, then no dfdl:assert annotations may appear in any of those locations. A dfdl:discriminator annotation may appear on a group reference or on the model group within the global group definition it refers to. But only one of those places, and similarly, if a discriminator appears in any of those places, then no dfdl:assert annotations may appear in any of those locations. Errata: Clarification about the execution order of DFDL Statements when they appear on an element reference or element declaration. DFDL Statement annotations for a given schema component are executed as follows: 1) all relevant DFDL statement annotations are gathered to form a single list which preserves schema-definition order. For a simpleType definition, the DFDL statement annotations found immediately on it are kept in schema-definition order, and are appended to the end of a list of those from any base simpleType definition. For an element declaration having simple type, the DFDL statement annotations found immediately on the declaration are appended to the end of the list of those from its simple type. For an element reference, DFDL statement annotations found immediately on the element reference are appended to the end of the list of those from the global element declaration it references. 2) given the combined list, the annotations are executed as follows: 1. before any parsing of the element, a dfdl:discriminator with testKind="pattern" is executed. 2. if there is no discriminator, then before any parsing of the element, all dfdl:asserts (there could be several) with testKind="pattern" are executed in the order they appear in the list of DFDL statements. 3. The element itself is parsed, or its inputValueCalc property is evaluated to create its value. 4. all newVariableInstance annotations are executed and new variables are placed into scope for the duration of these remaining steps. The statements are executed in the order they appear in the list of DFDL statements. 5. all setVariable annotations are executed. The statements are executed in the order they appear in the list of DFDL statements. 6. if a discriminator is present it is executed 7. if no discriminator is present, then assert annotations can be present, and they are executed. If there are multiple assert annotations the statements are executed in the order they appear in the list of DFDL statements. If the element reference or local element declaration is an array, then this evaluation is repeated for each occurrence of the array. Discussion: The above allows the default expressions associated with any statement to refer to the value of the element itself as "." However, there's this anomaly of syntax where things don't seem right: <dfdl:newVariableInstance ref="foo:bar" default="{....some expression ...}"/> creates a new variable for the scope of the entity it annotates. It's clear what this means if this annotation is placed on a sequence or choice. For the children of that sequence/choice the new variable instance is in effect. On a simpleType or element having simple type, similarly it is clear (in that case it's a very local variable, just for the expressions in other newVariableInstance statemsnts, setVariableStatements, and discriminators and assertions). The rub: on an element declaration (or reference) when there is a complex type, its not clear. <element name="foo"> <annotation><appinfo ....> <dfdl:newVariableInstance......>  </appinfo></annotation> <complexType> <sequence> <annotation><appinfo ....> <dfdl:assert>... according to rules above this cannot refer to the newVariableInstance...</dfdl:assert> </appinfo></annotation> .... </sequence> </complexType> </element> The above rules say the newVariableInstance isn't evaluated until AFTER the element is parsed, so the assert down inside on the sequence will NOT see this new variable even tho textually, the newVariableInstance annotation looks like it would be in scope over the sequence. Possible ways to avoid this oddity without messing up evaluation order: allow newVariableInstance only on simpleType, element declarations and element references having simpleType, group references, and sequence/choice. Disallow them on element declarations of complex type or on element references to those. The schema author can always introduce an extra tier of sequence to provide the exact behavior they need, and this otherwise error-prone issue can be avoided. -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

OGF DFDL WG Call Agenda 2012-10-30
by Steve Hanson 29 Oct '12

29 Oct '12

Please find agenda for call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

2 1

new action item needed: DFDL URN specification
by Mike Beckerle 29 Oct '12

29 Oct '12

Introduction: Using URLs as identifiers has caused no end of problems. E.g., in DFDL we have http://www.ogf.org/dfdl/dfdl-1.0/ as an identifier. W3C has http://www.w3.org/2001/XMLSchema and others. W3C has badly regretted establishing this convention, as they have farms of servers that do nothing but quickly return 404 errors to save network-aware applications the network-timeout delay that would otherwise occur. There are parties interested in exploiting DFDL who want DFDL schemas to NOT contain network URLs because it simply creates a concern about network access whenever a DFDL schema is inspected/used. Pre-Proposal: The new way to do this is with URNs which would look roughly like this: urn:ogf:dfdl:standard:dfdl-1.0. The whole point is that some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. The point is that it is a name in a managed namespace which cannot be confused with a network protocol URL. OGF is already establishing urn:ogf, and an ogf subgroup has already proposed urn:ogf:network for network resources. DFDL schemas aren't network resources so we don't want to be a substructure underneath network. Some other mechanism is used to establish correspondences between these and any resources in file systems, networks, or built-in to implementations. One such mechanism is called XML Catalog. Summary: An action item should be to specify DFDL urn, submit to OGF as a proposed namespace, and then produce errata/spec changes to specify its use. This requires a small design activity to specify a scheme for the sub-structure of the DFDL URNs (i.e., scheme for the stuff after urn:ogf:dfdl:...) where we want standard identifiers for versions of the standard, but we probably also want a few other things (e.g., I would like a space for implementations to identify themselves, i.e., an implementation-specific sub-area within our URNs.) Our existing URLs can be compatible (deprecated) practice vs the preferred URNs. -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

3 3

Re: [DFDL-WG] simpleType cannot contain assert?
by Mike Beckerle 29 Oct '12

29 Oct '12

The example is pretty simple. I have strings that need asserts to verify proper format. Nothing like working on a real format to put the small but critical issues into focus. I overlooked that you cannot put an assert on a global element. I read that too quickly, but there it is. I'm not sure what we were thinking here. Perhaps it was as simplistic as avoiding global constructs with ".." in the paths of expressions. But this just flies in the face of creating tidy schemas that avoid repetition of asserts or big complex expressions all over the place. And asserts are one of the very untidy things because they contain regular expressions, which are very unweildy and difficult to maintain and badly badly need to be centralized in any reasonable schema. So I'll amend my proposal: allow asserts on global element decls, and on simple type defs. That is, they are orthogonal in placement to whether the site is local or global. Here's the example: If I can put these as asserts on a simpleType, then I can abstract over them in a schema, using the same regex assert for many fields with different names. If I cannot, then I have no choice but to nest them inside a complex type, and all these simple string fields become complex types, which is adding a whole tier of elements to the schema. E.g. <simpleType name="dField" dfdl:ref="ex:dFieldDefaults"> <restriction base="xs:string"> </simpleType> then all over the schema..... <sequence dfdl:ref="dFieldListFormat"> <element name="Foo" type="ex:dField> <xs:annotation> <xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0/"> <dfdl:assert testKind="pattern" testPattern="((\p{L}|\d|\.|,|$|$|\?|!|@|#|\$|%|\^|&|\*|=|_|\+|\[|\]|\{|\}|\\|"|'|;|<|>|~|\|)|\-)(((\p{L}|\d|\.|,|$|$|\?|!|@|#|\$|%|\^|&|\*|=|_|\+|\[|\]|\{|\}|\\|"|'|;|<|>|~|\|)|\-)| )*((\p{L}|\d|\.|,|$|$|\?|!|@|#|\$|%|\^|&|\*|=|_|\+|\[|\]|\{|\}|\\|"|'|;|<|>|~|\|)|\-)|(\p{L}|\d|\.|,|$|$|\?|!|@|#|\$|%|\^|&|\*|=|_|\+|\[|\]|\{|\}|\\|"|'|;|<|>|~|\|)" message="Assertion failed for data_code" /> </xs:appinfo> </xs:annotation> </element> .... repeat ad nauseum for all other elements of type dField, which is EVERYTHING practically. This absolutely flies in the face of any good principles of abstraction. I want that regex, and the characteristics of data that have to obey it, captured in one place. I also really do not want to have to turn every element like Foo, which is logically just a string, into a complex type. What I need to be able to do is rotate that assertion over into the definition of dField. I can't do it with a pattern facet, because that doesn't affect parsing, and this assertion needs to guide speculative parsing. (Note: XML Schema allows patterns on simpleType defs and so allows proper abstraction of simple types that use pattern validation. DFDL is currently not consistent with this way of abstracting.) Now, currently the spec says I can't put an assert on a global element either, so I can't fix the above issue by doing this: <element name="dValue" type="xs:string"> ....big mombo assert with regex ... NOT ALLOWED HERE EITHER </element> <complexType name="dField"> <sequence> <element ref="ex:dValue"/> </sequence> </complexType> <sequence dfdl:ref="dFieldListDefaults"> <element name="Foo" type="dField"/> .... But I could push the assert down inside the complex type definition, where it would be on a local element decl, and if everywhere I share use of that complex type, then I can centralize the regex. What is this restriction possibly achieving? The assertion is still "trying to be on a global definition", we've just prevented it from being in a convenient place for the modeler. Also, as soon as you force me to use a complex type, now I'm stuck with the complex type restrictions on nillability, which are insufficient for my needs, so now I can't take advantage of mapping data containing non-empty nil indicators to xsi:nil nillability. I have to create a <element name="noValue" ....> and get into using choices, etc. I should note that precedent in the spec already allows asserts on a sequence/choice that is a global group definition and global group references. I believe this is just a happy accident of the XML Schema object model i.e., the fact that the sequence/choice object isn't the same object as the global definition object, it is contained inside the global group def object. The point: there is already a precedent for having to combine the lists of assertions from both defines and references to them. I think the rule is simple, asserts on references go after the asserts on the defines, but otherwise it is just merging the lists onto the reference schema component to be executed at the reference. The concern about global objeccts, that is about things having expressions/properties that can't be resolved isn't one that keeping assertions off global defs/decls will help. A global def/decl can contain a subset of DFDL properties such that it is useless in DFDL outside of some complementary referencing context. This is inherent in DFDL, and any system that allows separation of concerns when describing something complex including XML Schema itself, which has global types, groups, etc. all of which are useless without their referencing contexts. I would close by noting that this change, (allowing these additional locations for asserts), is backward compatible with our current spec, because it only adds new places these assertions are allowed. The right fix here: DFDL 'statements' (setVariable, assert, discriminator, even newVariableInstance) should be allowed on simpleType and on global element decls. Even newVariableInstance is useful there. A new variable is created, used in expressions in asserts/discriminators/setVariable/ other newVariableInstance, and then it goes out of scope immediately at the end of the element. The newVariableInstance is in fact our only way of really controlling the complexity of an expression. It lets you create a big complicated expression out of several variables each of which is bound to a sub-calculation, and so it is useful even when the scope is just the asserts/discriminators/setVariables/newVariableInstance statements of a single simple-typed element. newVariableInstance basically lets us create local variables for use in our expression language. Normal XPath doesn't have this. ...mike On Fri, Oct 26, 2012 at 4:11 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: > Mike, it's not an oversight, I'm sure it is for the same reason that you > can't put an assert on a global element. I think the rationale is that > asserts (and discriminators) are only allowed at annotation points where > all properties can be resolved. > > Please can you show an example of what you want to achieve? > > Regards > > Steve Hanson > Architect, Data Format Description Language (DFDL) > Co-Chair, *OGF DFDL Working Group* <http://www.ogf.org/dfdl/> > IBM SWG, Hursley, UK* > **smh(a)uk.ibm.com* <smh(a)uk.ibm.com> > tel:+44-1962-815848 > > > > From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> > To: dfdl-wg(a)ogf.org, > Date: 25/10/2012 17:47 > Subject: [DFDL-WG] simpleType cannot contain assert? > Sent by: dfdl-wg-bounces(a)ogf.org > ------------------------------ > > > > Is this just an oversight? > > I find it very tedious to model if I can't put the asserts on the > simpleTypes so that I don't have to repeat them over and over in the model. > > The composition rule here is simple. If an element has asserts, and its > simple type has asserts, both are executed, with the element's asserts run > after the simpleType's asserts. > > ...mike > > -- > Mike Beckerle | OGF DFDL WG Co-Chair > Tel: 781-330-0412 > -- > dfdl-wg mailing list > dfdl-wg(a)ogf.org > https://www.ogf.org/mailman/listinfo/dfdl-wg > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

3 3

OGF DFDL WG Call Minutes 2012-10-23
by Steve Hanson 29 Oct '12

29 Oct '12

Please find minutes from the above call on GridForge at https://forge.ogf.org/sf/docman/do/downloadDocument/projects.dfdl-wg/docman… Errata document is still at v010 here: http://forge.gridforum.org/sf/go/doc16280?nav=1 Regards Steve Hanson Architect, DFDL, IBM SWG Co-Chair, OGF DFDL Working Group Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

Re: [DFDL-WG] DFDL binaryCalendarRep pattern limitations
by Steve Hanson 29 Oct '12

29 Oct '12

DFDL WG agreed that the below behaviour is correct. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 From: Steve Hanson/UK/IBM To: dfdl-wg(a)ogf.org, Cc: Andrew Edwards/UK/IBM@IBMGB Date: 23/10/2012 10:57 Subject: DFDL binaryCalendarRep pattern limitations For discussion at next DFDL WG call. See summary below but key points are: - DFDL uses a calendar pattern to convert binary calendar 'packed', 'bcd' and 'ibm4690Packed' reps to a schema calendar type - No trimming/padding of binary reps takes place, the parser uses what was extracted from the data - A 'packed' rep will always present an odd number of digits (because of sign nibble) - A 'bcd' rep will always present an even number of digits - ICU gives an error if the number of digits presented to it exceeds the length of the calendar pattern - Therefore the onus is on the user to ensure that the calendar pattern matches the number of digits, eg, by adding leading zeros to the pattern Are we happy with that behaviour? Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/10/2012 10:49 ----- From: Andrew Edwards/UK/IBM To: Steve Hanson/UK/IBM@IBMGB, Date: 22/10/2012 15:36 Subject: DFDL binaryCalendarRep pattern limitations Hi Steve - a summary of the problem is below While adding support for IBM4690 packed representation for binaryNumberRep and binaryCalendarRep, it has become apparent that we may need to place a restriction on the calendarPattern property, depending on the choice of binaryCalendarRep. A problem surfaces in being able to reliably and reversibly distinguish a calendar value when the pattern length is incompatible to the packing type. If we use 'ibm4690packed' and a pattern that is of odd length, we end up matching an even-length string against an odd-length pattern and there isn't necessarily a well-defined defined answer. This is best understood with an example: - Consider a pattern with an odd number of characters, such as calendarPattern=yyyyMMddDDD - A value will require a bytestream of length 6 bytes, which would be serialised as 0x0{y}{y}{y}{y}{M}{M}{d}{d}{D}{D}{D} - For example, 2012-10-22 (day295) would be represented as 0x020121022295. If a parser is represented with this value and pattern, then it will try to match the string "020121022295" against "yyyyMMddDDD". ICU returns an error because the string is longer than the pattern and I can't say I blame it. Should it ignore the zero at the start, or the 5 at the end? Without understanding the pattern, a DFDL parser cannot know. ICU can't resolve the value as it has more than one group in the pattern so it can't resolve to one single solution. This becomes more problematic when we consider behaviour for a pattern of "DDD" and of "SSS" as they expect padding at different ends by default (a value of "0100" resolves as "DDD"=100days and "SSS"=0.010 seconds). For packed representation, the opposite problem occurs: There is always a sign nibble in the packed form, so we will always have a value made up of an odd number of digits. This can't match against all patterns of even length. The solution that we discussed was to require the calendar pattern to have a certain digit count depending on the choice of binaryCalendarRep, and allow the pattern to include number characters as a form of "padding". So for the pattern in the example above, this would have to be changed so that calendarPattern='0'yyyyMMddDDD to force it to have an even digit count. If that fully explains the problem, do you want to take it to the DFDL workgroup and check what the consensus opinion is? Cheers, Andy Andy Edwards - WebSphere Message Broker - DFDL Email: andy.edwards(a)uk.ibm.com Snail Mail: MP211, Hursley park, Hursley, WINCHESTER, Hants, SO21 2JN Tel int: 247222 Tel ext: +44 (0)1962 817222 Desk: DE2 U20 The Feynman problem solving Algorithm 1) Write down the problem 2) Think real hard 3) Write down the answer -- Murray Gell-mann in the NY Times Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

simpleType cannot contain assert?
by Mike Beckerle 25 Oct '12

25 Oct '12

Is this just an oversight? I find it very tedious to model if I can't put the asserts on the simpleTypes so that I don't have to repeat them over and over in the model. The composition rule here is simple. If an element has asserts, and its simple type has asserts, both are executed, with the element's asserts run after the simpleType's asserts. ...mike -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0

Clarification needed: pad/trim and delimited length
by Mike Beckerle 25 Oct '12

25 Oct '12

The spec says that pad characters are removed before escape scheme processing. However, in delimited context, I can't even determine the length of the field to trim off the padding unless I can do the escape scheme processing. This is either a Chicken-Egg, or the algorithm for parsing is substantially more complex due to padding. Comments? -- Mike Beckerle | OGF DFDL WG Co-Chair Tel: 781-330-0412

1 0