IBM has continued its review of the proposed additions to lengthKind and occursCountKind to simplify the modelling of MIL-STD-2045 formats. The email below carries on from an earlier email but has removed everything to do with bitOrder etc. New stuff is in blue. Regards Steve Hanson Architect, IBM DFDL Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh@uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 28/07/2014 11:31 ----- 1) Proposed new dfdl:lengthKind 'fixedLengthOrTerminated'. A new enum implies that it can be used in any scenario, so the following need to be specified. dfdl:terminator must be set and can not be empty string or contain ES on its own If xs:string or xs:hexBinary, can maxLength facet be used instead of dfdl:length? (Suggest no - this is variable length data so min/maxLength are for validation only). Can dfdl:length be an expression? (Suggest no unless specific use case identified) My use case needs only constants as the maximum, hence enum name contains "fixed" prefix, not "explicit". Any special rules for emptyValueDelimiterPolicy and nilValueDelimiterPolicy ? Since a terminator must be set, then these cannot be "none" or "initiator". SMH: Doesn't follow. Today, if I specify a terminator, it must be present, modulo EVDP/NVDP. So why is the same not true for the new enum? If we add a new enum, it has to work in a way that is consistent with other lengthKinds and not just for MIL-STD-2045 use cases. Use on complex element. Presumably dfdl:length is first used to extract a 'box' but within that box does parser immediately scan for the dfdl:terminator or does it descend into the complex type and parse the children, expecting to either consume all the box or to find the terminator at the end? (Suggest the latter). I have no use case that requires this for complex types at all. Perhaps we can dodge this by having it be simpleFixedLengthOrTerminated, and restricting it to simple types only. ? SMH: Perhaps, but that makes this lengthKind enum different from all the others, and that doesn't seem right. Use on complex element. Last child can not be dfdl:lengthKind 'endOfParent'. Scanning rules: Use of this new dfdl:lengthKind switches off any in-scope stack of terminating markup in force at that point. Put another way, when we are scanning for the dfdl:terminator, we are not looking for any markup from an outer scope. So there's plenty to think about with this new dfdl:lengthKind. A good rule for deciding whether a new dfdl:length or dfdl:occursCountKind should be added is whether it bends some other part of the spec out of shape. The new dfdl:lengthKind looks ok so far. However we *think* we have come up with an alternative model which is simpler than you one you state in the document. Example for field 'varstr' with max length 100: <xs:sequence dfdl:terminator="{if (fn:str-len(varstr) eq 100) then '%ES;' else '%DEL'}" ...> <xs:element name="varstr" type="xs:string" dfdl:lengthKind="pattern" dfdl:pattern="([^\x7F].\x7F)|(.{100})" ... /> </xs:sequence> Can't put dfdl:terminator with a self-referencing expression on the element. Might need fn:exists in the dfdl:terminator expression to handle optionality. Does that work? I don't think this will work as %ES isn't allowed in terminators. There is a proposal to allow it, but only when length kind is such that one is not scanning for delimiters (same restriction as for WSP*). Let's assume that we allow %ES for now. SMH: This has been incorporated as an update to erratum 2.148 and is the latest spec draft. One beauty of your idea here is that unparsing will "just work", so that's nice. But I think your pattern has a bug: I think it should be dfdl:pattern="[^\x7F]{0,99}(?=\x7F)| .{100}" This will not capture more than 99 characters prior to the DEL, and will not include the DEL as part of the string in the case where a DEL is found (uses lookahead in regex). Hence, the DEL will be available to be picked off as the terminator. Without this you end up with the DEL in the payload. With that I think your approach would work. So thanks for that idea. SMH: Yes my pattern was wrong, thanks for correcting. SMH: Also realised that the dfdl:terminator expression is illegal, as it looks downwards. The correct DFDL is: <xs:sequence ...> <xs:element name="varstr" type="xs:string" dfdl:lengthKind="pattern" dfdl:pattern="[^\x7F]{0,9}(?=\x7F)|.{10}" ... /> <xs:sequence dfdl:terminator="{if (fn:string-length(./varstr) eq 10) then '%ES;' else '%DEL'}" .../> </xs:sequence> I have tested this (using {if (fn:string-length(./varstr) eq 10) then '%WSP*;' else '%DEL;'} as %ES; not yet allowed in terminator) and it works ok both parse and unparse. It was noted that if the terminator expression was allowed to refer to the value of its own element then this could be simplified to: <xs:element name="varstr" type="xs:string" dfdl:lengthKind="pattern" dfdl:pattern="[^\x7F]{0,9}(?=\x7F)|.{10}" dfdl:terminator="{if (fn:string-length(.) eq 10) then '%ES;' else '%DEL'}" .../> Clearly this relaxation could only occur when lengthKind was not delimited. (That is, the same condition that we have proposed allowing %ES; for terminator/separator). But I think it also violates the known-to-exist rules ? Certainly IBM DFDL says it can't find '.' in the infoset when I tried this. So perhaps this is not a good idea. 2) Proposed new dfdl:occursCountKind 'prefixed'. The motivation here is to avoid the explosion of global groups needed for the hidden presence indicators. It was observed that a single global group could be used if the expression used a predicate when referring to the FPI element, though obviously that makes the schema very fragile. At first glance the new enum would appear to be symmetric with lengthKind 'prefixed', but on closer examination this is not true: Presumably the new enum would apply to optional elements and arrays. It would have to fit into the grammar thus: Array = [ [PrefixOccursCount Separator] EnclosedElement [ Separator EnclosedElement ]* [ Separator StopValue] ] PrefixOccursCount = SimpleNormalRep It would be wrong to couple the prefix more tightly to the first occurrence (by more tightly I mean like prefix length where the length occurs after the element's left framing region). When parsing, if the value is 0 then nothing else is expected in the data - zero occurrences, so no other DFDL properties are even examined. It must therefore occur ahead of all occurrences. If it is doing that, then it may as well have its own left and right framing, hence use of SimpleNormalRep rather than SimpleContent, and work with delimiters. However IBM questions the need for the enum as it can also be modelled using a choice of two sequences which, if you put the discriminator on the hidden FPI element itself, means you can get away with just two global groups. And you don't need outputValueCalc as you can just use defaults. ... <!-- Element unit_name --> <xs:choice> <xs:sequence> <xs:sequence dfdl:hiddenGroupRef="vmdfdl:gh_mil_std_2045_FPI_true" /> <xs:element name="unit_name" type="..." ... /> </xs:sequence> <xs:sequence dfdl:hiddenGroupRef="vmdfdl:gh_mil_std_2045_FPI_false" /> </xs:choice> <!-- Element unit_type --> <xs:choice> <xs:sequence> <xs:sequence dfdl:hiddenGroupRef="vmdfdl:gh_mil_std_2045_FPI_true" /> <xs:element name="unit_type" type="..." ... /> </xs:sequence> <xs:sequence dfdl:hiddenGroupRef="vmdfdl:gh_mil_std_2045_FPI_false" /> </xs:choice> ... <xs:group name="vmdfdl:gh_mil_std_2045_FPI_true" > <xs:sequence> <xs:element name="FPI" type="xs:boolean" default="true" ... > <dfdl:discriminator test="{. eq fn:true()}" </xs:element> </xs:sequence> </xs:group> <xs:group name="vmdfdl:gh_mil_std_2045_FPI_false" > <xs:sequence> <xs:element name="FPI" type="xs:boolean" default="false" ... > </xs:element> </xs:sequence> </xs:group> 3) Proposed new dfdl:occursCountKind 'repeatUntil'. It seems to IBM that the only practical effect of the new enum 'repeatUntil' is to simplify the discriminator. It doesn't remove it nor does it remove the need for the hidden FRI element. IBM does not see the benefit of the new enum in its proposed form. Further... If the above proposal is used for the FPI, the dfdl:occursIndex() branch of the discriminator simplifies to fn:true(). The FRI is local to the array element so, when parsing at least, there is no need for a globally unique group for each array. That simplifies the discriminator to the following and means you only need one global group for FRI. <dfdl:discriminator> if (dfdl:occursIndex() eq 1 then fn:true() else ../<array>[dfdl:occursIndex()-1]/vmfdfdl:gh_mil_std_2045_FRI <dfdl:discriminator> For that to work on unparsing there needs to be a generic way to set the (Boolean) FRI from within the hidden group. Something like dfdl:outputValueCalc="{dfdl:occursIndex() eq fn:count(..)}" There is a problem with this though. The property is on the FRI element so what does dfdl:occursIndex() return? The spec says it returns "the position of the current item within an array" but also says "this function may be used on non-array elements". I'm not clear what it would return for the latter case - does it return 1 or does it look back to its parent or ... ? Here we want the index of the parent. Perhaps this function needs to take an argument to be unambiguous, eg, . or .. or ../.., ie, it can only refer back up to the root. (In fact this problem applies whether or not there is a single FRI or one per array). A counter proposal... One way to really simplify this type of occurrence indicator is to consider it as part of the element, in the same way as a length prefix. This tight binding makes sense here, because there is an indicator per occurrence. dfdl:occursCountKind="stopIndicator' dfdl:occursStopIndicatorType="<type>" The stop indicator type must be derived from xs:boolean. True means the occurrence is the last. False means it is not. Or we can do it the other way round) The DFDL Boolean properties of the type can always be used to compensate. The parser would work a bit like it does for 'stopValue' - it keeps parsing speculatively until it finds an occurrence which indicates the end of the array - the difference being that in this case it is added to the infoset. The oddity about this is that it applies to arrays only and does not work with optional elements, so it can not be used with minOccurs = '0'. Grammar becomes: SimpleNormalRep = LeftFraming StopIndicator PrefixLength SimpleContent RightFraming ComplexNormalRep = LeftFraming StopIndicator PrefixLength ComplexContent ElementUnused RightFraming StopIndicator = SimpleContent Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU