- dfdl-wg - lists.ogf.org

RE: [dfdl-wg] Plumbing document
by mike.beckerle＠ascentialsoftware.com 09 Dec '04

09 Dec '04

Comments on the document. It would be most helpful if all sections were numbered so that comments can be made specific, but let me do it this way for now: Section 1.Function.Use About "runtimeOccurs". I think the prefix "runtime" is part of recognizing that there are static parameters and dynamic parameters. We think of byteOrder as something static or usually static, though in reality, it can vary based on an indicator found in the file, witness the UTF-16 character set which can have a byte-order-mark at the beginning to indicate whether it is really UTF-16BE or UTF-16LE. The question is whether the separation of static and dynamic as the usual usage pattern for each parameter is tight enough that we feel comfortable putting something like "runtime..." in front of the name. In fact the parameters runtimeOccurs and runtimeValue probably should be renamed to occurs and value to reflect that they can be either static or dynamic just like byteOrder can be. I like better the proposal you have in section 3.Parameter Properties where you have two states for each parameter, dynamic static, and scoped not-scoped. I'd rather not encode the status of the parameter with respect to these two states in its name. Section 2 I want to noodle with the mathematical semantics of reader/writer/filter a bit here. I think this is all not crisp enough yet. In particular, I want to break streams up into sets of values, and separate out keeping track of cursor position. This will let us make absolutely clear what is going on with what I called "length protocols" in previous emails. So I would redefine reader to take in an input (set of values with positions), a cursor position, and to compute a value, and a new cursor position, but not to have any side-effects, i.e., a pure function. A writer takes a stream (set of values with positions), a current position, and a value, and produces a stream (with more members in the set) and a new position. I need to work on this more and present it back to you all. Section 3 The scoping scheme you describe here makes sense to me. However, I worry about order sensitivity. Suppose I moved the top level bigEndian annotation to a position textually after the definition of xType. Would xType now be without any annotations? Or does order not matter? Order sensitivity of this kind is consistent with programming-language semantics which generally considers both order and nesting level together when considering scope of a declaration. I believe we will want the ability to do both: 1) introduction of global defaults for rep parameters 2) introduction of reusable complexType definitions which *take no position* on rep parameters such that their use points inherit local control over rep. To do this in your example I'd have to be able to get the xType's definition out of the scope of the global declarations somehow. Just switching the order around would do it if we're comfortable with that. > -----Original Message----- > From: Martin Westhead [mailto:martinwesthead@yahoo.co.uk] > Sent: Wednesday, December 08, 2004 3:16 PM > To: dfdl-wg(a)gridforum.org > Subject: [dfdl-wg] Plumbing document > > Hi Folks, > > Sorry you didn't get a round up Susan, I'm afraid we were not > prepared for it. I was on the call this morning ready to > summarise events but I was the only one on. > > Based on discussions, yesterday afternoon I have drawn up the > attached document. The starting point for this was stuff that > was agreed but I have tried to flesh it out and push the > ideas in various directions. > > I have tried to suggest a what would be required to define > and use each of our various operation elements. I have also > proposed a possible solution to the scoping problem. > > This document may not make much sense to those who were not > at the face to face and for that reason I do not propose to > put it on Grid Forge yet. It should provide some material for > our GGF13 submission. > > It would, of course be very useful if those at the F2F could > take a look and comment before our collective memories of > what we discussed fade. > > Thanks, > > Martin > > (Doc is in PDF because I'm not confident that OpenOffice -> > Word will preserve the diagrams. If anyone wants to make > detailed comments I'll send them a Word version too.) >

1 0

F2F Meeting Status Report
by mike.beckerle＠ascentialsoftware.com 08 Dec '04

08 Dec '04

Attached is my status on this F2F. This ".doc" file seems to work in both MS Word and OpenOffice, though the heading numbers do change somewhat. Comments, feedback, things I forgot are all welcome. I'll post to grid forge once people affirm that I have everything captured, or enough anyway. ...mikeb

1 0

Plumbing document
by Martin Westhead 08 Dec '04

08 Dec '04

Hi Folks, Sorry you didn't get a round up Susan, I'm afraid we were not prepared for it. I was on the call this morning ready to summarise events but I was the only one on. Based on discussions, yesterday afternoon I have drawn up the attached document. The starting point for this was stuff that was agreed but I have tried to flesh it out and push the ideas in various directions. I have tried to suggest a what would be required to define and use each of our various operation elements. I have also proposed a possible solution to the scoping problem. This document may not make much sense to those who were not at the face to face and for that reason I do not propose to put it on Grid Forge yet. It should provide some material for our GGF13 submission. It would, of course be very useful if those at the F2F could take a look and comment before our collective memories of what we discussed fade. Thanks, Martin (Doc is in PDF because I'm not confident that OpenOffice -> Word will preserve the diagrams. If anyone wants to make detailed comments I'll send them a Word version too.)

1 0

MikeB's random notes from F2F
by mike.beckerle＠ascentialsoftware.com 08 Dec '04

08 Dec '04

This is not a meeting summary. I'll work on that on the plane today and send it subsequently, but people asked about having these notes. This is truly random note taking. ...mikeb --------------------------------------------------------------- Starting Points: * xml xsd - what I already have, but not what I want - what I want * Cobol/C-struct + system and compiler details * data dictionary (ad-hoc spreadsheet or text description) * example data files only Some degree of structural similarity required. Reasonably compatible. You could express any trasnformation at all, but the intent is to make easy those where structural similarity is present. Q: is this just a taste and style issue? ------------------------------------------------ - desirable to have symmetric read/write capability given a DFDL descriptor - all built-in types and reps should implement both directions - choices and use of runtimeValue expressions can create non-invertible parsers - mechanisms can allow explicit introduction of the output formatting properties needed symmetric with the input parsing properties ----------------------------------------------- Ways to refer to one element from the dfdl annotations of another: 1) use value of other element in runtimeValue expression. 2) use value of other element as source for dfdl read conversion 3) use value of other element as a parameter for dfdl read conversion (note 2 and 3 are the same if the source is just another parameter) ----------------------------------------------- Hypothesis: no XSD syntax is needed inside DFDL rep annotations. Can instead reference a type name, and name elements within it. --------------------------------------------- Choice groups imply the need to have an additional element to provide a name for the choice. This is required only when the alternatives of the choice contain a single value. -------------------------------------------- Key topic: bining and passing mechanisms for property values aka parmeters to read/write conversions Parameterization and binding examples 1) Mime type - image format logical model looks like bmp black box read conversion 2) complex number with 2 possible component order, realFirst or imaginaryFirst white box complexComponentOrder is the parameter ------------------------------------------------------- Agreed: "transforms" will be called readers and writers, collectively converters and conversions -------------------------------------------------------- Proposal for parking lot: discontiguous representations. E.g., file full of variable length strings where the length fields are all first, then all the contents. -------------------------------------------------------- XSLT - has variables and things we can use as constructs. E.g., they use this idiom <xsl:value-of name="variable" select="...."/> and equivalently <xsl:value-of name="variable">....</xsl:value-of> --------------------------------------------------------- Issue: when annotations are added on to an element, can we validate that only relevant properties are asserted for that element? Is it desirable to insure that only relevant properties are asserted, or should irrelevant properties simply be ignored? Position - rule out irrelevant attributes improves validity checking, catches errors earlier. E.g., I keep changing the byteOrder setting, but nothing is changing in the data I'm reading (turns out it's because byte order is irrelevant, but if nothing was checking that nothing would help you find that out.) Position - tolerate irrelevant attributes improves flexibility (e.g., if you change the overall representation, you don't have to edit all the other properties that no longer apply. A single file of DFDL can capture characteristicts of more than one representation (at least one text and one binary flavor, though this doesn't generalize.) ----------------------------------------------------------- Issue: parameterization of transforms seems like the OMG DT model and the transform descriptions (alan's proposal) are very very close conceptually, but exactly how isn't entirely clear. ----------------------------------------------------------- Preprocessing an attribute called source (and presumably another called target) ---------------------------------------------------------- 5 kinds of operations reader writer filter change filter function known signatures we can chain them together conceptually think of this as pull model, or perhaps the DFDL expressions don't take any position on whether the implementation is pull or push. should be a way to create pull-model code in a programming language and use it as an augmentation of the DFDL system. could be ways to also adapt push model code, or other schemes like stateful threads. Where can these go in DFDL? - readers and writers go on elements - filters go on a special construct for creating sources or targets from other sources or targets I/O asymetries - using filters you are discarding information, so it affects ability to exactly reproduce output. Box and arrow diagrams using these function types can be used to provide a semantics for DFDL. ----------------------------------------------------- <element name="charstream" type="dfdl:sourceStream"> <annotation><appinfo source="..."> <dfdl:sourceStreamTD> <charset>utf-8</charset> <source>byteStream</source> <filter>bytesToChars</filter> </dfdl:sourceStreamTD> </appinfo></annotation> </element> <element name="s" type="dfdl:sourceStream"> <annotation><appinfo source="..."> <dfdl:sourceStreamTD> <filter>replaceRegexp("...regexp for C-comments...", "")</filter> <source>charstream</source>> </dfdl:sourceStreamTD> </appinfo></annotation> </element> <element name="t" type="dfdl:targetStream"> <annotation><appinfo source="..."> <dfdl:targetStreamTD> <charset>utf-8</charset> <target>outbyteStream</target> <filter>charsToBytes</filter> </dfdl:targetStreamTD> </appinfo></annotation> </element> <element name="toplevel"> <annotation><appinfo source="..."> <dfdl:instanceTD> <source>s</source> <target>t</target> <repType>text</repType> </dfdl:instanceTD> </appinfo></annotation> <sequence> <element name="len" type="int"> <annotation><appinfo source="..."> <intTD> <terminator>\p{newline}</terminator> </intTD> </appinfo></anntation> </element> <element name="val" type="int" minOccurs="0" maxOccurs="unbounded"> <annotation><appinfo source="..."> <intTD> <arrayTD> <storedLength>../len</storedLength> <terminator>\p{newline}</terminator> <separator>\p{space}</separator> </arrayTD> <numbase>10</numBase> <reader name="myIntReader"> <numberOfBits>13</numberOfBits> </reader> </intTD> </appinfo></anntation> </element> </sequence> </element> -------------------------------------------------------- Still open issues: 1) scoping of property definitions. Useful or source of bad interactions? 2) how to organize model of the properties for the types - suman and mike in rough agreement. 3) ----------------------------------------------

1 0

Wrap up call from F2F today?
by Susan Malaika 07 Dec '04

07 Dec '04

Will there be a wrap-up call from F2F today? We did not see any callin phone number posted. Susan Malaika

1 0

Fw: [dfdl-wg] Annotation complexity
by Steve Hanson 02 Dec '04

02 Dec '04

Some thoughts on this from one of the IBMers on the XML schema WG. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 02/12/2004 10:05 ----- Sandy Gao/Toronto/IBM@I BMCA To Steve Hanson/UK/IBM@IBMGB 30/11/2004 14:58 cc Suman Kalia/Toronto/IBM@IBMCA Subject Re: Fw: [dfdl-wg] Annotation complexity(Document link: Steve Hanson) > One more thought in this area. XML Schema itself does not have the notion of inheriting properties from a parent object. I think by "parent object" you mean the containing component, instead of something like the "base type". I don't think schema has a general principle on this. The only case I can think of where things are inherited (or defaulted using values from parent) is for blockDefault, finalDefault, elementFormDefault and attributeFormDefault attributes on the <schema> element. As for your specific example, I don't know whether the Schema WG has considered it (my involvement in schema doesn't go that far back), but my intuition here is that I don't think that's something schema would want to do. I think defaults are for the common cases, but it's not that common for all elements inside a complex type to share the same type. Don't know whether that answered your question. Let me know if there's anything more I can do. Cheers, Sandy Gao Software Developer, IBM Canada (1-905) 413-3255 sandygao(a)ca.ibm.com ----- Forwarded by Steve Hanson/UK/IBM on 30/11/2004 11:39 ----- Steve Hanson/UK/IBM To 23/11/2004 09:38 Sandy Gao/Toronto/IBM cc Suman Kalia/Toronto/IBM@IBMCA Subject Fw: [dfdl-wg] Annotation complexity Hi Sandy There is a Global Grid Forum (GGF) initiative underway called Data Format Description Language (DFDL) which aims to model non-XML data using XML schema for the logical structure and xsd:appinfo annotations for the physical 'bitstream' representation information. IBM is involved in this (myself and Suman Kalia) because the IBM message broker product (WVBI-MB) already does exactly this in its 'message model', so we are using our experience in this area to help influence this initiative. I'd like to ask your opinion on something quite specific. See the mail directly below. Did the XML Schema WG think about the possibility of providing an inheritance or defaulting capability as in my example? And if so, what were the precise reasons for excluding it? I'm asking because several folk in the DFDL WG are keen on having this as a way of making a DFDL annotated schema more compact. If you are interested in finding out more, please visit http://forge.gridforum.org/projects/dfdl-wg/ Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/11/2004 09:30 ----- Steve Hanson/UK/IBM To 23/11/2004 09:29 dfdl-wg(a)gridforum.org cc Subject Fw: [dfdl-wg] Annotation complexity One more thought in this area. XML Schema itself does not have the notion of inheriting properties from a parent object. For example, if I have a complex type that contains only integer elements, there is no facility to factor out the xsd:type attribute from all the elements and instead set it as an attribute of the complex type, thereby making the schema more compact. Was this a deliberate decision? I'll see if I can find out. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/11/2004 09:27 ----- Steve Hanson/UK/IBM To 22/11/2004 11:24 dfdl-wg(a)gridforum.org cc Subject RE: [dfdl-wg] Annotation complexity (Document link: Steve Hanson) > Another approach would be to allow annotation locations to be > specified using XPath. Since XPath can specify both specific > locations and large groups of locations this could be quite > powerful. We used to have a similar concept in the IBM broker message model, we called it 'element qualifier'. A top level global element could carry a reference (XPATH) to an element it ultimately contained, for the purposes of overriding certain properties, including default value. We found that: a) Their use made the model much harder to understand b) The code we had to write was considerably more complex c) No customer ever admitted to using the capability We withdrew the facility with no deprecation warning in V5 (June 2003) and so far nobody has shouted. Was this the kind of thing you are proposing? Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 "Myers, James D" <jim.myers(a)pnl.go v> To Sent by: dfdl-wg(a)gridforum.org owner-dfdl-wg@ggf cc .org Subject RE: [dfdl-wg] Annotation complexity 19/11/2004 21:03 > > Myers, James D wrote: > > I guess I'm not sure how restricting annotations to elements solves > > things. > > Sorry, I am finding it more difficult to be precise than I > thought with > this. What I mean here is _leaf_ elements/attributes - which > I think can > be defined as elements with simple type descriptions. I think if you > stick to these its unambiguous...no? I agree, I think - if anotations are only on simple types. That would eliminate the element hierarchy, though - does it eliminate the type hierarchy - simple element in base type has an annotation that gets inherited by the same simple element in a derived type. I think this would be unambiguous, but perhaps hard to find (surprise the global dfdl default for endianess doesn't apply because some subtype three levels back had an annotation) > > I think Mike has examples of annotations that he would like higher in > the tree but with this extreme position there would be no inheritance. Except through type derivation...? > > Two other thoughts occur: > - I think definitions applied to a data source are > orthogonal to this > scoping issue. We still need to understand precedence but that seems > relatively easy to resolve. Definitions on sources: I know we've talked about this (!!! - the whole outside-in or inside out representations...), but I think the only mechanism for this in the examples is to assign a source to an element and annotate that element with, for example, endianness. For some types of sources, e.g. other layers, that have an internal structure, any annotation of the source itself would essentially become an inheritable annotation on the elements within it. So - we could try to define such a mechanism to apply to sources, but I'm not sure it would be applicable to all without recreating some of the inheritance issues. > > - Another approach would be to allow annotation locations to be > specified using XPath. Since XPath can specify both specific > locations > and large groups of locations this could be quite powerful. Hmmm. Yes, as long as xpaths can only be defined from a single root, i.e. not from within type definitions relative to the type (not talking about layers here). Otherwise we don't get rid of inheritance/precedence issues. At a minimum, this approach might help collect all of the defaults together at the top level for readability, but it might separate defaults from reusable types. Jim

1 0

Fw: [dfdl-wg] Discussion forums - try them out
by Steve Hanson 01 Dec '04

01 Dec '04

Mike - working now. It looks a very useful facility. It might be worth posting some of our existing discussion threads up there. I'll see if I get time to do that tomorrow. I added one append on the F2F. Suman, Susan - suggest you add yourselves as DFDL group members via the web page http://forge.gridforum.org/projects/dfdl-wg/ - I think that was what gave me the correct access in the end. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 01/12/2004 16:16 ----- Steve Hanson/UK/IBM To 01/12/2004 10:07 mike.beckerle(a)ascentialsoftware.com cc Subject Fw: [dfdl-wg] Discussion forums - try them out Mike - I get this error when I follow the link Error Permission denied. Exiting with Error Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 01/12/2004 10:06 ----- mike.beckerle@asc entialsoftware.co m To Sent by: dfdl-wg(a)gridforum.org owner-dfdl-wg@ggf cc .org Subject [dfdl-wg] Discussion forums - try 01/12/2004 00:42 them out https://forge.gridforum.org/forum/forum.php?forum_id=455 We have discussion forums now. I added one about the agenda for our F2F. Please post some comments and we'll see how it works. ...mikeb

1 0

Discussion forums - try them out
by mike.beckerle＠ascentialsoftware.com 01 Dec '04

01 Dec '04

https://forge.gridforum.org/forum/forum.php?forum_id=455 We have discussion forums now. I added one about the agenda for our F2F. Please post some comments and we'll see how it works. ...mikeb

1 0

RE: [SPAM] RE: [dfdl-wg] BSDF
by mike.beckerle＠ascentialsoftware.com 24 Nov '04

24 Nov '04

Good find Jim, One thing we can look at from these guys is their handling of bit-packed fields. I've kind of been hand-waving the whole bit-level thing recently as less important than "hard issues" of compositionality and layering, but we do have to face this issue eventually. Other than that, from my skim I don't see how we don't end up just subsuming what they do. I expect there's quite a few efforts out there to add some sort of dense-packed format description capability to XML. Until DFDL really "comes out" I think we'll see more and more of this. In some sense this bodes well for the eventual success of DFDL. In every case I suggest we reach out to them to be sure they are aware of the DFDL effort, and then they can decide to contribute or not, but at least they won't be working in a vaccum. At some point we'll badly need poeple to review specs and consider if DFDL meets their needs. ...mikeb _____ From: Myers, James D [mailto:jim.myers@pnl.gov] Sent: Tuesday, November 23, 2004 10:39 AM To: dfdl-wg(a)gridforum.org Subject: [SPAM] RE: [dfdl-wg] BSDF I ran across another community working in the same space as DFDL: BSDL as part of MPEG21 (someone at the NARA conference mentioned it). From a quick look at what they have, it appears to be inverse DFDL with plans to make DFDL next, i.e. they go from model to output binary format, but not vice versa. Similar schema approach, etc. with some comments on things in schema that they don't allow. I haven't found a spec doc yet, just a paper and some presentations you can google for. Jim <http://www2002.org/CDROM/alternate/334/> Bitstream Syntax Description Language: Application of XML-Schema ... ... Language (BSDL) that embraces such XML bitstream descriptions. ... 2/; [13] IEEE Standard for Binary Floating-Point ... Final Draft Amendment 4: Streaming video profile ... www2002.org/CDROM/alternate/334/

1 0

RE: [dfdl-wg] Transform examples
by mike.beckerle＠ascentialsoftware.com 24 Nov '04

24 Nov '04

To this set I would add: - Parse (or Parser, with the output-direction to be called unparse/unparser) > -----Original Message----- > From: Steve Hanson [mailto:smh@uk.ibm.com] > Sent: Wednesday, November 24, 2004 6:34 AM > To: dfdl-wg(a)gridforum.org > Subject: Fw: [dfdl-wg] Transform examples > > > > > > Here's a candidate list of words. > > - Transform > - Map > - Converter > - Conversion > - Translation > - Render > - Representation > - Formatter > - Cast > > We could also consider prefixing a word with 'Physical', eg, > 'PhysicalRender' > > Regards, Steve > > > ----- Forwarded by Steve Hanson/UK/IBM on 24/11/2004 11:08 ----- > > > Martin Westhead > > <martinwesthead@y > > ahoo.co.uk> > To > Sent by: "Chappell, Alan R" > > owner-dfdl-wg@ggf > <chappella(a)BATTELLE.ORG>, > .org dfdl-wg(a)gridforum.org > > > cc > > > 22/11/2004 17:31 > Subject > Re: [dfdl-wg] > Transform examples > > > > > > > > > > > > > > > > > Hi Alan, > > Thanks for this. Some thoughts/questions/comments that > occurred to me looking at your example. > > First a general point: > > - On one of the calls which I think was just me and the > IBMers they pointed out quite strongly that the ther > "transformation" has other connotations that make it very > confusing here, Mike also agrees. I think we need to choose a > different name to avoid such confusion. Mike suggested > "converters"/"convertions" which is suitably general. I also > thought about "casting" but I think that's perhaps too > specific. Either way it would be nice to choose something or > our discussions will be rather difficult to follow. > > (I'll continue to use transform for this email...) > > On the example: > - I was expecting the definition of the input and output > types to the transform should have equal status. Here it > seems that the input is specified implicitly in the parameter > set it will accept. > > - Why does asciiToInt read byte[]? I would expect it to > read at least char[] (or probably xs:string, since there is > no xs:char). The intent would be that you could then compose > it with a character converter of your choice. > > - In general I think the typing and compositional > properties of the transformations are the nut that requires > cracking here. My proposal is that the types should be > specified in XML data model at all levels so instead of > byte[] you would have: > > <xs:sequence> > <xs:element name="anon" type="xs:byte" minOccurs="0" > maxOccurs="unbounded"/> > </xs:sequence> > > that's all for now... > > Cheers, > > Martin > > > > > > > > > Chappell, Alan R wrote: > > Third try... No zip, just the 3 files important to the simple > > transform example.... > > > > > ---------------------------------------------------------------------- > > -- > > *From:* Chappell, Alan R > > *Sent:* Friday, November 19, 2004 1:39 PM > > *To:* dfdl-wg(a)gridforum.org > > *Subject:* *MJ-REJECTED* Transform examples > > > > Second try on sending these examples. I've cut the set down > to the 3 > > important files so hopefully it will get through this time. > > > > > ---------------------------------------------------------------------- > > -- > > *From:* Chappell, Alan R > > *Sent:* Thursday, November 18, 2004 8:47 AM > > *To:* dfdl-wg(a)gridforum.org > > *Subject:* *MJ-REJECTED* Transform examples > > > > Here is the example I mentioned yesterday. Look particularly at > > dfdltransforms.xsd, BasicAsciiIntExp.xsd, and > BasicBinIntExp.xsd. Note > > the "Exp" on those last two files indicate that they are > expansions of > > the information in the original versions of those files. > These make a > > first stab at giving a fully verbose description of the > structure and > > the transforms, i.e., it's working towards the canonical > > representation Martin talked about yesterday. The "dfdltransforms" > > gives the definitions of transforms and their components. > > > > There are lots of things that can be improved here. > > <<dfdl-examples.zip>> > > Alan R. Chappell > > chappella(a)battelle.org > > > > Pacific Northwest National Laboratory > > Battelle Seattle Research Center > > (206) 528-3228 > > > > >

1 0