- dfdl-wg - lists.ogf.org

Fw: [dfdl-wg] Annotation complexity
by Steve Hanson 16 Mar '05

16 Mar '05

Some thoughts on this from one of the IBMers on the XML schema WG. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 02/12/2004 10:05 ----- Sandy Gao/Toronto/IBM@I BMCA To Steve Hanson/UK/IBM@IBMGB 30/11/2004 14:58 cc Suman Kalia/Toronto/IBM@IBMCA Subject Re: Fw: [dfdl-wg] Annotation complexity(Document link: Steve Hanson) > One more thought in this area. XML Schema itself does not have the notion of inheriting properties from a parent object. I think by "parent object" you mean the containing component, instead of something like the "base type". I don't think schema has a general principle on this. The only case I can think of where things are inherited (or defaulted using values from parent) is for blockDefault, finalDefault, elementFormDefault and attributeFormDefault attributes on the <schema> element. As for your specific example, I don't know whether the Schema WG has considered it (my involvement in schema doesn't go that far back), but my intuition here is that I don't think that's something schema would want to do. I think defaults are for the common cases, but it's not that common for all elements inside a complex type to share the same type. Don't know whether that answered your question. Let me know if there's anything more I can do. Cheers, Sandy Gao Software Developer, IBM Canada (1-905) 413-3255 sandygao(a)ca.ibm.com ----- Forwarded by Steve Hanson/UK/IBM on 30/11/2004 11:39 ----- Steve Hanson/UK/IBM To 23/11/2004 09:38 Sandy Gao/Toronto/IBM cc Suman Kalia/Toronto/IBM@IBMCA Subject Fw: [dfdl-wg] Annotation complexity Hi Sandy There is a Global Grid Forum (GGF) initiative underway called Data Format Description Language (DFDL) which aims to model non-XML data using XML schema for the logical structure and xsd:appinfo annotations for the physical 'bitstream' representation information. IBM is involved in this (myself and Suman Kalia) because the IBM message broker product (WVBI-MB) already does exactly this in its 'message model', so we are using our experience in this area to help influence this initiative. I'd like to ask your opinion on something quite specific. See the mail directly below. Did the XML Schema WG think about the possibility of providing an inheritance or defaulting capability as in my example? And if so, what were the precise reasons for excluding it? I'm asking because several folk in the DFDL WG are keen on having this as a way of making a DFDL annotated schema more compact. If you are interested in finding out more, please visit http://forge.gridforum.org/projects/dfdl-wg/ Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/11/2004 09:30 ----- Steve Hanson/UK/IBM To 23/11/2004 09:29 dfdl-wg(a)gridforum.org cc Subject Fw: [dfdl-wg] Annotation complexity One more thought in this area. XML Schema itself does not have the notion of inheriting properties from a parent object. For example, if I have a complex type that contains only integer elements, there is no facility to factor out the xsd:type attribute from all the elements and instead set it as an attribute of the complex type, thereby making the schema more compact. Was this a deliberate decision? I'll see if I can find out. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 23/11/2004 09:27 ----- Steve Hanson/UK/IBM To 22/11/2004 11:24 dfdl-wg(a)gridforum.org cc Subject RE: [dfdl-wg] Annotation complexity (Document link: Steve Hanson) > Another approach would be to allow annotation locations to be > specified using XPath. Since XPath can specify both specific > locations and large groups of locations this could be quite > powerful. We used to have a similar concept in the IBM broker message model, we called it 'element qualifier'. A top level global element could carry a reference (XPATH) to an element it ultimately contained, for the purposes of overriding certain properties, including default value. We found that: a) Their use made the model much harder to understand b) The code we had to write was considerably more complex c) No customer ever admitted to using the capability We withdrew the facility with no deprecation warning in V5 (June 2003) and so far nobody has shouted. Was this the kind of thing you are proposing? Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848 "Myers, James D" <jim.myers(a)pnl.go v> To Sent by: dfdl-wg(a)gridforum.org owner-dfdl-wg@ggf cc .org Subject RE: [dfdl-wg] Annotation complexity 19/11/2004 21:03 > > Myers, James D wrote: > > I guess I'm not sure how restricting annotations to elements solves > > things. > > Sorry, I am finding it more difficult to be precise than I > thought with > this. What I mean here is _leaf_ elements/attributes - which > I think can > be defined as elements with simple type descriptions. I think if you > stick to these its unambiguous...no? I agree, I think - if anotations are only on simple types. That would eliminate the element hierarchy, though - does it eliminate the type hierarchy - simple element in base type has an annotation that gets inherited by the same simple element in a derived type. I think this would be unambiguous, but perhaps hard to find (surprise the global dfdl default for endianess doesn't apply because some subtype three levels back had an annotation) > > I think Mike has examples of annotations that he would like higher in > the tree but with this extreme position there would be no inheritance. Except through type derivation...? > > Two other thoughts occur: > - I think definitions applied to a data source are > orthogonal to this > scoping issue. We still need to understand precedence but that seems > relatively easy to resolve. Definitions on sources: I know we've talked about this (!!! - the whole outside-in or inside out representations...), but I think the only mechanism for this in the examples is to assign a source to an element and annotate that element with, for example, endianness. For some types of sources, e.g. other layers, that have an internal structure, any annotation of the source itself would essentially become an inheritable annotation on the elements within it. So - we could try to define such a mechanism to apply to sources, but I'm not sure it would be applicable to all without recreating some of the inheritance issues. > > - Another approach would be to allow annotation locations to be > specified using XPath. Since XPath can specify both specific > locations > and large groups of locations this could be quite powerful. Hmmm. Yes, as long as xpaths can only be defined from a single root, i.e. not from within type definitions relative to the type (not talking about layers here). Otherwise we don't get rid of inheritance/precedence issues. At a minimum, this approach might help collect all of the defaults together at the top level for readability, but it might separate defaults from reusable types. Jim

1 0

Call today
by mike.beckerle＠ascentialsoftware.com 16 Mar '05

16 Mar '05

I will be on the call today. I can give a status report on the prototype we're doing here. ...mikeb

1 0

Interesting news: IBM Acquires Ascential
by mike.beckerle＠ascentialsoftware.com 14 Mar '05

14 Mar '05

Press Release IBM to Acquire Ascential Software Extends Leadership in Helping Customers Integrate, Structure and Manage Information ARMONK, NY and WESTBORO, Mass.-March 14, 2005-IBM (NYSE: IBM) and Ascential Software Corporation (Nasdaq: ASCL) today announced the two companies have entered into a definitive agreement for IBM to acquire the equity of Ascential Software, a publicly held company based in Westboro, Mass., in an all cash transaction at a price of approximately $1.1 billion or $18.50 per share. The acquisition is subject to Ascential Software shareholder and regulatory approvals and other customary closing conditions, and is expected to close in the second quarter of 2005. Ascential Software is a leading provider of enterprise data integration software. Customers worldwide use Ascential's data integration software to build enterprise data warehouses, power business intelligence systems, consolidate enterprise applications, create and manage master repositories of critical business information, and enable on demand data access. Ascential Software's integrated portfolio helps companies solve their most complex and demanding data integration challenges to streamline operations, improve customer support, expedite business transactions, and make better strategic decisions. The acquisition complements and strengthens IBM's fast-growing information integration business, a key part of the company's information management efforts and a unit that produced triple-digit growth in 2004. Ascential Software also grew rapidly in 2004, with a reported 46 percent total revenue increase to $271.9 million. "Information integration is an important enabler of an on demand business strategy and customers are increasing their investments in software that allows them to rapidly analyze, consolidate and extract value from their business data," said Steve Mills, senior vice president and group executive, IBM Software Group. "The acquisition of Ascential Software expands IBM's open information integration platform and strengthens our ability to help customers create an environment that delivers the data they need, in the right form, to the right location, and at the right time. That's the mark of an on demand business." The acquisition announced today addresses a key customer challenge -- applying technology to respond much more quickly to changing market conditions. For example, some of the largest retailers use Ascential's software to gather, standardize and structure sales information from multiple channels (for example, Internet, catalogs, storefronts), providing a single view of vendors and customers from disparate systems, and make rapid inventory and pricing adjustments in response to changing market demands. Enabling customers to react and respond with speed to rapidly emerging business opportunities is an example of IBM's on demand business strategy. Ascential Software's ability to quickly gather, move and enhance the quality of large amounts of data complements IBM's WebSphere Information Integrator product portfolio. WebSphere Information Integrator software enables customers to centrally manage and access data that is stored across a variety of structured and unstructured sources, from IBM and non-IBM vendors, in real-time. This combination will help customers solve numerous integration problems. For example, a company trying to consolidate data from multiple ERP systems into a single system could leverage WebSphere Information Integrator to access various mainframe or distributed sources for profiling and assessment, and then use Ascential Software's data migration and transformation capabilities to integrate the data. IBM is acquiring Ascential Software to extend the industry's broadest and deepest offering for standards-based business integration across heterogeneous environments. With more than 3,000 customers and partners across all major industries, Ascential Software is widely viewed as a leader in enterprise data integration. IDC, a global market intelligence and advisory firm in the information technology and telecommunications industries, projects that worldwide data integration spending will increase from $9.3 billion in 2003 to $13.6 billion in 2008. IDC also reports that the top 3 business issues most organizations say they need to be addressed through integration are to respond faster to changing business needs, improve operational productivity of employees, and provide better service to customers.1 "The integration market is changing and evolving as customers demonstrate a preference for broad solutions that meet enterprise integration requirements," said Peter Gyenes, chairman and chief executive officer, Ascential Software. "IBM shares our understanding that data integration is a strategic component of information and process integration and offered us this opportunity to combine and form a stronger competitor in the large and growing integration market. At the same time, this transaction offers Ascential Software shareholders solid appreciation over recent trading prices of our stock, without the execution and business risk." Once the acquisition is completed, IBM will: * Establish Ascential Software's operations as a business unit within IBM's Information Management software division led by General Manager Janet Perna. * Incorporate Ascential Software technology and solutions into IBM's Information Management and Software Group offerings. * Market and sell Ascential Software products through IBM's and Ascential Software's worldwide sales channels and Business Partners. * Build upon the companies' existing partnership, which includes more than 550 joint customers worldwide. Ascential Software already integrates with IBM WebSphere Business Integration software as part of a service-oriented architecture (SOA), providing complex data movement and transformation within a process step, thereby helping customers speed time-to-market and increase their ROI. This acquisition, combined with IBM's middleware portfolio, strengthens IBM's leadership in key on demand initiatives such as business intelligence, business performance management, business transformation, multi-channel commerce, RFID, merger and acquisition consolidation, master data management and regulatory compliance. The Ascential Software technology also furthers IBM's commitment to open computing through its ability to support both IBM and non-IBM data sources, including those of Oracle, Microsoft and SAP. About IBM IBM is the world's largest information technology company, with 80 years of leadership in helping businesses innovate. Drawing on resources from across IBM and key Business Partners, IBM offers a wide range of services, solutions and technologies that enable customers, large and small, to take full advantage of the new era of e-business. For more information about IBM, visit <http://www.ibm.com/> www.ibm.com. About Ascential Software Corporation Ascential Software Corporation (Nasdaq: ASCL) is the leader in enterprise data integration. Customers and partners worldwide use the Ascential Software Enterprise Integration Suite to confidently transform data into accurate, reliable and complete business information to improve operational performance and decision-making across every critical business dimension. Our comprehensive end-to-end solutions provide on demand data integration complemented by our professional services, industry expertise, and methodologies. Ascential Software is headquartered in Westboro, Mass., and has more than 3,000 customers and partners globally across such industries as financial services and banking, insurance, healthcare, retail, manufacturing, consumer packaged goods, telecommunications and government. For more information call 1-800-966-9875 (or 1-508-366-3888 from outside the U.S. or Canada) or visit the Ascential Software website at www.ascential.com. NOTE TO INVESTMENT COMMUNITY: Financial analysts can find more information regarding this acquisition at: www.ibm.com/investor. Ascential Software management will host a conference call at 8:00 AM (EST) today to discuss this important announcement. The conference call will be broadcast live through a link on the Investor Relations page on the Ascential Software web site at www.ascential.com/investors <http://www.ascential.com/investors> . For those who cannot attend the live broadcast, a replay will be available on the web site at www.ascential.com/investors <http://www.ascential.com/investors> or by calling (800) 945-1634 beginning about two hours after the call ends. No passcode is required for the replay. 1 IDC Worldwide Data Integration Spending 2004-2008 Forecast, July 2004 (IDC #31546), Steve McClure IBM, DB2, and IBM are trademarks of International Business Machines Corporation in the United States, other countries, or both. Other company, product or service names may be trademarks or service marks of others. ADDITIONAL INFORMATION ABOUT THE MERGER AND WHERE TO FIND IT In connection with the proposed merger and required stockholder approval, Ascential Software Corporation (the "Company") will file with the SEC a preliminary proxy statement and a definitive proxy statement that will contain important information about the Merger. Investors and security holders of the Company are urged to read the proxy statement, and any other relevant materials filed by the Company because they contain, or will contain, important information about the Company and the Merger. All documents filed by the Company with the SEC, may be obtained for free at the SEC's website at www.sec.gov. In addition, the documents filed with the SEC by the Company may be obtained free of charge by directing such request to: David Roy, Vice President of Investor Relations, (800) 966-9875 or from the Company's website at www.ascential.com. The Company and its executive officers and directors may be deemed to be participants in the solicitation of proxies from the stockholders of the Company in favor of the acquisition. Information about the executive officers and directors of the Company and their ownership of the Company's common stock is set forth in the proxy statement for the Company's 2004 Annual Meeting of Shareholders, which was filed with the SEC on April 27, 2004. Investors and security holders may obtain more detailed information regarding the direct and indirect interests of the Company and its respective executive officers and directors in the acquisition by reading the preliminary and final proxy statements regarding the acquisition, which will be filed with the SEC. Safe Harbor Statement under the Private Securities Litigation Reform Act of 1995: This release contains forward-looking statements based on current expectations or beliefs, as well as a number of assumptions about future events, and these statements are subject to important factors and uncertainties that could cause actual results to differ materially from those described in the forward-looking statements. The forward-looking statements in this release address a variety of subjects including, for example, the functionality, characteristics, quality and performance capabilities of Ascential Software's products and technology; results achievable and benefits attainable through deployment of Ascential Software's products and provision of services; the ability of Ascential Software's products to help companies streamline operations, improve customer support, expedite business transactions, and make better tactical and strategic decisions; and improvements in operational performance and decision-making. The following factors, among others, could cause actual results to differ materially from those described in these forward-looking statements: the ability of the company to expand its market share; growth rates for the enterprise data integration software market; general business conditions in the software industry, the technology sector, and in the domestic and international economies; rapid technological change in the markets served by the company; dependence on international operations; global and geopolitical instability; compliance costs associated with regulations to which the company is subject, including the Sarbanes-Oxley Act of 2002, and the impact of such regulations; and difficulties that the company may experience integrating technologies, operations and personnel of completed or future acquisitions. For a detailed discussion of these and other cautionary statements, please refer to the filings made by Ascential Software with the Securities and Exchange Commission, including, without limitation, the most recent Quarterly Report on Form 10-Q. Ascential Software disclaims any intent or obligation to update any forward-looking statements made herein to reflect any change in Ascential Software's expectations with regard thereto or any change in events, conditions, or circumstances on which such statements are based. Ascential Software is not responsible for statements attributed to third parties within this release or in any materials referenced herein. Partnerships described herein shall be deemed to be strategic alliances only, and shall not imply the power of either partner to bind the other partner or to incur obligations on behalf of the other partner.

1 0

Some things that need discussion
by Steve Hanson 11 Mar '05

11 Mar '05

Here is a list of things that are important to resolve for DFDL but which I don't recall seeing discussed or in a spec. 1) Rules for application of default values. XML Schema has rules for default value application when handling XML instance documents (they are different for elements and attributes). These rules could/should apply in some non-XML circumstances but are not applicable to others. I think we need to agree the rules that apply to different non-XML circumstances on both input and output. For example, - Fixed length data (eg, COBOL) - here the data must be present on input, default values could be added if missing on output - When a separator is present - this changes things as a double separator can occur - but does this indicate missing or empty? Customers use both semnatics, and Schema rules distinguish the two cases. - When an initiator is present - here missing is observably different from empty, so we could probably use Schema rules here. I have a draft proposal I am working on for this which I could share. 2) Properties - on object or on object inclusion. I didn't see anywhere in Mike's properties spec that said a property occurred on an element/attribute per se, or on its use in a structure. Eg, offset is clearly something that is only applicable to a local element or element reference, you would not put it on a global element. But some other properties are perhaps not so clear cut. 3) Mike's properties spec does not impose any restrictions on the use of different ways of identifying an element in the bitstream. Examples: - Optionality - are eg COBOL fixed length elements allowed to be optional - if so how can you identify one is missing. Here the IBM model mandates that maxOccurs must appear. - Unordered content - what should a DFDL parser do when faced with an xsd:all group - unless an initiator is present, should xsd:all be treated as an xsd: sequence when parsing? 4) Wildcards and 'self-defining' content - what are the rules that apply here? While this might seem unimportant if your starting point is a fixed file, in the messaging world this is frequently encountered - eg HL7 or X12 users will agree their own private extensions to the standard and add extra data, we must be able to model and parse/write this. 5) Truncation/omission rules when separators are involved - we have some extra options in the IBM model and several parsing rules, which we find necessary to cope with our user's CSV style messages. I should note at this point that the IBM model has the concept of 'separation type' which is a property of a group. It stipulates that all members of that group follow a certain pattern - examples are 'fixed length', 'separated', 'tagged and separated', 'use a regular expression'. We have found this a convenient way to define rules for default value, unordered content, wildcard and open content, etc, processing. These 'separation types' can be considered as specializations of a general case where the members of a group do not all follow a pattern. Clearly we need to define rules for the general case, but I think we should also consider whether such specializations are a useful addition to the DFDL model. I would observe that the majority of customers element groups to indeed follow one pattern or another (eg, COBOL - fixed length, CSV - separated). Apologies if any of these have been discussed already, eg at the f-2-f, or prior to GGF12, and I have missed it. If it is more convenient we could start up a series of discussion documents on the forum rather than the usual e-mail chain. I was certainly having trouble tracking the various mails about multi-dimensional arrays. Regards, Steve Steve Hanson WebSphere Business Integration Brokers, IBM Hursley, England Internet: smh(a)uk.ibm.com Phone (+44)/(0) 1962-815848

1 0

dfdl.info
by mike.beckerle＠ascentialsoftware.com 10 Mar '05

10 Mar '05

Ascential has parked the dfdl.info domain as well for our future use. ...mikeb

1 0

RE: [dfdl-wg] How to handle multi-dimensional arrays - version 2
by Myers, James D 07 Mar '05

07 Mar '05

> > Ok, so I guess your point I can summarize like this: > > MD arrays shouldn't need special treatment. We should be able > to compose what we need from lower levels that have only a > small number of primitives. Rich functionality must be able > to be built up from this. If we can't so this, that is, if > the bottom level has to have lots of elaborate constructs, > then something is wrong with the composition model. yes... if the goal is to provide a fine-grained XML-model as in your example. If there's a need to expose an MD array as a single XML element, then we need a new datatype. Otherwise I think treating MD arrays as special starts mixing types and reader params in ways that limit composition. Jim

1 0

RE: [dfdl-wg] How to handle multi-dimensional arrays - version 2
by Mike Beckerle 07 Mar '05

07 Mar '05

Ok, so I guess your point I can summarize like this: MD arrays shouldn't need special treatment. We should be able to compose what we need from lower levels that have only a small number of primitives. Rich functionality must be able to be built up from this. If we can't so this, that is, if the bottom level has to have lots of elaborate constructs, then something is wrong with the composition model. I'll give this some more thought. ...mikeb > -----Original Message----- > From: owner-dfdl-wg(a)ggf.org [mailto:owner-dfdl-wg@ggf.org] On > Behalf Of Myers, James D > Sent: Thursday, March 03, 2005 5:21 PM > To: dfdl-wg(a)gridforum.org > Subject: RE: [dfdl-wg] How to handle multi-dimensional arrays > - version 2 > > I sort-of agree :-) I think the distinctions I'm making are > subtle, but important with repect to composability/layers, > but don't shift what you can do in the multidimensional array > case from where you're trying to go. And if that doesn't make > your eyes cross and cause fits, read on... > > Why haven't you haven't felt it necessary to define an XSD > for vectors beyond putting dfdl:runtimeoccurs limits on how > many to pull from a stream? In this case, the runtimeoccurs > param is a param of the reader that populates a 'normal' XSD > sequence with 'normal' XSD elements. For multidimensional > arrays, the runtimeoccurs parameters for each dimension are > now becoming part of the model rather than parameters of the reader. > I don't know if I like that, but, if we do it, why not do it > everywhere and make, for example, dfdl:byteorder an > attribute on all ints and floats that are read? Of course, a > byteorder attribute would only be available if you actually > came from a binary stream, which may be defined elsewhere > (some enclosing node, another layer). To me, any of this > starts to mix the model and the method used to read the > model, which gets back to the issue of how independent are > readers/ does creating a new reader imply creating a new subtype, etc. > > I guess I'd rather see the concept of multidimensional arrays as > follows: there is not, in fact a multidimensional array on > disk/stream, just a serialized sequence of > ints/floats/whatever. But, to assist the user in interpreting > this flat data as a multidimensional array, we want DFDL to > make index info available and, rather than just making a > single cursor count available and requiring users to do math > to have indexes that don't start at 0 (or 1 - whatever) or > support multiple dimensions, we provide some convenience > mechanisms that can report an index or indexes that cycle > from user defined mins and maxes as elements are read, which > can be used to decorate elements with attributes or be used > in conditional logic, etc. This would preserve the separation > of reader from model, at the expense of saying that indexes > like this are different/ are not like all the dfdl reader > parameters that might be in the current context. > > > > What makes all this confusing for DFDL is that we have some > > representations that are complex enough to need layered multi-step > > descriptions, and once you have that, there's no stopping you from > > using it to do all sorts of transformation from one format > to another. > > So it feels like you can have your cake and eat it too, which is to > > say you can pick your XML Schema and populate it from quite > > differently structured data. And that is probably true, but at the > > bottom level of the stack of layers you have to have a > vocabulary and > > model for directly describing the structure of the data so > as to get > > the whole ball rolling. And at this bottom layer, the needs of > > describing the data format completely dictate what the > schema is like. > > I would solve this by just saying that, at the bottom layer, > there are no single or multidimensional arrays, just > sequences of base types, and that any concept of dimensions > is fabrication created by the user (a very common and > convenient one we might want special support for...). > > The only reason I think we would need a multidimensional > array type in DFDL is if we wanted to directly read m*n bytes > and create a single XML element representing the entire array > that would then have some accessor methods to get a value for > a particular x,y offset pair. I'm not sure what kind of > analogy will make sense to people here, but I see a similar > argument for floats from strings: if you want to create a > float from a sequence of characters, you need a float type. > If you just want to prescribe a standard way to model a > sequence of characters representing a float so that we can > consistently label the mantissa and exponent chars regardless > of storage order, you're not really defining a new float type > in XSD. Instead, your exposing the semantics created/inferred > by the reader as standardized annotations of the existing char (or > string) type (with the annotations being potential or > required depending on whether you let me shut them off or not). > > Jim > >

1 0

RE: [dfdl-wg] How to handle multi-dimensional arrays - version 2
by Myers, James D 03 Mar '05

03 Mar '05

I sort-of agree :-) I think the distinctions I'm making are subtle, but important with repect to composability/layers, but don't shift what you can do in the multidimensional array case from where you're trying to go. And if that doesn't make your eyes cross and cause fits, read on... Why haven't you haven't felt it necessary to define an XSD for vectors beyond putting dfdl:runtimeoccurs limits on how many to pull from a stream? In this case, the runtimeoccurs param is a param of the reader that populates a 'normal' XSD sequence with 'normal' XSD elements. For multidimensional arrays, the runtimeoccurs parameters for each dimension are now becoming part of the model rather than parameters of the reader. I don't know if I like that, but, if we do it, why not do it everywhere and make, for example, dfdl:byteorder an attribute on all ints and floats that are read? Of course, a byteorder attribute would only be available if you actually came from a binary stream, which may be defined elsewhere (some enclosing node, another layer). To me, any of this starts to mix the model and the method used to read the model, which gets back to the issue of how independent are readers/ does creating a new reader imply creating a new subtype, etc. I guess I'd rather see the concept of multidimensional arrays as follows: there is not, in fact a multidimensional array on disk/stream, just a serialized sequence of ints/floats/whatever. But, to assist the user in interpreting this flat data as a multidimensional array, we want DFDL to make index info available and, rather than just making a single cursor count available and requiring users to do math to have indexes that don't start at 0 (or 1 - whatever) or support multiple dimensions, we provide some convenience mechanisms that can report an index or indexes that cycle from user defined mins and maxes as elements are read, which can be used to decorate elements with attributes or be used in conditional logic, etc. This would preserve the separation of reader from model, at the expense of saying that indexes like this are different/ are not like all the dfdl reader parameters that might be in the current context. > What makes all this confusing for DFDL is that we have some > representations > that are complex enough to need layered multi-step > descriptions, and once > you have that, there's no stopping you from using it to do > all sorts of > transformation from one format to another. So it feels like > you can have > your cake and eat it too, which is to say you can pick your > XML Schema and > populate it from quite differently structured data. And that > is probably > true, but at the bottom level of the stack of layers you have > to have a > vocabulary and model for directly describing the structure of > the data so as > to get the whole ball rolling. And at this bottom layer, the needs of > describing the data format completely dictate what the schema > is like. I would solve this by just saying that, at the bottom layer, there are no single or multidimensional arrays, just sequences of base types, and that any concept of dimensions is fabrication created by the user (a very common and convenient one we might want special support for...). The only reason I think we would need a multidimensional array type in DFDL is if we wanted to directly read m*n bytes and create a single XML element representing the entire array that would then have some accessor methods to get a value for a particular x,y offset pair. I'm not sure what kind of analogy will make sense to people here, but I see a similar argument for floats from strings: if you want to create a float from a sequence of characters, you need a float type. If you just want to prescribe a standard way to model a sequence of characters representing a float so that we can consistently label the mantissa and exponent chars regardless of storage order, you're not really defining a new float type in XSD. Instead, your exposing the semantics created/inferred by the reader as standardized annotations of the existing char (or string) type (with the annotations being potential or required depending on whether you let me shut them off or not). Jim

1 0

RE: [dfdl-wg] How to handle multi-dimensional arrays - version 2
by mike.beckerle＠ascentialsoftware.com 03 Mar '05

03 Mar '05

I'd like to propose that the kind of thing you are talking about below Jim, where you get to decide what the presentation of the data is, is always done using layers. The point is that this lets us pick a base-level DFDL/XSD schema that is the one implied by the physical representation of the data, and use that as the starting point for layer transformations. Martin and I discussed this privately last fall at some point as the "XML all the way down" approach. I.e., at each layer, the layer above gets to think of the next layer down as if it were XML described by some XSD. The trick is the bottom-most-layer. Here there is no distinction between the shape of the actual physical representation and the shapes described in the XSD. If the physical rep has certain nested hierarchy, then the base-level XSD describes exactly that hierarchy and not any sort of mapping of it to something different in shape. This lowest level one-to-one mapping is how we get the induction of layer transformations started, and it allows for the first such transformation to be no different conceptually from any higher level. All transformations appear to be XML to XML conceptually, rather than the bottom one being something-else to XML, followed by all higher-level XML to XML ones. Now, the alternative is to have the lowest-level model be something different from XSD, and I'm sure this could be made workable too, but I thought we were all reconciled that any such model would be so similar to XSD, having parent-child hierarchy, choice, types, names, dimensions, etc. that making it different from XSD would just be confusing. If you're with me so far, then the only nub was that XSD doesn't directly model multi-dimensional arrays. Which brings us to the topic of the day. So with that I've been forging ahead on the path of worrying only about the lowest-layer schema where it is intended to correspond directly to the structure of the data, and there I'd like to suggest that multi-dimensional arrays have a base-level model in DFDL that looks like the one I proposed first over a week ago, with attributes and all. E.g., like this: <a x="5" y="7">4.32</a> <a x="5" y="8">2.212</a> This is a suitable representation for doing layered transformation into whatever other form you'd like, as it lets xpath-like expressions retrieve individual array elements by index positions. Of course if you actually convert your data into XML you'd pay the cost of these attributes in the text size, but if you're using a DFDL API-based system, then the implementation would be free to respond to requests for the "..../a/@x" attribute which return 5 (in the above example) without actually using any storage for that attribute. > -----Original Message----- > From: Myers, James D [mailto:jim.myers@pnl.gov] > Sent: Thursday, March 03, 2005 10:38 AM > To: dfdl-wg(a)gridforum.org > Subject: RE: [dfdl-wg] How to handle multi-dimensional arrays > - version 2 > > > > > > > Here's a slightly different formulation of the > multi-dimension stuff. > > > > 1) no longer dictates the XSD for representing the array. > > This cuts both ways since you no longer really have an XSD > model for > > multi-dimensional arrays. That is. It is up to the author > of the DFDL > > Schema to insure the needed information about the array > (coordinates > > of each element) make it in to the logical model in a useful way. > > I didn't realize we were proposing to extend the XML schema > to have a multidimensional array type, versus providing a way > for DFDL to read and internally represent a multidimensional > array. The latter seems descriptive and the former prescriptive. > We have a different sense of descriptive and prescriptive. To me descriptive does NOT mean you get to choose and populate an arbitrary XML schema from data of any shape/form. To me "descriptive" means you tell me what your data looks like. But the schema you use to tell me that format is structured directly as a function of what the data format is like. That is, the Schema you provide must be written in a way which informs me (I'm the DFDL processor here) about what the data looks like. The schema is highly constrained in its shape/form by what the data's representation is. One of the members of my team describes this as "it's an input schema, not an output schema." It's a little confusing because DFDL will "prescribe" the schema, but does not "prescribe" the format of the data described by that schema. Rather it uses the "prescribed schema" to "describe" the data format. You do have freedom in what your data format actually is, you don't have freedom to choose what the description of it looks like. I think neither prescriptive nor descriptive data format says anything about the data transformation problem, which is is "how to I get data into a specific form". That's just plain different, and I claim it is the same whether the data started out in XML or in a file described by DFDL. Given data with a certain logical structure transform it into a different logical structure. Use Xquery or XSLT or a variety of non-XML oriented technologies to do so. What makes all this confusing for DFDL is that we have some representations that are complex enough to need layered multi-step descriptions, and once you have that, there's no stopping you from using it to do all sorts of transformation from one format to another. So it feels like you can have your cake and eat it too, which is to say you can pick your XML Schema and populate it from quite differently structured data. And that is probably true, but at the bottom level of the stack of layers you have to have a vocabulary and model for directly describing the structure of the data so as to get the whole ball rolling. And at this bottom layer, the needs of describing the data format completely dictate what the schema is like. Bringing this back to multidimensional arrays, it seems we have this choice: 1) add some new construct to XSD to support base-level description of multi-dimensional arrays 2) choose constructs already present in XSD and use them in a cannonical or "base" way to describe multi-dimensional arrays Currently I'm advocating (2), since there is enough expressive power there. XML/XSD basically can provide an arbitrary map from element names and attribute names and values onto values, and a multi-dimensional array is a subset of this kind of mapping functionality. So we define a canonical way of using these constructs to describe MD arrays and we're done. > > > > 2) I added in the complexity of calculating the array size, > actually > > the lower and upper bounds of each dimension, dynamically based on > > data. This makes the example more real. > > > > This still works out pretty well. I'm still pondering > whether I like > > this better or not. I'm thinking about perhaps some sort of pseudo > > attributes which are guaranteed to be put into XML if you actually > > render to XML, but where a DFDL API-based implementation can choose > > not to realize them. > > I think this example removes some of the prescriptive nature > of the first one, but I'd like to be able to format my array > however I want, e.g. as > > <row><elem>3</elem><elem>2</elem></row> > <row><elem>5</elem><elem>6</elem></row> > ... > > Or even > > <states><state>Alabama</state><state>Alaska</state></states> > <population><pop>34.2</pop><pop>10.6</pop></population> > .... > > (an array containing state names, population and other data, > perhaps serialized in the file as all info for each state together). > > If DFDL could separate the reading of such an array from how > it is output in the schema, I could do any of this. Having > multiple layers is a start - DFDL reads the array in to > something that is addressable along the lines Mike proposes > and then the contents of that layer are referenced via xpath > to provide values in some structure I define in XSD. The only > piece missing (I think) is that we haven't yet defined how to > access iterators, i.e. if I have an element <elem minoccurs="1" > maxoccurs = "5"> , how can I say that element n (n = 1...5) > has dfdl:runtimevalue <a x="n" y="1">, which would put just > the first column of a into the element sequence. If, in > Mike's example, I could define the x and y dimensions > independent of an array-reading context, just so I can use > them in value references for dfdl:runtimevalue elements, I > think we'd be all set. > > This type of capability would allow all sorts of useful > things - including the array to set of vectors conversions > outlined here as well as subsampling, expansion/contraction > of sprase arrays (where the array is stored as a sequence of > x,y, value triples for only nonzero elements), etc. > > > One other minor point - if the order of x and y in the DFDL > file is important (as it is in the example), do we need a > <dfdl:array storageOrder="firstDimensionChangesFirst"> > option? OR can we just list y first and then x? > > Jim > >

1 0

FYI: Oracle article on a similar approach
by Myers, James D 03 Mar '05

03 Mar '05

http://www.sys-con.com/story/?storyid=48039 "Enterprise applications such as banking, healthcare, and so on still use flat files to import/export data between applications. Flat files contain machine-readable data that is typically encoded in printable characters. There is a growing need for these applications to interact with XML-aware applications and Web services, and to satisfy this need these applications must convert flat file data to an XML format..." Jim

1 0