February 2013 - dfdl-wg - lists.ogf.org

Action 200 - Terminology - Git for central repo of DFDL schemas at github (was Re: Git commit policies)
by Mike Beckerle 08 Feb '13

08 Feb '13

(re-send: neglected to send to whole workgroup mailing list) I'll send a couple emails on this subject. The initial big git hurdle is terminology being so different from other SCM. E.g., index - the set of files git is tracking. If you create a new file, by default git is not tracking it. Until you "add it to the index" it is very easy to lose it. sandbox - git doesn't use this term. The equivalent is a set of 4 things: * your local repository * commits - the deltas in your local repository: includes what you originally fetched/pulled plus any commits you have done to it * any files that have been "added to the index" but not committed yet. * any untracked files (that you haven't yet added to the index) - git requires you be careful about these. commit - a purely local operation of taking all files that have changes and are being tracked (i.e., new files have been added to the index), and creating a change-set out of them. Does NOT interact with a central repository at all. checkout - purely local operation which switches branches in your local repository - does NOT interact with a central anything. Simplest policy to stay out of trouble is to always add any new files to index and commit all outstanding changes before you do this. fetch - get changes from central repository, but don't combine/merget/change what is in current 'sandbox' to include them yet. Safe to do this operation whenever. pull - fetch plus merge the changes. We tend to avoid this and use fetch then rebase. merge - locally merge changes. We tend to avoid this and use rebase. rebase - a merge which replays your changes on top of another commit point (typically a master branch head), so as to make it look like all the changes were done in one linear sequence rather than by fork/join splitting and merging. This is often much easier to deal with than fork/join merging. push - moves changes to another repository. Doesn't merge anything. (Must already have been merged locally.) On Tue, Jan 29, 2013 at 10:46 AM, Tim Kimber <KIMBERT(a)uk.ibm.com> wrote: > > http://stackoverflow.com/questions/3505409/does-there-exist-git-commit-poli… > > http://git-scm.com/book/ch5-1.html > > regards, > > Tim Kimber, DFDL Team, > Hursley, UK > Internet: kimbert(a)uk.ibm.com > Tel. 01962-816742 > Internal tel. 37246742 > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0

Action 200 - Workflows - Git for central repo of DFDL schemas at github (was Re: Git commit policies)
by Mike Beckerle 08 Feb '13

08 Feb '13

Please look at the terminology email I sent prior along with this one. The simplest possible one-branch-only straight-line only git workflow is described below: git clone - this is a one time operation to create local repository copy from central one Note: do NOT put this clone repository directory INSIDE your eclipse workspace, nor should you point your eclipse at a directory inside this local repository as its workspace. Keep things separate. I have two git repositories cloned on my machine. One is in ~/git/fouo another is in ~/daffodil/git/daffodil. My eclipse workspace is elsewhere. Eclipse (with eGit) will happily point at a project directory inside the repository, but keep its settings and such elsewhere. (Project specific settings end up in the project of course. They have to. Treat them in git like any other files/directories.) The "git" directory name is just a convention, the repository is usually the directory beneath that. (It will have a ".git" hidden directory in it.) You will have to be cd to this repository directory to issue any of the git commands. Here's the repeating workflow: 1. edit files, test stuff in the usual ways (nothing to do with git) 2. git status - shows you list of untracked files, and changes that are ready to commit (use this command often) 3. git add - any new (i.e., untracked) files so git knows to track them 4. git commit - locally packages changes into a unit/change-set 5. repeat from step 1 as many times as you want. When you want to publish your changes for others continue with the next step. 6. optional step - zip/tar your local repository for safe keeping (for the paranoid - or git newby) 7. git fetch - pulls all outstanding changes from central repository (doesn't merge anything) 8. git rebase - begins the merge-the-code sub-loop here. Rebase replays your commits one by one on top of latest from central repo. 1. if rebase finished with no conflicts, leave this sub-loop 2. edit files to fix any conflicts (nothing to do with git) 3. git add - conflicted files that you merged have to be re-added (tells git you have merged them) 4. git rebase --continue - carry out more replay of your committed changes 5. go to step 5.1 (top of this sub loop) 9. fix remaining problems that aren't just merge conflicts. Test. (nothing to do with git) 10. git commit - packages up anything changed as part of the merge. 11. Decide - your local repository is now current with latest and greatest with your changes on top 1. If you want to continue working locally, go to step 1. 2. If your stuff is at a good point and you want to push your changes to the central repository for others to see, then go to the next step. 12. git push - move your changes to the central repository. 13. Note prior step will probably succeed, but if someone beats you to the push, you will be rejected, so repeat from step 7. 14. You are done. Repeat for your next changes. The above workflow achieves something similar to subversion with no branches being created. You have to have your changes merged (we call it rebased) properly on top of the latest and greatest before you are able to push your changes back to the central repository. A useful technique: git fetch and git rebase really often even before you intend to push your changes (see step 11 above). This keeps your state close to that of the central repository and keeps the amount of conflict resolution low. In my experience, it is possible to lose local changes and files using git - usually this happens if they are untracked files. A git expert can *usually* get things back for you, but when it doubt: just zip/tar up a copy of your whole local repository. That preserves *everything* allowing you to go back. I always do this before a complex merge/rebase. It is easier for me to back out of something by just deleting the whole repository and un-tar of my save than to figure out the git archaeology needed to back up to the state I want. The daffodil open source project uses a more complex git workflow that accomodates a code-review cycle. This is described here on our Wiki: https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Development+Wo… Searching the internet for "git cheat sheet", I found this one is the simplest but quite a few of them are useful. http://nakedstartup.com/2010/04/simple-daily-git-workflow (Scroll down. It's below the fold in a Scribd slide) There are useful, but all imperfect, graphical git tools. I use eGit mode in Eclipse a great deal. Others use gitg or qgit, etc. When I say they are imperfect, it's because it is *not* in my experience, possible to "live entirely" in one of these graphical tools. You have to use the command line git sometimes. The daffodil project workflow depends on this. Here are the cautions I would give to anyone new to git. 1) commit often. Think of it almost like save. 2) never do any operation while you have uncommitted or untracked files in your sandbox. Commit first, then do whatever your operation is. 3) the command line 'git status' operation is your friend. Something I learned recently: git doesn't track empty directories. Add a ".keep" file to them to keep them around. On Tue, Jan 29, 2013 at 10:46 AM, Tim Kimber <KIMBERT(a)uk.ibm.com> wrote: > > http://stackoverflow.com/questions/3505409/does-there-exist-git-commit-poli… > > http://git-scm.com/book/ch5-1.html > > regards, > > Tim Kimber, DFDL Team, > Hursley, UK > Internet: kimbert(a)uk.ibm.com > Tel. 01962-816742 > Internal tel. 37246742 > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0

what do we allow in DFDL expressions after "/"
by Mike Beckerle 07 Feb '13

07 Feb '13

In reexamining the grammar for the DFDL expression language, I have the following question In a path Step/Step/Step[Index] What can Step be? I know it can be ".." or parent::QName or child::QName or just an NCName or QName, Those are all what the XPath grammar calls AxisStep. anything else? The grammar can be changed in a very small way from the original XPath grammar if the above are the only possibilities. the clause StepExpr ::= FilterExpr | AxisStep can just be changed to StepExpr ::= AxisStep -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 3

OGF DFDL WG Call Minutes 2013-02-05
by Steve Hanson 05 Feb '13

05 Feb '13

Please find minutes from the above call on Redmine at http://redmine.ogf.org/dmsf_files/13001?download= Errata document is now at v011.3 here: http://redmine.ogf.org/dmsf_files/8104?download= Regards Steve Hanson Architect, IBM DFDL, Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

OGF DFDL WG Call Agenda 2013-02-05
by Steve Hanson 04 Feb '13

04 Feb '13

Please find agenda for call on Redmine at http://redmine.ogf.org/dmsf_files/13001?download= Regards Steve Hanson Architect, IBM Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848

1 0

dfdl:checkConstraints
by Mike Beckerle 04 Feb '13

04 Feb '13

The error case of an argument that is an XPath to a complex type isn't discussed. Should this return true (hey, a complex element can't have facets, so there are none to be violated), false (errors are false), or cause a processing error or schema definition error? Most conservative choice here is a schema definition error. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

4 5

Action 193: Second draft for errata for RegEx
by Steve Hanson 04 Feb '13

04 Feb '13

The errata for action 193 has been updated below, please review for next WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ----- Forwarded by Steve Hanson/UK/IBM on 04/02/2013 12:50 ----- From: Mike Beckerle <mbeckerle.dfdl(a)gmail.com> To: Steve Hanson/UK/IBM@IBMGB, Cc: dfdl-wg(a)ogf.org Date: 28/01/2013 13:49 Subject: Re: [DFDL-WG] Action 193: First draft for errata for RegEx To me this is excellent work, much appreciated. I'd like to be much more directing about the non-portable constructs. We should decide among only these choices: 1) the non-portable constructs are disallowed. It is an SDE to use them. The check is required for all compliant DFDL implementations (that implement regular expressions at all.) 2) the non-portable constructs are allowed, but not recommended, and DFDL implementations are *required* to issue non-portability warnings if these constructs are used. Not checking this, hoping for the best, user-beware, is a bad idea. A scanner to find these syntaxes and disallow them is pretty easy to write. Regular expressions are, by their very nature, not very rich. You implement an escape scheme, anything else you scan for appearance of the offending constructs. Ironically, it's something that can be done with a regular expression itself. ...mike On Mon, Jan 28, 2013 at 8:22 AM, Steve Hanson <smh(a)uk.ibm.com> wrote: Here's a draft errata for action 193, for review on the next WG call. Regards Steve Hanson Architect, Data Format Description Language (DFDL) Co-Chair, OGF DFDL Working Group IBM SWG, Hursley, UK smh(a)uk.ibm.com tel:+44-1962-815848 ============================================================ Section 24 to read as follows: A DFDL regular expression may be specified for the dfdl:lengthPattern format property and the dfdl:testPattern attribute of the dfdl:assert and dfdl:disciminator annotations. DFDL regular expressions do not interpret DFDL entities. A DFDL regular expression is defined by a set of valid pattern characters. For portability, a DFDL regular expression pattern is restricted to the inclusive subset of the ICU regular expression [ICURE] and the Java(R) 7 regular expression [JAVARE] with the Unicode flags UNICODE_CASE and UNICODE_CHARACTER_CLASS turned on. The following regular expression constructs are not common to both ICU and Java(R) 7 and it is a schema definition error if any are used in a DFDL regular expression: *Construct* *Meaning* *Notes* \N{UNICODE CHARACTER NAME} Match the named character ICU only \X Match a Grapheme Cluster ICU only \Uhhhhhhhh Match the character with the hex value hhhhhhhh. ICU only (?# ... ) Free-format comment ICU only (?w-w) UREGEX_UWORD - Controls the behaviour of \b in ICU only a pattern. (?d-d) UNIX_LINES - Enables Unix lines mode. Java 7 only (?u-u) UNICODE_CASE - Enables Unicode-aware case folding. Java 7 only (1) (?U-U) UNICODE_CHARACTER_CLASS - Enables the Unicode Java 7 only (2) version of Predefined character classes and POSIX character classes. (?imsx-imsx:X) X, as a non-capturing group with the given flags. Java 7 only Note that the flags i,s,m,x are valid, but appending :X to the flag is not. Notes: (1) Implementations using Java 7 must set flag UNICODE_CASE by default to match ICU: (2) Implementations using Java 7 must set flag UNICODE_CHARACTER_CLASS by default to match ICU: Additionally, the behaviour of the word character construct (\w) is not consistent in ICU and Java 7. In Java 7 \w is [\p{Alpha}\p{gc=Mn}\p{gc=Me}\p{gc=Mc}\p{Digit}\p{gc=Pc}], which is a larger set than ICU where \w is [\p{Ll}\p{Lu}\p{Lt}\p{Lo}\p{Nd}]. The use of \w is not recommended in DFDL regular expressions in conjunction with Unicode encodings, and an implementation must issue a warning if such usage is detected. Character properties are detailed by the Unicode Regular Expressions [UNICODERE]. Section 30 to add: [ICURE] - http://userguide.icu-project.org/strings/regexp [JAVARE] - http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html [UNICODERE] - http://www.unicode.org/reports/tr18/ Section 30 to remove: [PERLRE] - http://perldoc.perl.org/perlre.html#Extended-Patterns [JAVARE] - http://download.oracle.com/javase/1.4.2/docs/api/java/util/regex/Pattern.ht… Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- dfdl-wg mailing list dfdl-wg(a)ogf.org https://www.ogf.org/mailman/listinfo/dfdl-wg -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

1 0

occursCountWIthDefault when parsing?
by Mike Beckerle 01 Feb '13

01 Feb '13

Same question different function. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0

dfdl:contentLength and dfdl:valueLength when parsing?
by Mike Beckerle 01 Feb '13

01 Feb '13

Do these functions work when parsing? or only when unparsing? -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

1 0

Use of xs:fixed and xs: prefix generally on XSD attributes
by Mike Beckerle 01 Feb '13

01 Feb '13

This may be a nit, but in prose when we want to distinguish an xsd attribute from a dfdl attribute or property, we prefix it with xs: as in xs:fixed, xs:nillable, etc. However, this is technically incorrect as those attributes are in no namespace. (XML Schema of XSD itself uses the default attributeFormDefault which is 'unqualified'.) A reader might think we are referring to some XSD construct they haven't heard of such as an <xs:nillable...> XSD element, but which doesn't exist. I am ok with keeping things as they are, or we can change to language like "XSD's nillable attribute" and "XSD's fixed attribute", and so forth. -- Mike Beckerle | OGF DFDL Workgroup Co-Chair | Tresys Technology | www.tresys.com

2 1