Andrew Grimshaw wrote:
In the data staging elements there is a creation flag that indicates whether to over-write or append.
Often, one only wants to overwrite if the target and the source are different, for example if I am using a large data set that changes infrequently or several jobs on the same resource will use the "same" input file.
Is there a way to do conditional stage in?
My understanding is that the implementation is free to use some mechanism outside the JSDL scope to optimize downloads. From the JSDL perspective (not that this is the only valid perspective, of course) it doesn't matter all that much. Looking at the latest version of the spec though, I see that the value "jsdl:dontOverwrite" could be interpreted to mean "transfer only if not pre-existing". Of course, at that point you then have to worry about whether the two ends refer to the same data, but if you're piling the data into some job-specific dir that's IMO not a huge issue.
I know the document says
"More complex file transfers, for example, conditional transfers based on job termination status are out of scope. "
But we're talking about a major performance optimization.
Well, perhaps. But if we add it, we have to worry about systems that don't have a mechanism for doing this sort of thing in the first place. Indeed, the JSDL 1.0[*] spec leaves out many things that could be major performance wins (e.g. compressed data transfers) either because they're complicated in their own right, or because we decided to get a spec going *this* decade as opposed to the next one. :^) None of which means that we will not go back and revisit these issues once there is some more data and experience reports available on actual deployed implementations. In particular, as it is a spec put together by mainly compute guys, we know that the data-related part is in need of fleshing out in the future. We've not reached the end of the story. Maybe just the end of the chapter instead. ;^) Donal. [*] As a side note, it should be done just about now as I write this. :)