[jsdl-wg] Regarding Data Staging Elements in JSDL

[Alexander Papaspyrou]

Donal,

Donal K. Fellows schrieb:
> Gargi B Dasgupta wrote:
>> My question is:  In case of a workflow, with multiple job steps, and 
>> data transferring in and out between these steps,
>> what is the natural home for these data staging elements.
>>
>> Is it in the workflow or in the individual JSDL elements. Will the Data 
>> Staging elements remain in JSDL and then transfers needed in the 
>> workflow be inferred from these JSDLs. Or will the staging be explicitly 
>> defined in the flow.
> 
> Excellent question.
> 
> Ideally, I'd imagine that they'd be in the flow, especially as transfer
> of a large dataset could take a long time (and considerable resources)
> in itself. JSDL deliberately doesn't do this, and this is because it
> doesn't assume the presence of an outer flow language (some use-cases
> specifically require data transfers in order for even a simple job to
> run, so they have to be there). If you've got such an outer language, do
> put the transfers there (though you could also use a JSDL document
> containing just a DataStaging section to describe the transfers,
> resulting in the flow language just joining up a bunch of JSDL docs).

yes, but handling stuff completely outside impairs the possibility of
using JSDL filesystems, which is a very interesting feature especially
for POSIX jobs along with using environment variables.

Maybe it makes sense to handle file naming and referencing in JSDL while
keeping track of workflow I/O dependencies outside, probably doubling
something.

Regards,
-- 
Dipl.-Inform. Alexander Papaspyrou  |  44221 Dortmund, NRW (Germany)
Robotics Research Institute         |  phone  : +49(231)755-5058
Information Technology Section      |  fax    : +49(231)755-3251
Dortmund University                 |  web    : http://it.irf.de/~alexp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 187 bytes
Desc: OpenPGP digital signature
Url : http://www.ogf.org/pipermail/jsdl-wg/attachments/20070412/1a0eda80/attachment.pgp