[jsdl-wg] Question on file stage-in (fwd)

[Andrew Grimshaw]

All,
Is there a way to do condition statements in JSDL, in other words the
implementation that is "executing" the JSDL can choose which of several
options?

Andrew

-----Original Message-----
From: owner-jsdl-wg at ggf.org [mailto:owner-jsdl-wg at ggf.org] On Behalf Of
Steve McGough
Sent: Thursday, June 09, 2005 7:07 AM
Cc: JSDL WG
Subject: Re: [jsdl-wg] Question on file stage-in (fwd)

I think at this point we should remember that JSDL 1.0 is a minimum 
specification. All consumers need to be able to deal with the elements 
we define.

I do agree with Andrew that optimised transfers are very useful and it's 
a prime example of an extension to JSDL that should be done. Then at 
some point in the future you'll have services which handle the useful 
extensions and some which don't. People will submit jobs to services 
which have the extensions they desire - though could use the others if 
they have to. This will hopefully cause services which don't have the 
extensions to upgrade or "mimic" the features so they do get jobs. This 
is something I've already seen signs of.

steve..

Donal K. Fellows wrote:

> Andrew Grimshaw wrote:
>
>> In the data staging elements there is a creation flag that indicates 
>> whether
>> to over-write or append.
>>
>> Often, one only wants to overwrite if the target and the source are
>> different, for example if I am using a large data set that changes
>> infrequently or several jobs on the same resource will use the "same" 
>> input
>> file.
>> Is there a way to do conditional stage in?
>
>
> My understanding is that the implementation is free to use some
> mechanism outside the JSDL scope to optimize downloads. From the JSDL
> perspective (not that this is the only valid perspective, of course) it
> doesn't matter all that much. Looking at the latest version of the spec
> though, I see that the value "jsdl:dontOverwrite" could be interpreted
> to mean "transfer only if not pre-existing". Of course, at that point
> you then have to worry about whether the two ends refer to the same
> data, but if you're piling the data into some job-specific dir that's
> IMO not a huge issue.
>
>> I know the document says
>>
>> "More complex file transfers, for example, conditional transfers 
>> based on
>> job termination status are out of scope. "
>>
>> But we're talking about a major performance optimization.
>
>
> Well, perhaps. But if we add it, we have to worry about systems that
> don't have a mechanism for doing this sort of thing in the first place.
> Indeed, the JSDL 1.0[*] spec leaves out many things that could be major
> performance wins (e.g. compressed data transfers) either because they're
> complicated in their own right, or because we decided to get a spec
> going *this* decade as opposed to the next one. :^) None of which means
> that we will not go back and revisit these issues once there is some
> more data and experience reports available on actual deployed
> implementations. In particular, as it is a spec put together by mainly
> compute guys, we know that the data-related part is in need of fleshing
> out in the future.
>
> We've not reached the end of the story. Maybe just the end of the
> chapter instead. ;^)
>
> Donal.
> [*] As a side note, it should be done just about now as I write this. :)
>
>

-- 
------------------------------------------------------------------------
Dr A. Stephen McGough
------------------------------------------------------------------------
Research Associate,   Imperial College London,  Department of Computing,
180 Queen's Gate,    London SW7 2BZ, UK
tel: +44 (0)207-594-8310                        fax: +44 (0)207-581-8024
------------------------------------------------------------------------
Assistant Warden,  Brabazon House,  Pimlico,   5 Moreton Street,  London
SW1V 2PN, UK       tel: +44 (0)207-828-4733     fax: +44 (0)207-233-8105
------------------------------------------------------------------------