[gsm-wg] Draft Charter for Data Movement Interface Standardization WG

[William E. Allcock]

All,

Sorry for the SPAM, but I sent this to the "likely suspects" who might
be interested.  I have a proposed BOF (waiting for AD approval) to
discuss standardizing an interface for invoking data movement.  There
are several of them out there already.  CERN has the File Transfer
System (FTS), the gsm-wg has SRM copy, Globus has the Reliable File
Transfer (RFT) service, etc..  I don't think there will be any argument
that there is a need for such standardization, the hard part will be
scoping the extent of what we will work on.  For instance, all the
examples above are file based, but ideally, this interface would work
for any data that can be addressed.

I expect that that the BOF will be centered around scoping the working 
group, but I think we should (and approval of the BOF depends on) 
getting some initial discussion around the scope.  So... here it goes:

I think the obvious thing is that it needs to be able to have the basic 
functionality presented by FTS, RFT, and SRM-copy, however the devil is 
in the details, so I will break this up into "blocks of functionality":

Lets start with naming.  What will this service accept as valid names 
for entities that it will move?  URLs? EPRs? Will logical file names be 
accepted or should they be translated outside this service?

Related to the naming is what type of data will this service move? 
Files? video streams?  the output of simulations? the output of database 
queries?  Can we make this a service that any service that wants to move 
data can simply invoke it?  Note that I am differentiating data from 
messages.  You would not use this to send the result from a service that 
summed a bunch of numbers, that would simply be a SOAP response... IMHO :-).

Can we make a generic module that would allow this functionality to be 
applied to any service that exposes the byte-io interface?  Does that 
affect the interface or is it just an implementation issue?

Can we make this service transport mechanism agnostic?  both application 
transport (GridFTP vs HTTP vs ...) as well as network transport (TCP vs 
UDP vs UDT vs ...).  My concern here is that I am not sure SOAP has the 
functionality we need.  To do this, I wonder if we need the equivalent 
of a union in C, so that the parameters specified are based on the 
transport(s) chosen.  For instance, if you use TCP you need to specify a 
buffer size, but not for UDP.  GridFTP specifies streams and data 
channel authentication, but HTTP does not.

What about security / authorization.  This is a broad category and we 
should push as much as possible outside of scope via callouts and Policy 
Enforcement Points (PEPs), but what about delivery guarantees such as AT 
MOST ONCE, AT LEAST ONCE, EXACTLY ONCE, non-repudiation, etc.?  I know 
Dieter has a set of use cases that require some of this type delivery 
guarantee functionality.

A potentially contentious issue is whether or not these services will 
use WSRF and notifications to expose (push from the service) or methods 
to query the state (pull from the service).  Hopefully, we can find a 
way to make each optional.

If we start making many optional parts to the interface, it will make 
what is exposed as service metadata for brokering will become more 
important.  I would propose that we should make at a minimum a 
recommendation for what facts about the service should be exposed.

All of the existing services accept "bulk" inputs, i.e., move these 100 
files.  This can be a problem when the requests become very large due to 
de-serialization.  Should we provide a "chunking" interface so that 
requests can be of unlimited size?

Please feel free to make comments on the above and more importantly 
suggest other important issues we need to address.

btw, once we have a mail list of our own we will quit spamming the other 
lists :-).

Bill
-- 
William E. Allcock
Argonne National Laboratory
Bldg 221, Office C-115A
9700 South Cass Ave
Argonne, IL 60439-4844
Office Phone:  +1-630-252-7573
Office Fax:      +1-630-252-1997
Cell Phone:      +1-630-854-2842