[ogsa-dmi-wg] [Fwd: RE: OGSA-DMI approaches and feedback (& EGI)]

[Steve Crouch]

See below for the reply from Steven about our current work and linking 
in with the EGI and their requirements...

Cheers
Steve

-------- Original Message --------
Subject: RE: OGSA-DMI approaches and feedback (& EGI)
Date: Mon, 29 Mar 2010 15:14:00 +0200
From: Steven Newhouse <Steven.Newhouse at cern.ch>
To: s.crouch at omii.ac.uk <s.crouch at omii.ac.uk>
CC: david meredith <david.meredith at stfc.ac.uk>,        Steven Newhouse 
<steven.newhouse at egi.eu>
References: <4BACDE02.8090703 at omii.ac.uk>

Hi Steve,

> When you have a moment, I would greatly appreciate any feedback you
> could offer on some of our current OGSA-DMI work, particularly from the
> point of view of EGI and its requirements...

At a high-level yes... but I regret I do not have the bandwidth for a 
detailed examination.

Clearly, data is becoming increasingly important. I say data instead of 
files. Clearly, underpinning the data there is a need to move 
collections of files about. There are three interfaces here - 
coordination, transport & storage. Support of SRM that plugs whatever 
transport layer is used down into the underlying storage layer is 
important considering the diversity of the different storage systems. 
Transport - for files I'm not sure there is anything better than 
GridFTP. At the coordination layer there are implementations - FTS and 
RFT - I'm not sure if anything has appeared beyond DMI in this space 
over the last few years.

> What has emerged is the common need for specifying bulk data copying,
> from potentially multiple sources to multiple sinks, using different
> protocols and credentials per copy.

Keep in mind that files are only one of the sources of data. I can see 
scenarios where you might want to extract and replicate structured data 
sources (or subsets of the data) at different sites around the 
infrastructure so that they are available locally for intensive computation.

> Attached is a presentation given by David Meredith at OGF28, that
> provides an overview of our current efforts, and provides more detail.
> If you could comment or provide suggestions on the slides that would be
> most helpful.

More generally where does this DMI activity now sit in relation to the 
functionality offered by iRODS? How would accessing data through this 
interfaces (which is seeing some widescale deployment) work?

> Also definitely worth mentioning, there is an OMII-UK CSP (remember
> them? :) called DataMINX.  It's funded until the end of this year and
> with David Meredith as PI, aims to deliver a scalable data transfer
> service (DTS), based on these open standards and core requirements.

That's interesting. As you know EGI will be offering an open process in 
incorporating services from national and European projects into its 
Unified Middleware Distribution - which may be virtual inclusion through 
a software repository or physical inclusion in a distribution.

> Also, we'd ideally like to keep tabs on and investigate such data
> requirements that emerge from EGI as it progresses - do you have any
> thoughts on this?

Specifically at the moment - no. But the EC this morning mentioned data 
as a big concern for them (and therefore by definition EGI) and the 
frequent ESFRI meetings that I go to mention this as a concern. I can 
imagine a 'data session' taking place at the EGI technical forum in 
September in Amsterdam to try and carry forward some activity here.

Steven


-- 
Dr Stephen Crouch,
Software Architect,
OMII-UK,
School of Electronics and Computer Science,
Room 4067, Level 4, Building 32,
University of Southampton,
Highfield, Southampton SO17 1BJ
Tel: +44 (0)23 8059 8787        EMail: s.crouch at omii.ac.uk
Fax: +44 (0)23 8059 3045        WWW: http://www.ecs.soton.ac.uk/~stc