[ogsa-dmi-wg] DMI Telcon Notes

[Mario Antonioletti]

Hi,
    Ok, here are my notes from today's call. Please feel free to 
supplement/correct. Not a great idea to chair and take notes at
the same time when more than 2 or 3 people ... at least that is my 
excuse,

Mario

---

DMI Telcon 22nd June 2009
=========================

Present:
 	Mario Antonioletti, EPCC
 	David Meredith, STFC
 	Shahbaz Memon, FZJ
 	Gerson Galang, ARCS
 	Steven Crouch, OMII-UK
 	Allen Luniewski, IBM
 	Alex Arana, Intersect
 	Peter Turner, University of Sidney

Agenda:

- Introductions
- Agenda bashing
- DTS progress update
- Relationship of progress to JSDL and DMI specs
- Consequential anticipated extension requests for JSDL and DMI
- Planning

----

Introduction
------------

Mario Antonioletti is one of the chairs of OGSA-DMI. Took the position
after Michel Drescher had to step down. Keen to see adoption of DMI to
take place.

Dave Meredith STFC Daresbury Laboratory - interested in DMI from the
Data Minx project and NGS perspectives.

Gerson Galang is a developer based in Melbourne, working in Data-MINX,
interested in DMI and JSDL and has been looking at the schemas.

Steve Crouch, from OMII-UK, pursuing this work from the JSDL
route. Have been involved with Peter, Gerson and David for a while now.

Allen Luniewski, has been with OGF for many years now. One of the
instigators for this work going on within OGF. Co-chaired the data
architecture working group that motivated DMI and has been involved in
this ever since.

Alex Arana is looking at this work from the Data-MINX perspective -
has been looking at the specs.

Peter Turner from the University of Sidney, is a Crystallographer - but
now providing data management and access for open neutron source and
the Australian Synchrotron.

Shahbaz Memon is from FZJ and is a co-chair of the DMI working group.
He has been implementing a WSRF implementation of DMI.

DTS Progress Update
-------------------

Alex - was in the UK in late April. Attended the OMII-UK
Collaborations Workshop - got a design for a heavy lifting data moving
that will become part of the data-MINX project
(http://tinyurl.com/ppt6gr). Looked at various technologies. Came up
with the draft version of the current schema from a JSDL spec. Doing
the bits of the architecture. Have defined 3 major components:

o client - responsible for sending requests to a DTS web service
o DTS web service for managing data transfers, use JMS
o DTS worker node(s) - receive and process messages.

Have implemented a "thin red line" - an end-to-end implementation that
satisfies the most basic use case.

Thus have a 3 tier architecture:

    DTS client ---> DTS WS ---> DTS worker node

Have JMS queues and topics for transmitting/submitting job requests.

DTS worker node are at the end - each does a unique job, DTS worker
node cloud is responsible for picking up messages. When one arrives
the DTS worker node works on this. Fulfills request. Provisions for
management of requests.

JMS good for this as it scales - can move data between different types
of protocol - SRB, ftp, etc ... add worker to process messages. Thus
JMS - allows for horizontal scaling. Lots of flexibility in how you
confer the jobs through JMS.

Protocol between client and server is SOAP.
Interested in a document/literal approach.


Want to be able to express multiple data transfers within one requests
and manage these from one service.

David Meredith has put in a proposal to use a wrapped data transfer
request that could encapsulate multiple pairwise sources and sinks as
a starting position to be able to do this:

http://www.ogf.org/pipermail/ogsa-dmi-wg/2009-June/000488.html

At the moment DMI cannot handle multiple transfers from one
service instance. This would be an extension to the existing
functionality but probably needs some refinement.

There was some question as to the nature of these. If one transaction 
fails will the whole data transfer fail? Is it best effort? How do
you cope with the lack of transactional support for file operations
in most OSs? There are a large number of questions that need to
be addressed.

The other proposal within the mail message to add extensibility points
to various elements seemed acceptable.

Mario was in favour of trying to modify the spec accordingly if
it would increase uptake/adoption. Need to think carefully about
what the implication of any changes will be though and see what
the processes are.

Planning
--------

Mario will try to contact with the OGF editor to see how easy it will
be to put changes through. Allen will check to see whether a number of
distinct directories can be transferred from one source using a single
request to a DTF (and thus managed by a single DTI). All to look at
and comment on David's proposal and see how/whether they can be
improved.

Date of next call: 8am  UK time (GMT+1), 13 July 2009

+-----------------------------------------------------------------------+
|Mario Antonioletti:EPCC,JCMB,The King's Buildings,Edinburgh EH9 3JZ.   |
|Tel:0131 650 5141|mario at epcc.ed.ac.uk|http://www.epcc.ed.ac.uk/~mario/ |
+-----------------------------------------------------------------------+
-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.