[gin-data] SRB-gLite interop - success!! (Was: Re: On SRB interoperability with gLite)

[Jensen, J (Jens)]

Hi all.

You will recall, rather than adapting SRM directly to SRB or vice
versa, my proposal was that much would be gained simply by being
able to transfer files from one to the other.

(Of course nothing wrong with writing adapters and stuff, I mainly
wanted to see how far we could get with what we've got today,
low hanging fruit and all that.)

Test subjects were our local Tier 1 dCache (the disk one, not the
tape one), and a test SRB server set up for a customer.  Tested
with globus-url-copy and FTS.  Roger Downing and Adil Hasan set
up the SRB stuff for me, and had to do something to map the
DN to my account.

To g-u-c a local file to and from dCache (a GridFTP door), you
need to know the pool name in advance, or ask the SRM for one.  You
can copy files to the door, don't have to go via the SRM (it's like
the DCAP door in that sense).

g-u-c local file to and from SRB works.  In addition I can Sget a
file I copied in with g-u-c, and g-u-c a file out which I put in
with Sput.  So the gridftp interface works.

The interesting thing is 3rd party copying, of course.  It works!
As long as you remember -nodcau (switching off data channel
authentication).  Tried with 1 and 4 streams.  Both ways: from
dCache to SRB, and SRB to dCache.  Didn't do performance testing,
though.

Next step was FTS.  Matt Hodges from RAL Tier1 set it up for me:
<MH> "I added:

      <service name="gsiftp://kisumu.esc.rl.ac.uk:2670">
        <parameters>
          <endpoint>gsiftp://kisumu.esc.rl.ac.uk:2670</endpoint>
          <type>GridFTP</type>
          <version>1</version>
          <site>RAL-LCG2</site>
          <wsdl>unset</wsdl>
          <volist>
            <vo>dteam</vo>
          </volist>
        </parameters>
      </service>

to the /opt/glite/etc/services.xml file, and restarted the dteam VO
agent, and the STAR-RALLCG2 channel agent (on which transfers from RAL
dCache to this destination should be assigned); no need to add a
channel for this."
</MH>

Then it worked: we could use glite-transfer-submit to transfer
files from the dCache _SRM_ to SRB, and back again.  Note it
talks to the SRM on the dCache side, so you don't need to know
which pool to talk to; dCache automatically does the Right Thing.

jensen at lcgui0357[1]154% glite-transfer-submit -p wopwopwopwop \                                   
-s https://lcgfts0421.gridpp.rl.ac.uk:8443/glite-data-transfer-fts/services/FileTransfer \
srm://dcache.gridpp.rl.ac.uk:8443//pnfs/gridpp.rl.ac.uk/data/dteam/canned/canned1k \
gsiftp://kisumu.esc.rl.ac.uk:2670/tahdszone/home/testahds2.tahds/zugrapyrk

[note this is not my normal myproxy password, and I have removed
the credential before sending this email...!]

This magic works because FTS still supports the "Classic SEs".

Next step is to give the SRB an information system to pretend it
is really a Classic SE.  We haven't had time to play with that
yet, but the real test is whether it then works with lcg-utils
and GFAL.

The step after that is to investigate Reagan's suggestions
- earlier Reagan had suggested other ways to get SRB and SRM
to interoperate.  I need to discuss this with Adil.

Regards,
--jens (with Matt and Roger and Adil)


Jensen, J (Jens) wrote:
> Hi Erwin.  Ah, no, unfortunately I have been out of the office
> most of the last couple of weeks - and will be out the next
> two as well.
>
> But I did delegate parts of the investigations to others,
> as mentioned below.
>
> Matt reported success with using FTS to do GridFTP transfers,
> although he was unsure about the support for this.  I haven't
> had time to catch up with Adil about the actual transfer, and
> we also need to test Reagan's suggestions.
>
> I'll book some time to follow up later in June.
>
> Hope that's ok.
> Cheers
> 			--jens
>
>
> -----Original Message-----
> From: Erwin Laure [mailto:Erwin.Laure at cern.ch] 
> Sent: 04 June 2007 15:15
> To: Jensen, J (Jens)
> Cc: gin-data at ogf.org
> Subject: Re: [gin-data] On SRB interoperability with gLite
>
> Hi Jens,
>
> Did you already do some of the tests you'd proposed below?
>
> Cheers,
>
> -- Erwin
>
> Jensen, J (Jens) wrote:
>   
>> Just a quick update on the message below.
>>
>> I did get feedback from Erwin, but not from anyone else.  If
>> someone is taking it off to discuss on srb lists, could you
>> summarise the result to gin-data please.  Ta.
>>
>> Someone once told me that SRB has a GridFTP server, so I checked
>> with Adil Hasan and he says his, yes, and his team got it working.
>>
>> As I see it, it is essential to be able to drive the SRB via
>> GridFTP to avoid copying files out to the client.  I.e., if copying
>> files from SRM A to SRB B from Client C, then you want the data
>> to travel from A to B, not via C.
>>
>> If this is the case, then perhaps the FTS can drive the SRB like
>> it does Classic SEs (I am told some countries still have Classic
>> SEs :-)
>>
>> STFC - our organisation, RAL's research council, has got all the
>> experience in-house, so we can play with it a bit and see what
>> happens.
>>
>> I asked Adil to look into the GridFTP-for-SRB thing, and Matt
>> Hodges from Tier 1 to test the FTS-to-GridFTP.  We can then plug
>> it together and see if it works trivially.
>>
>> Regards,
>> 			--jens
>>
>>