[dmis-bof] Vote for submission of Charter

[Arie Shoshani]

Hi, Bill,

See embedded responses.

Arie

William E. Allcock wrote:
>> 1) Regarding the sentence:
>> "Setting up a data movement includes the selection of a transport 
>> protocol, for example GridFTP, and parameters for 
>> reliability, timing, 
>> scheduling, resource usage, accounting, billing, etc. The 
>> Working Group 
>> will explore existing mechanisms to reach such agreement, e.g. 
>> WS-Agreement and use them where appropriate."
>>     
>
> Perhaps we need to reword this section, but I think the intent of this part
> was in response to comments about providing information for the service to
> make prioritization decisions amongst files within a request and between
> requests.  So, reliability might be parameters associated with number of
> retries, backoff, etc.; timing could be fail if it takes more than x hours;
> scheduling could be have it done by x time; resource usage could be use no
> more that 50 Mbs; accounting/billing could be don't exceed z cost.
>   
I see.  So, you expect other layers (components, services) to provides 
such parameters to the DMIS, right?
> The issue is that once something has decided the files need to move (like
> SRM) and it gives the service a request for say 100 files, the service
> could, in theory, move those in any order within a request (perhaps we
> should have an option for in order...), or may need to prioritize between
> requests.  So, a user might want to put priorities within a request (though
> I could argue against that), but they certainly might put priorities between
> requests of their own.  
OK.  This makes sense to me.
> This service is intended to be a file movement
> service, along the lines of the existing RFT, FTS, etc..  We have no
> intention of doing SRM type functionality.  It is something SRM might call.
>   
Good.  This is where I found the above sentence unclear.  It sounded to 
me like the DMIS will be responsible to call all the other services and 
coordinate various activities including allocating and reserving space.  
But, as long as it is intended as a "file movement service" without 
expectations that it will perform space negotiations, etc. it sounds fine.
> So, I would argue that we need verbiage that says we will want extra
> information available to make prioritization decisions within the service,
> and the information is of the type listed.  Do you have a suggestion for a
> wording change that makes it clearer?
>   
I'll try a little later, by the end of the day ...
>   
>> Thus, in the setup phase, all kinds of services will have to be 
>> contacted, such as components that manage resources, components that 
>> perform scheduling, keep track of accounting, etc.  For example, to 
>> reserve storage space, the data movement service may want to interact 
>> with SRMs.  I think this is a huge undertaking, and too large of a 
>> scope.  Even if you go through WS-agreement, it will have to interact 
>> with various resource managers, accounting, billing, etc.  
>> You may want 
>> to leave all the setup to a separate component, like 
>> WS-agreement, and 
>> then focus on the data movement part.
>>     
>
> WS-Agreement is not a component, it is just a pattern for negotiating
> agreements, but as I noted above, our intent is that this is just a file
> movement service.  The one point I might argue on this would be reporting to
> such services .vs. negotiating with them.  For instance, if there were some
> standard message that could be used to report "resource consumption", it
> might be useful to include some kind of EPR or such which the data movement
> service could inform about the resources consumed during the request.  I am
> not sure this is a good idea, nor that fits in the WS-I basic profile
> because it is a sort of notification, but it is a thought.  In the WSRF
> rendering, you have the alternative of exposing such info as an RP and the
> interested service could be directed to subscribe to that RP.
>   
Again, I am confused about term"resource consumption".  A DMIS can 
report on file movement, how many files were moved successfully, total 
size moved, hoe many to go, etc.  It cannot report on storage consumed, 
because it does not know what is done with the files after they are 
moved (for example they may be used quickly and removed, or archived).  
Am I missing something?

>> 2) A related point, in the "Seven questions: Evaluation 
>> Criteria (from 
>> GFD.3)" document it says:
>> "The most direct overlap would be with the gsm-wg, but they are 
>> participating in this group and will, presumably, use what is 
>> developed 
>> here for the transport portion of their interface."
>>
>> Yes, it makes sense for SRMs to make use of powerful 
>> transport services, 
>> since SRMs rely on transport services.  But, when a request 
>> is made to 
>> an SRM, they take care of managing storage space, keeping 
>> track of usage 
>> and accounting, as well as perform scheduling, so it does not 
>> make sense 
>> for the transport services to negotiate storage allocations 
>> (including 
>> lifetime), scheduling, etc, again, as is suggested in the 
>> "setup phase" 
>> above.
>>     
>
> I think I addressed this above.  We have no intention of the data movement
> service to be doing storage allocation.  I strongly disagree with you though
> that it does not make sense for the service to be able do scheduling of its
> own data movement tasks.
>   
I do not disagree with you.  I did not mean to say that the DMIS should 
not be able to do its own scheduling of data movement.  What I meant is 
that if a request such as srmCopy (which later could be renamed) is 
made, and the SRMs reserved space and make their own scheduling 
decisions, that this is now passed to the DMIS.  For example, if you 
include in the DMIS space allocation, then we get into a circular 
situation: call SRM which calls DMIS which calls SRM.
>   
>> 3) In the charter document, it says:
>> "To accomplish 3^rd party data transfer, a uniform, yet 
>> abstract naming 
>> scheme for resources (data in general, files in particular) 
>> is required. 
>> This working group will provide such abstract uniform naming scheme."
>>
>> I think this was the goal of the GFS-WG group, to the extent that I 
>> understand what they doing.  It might be good to coordinate 
>> with them.  
>> Also, isn't the issue if a uniform name space (also referred to as 
>> logical namespace) and its mapping into physical names fall 
>> into the RLS 
>> domain (and future related activities).  Do you really want to keep 
>> track of this mapping as part of the data movement service?  
>> Perhaps I 
>> don't understand the concept.  Please clarify.
>>     
>
> I don't believe we intend to create any namespace tech, but use what others
> are creating.  This sentence is there because we want this service to be
> able to move any named piece of data, not just a file.  We just want to make
> sure that, where possible, we keep data other than files in mind and allow
> for that in our schema, WSDL, etc.
>   
Well, in this case there is probably some overlap with the DAIS activity 
(or whatever they call it now).  It is interesting how far one can use 
the file paradigm to include other types of data, or vice versa.  Again, 
I am not sure it is worth pushing this item into your agenda.  I am 
neutral on this.
> As to someone to handle the srm_copy WSDL, in theory anyone who can read
> WSDL could do that, but it will be the ensuing discussion that will be more
> important, and where the SRM communities viewpoint would be useful.  Could
> whoever is leading the gsm WG meeting handle this?  Or is there not going to
> be a gsm WG meeting at this GGF? 
>   
So far, we chose to skip GGF17 as far as GSM is concerned.  But, perhaps 
we could prepare some slides and somebody like James Casey could present 
that. 

Timur, if you are reading this email - are you going to GGF in Tokyo?
> It would be best if someone was present from all the concerned parties, but
> if not, there will be discussions via email and future GGFs.  I don't really
> see an option other than proceeding as best we can with those that can make
> it.
>   
I understand.

Arie