[dmis-bof] Vote for submission of Charter

[Michel Drescher]

Hi Ari, others,

On 20 Mar 2006, at 22:05, Arie Shoshani wrote:

> Hi, Bill,
>
> See embedded responses.

ditto.

> William E. Allcock wrote:
>>> 1) Regarding the sentence:
>>> "Setting up a data movement includes the selection of a transport  
>>> protocol, for example GridFTP, and parameters for reliability,  
>>> timing, scheduling, resource usage, accounting, billing, etc. The  
>>> Working Group will explore existing mechanisms to reach such  
>>> agreement, e.g. WS-Agreement and use them where appropriate."
>>>
>>
>> Perhaps we need to reword this section, but I think the intent of  
>> this part
>> was in response to comments about providing information for the  
>> service to
>> make prioritization decisions amongst files within a request and  
>> between
>> requests.  So, reliability might be parameters associated with  
>> number of
>> retries, backoff, etc.; timing could be fail if it takes more than  
>> x hours;
>> scheduling could be have it done by x time; resource usage could  
>> be use no
>> more that 50 Mbs; accounting/billing could be don't exceed z cost.
>>
> I see.  So, you expect other layers (components, services) to  
> provides such parameters to the DMIS, right?

Perhaps, perhaps not. This is simply out of scope from where the  
service gets its information. However, before a data movement can  
take place, all surrounding parameters need to be clarified: A data  
movement inflicts temporary changes in the serivce's environment:  
bandwidth usage, system load, etc. When requesting a data movement, I  
have certain expectations that I would like to be met. It is simply  
not always true that a service can meet these expectations; for  
example if I request a data transfer to be carried out using GridFTP  
but the service supports only MTOM, then the request would fail.

This section expresses that the DMIS WG wants to have the parameters  
that directly affect the data movement itself sorted out before the  
data movement will actually take place. (Don't know it this sentence  
makes it any clearer, but on the other hand, I don't know how to  
express this better.)

THe DMIS WG will start with a relatively simple model to meet this  
use case. It it turns out that this is a more fundamental issue, we  
always have the option to re-charter and spin off another Working  
Group. In fact, this strategy was discussed at the GGF16 BoF.

>> The issue is that once something has decided the files need to  
>> move (like
>> SRM) and it gives the service a request for say 100 files, the  
>> service
>> could, in theory, move those in any order within a request  
>> (perhaps we
>> should have an option for in order...), or may need to prioritize  
>> between
>> requests.  So, a user might want to put priorities within a  
>> request (though
>> I could argue against that), but they certainly might put  
>> priorities between
>> requests of their own.
> OK.  This makes sense to me.
>> This service is intended to be a file movement
>> service, along the lines of the existing RFT, FTS, etc..  We have no
>> intention of doing SRM type functionality.  It is something SRM  
>> might call.
>>
> Good.  This is where I found the above sentence unclear.  It  
> sounded to me like the DMIS will be responsible to call all the  
> other services and coordinate various activities including  
> allocating and reserving space.  But, as long as it is intended as  
> a "file movement service" without expectations that it will perform  
> space negotiations, etc. it sounds fine.

Yes, 'bull's eye".

>> So, I would argue that we need verbiage that says we will want extra
>> information available to make prioritization decisions within the  
>> service,
>> and the information is of the type listed.  Do you have a  
>> suggestion for a
>> wording change that makes it clearer?
>>
> I'll try a little later, by the end of the day ...
>>
>>> Thus, in the setup phase, all kinds of services will have to be  
>>> contacted, such as components that manage resources, components  
>>> that perform scheduling, keep track of accounting, etc.  For  
>>> example, to reserve storage space, the data movement service may  
>>> want to interact with SRMs.  I think this is a huge undertaking,  
>>> and too large of a scope.  Even if you go through WS-agreement,  
>>> it will have to interact with various resource managers,  
>>> accounting, billing, etc.  You may want to leave all the setup to  
>>> a separate component, like WS-agreement, and then focus on the  
>>> data movement part.
>>>
>>
>> WS-Agreement is not a component, it is just a pattern for negotiating
>> agreements, but as I noted above, our intent is that this is just  
>> a file
>> movement service.  The one point I might argue on this would be  
>> reporting to
>> such services .vs. negotiating with them.  For instance, if there  
>> were some
>> standard message that could be used to report "resource  
>> consumption", it
>> might be useful to include some kind of EPR or such which the data  
>> movement
>> service could inform about the resources consumed during the  
>> request.  I am
>> not sure this is a good idea, nor that fits in the WS-I basic profile
>> because it is a sort of notification, but it is a thought.  In the  
>> WSRF
>> rendering, you have the alternative of exposing such info as an RP  
>> and the
>> interested service could be directed to subscribe to that RP.
>>
> Again, I am confused about term"resource consumption".  A DMIS can  
> report on file movement, how many files were moved successfully,  
> total size moved, hoe many to go, etc.  It cannot report on storage  
> consumed, because it does not know what is done with the files  
> after they are moved (for example they may be used quickly and  
> removed, or archived).  Am I missing something?

Yes, a DMIS can and should report on the resources it has consumed to  
fulfil a data movement request, e.g. bandwidth usage, CPU cycles(?),  
total bytes moved, etc. It is a different set of resources a storage  
management system may report, though, but still a report on which a  
bill/invoice may be filed later-on.

>>> 2) A related point, in the "Seven questions: Evaluation Criteria  
>>> (from GFD.3)" document it says:
>>> "The most direct overlap would be with the gsm-wg, but they are  
>>> participating in this group and will, presumably, use what is  
>>> developed here for the transport portion of their interface."
>>>
>>> Yes, it makes sense for SRMs to make use of powerful transport  
>>> services, since SRMs rely on transport services.  But, when a  
>>> request is made to an SRM, they take care of managing storage  
>>> space, keeping track of usage and accounting, as well as perform  
>>> scheduling, so it does not make sense for the transport services  
>>> to negotiate storage allocations (including lifetime),  
>>> scheduling, etc, again, as is suggested in the "setup phase" above.
>>>
>>
>> I think I addressed this above.  We have no intention of the data  
>> movement
>> service to be doing storage allocation.  I strongly disagree with  
>> you though
>> that it does not make sense for the service to be able do  
>> scheduling of its
>> own data movement tasks.
>>
> I do not disagree with you.  I did not mean to say that the DMIS  
> should not be able to do its own scheduling of data movement.  What  
> I meant is that if a request such as srmCopy (which later could be  
> renamed) is made, and the SRMs reserved space and make their own  
> scheduling decisions, that this is now passed to the DMIS.  For  
> example, if you include in the DMIS space allocation, then we get  
> into a circular situation: call SRM which calls DMIS which calls SRM.

The model would be hierarchical: SRM utilises DMIS to carry out the  
data movement, passing on its constraints down to a DMIS (like "Hey,  
service, I need to have the following files transferred with such and  
such constraints, can you do that?"). If you have only one DMIS in  
your domain, that this is usually a piece of cake, basically  
obliterating the need of some sort of negotiating/agreement phase.  
But consider a horde of DMISs, then you may need or want to contact  
more than one DMIS and see which one may serve you best.


Cheers,
Michel