[Nmc-wg] Call 6/10 @ 15:00 UTC

[Michael Bischoff]

Hello Roman,

On Fri, Jun 18, 2010 at 4:08 PM, romradz at man.poznan.pl
<romradz at man.poznan.pl> wrote:
>
>
> Thanks Michael. In general I think it would be perfect if NMC could support
> both approaches of data transfer (streaming and separate data channel).
>
> btw. could you write what's the difference between Flow MA (or Flow
> Subscription MA) and Flow Subscription MP. It's not clear to me (the picture
> https://wiki.man.poznan.pl/perfsonar-mdm/images/perfsonar-mdm/2/28/JRA1_Netflow_Subscription_MA_-_design_-_v0.1.png
> is confusing).

I understand why it's confusing, this is in part because it exposes
internals (nfcapd/nfdump) in other part because it saids 'flow
subscription MA' this should be MP instead of MA

The general difference is the same as with all MP vs MA:
the MP gives near realtime stream of flowdata.
where the MA exposes an archive(allows you to analyse the past) of
flowdata can be queried(make specific selections, aggregate
information) to return a result.

The specific difference is that the MP provides a stream of netflow
data(that can be filtered) where the mp provides it's data in a
perfsonar/nmc way. (example can be found here
http://svn.geant.net/GEANT/SA2/ps-java-services/trunk/ps-mdm-flowsa-ma/doc/samples/SetupDataResponse-rawFlows.xml
notice that there can be hundreds or thousands of datum elements
depending on the query - this is also a typical case where streaming
as proposed can greatly help - the control channel approach doesn't
fit here)

the mp and ma can complement each other, the output of the MP can be
used as input for a MA because the MP allows filtering this could
yield quite powerful results. Which the diagram tries the illustrate
but doesn't quite get there. (there's more to be said here but I feel
that we would get too off track.)

> ... and do you have examples of control NMC messages you were using for
> setting up tunnel(s).

There in the svn:
http://svn.geant.net/GEANT/SA2/ps-java-services/trunk/ps-mdm-flowsub-mp/doc/examples/

> Roman
>
>
> On Wed, 16 Jun 2010, Michael Bischoff wrote:
>
>> Well the old, and really old is still on the wiki:
>>
>> https://wiki.man.poznan.pl/perfsonar-mdm/index.php/Flow_Subscription_MP/Implementation_Details
>>
>> The (over?) simplified rundown is as follows:
>>
>> ----------------------------------------------------
>> On perfsonar layer(control channel):
>> Subscribe message( subject(what do you want a stream of) +
>> target(setup tunnel to where) ->
>> <- key
>> ...
>> keepalive(with key) ->
>> <- ok
>> ...
>> Unsubscribe ->
>> <- clean tear down
>> ----------------------------------------------------
>> On the transport layer(data channel):
>> (pre condition -> subscribe request)
>> perfsonar service creates a tunnel to target.
>> write data to service end of tunnel
>>
>> Now an extension upon the NMC-base that would standardize the
>> 'persfonar as control channel + data channel' would
>> need to ratify a couple of things like tunnel type negotiation(current
>> zebadee is simply assumed and used), some
>> definition over what protocols are allowed(they should probably be
>> 'open') and other things that I can't remember right
>> now.
>>
>> papers describing different implementations(non-perfsonar/nmc) should
>> be widely available.
>> The short, as most of the papers I've seen also seem to comment, is
>> that setting up a separate data channel is
>> costly and one needs to be able to justify this cost. Such an approach
>> is only useful in cases where you know
>> beforehand that there is going to be a lot of data. Other issue is the
>> cost of converting data because the data-
>> channel can 'speak' anything it wants it doesn't require conversion
>> which might be, a big enough safe that it offsets
>> the cost of setting up the separate channel. a Perfsonar specific
>> advantage is that it allows for smoother transition
>> to perfsonar as you can utilize perfsonar to discover services but
>> still receive in know format and process it using
>> (non-perfsonar) tools that are already available. (this avoids the all
>> or nothing barrier)
>>
>> Where there any specifics you wanted to know?
>>
>> Best regards,
>>
>> Michael
>>
>>
>> On Wed, Jun 16, 2010 at 1:18 PM, romradz at man.poznan.pl
>> <romradz at man.poznan.pl> wrote:
>>>
>>> Hi Michael,
>>>
>>> is there the documentation of control channel approach which was
>>> implemented
>>> in flow subscription MA?
>>>
>>> Cheers,
>>> Roman
>>>
>>>
>>> On Fri, 11 Jun 2010, Michael Bischoff wrote:
>>>
>>>> I guess we postponed or something went really wrong..
>>>> I've set up a doodle so people can indicate their availability for next
>>>> week:
>>>> http://doodle.com/m7vh8pd6exbaqf29
>>>>
>>>> A proposal for streaming in NMC is here:
>>>> (mind you in still a work in progress/draft and it hasn't been proof
>>>> read/spell checked)
>>>> https://docs.google.com/a/controplex.com/View?id=dpn9385_1fmtsx7g9
>>>> In it's current form should be good enough for gettign the discussion
>>>> started.
>>>>
>>>> Best regards,
>>>>
>>>> Michael
>>>> On Thu, Jun 10, 2010 at 4:03 PM, Michael Bischoff
>>>> <Michael.bischoff at controplex.nl> wrote:
>>>>      That is a possibility I suppose, I don't know about
>>>> the availability
>>>> of others.
>>>> Who is going to be available as it currently stands?
>>>>
>>>>
>>>> On Thu, Jun 10, 2010 at 3:14 PM, romradz at man.poznan.pl
>>>> <romradz at man.poznan.pl> wrote:
>>>>
>>>>      Hi,
>>>>
>>>>      it will be very difficult to me to join the meeting today :( Can we
>>>> postpone it?
>>>>
>>>>      Roman
>>>>
>>>>
>>>> On Tue, 8 Jun 2010, Jason Zurawski wrote:
>>>>
>>>> All;
>>>>
>>>> There will be an NMC call this thursday at 15:00 UTC.  The meeting will
>>>> be run by Michael, and the agenda is to discus altering the protocol to
>>>> allow streaming (the response in particular).  The notes and agenda will
>>>> be posted here:
>>>>
>>>>
>>>>
>>>> https://forge.gridforum.org/sf/wiki/do/viewPage/projects.nmc-wg/wiki/20100610Notes
>>>>
>>>> Thanks;
>>>>
>>>> -jason
>>>> _______________________________________________
>>>> Nmc-wg mailing list
>>>> Nmc-wg at ogf.org
>>>> http://www.ogf.org/mailman/listinfo/nmc-wg
>>>>
>>>>
>>>>
>>>>
>>>
>