Re: [Nmc-wg] Call 6/10 @ 15:00 UTC
Thanks Michael. In general I think it would be perfect if NMC could support both approaches of data transfer (streaming and separate data channel). btw. could you write what's the difference between Flow MA (or Flow Subscription MA) and Flow Subscription MP. It's not clear to me (the picture https://wiki.man.poznan.pl/perfsonar-mdm/images/perfsonar-mdm/2/28/JRA1_Netf... is confusing). ... and do you have examples of control NMC messages you were using for setting up tunnel(s). Roman On Wed, 16 Jun 2010, Michael Bischoff wrote:
Well the old, and really old is still on the wiki: https://wiki.man.poznan.pl/perfsonar-mdm/index.php/Flow_Subscription_MP/Impl...
The (over?) simplified rundown is as follows:
---------------------------------------------------- On perfsonar layer(control channel): Subscribe message( subject(what do you want a stream of) + target(setup tunnel to where) -> <- key ... keepalive(with key) -> <- ok ... Unsubscribe -> <- clean tear down ---------------------------------------------------- On the transport layer(data channel): (pre condition -> subscribe request) perfsonar service creates a tunnel to target. write data to service end of tunnel
Now an extension upon the NMC-base that would standardize the 'persfonar as control channel + data channel' would need to ratify a couple of things like tunnel type negotiation(current zebadee is simply assumed and used), some definition over what protocols are allowed(they should probably be 'open') and other things that I can't remember right now.
papers describing different implementations(non-perfsonar/nmc) should be widely available. The short, as most of the papers I've seen also seem to comment, is that setting up a separate data channel is costly and one needs to be able to justify this cost. Such an approach is only useful in cases where you know beforehand that there is going to be a lot of data. Other issue is the cost of converting data because the data- channel can 'speak' anything it wants it doesn't require conversion which might be, a big enough safe that it offsets the cost of setting up the separate channel. a Perfsonar specific advantage is that it allows for smoother transition to perfsonar as you can utilize perfsonar to discover services but still receive in know format and process it using (non-perfsonar) tools that are already available. (this avoids the all or nothing barrier)
Where there any specifics you wanted to know?
Best regards,
Michael
On Wed, Jun 16, 2010 at 1:18 PM, romradz@man.poznan.pl <romradz@man.poznan.pl> wrote:
Hi Michael,
is there the documentation of control channel approach which was implemented in flow subscription MA?
Cheers, Roman
On Fri, 11 Jun 2010, Michael Bischoff wrote:
I guess we postponed or something went really wrong.. I've set up a doodle so people can indicate their availability for next week: http://doodle.com/m7vh8pd6exbaqf29
A proposal for streaming in NMC is here: (mind you in still a work in progress/draft and it hasn't been proof read/spell checked) https://docs.google.com/a/controplex.com/View?id=dpn9385_1fmtsx7g9 In it's current form should be good enough for gettign the discussion started.
Best regards,
Michael On Thu, Jun 10, 2010 at 4:03 PM, Michael Bischoff <Michael.bischoff@controplex.nl> wrote: That is a possibility I suppose, I don't know about the availability of others. Who is going to be available as it currently stands?
On Thu, Jun 10, 2010 at 3:14 PM, romradz@man.poznan.pl <romradz@man.poznan.pl> wrote:
Hi,
it will be very difficult to me to join the meeting today :( Can we postpone it?
Roman
On Tue, 8 Jun 2010, Jason Zurawski wrote:
All;
There will be an NMC call this thursday at 15:00 UTC. The meeting will be run by Michael, and the agenda is to discus altering the protocol to allow streaming (the response in particular). The notes and agenda will be posted here:
https://forge.gridforum.org/sf/wiki/do/viewPage/projects.nmc-wg/wiki/2010061...
Thanks;
-jason _______________________________________________ Nmc-wg mailing list Nmc-wg@ogf.org http://www.ogf.org/mailman/listinfo/nmc-wg
Hello Roman, On Fri, Jun 18, 2010 at 4:08 PM, romradz@man.poznan.pl <romradz@man.poznan.pl> wrote:
Thanks Michael. In general I think it would be perfect if NMC could support both approaches of data transfer (streaming and separate data channel).
btw. could you write what's the difference between Flow MA (or Flow Subscription MA) and Flow Subscription MP. It's not clear to me (the picture https://wiki.man.poznan.pl/perfsonar-mdm/images/perfsonar-mdm/2/28/JRA1_Netf... is confusing).
I understand why it's confusing, this is in part because it exposes internals (nfcapd/nfdump) in other part because it saids 'flow subscription MA' this should be MP instead of MA The general difference is the same as with all MP vs MA: the MP gives near realtime stream of flowdata. where the MA exposes an archive(allows you to analyse the past) of flowdata can be queried(make specific selections, aggregate information) to return a result. The specific difference is that the MP provides a stream of netflow data(that can be filtered) where the mp provides it's data in a perfsonar/nmc way. (example can be found here http://svn.geant.net/GEANT/SA2/ps-java-services/trunk/ps-mdm-flowsa-ma/doc/s... notice that there can be hundreds or thousands of datum elements depending on the query - this is also a typical case where streaming as proposed can greatly help - the control channel approach doesn't fit here) the mp and ma can complement each other, the output of the MP can be used as input for a MA because the MP allows filtering this could yield quite powerful results. Which the diagram tries the illustrate but doesn't quite get there. (there's more to be said here but I feel that we would get too off track.)
... and do you have examples of control NMC messages you were using for setting up tunnel(s).
There in the svn: http://svn.geant.net/GEANT/SA2/ps-java-services/trunk/ps-mdm-flowsub-mp/doc/...
Roman
On Wed, 16 Jun 2010, Michael Bischoff wrote:
Well the old, and really old is still on the wiki:
https://wiki.man.poznan.pl/perfsonar-mdm/index.php/Flow_Subscription_MP/Impl...
The (over?) simplified rundown is as follows:
---------------------------------------------------- On perfsonar layer(control channel): Subscribe message( subject(what do you want a stream of) + target(setup tunnel to where) -> <- key ... keepalive(with key) -> <- ok ... Unsubscribe -> <- clean tear down ---------------------------------------------------- On the transport layer(data channel): (pre condition -> subscribe request) perfsonar service creates a tunnel to target. write data to service end of tunnel
Now an extension upon the NMC-base that would standardize the 'persfonar as control channel + data channel' would need to ratify a couple of things like tunnel type negotiation(current zebadee is simply assumed and used), some definition over what protocols are allowed(they should probably be 'open') and other things that I can't remember right now.
papers describing different implementations(non-perfsonar/nmc) should be widely available. The short, as most of the papers I've seen also seem to comment, is that setting up a separate data channel is costly and one needs to be able to justify this cost. Such an approach is only useful in cases where you know beforehand that there is going to be a lot of data. Other issue is the cost of converting data because the data- channel can 'speak' anything it wants it doesn't require conversion which might be, a big enough safe that it offsets the cost of setting up the separate channel. a Perfsonar specific advantage is that it allows for smoother transition to perfsonar as you can utilize perfsonar to discover services but still receive in know format and process it using (non-perfsonar) tools that are already available. (this avoids the all or nothing barrier)
Where there any specifics you wanted to know?
Best regards,
Michael
On Wed, Jun 16, 2010 at 1:18 PM, romradz@man.poznan.pl <romradz@man.poznan.pl> wrote:
Hi Michael,
is there the documentation of control channel approach which was implemented in flow subscription MA?
Cheers, Roman
On Fri, 11 Jun 2010, Michael Bischoff wrote:
I guess we postponed or something went really wrong.. I've set up a doodle so people can indicate their availability for next week: http://doodle.com/m7vh8pd6exbaqf29
A proposal for streaming in NMC is here: (mind you in still a work in progress/draft and it hasn't been proof read/spell checked) https://docs.google.com/a/controplex.com/View?id=dpn9385_1fmtsx7g9 In it's current form should be good enough for gettign the discussion started.
Best regards,
Michael On Thu, Jun 10, 2010 at 4:03 PM, Michael Bischoff <Michael.bischoff@controplex.nl> wrote: That is a possibility I suppose, I don't know about the availability of others. Who is going to be available as it currently stands?
On Thu, Jun 10, 2010 at 3:14 PM, romradz@man.poznan.pl <romradz@man.poznan.pl> wrote:
Hi,
it will be very difficult to me to join the meeting today :( Can we postpone it?
Roman
On Tue, 8 Jun 2010, Jason Zurawski wrote:
All;
There will be an NMC call this thursday at 15:00 UTC. The meeting will be run by Michael, and the agenda is to discus altering the protocol to allow streaming (the response in particular). The notes and agenda will be posted here:
https://forge.gridforum.org/sf/wiki/do/viewPage/projects.nmc-wg/wiki/2010061...
Thanks;
-jason _______________________________________________ Nmc-wg mailing list Nmc-wg@ogf.org http://www.ogf.org/mailman/listinfo/nmc-wg
participants (2)
-
Michael Bischoff -
romradz＠man.poznan.pl