[Ndn-lib] PSync PartialProducer on multiple nodes

Lan Wang (lanwang) lanwang at memphis.edu
Tue Jan 29 15:36:42 PST 2019

On Jan 29, 2019, at 5:22 PM, Junxiao Shi <shijunxiao at email.arizona.edu<mailto:shijunxiao at email.arizona.edu>> wrote:

Hi Lan

As I said, this approach is not scalable: it requires one Consumer for each participant, which is expensive in terms of traffic overhead and memory usage.
Why not merge FullSync and PartialSync so that multiple producers can publish their user nodes on the same sync prefix?

That’s something we can think about, but they do work differently (the PartialSync interest carries the producer’s previous IBF when the subscriber last sync’ed with the producer (not the subscriber’s own IBF), while the Fullsync interest carries the node’s own IBF.  I don’t have a clear idea of how to combine them or if it’s possible right now.

I see, the reason is down in the protocol. It is indeed impossible to combine them because they need different IBFs.

I think the solution is to provide a higher level facility in the library that supports PartialSync from multiple producers. Inside this facility:

  1.  It runs one instance of FullSync to maintain the roster. Every producer and consumer participate in this FullSync group.
  2.  Each producer has one UserNode in the FullSync group. If two producers are serving the same dataset, they should participate FullSync under the same UserNode. Consumers do not have UserNodes in the FullSync group.

I am not sure if we allow pure consumer nodes (no UserPrefix) in FullSync mode.

  1.  Each producer runs one instance of PartialSync. Whenever the list of UserNodes in PartialSync changes, the producer increments the sequence number of its UserNode in FullSync.

  1.  Consumer sends hello to a producer whenever the producer's FullSync UserNode has a new sequence number.

This would accommodate a PartialSync UserNode to be served from any producer. PartialSync UserNodes can even be moved between producers.
One potential problem is what to do when multiple distinct producers have the same PartialSync UserNode.

If they have different data, they should not have the same user prefix.  If they do have the same data (e.g., they replicate each other’s data), then PartialSync can handle these producers having the same user prefix.

If you are concerned about the number of consumers.  We can change the code to allow one consumer to subscribe to multiple different sync prefixes.  That should not be too difficult an implementation change.

If the library has the higher level facility as described above, number of consumers isn't really an issue because it's hidden from the application.

I think this is a good direction.  I suggest creating a Redmine issue for further discussion.


Yours, Junxiao

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-lib/attachments/20190129/235f2716/attachment-0001.html>

More information about the Ndn-lib mailing list