[Nfd-dev] [Ndn-app] Close NFD backdoor

[Junxiao Shi]

Hi Beichuan

My proposal does not store application's Data packets into the
ContentStore. The In-Forwarder Data Storage is separate from ContentStore,
although packets can be shared via shared_ptr.

Using In-App Storage and not caching Data in ContentStore is compared to my
proposal as follows.

Benefits:

* less memory usage by forwarder, which means less memory management
overhead in forwarder process and thread
* app is able to enumerate stored Data packets
* deletion is easier

Drawbacks:

* more inter-process communication and encoding/decoding overhead

Same:

* no duplication
* PIT states are generated. Note: in NFD, PIT entry is created even if
Interest can be satisfied by ContentStore, because this is needed for loop
detection and measurements. Not caching in ContentStore doesn't increase
PIT states.

I agree that using In-App Storage and not caching Data in ContentStore is
better than In-Forwarder Data Storage.

Yours, Junxiao
On Sep 9, 2014 10:49 PM, "Beichuan Zhang" <bzhang at cs.arizona.edu> wrote:

> I feel things are becoming too complicated. Here’s my opinion:
>
> The backdoor, i.e., caching unsolicited data from local apps, was a hack
> to make some apps work quickly. It is not the proper way to write the app
> nor the forwarder. Apps should take care of their own data (think e2e
> argument). They should work with or without local CS, or with any size of
> local CS. In other words, they shouldn’t rely on the backdoor.
>
> The most effective way to take care of one’s data is to store and manage
> them in the app itself. So an in-app storage mechanism such as
> MemoryContentCache is the way to go. An app can also use a database, the
> file system, or any other way to manage its data.
>
> If data duplication, one copy in app and one copy in CS, is a concern, the
> CS can be made not to cache data coming from local apps. Caching such data
> doesn’t improve user-perceived performance much anyway.
>
> Solving app’s data management problem in CS will only make the forwarder
> unnecessarily complicated, and I don’t see any clear gain from doing that.
>
> Beichuan
>
>
> On Sep 9, 2014, at 9:50 PM, Junxiao Shi <shijunxiao at email.arizona.edu>
> wrote:
>
> Dear folks
>
> Below is a summary of the problem, and a comparison of three solutions.
> "In-Forwarder Storage" is my proposed solution.
>
> Yours, Junxiao
>
> BackgroundContentStore concept
>
> ContentStore is an element of NFD which serves as an *opportunistic cache*.
> Incoming Data packets have the opportunity to be admitted into the
> ContentStore. Data packets are not persistent in ContentStore: they can be
> evicted when ContentStore is full.
>
> Admission to the ContentStore is a two step process:
>
>    1. Forwarding pipelines must decide whether a Data packet *SHOULD* be
>    admitted. A Data packet should be admitted if it's believed to be
>    legitimate (ie. not part of an attack).
>    2. ContentStore must decide whether there is space for this Data
>    packet.
>
> Data packets in the ContentStore can be used to satisfy incoming
> Interests.
> When an incoming Interest contains *MustBeFresh=yes* selector, only
> *non-stale* Data packets can be returned. A Data packet is *stale* if it
> has been in the ContentStore for longer than*FreshnessPeriod*; note that
> if a Data packet is admitted while the same Data packet already exists in
> the ContentStore, the admission timestamp is reset, so this is equivalent
> to deleting the old Data packet and admitting the new one.
>
> Retention in the ContentStore is not guaranteed. The ContentStore is
> permitted to *evict* any Data packet at any time, regardless of its
> origin, staleness, admission time, etc.
> ContentStore in NFD 0.2.0
>
> *Admission*: *Incoming Data pipeline* determines whether a Data packet
> *SHOULD* be admitted. If a Data packet satisfies one or more PIT entries,
> it should be admitted; this is regardless of whether an Interest has been
> forwarded to the origin of this Data packet. Otherwise, the Data packet is
> said to be *unsolicited*, and decision is given to *Data unsolicited
> pipeline*.
>
> Generally, unsolicited Data needs to be dropped as it poses security risks
> to the forwarder. However, there are cases when unsolicited Data packets
> needs to be accepted to the ContentStore. In particular, the current
> implementation allows any unsolicited Data packet to be cached if this Data
> packet arrives from a local Face.
>
> Once forwarding pipelines decide a Data should be admitted, ContentStore
> always admits it. If the ContentStore is full and a new Data packet needs
> to be admitted, one old Data packet is evicted.
>
> *Eviction*: The ContentStore logically maintains three FIFO queues:
>
>    1. unsolicited Data
>    2. stale Data
>    3. fresh Data
>
> The front Data packet in the first non-empty queue is evicted.
> Data Generation in ApplicationData Generation Misaligned with Interest
> Arrival
>
> NDN content distribution is receiver-driven. The producer cannot push Data
> packets into the network; it must wait for consumer to express Interests,
> before responding with Data packets.
>
> However, *Data generation can happen before Interest arrival*.
>
>    - a video camera captures a consequtive video stream, and generates
>    Data packets as a frame is captured
>    - a NFD StatusDataset
>    <http://redmine.named-data.net/projects/nfd/wiki/StatusDataset> publisher
>    generates all segments of a StatusDataset in response to a request, not one
>    segment at a time
>    - a ChronoChat <https://github.com/bruinfish/ChronoChat> client
>    generates a Data packet when the user enters a message, before other users
>    discover the existence of this Data packet (through synchronization) and
>    retrieve it
>    - a NLSR <https://github.com/named-data/NLSR> daemon generates a Data
>    packet for a Link State Announcement, before other routers discover the
>    existence of this Data packet (through synchronization) and retrieve it
>
> Producer is not allowed to send those Data packets into the network before
> Interest arrival. Therefore, those Data packets must be stored *somewhere*
> .
> Latest Version and Short Freshness Period
>
> The most useful Data packets are those representing the latest version of
> some information. The producer has the final authority: when an incoming
> Interest asks for the latest version, the producer would respond with the
> latest version.
>
> *FreshnessPeriod* field is an indication about how long the Data packet
> may remain the latest version. Typically, the producer sets this field to a
> duration in which it believes a newer version won't be generated. However,
> two situations cause the producer to prefer a shorter *FreshnessPeriod*:
>
>    - *FreshnessPeriod* is per network node: if a Data packet is near the
>    end of its freshness period at one node, and it's delivered to another
>    node, the admitted Data packet on the second node would remain
>    *non-stale* for the entire *FreshnessPeriod*. This means, if there are
>    N hops between consumer and producer, in the worst case, the Data may
>    appear fresh to consumer for up to N times*FreshnessPeriod*.
>    - The producer is sometimes unable to predict how long a version can
>    remain to be the latest version.
>
> Problem
>
> The "NFD backdoor" is a *feature*: NFD allows unsolicited Data from a
> local face to be admitted to the ContentStore.
>
> This feature allows NDN applications to "pre-publish" Data packets when
> future Interests are anticipated. At the other end, the NDN applications
> themselves do not retain the Data packet.
>
> The drawback of this approach is: *retention is not guaranteed*.
> Although pre-published Data packets are admitted to the ContentStore, they
> are tagged as unsolicited. Unsolicited Data packets are the first to be
> evicted from ContentStore when ContentStore is full. Therefore, when there
> are enough solicited Data packets in the ContentStore, unsolicited Data
> packets can hardly be retained in the ContentStore.
> If a Data packet is evicted, the Interest would be forwarded to the
> application. The application either has to generate the Data again, or is
> unable to respond at all.
>
> Another problem of application not storing generated Data packets is: *a
> Data packet may remain to be the latest version beyond FreshnessPeriod*.
> The ContentStore can only use *non-stale* Data to satisfy an Interest
> with *MustBeFresh=yes*. If the producer sets a short *FreshnessPeriod*,
> the Interest would be forwarded to the application, even if the Data packet
> is still the latest version.
> The application would have to generate (and sign) the Data again.
> Necessity of Application-Controllable Data Storage
>
> Both problems have a common solution: a Data storage controlled by
> application.
>
> This Data storage should have the following characteristics:
>
>    - Data packets *useful* to a producer application are stored. A Data
>    packet is *useful*, if the producer application would generate the
>    same again when an Interest asks for it.
>    - Producer can explicitly delete Data packets that are no longer
>    *useful*. The most common time to delete Data packets is when a new
>    version is generated.
>    - Producer should delete Data packets when the Data storage is close
>    to its capacity. Such deletions can either be explicit, or be delegated to
>    an eviction policy.
>    - The Data storage is not a cache.
>
> Solutions
>
> The application-controllable Data storage can either be placed in the
> application, or be placed in the forwarder.
> In-App Storage
>
> In-App Storage
> <http://redmine.named-data.net/projects/ndn-cxx/wiki/InMemoryStorage> provides
> an application-controllable Data storage within the application. The module
> is typically provided by the library.
>
> Benefits:
>
>    - keep the forwarder simple
>    - application has a chance to inspect the Data again before using it
>    to satisfy incoming Interest
>
> Drawbacks:
>
>    - duplication: when a Data packet is used to satisfy an Interest, it
>    would have copies in both In-App Storage and ContentStore
>    - risk of Data loss: when a Data packet is used to satisfy an
>    Interest, if application chooses to delete the copy in In-App Storage, the
>    Data would be lost when the copy in ContentStore is evicted
>
> Managed ContentStore
>
> Managed ContentStore
> <http://redmine.named-data.net/projects/nfd/wiki/ManagedContentStore> provides
> an application-controllable Data storage as part of the ContentStore. A
> Data packet is explicitly inserted to the ContentStore, and the
> ContentStore guarantees to retain the Data packet until it becomes *stale* or
> the application requests to delete it.
>
> Benefits:
>
>    - no duplication
>
> Drawbacks:
>
>    - ContentStore is no longer a cache; eviction policy becomes more
>    complex
>    - application must refresh the stored Data packet, if it remains the
>    latest version beyond *FreshnessPeriod*; setting a large
>    *FreshnessPeriod* is still infeasible because in-network caches may
>    retain the Data packet for too long, this means refreshing would be very
>    frequent
>    - higher deletion cost: deletion needs a command
>    - inability to enumerate: unlike In-App Storage, the application is
>    unable to enumerate stored Data packets
>
> In-Forwarder Storage
>
> An application-controllable Data storage can be placed in the forwarder,
> but out of the ContentStore.
>
>    - The application sends a command to create an
>    application-controllable Data storage in the forwarder.
>       - The Data storage is logically attached to the face of this
>       application. Its capacity and replacement policy are determined when it's
>       created. Its destroyed when face is closing.
>    - To insert a Data packet to the Data storage, the application must
>    explicitly set a field in the LocalControlHeader.
>       - Insertion to the Data storage is independant from forwarding
>       pipelines, but the application can elect to deliver the same Data packet
>       into forwarding pipelines.
>       - *Data unsolicited pipeline* would not admit unsolicited Data
>       packets to the ContentStore, except in special environments such as vehicle
>       networks.
>    - The application can scheduled the deletion of a Data packet during
>    insertion.
>       - The deletion timer can be set differently from *FreshnessPeriod*.
>    - The application can delete Data packets by sending a command.
>       - Command parameter is a Name prefix. All stored Data packets under
>       this prefix are deleted.
>    - Stored Data packets are considered only if an Interest is to be
>    forwarded to the application.
>       - When an Interest is to be forwarded to the application, the Data
>       storage attached to the face is queried. If a matching Data packet is
>       found, the Interest is not sent, and the Data packet is treated as if it's
>       received from the application.
>
> Benefits:
>
>    - no duplication: if a Data packet appears in both
>    application-controllable Data storage and the ContentStore, it has only one
>    in-memory copy, with references from two places
>    - less inter-process communication and encoding/decoding, when most
>    stored Data packets are requested more than once
>    - no risk of cache poisoning: the application-controllable Data
>    storage is not considered until an Interest is to be forwarded to the
>    application
>    - no refreshing needed
>
> Drawbacks:
>
>    - higher deletion cost: deletion needs a command
>    - inability to enumerate: unlike In-App Storage, the application is
>    unable to enumerate stored Data packets
>    - address space consumption: stored Data packets are mapped in
>    forwarder's address space, and address space is limited on 32-bit platforms
>
> <contentstore-appstorage_20140909.md>
> _______________________________________________
> Ndn-app mailing list
> Ndn-app at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-app
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/nfd-dev/attachments/20140909/a0d24cf7/attachment.html>