[Nfd-dev] [Ndn-app] Close NFD backdoor

Tue Sep 9 22:49:04 PDT 2014

I feel things are becoming too complicated. Here’s my opinion:

The backdoor, i.e., caching unsolicited data from local apps, was a hack to make some apps work quickly. It is not the proper way to write the app nor the forwarder. Apps should take care of their own data (think e2e argument). They should work with or without local CS, or with any size of local CS. In other words, they shouldn’t rely on the backdoor.

The most effective way to take care of one’s data is to store and manage them in the app itself. So an in-app storage mechanism such as MemoryContentCache is the way to go. An app can also use a database, the file system, or any other way to manage its data.

If data duplication, one copy in app and one copy in CS, is a concern, the CS can be made not to cache data coming from local apps. Caching such data doesn’t improve user-perceived performance much anyway.

Solving app’s data management problem in CS will only make the forwarder unnecessarily complicated, and I don’t see any clear gain from doing that.

Beichuan

On Sep 9, 2014, at 9:50 PM, Junxiao Shi <shijunxiao at email.arizona.edu> wrote:

> Dear folks
> 
> Below is a summary of the problem, and a comparison of three solutions.
> "In-Forwarder Storage" is my proposed solution.
> 
> Yours, Junxiao
> 
> Background
> 
> ContentStore concept
> 
> ContentStore is an element of NFD which serves as an opportunistic cache. Incoming Data packets have the opportunity to be admitted into the ContentStore. Data packets are not persistent in ContentStore: they can be evicted when ContentStore is full.
> 
> Admission to the ContentStore is a two step process:
> 
> Forwarding pipelines must decide whether a Data packet SHOULD be admitted. A Data packet should be admitted if it's believed to be legitimate (ie. not part of an attack).
> ContentStore must decide whether there is space for this Data packet.
> Data packets in the ContentStore can be used to satisfy incoming Interests. 
> When an incoming Interest contains MustBeFresh=yes selector, only non-stale Data packets can be returned. A Data packet is stale if it has been in the ContentStore for longer thanFreshnessPeriod; note that if a Data packet is admitted while the same Data packet already exists in the ContentStore, the admission timestamp is reset, so this is equivalent to deleting the old Data packet and admitting the new one.
> 
> Retention in the ContentStore is not guaranteed. The ContentStore is permitted to evict any Data packet at any time, regardless of its origin, staleness, admission time, etc.
> 
> ContentStore in NFD 0.2.0
> 
> Admission: Incoming Data pipeline determines whether a Data packet SHOULD be admitted. If a Data packet satisfies one or more PIT entries, it should be admitted; this is regardless of whether an Interest has been forwarded to the origin of this Data packet. Otherwise, the Data packet is said to be unsolicited, and decision is given to Data unsolicited pipeline.
> 
> Generally, unsolicited Data needs to be dropped as it poses security risks to the forwarder. However, there are cases when unsolicited Data packets needs to be accepted to the ContentStore. In particular, the current implementation allows any unsolicited Data packet to be cached if this Data packet arrives from a local Face.
> 
> Once forwarding pipelines decide a Data should be admitted, ContentStore always admits it. If the ContentStore is full and a new Data packet needs to be admitted, one old Data packet is evicted.
> 
> Eviction: The ContentStore logically maintains three FIFO queues:
> 
> unsolicited Data
> stale Data
> fresh Data
> The front Data packet in the first non-empty queue is evicted.
> 
> Data Generation in Application
> 
> Data Generation Misaligned with Interest Arrival
> 
> NDN content distribution is receiver-driven. The producer cannot push Data packets into the network; it must wait for consumer to express Interests, before responding with Data packets.
> 
> However, Data generation can happen before Interest arrival.
> 
> a video camera captures a consequtive video stream, and generates Data packets as a frame is captured
> a NFD StatusDataset publisher generates all segments of a StatusDataset in response to a request, not one segment at a time
> a ChronoChat client generates a Data packet when the user enters a message, before other users discover the existence of this Data packet (through synchronization) and retrieve it
> a NLSR daemon generates a Data packet for a Link State Announcement, before other routers discover the existence of this Data packet (through synchronization) and retrieve it
> Producer is not allowed to send those Data packets into the network before Interest arrival. Therefore, those Data packets must be stored somewhere.
> 
> Latest Version and Short Freshness Period
> 
> The most useful Data packets are those representing the latest version of some information. The producer has the final authority: when an incoming Interest asks for the latest version, the producer would respond with the latest version.
> 
> FreshnessPeriod field is an indication about how long the Data packet may remain the latest version. Typically, the producer sets this field to a duration in which it believes a newer version won't be generated. However, two situations cause the producer to prefer a shorter FreshnessPeriod:
> 
> FreshnessPeriod is per network node: if a Data packet is near the end of its freshness period at one node, and it's delivered to another node, the admitted Data packet on the second node would remain non-stale for the entire FreshnessPeriod. This means, if there are N hops between consumer and producer, in the worst case, the Data may appear fresh to consumer for up to N timesFreshnessPeriod.
> The producer is sometimes unable to predict how long a version can remain to be the latest version.
> Problem
> 
> The "NFD backdoor" is a feature: NFD allows unsolicited Data from a local face to be admitted to the ContentStore.
> 
> This feature allows NDN applications to "pre-publish" Data packets when future Interests are anticipated. At the other end, the NDN applications themselves do not retain the Data packet.
> 
> The drawback of this approach is: retention is not guaranteed. 
> Although pre-published Data packets are admitted to the ContentStore, they are tagged as unsolicited. Unsolicited Data packets are the first to be evicted from ContentStore when ContentStore is full. Therefore, when there are enough solicited Data packets in the ContentStore, unsolicited Data packets can hardly be retained in the ContentStore. 
> If a Data packet is evicted, the Interest would be forwarded to the application. The application either has to generate the Data again, or is unable to respond at all.
> 
> Another problem of application not storing generated Data packets is: a Data packet may remain to be the latest version beyond FreshnessPeriod. 
> The ContentStore can only use non-stale Data to satisfy an Interest with MustBeFresh=yes. If the producer sets a short FreshnessPeriod, the Interest would be forwarded to the application, even if the Data packet is still the latest version. 
> The application would have to generate (and sign) the Data again.
> 
> Necessity of Application-Controllable Data Storage
> 
> Both problems have a common solution: a Data storage controlled by application.
> 
> This Data storage should have the following characteristics:
> 
> Data packets useful to a producer application are stored. A Data packet is useful, if the producer application would generate the same again when an Interest asks for it.
> Producer can explicitly delete Data packets that are no longer useful. The most common time to delete Data packets is when a new version is generated.
> Producer should delete Data packets when the Data storage is close to its capacity. Such deletions can either be explicit, or be delegated to an eviction policy.
> The Data storage is not a cache.
> Solutions
> 
> The application-controllable Data storage can either be placed in the application, or be placed in the forwarder.
> 
> In-App Storage
> 
> In-App Storage provides an application-controllable Data storage within the application. The module is typically provided by the library.
> 
> Benefits:
> 
> keep the forwarder simple
> application has a chance to inspect the Data again before using it to satisfy incoming Interest
> Drawbacks:
> 
> duplication: when a Data packet is used to satisfy an Interest, it would have copies in both In-App Storage and ContentStore
> risk of Data loss: when a Data packet is used to satisfy an Interest, if application chooses to delete the copy in In-App Storage, the Data would be lost when the copy in ContentStore is evicted
> Managed ContentStore
> 
> Managed ContentStore provides an application-controllable Data storage as part of the ContentStore. A Data packet is explicitly inserted to the ContentStore, and the ContentStore guarantees to retain the Data packet until it becomes stale or the application requests to delete it.
> 
> Benefits:
> 
> no duplication
> Drawbacks:
> 
> ContentStore is no longer a cache; eviction policy becomes more complex
> application must refresh the stored Data packet, if it remains the latest version beyond FreshnessPeriod; setting a large FreshnessPeriod is still infeasible because in-network caches may retain the Data packet for too long, this means refreshing would be very frequent
> higher deletion cost: deletion needs a command
> inability to enumerate: unlike In-App Storage, the application is unable to enumerate stored Data packets
> In-Forwarder Storage
> 
> An application-controllable Data storage can be placed in the forwarder, but out of the ContentStore.
> 
> The application sends a command to create an application-controllable Data storage in the forwarder.
> The Data storage is logically attached to the face of this application. Its capacity and replacement policy are determined when it's created. Its destroyed when face is closing.
> To insert a Data packet to the Data storage, the application must explicitly set a field in the LocalControlHeader.
> Insertion to the Data storage is independant from forwarding pipelines, but the application can elect to deliver the same Data packet into forwarding pipelines.
> Data unsolicited pipeline would not admit unsolicited Data packets to the ContentStore, except in special environments such as vehicle networks.
> The application can scheduled the deletion of a Data packet during insertion.
> The deletion timer can be set differently from FreshnessPeriod.
> The application can delete Data packets by sending a command.
> Command parameter is a Name prefix. All stored Data packets under this prefix are deleted.
> Stored Data packets are considered only if an Interest is to be forwarded to the application.
> When an Interest is to be forwarded to the application, the Data storage attached to the face is queried. If a matching Data packet is found, the Interest is not sent, and the Data packet is treated as if it's received from the application.
> Benefits:
> 
> no duplication: if a Data packet appears in both application-controllable Data storage and the ContentStore, it has only one in-memory copy, with references from two places
> less inter-process communication and encoding/decoding, when most stored Data packets are requested more than once
> no risk of cache poisoning: the application-controllable Data storage is not considered until an Interest is to be forwarded to the application
> no refreshing needed
> Drawbacks:
> 
> higher deletion cost: deletion needs a command
> inability to enumerate: unlike In-App Storage, the application is unable to enumerate stored Data packets
> address space consumption: stored Data packets are mapped in forwarder's address space, and address space is limited on 32-bit platforms
> <contentstore-appstorage_20140909.md>_______________________________________________
> Ndn-app mailing list
> Ndn-app at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-app

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/nfd-dev/attachments/20140909/017bd743/attachment.html>