From felix at rabe.io  Wed Oct  1 10:16:23 2014
From: felix at rabe.io (Felix Rabe)
Date: Wed, 01 Oct 2014 19:16:23 +0200
Subject: [Ndn-interest] FreeNode IRC channel: #named-data
Message-ID: <542C36E7.3080700@rabe.io>

Hi

I've just opened *#named-data* on freenode.net for use by the NDN 
project. (I'll gladly hand over op rights.) If you are not in the middle 
of deep research, feel free to join me there.

Actually, I'm online now and got some questions, so please join :)

Someone already got the #ndn channel there, but I don't know who or 
whether it is for NDN discussion.

- Felix
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141001/67204930/attachment.html>

From jburke at remap.ucla.edu  Fri Oct  3 19:17:32 2014
From: jburke at remap.ucla.edu (Burke, Jeff)
Date: Sat, 4 Oct 2014 02:17:32 +0000
Subject: [Ndn-interest] Selector Protocol over exact matching
In-Reply-To: <D049953D.33672%Ignacio.Solis@parc.com>
Message-ID: <D054A247.175B49%jburke@remap.ucla.edu>


Sorry about the delay in returning to this.  The idea of a protocol that
provides these features at a higher layer definitely draws out the design
issues; thanks for writing it out.

A couple of questions that occur to me about the approach:

- Data names become about the query rather than about the data; this seems
to be a fundamental design difference.  If the response to each interest
includes a hash of the query, it can't be used to fulfill different
queries that would have the same result. So some caching benefit is lost.
(This seems like the basic tradeoff of exact match vs lpm+selectors.)

- In the case where there are multiple publishers with different answers
to the same selector protocol query, they would return different data for
the same request.   In the current NDN case, this is disambiguated because
the response includes the name of the data itself.  But in the selector
protocol case, couldn't you have many answers to the question all sharing
the same name?  

- Further, without exclusions and without more specific names, how do you
retrieve multiple answers to the same query?  Let's assume that in some
cases this is a desired feature.

- How do you envision applications interacting with the selector protocol?
 Would they implement it directly or publish "normally named" content to a
stack that then implements the protocol?



Thanks,
Jeff

On 9/25/14, 1:10 AM, "Ignacio.Solis at parc.com" <Ignacio.Solis at parc.com>
wrote:

>On 9/25/14, 9:17 AM, "Marc.Mosko at parc.com" <Marc.Mosko at parc.com> wrote:
>>In the CCNx 1.0 spec, one could also encode this a different way.  One
>>could use a name like ?/mail/inbox/selector_matching/<hash of payload>?
>>and in the payload include "exclude_before=(t=version, l=2, v=279) &
>>sort=right?.
>
>
>I want to highlight this.
>
>There is a role that selectors can play in a network.  However, our
>biggest issue with selectors is that they are mandated at the forwarder
>level.  This means that every node must support selectors.
>
>We want to make sure that the core protocol is simple and efficient.
>Exact matching gives us that.  If you?re interested in selector matching
>and searching, then create that protocol over exact matching.
>
>Marc just described a simple ?Selector Protocol", basically:
>- Encode selectors (or any query you want) in the interest payload.
>- Add a name segment to indicate that this is a selector based query
>- Add a name segment to uniquely identify the query (a hash of the payload
>for example)
>
>Example:
>name    = /mail/inbox/list/selector_matching/<hash of interest payload>
>
>payload = version > 100
>
>Topology:
>
>A ?? B ?? C
>
>A and C run the Selector Protocol
>B does not run the Selector Protocol
>
>Now: 
>- Any node that does not understand the Selector Protocol (B) forwards
>normally and does exact matching.
>- Any node that understands the Selector Protocol (C) can parse the
>payload to find a match.
>
>If no match is found, forward the interest.
>If a match is found, create a reply.
>
>The reply can contain 2 types of data:
>- Structured data with links to the actual content objects
>- Encapsulated content objects
>
>So, in our example, the Selector Protocol reply could be:
>
>name = /mail/inbox/list/selector_matching/<hash of interest payload>
>payload =
>  [  matching name = /mail/inbox/list/v101 ]
>  [  embedded object < name = /mail/inbox/list/v101, payload = list,
>signature = mail server > ]
>signature = responding cache
>
>
>
>A few notes:
>- Malicious nodes could inject false replies.  So, if C is malicious, it
>can insert a reply linking to some random object or just return junk.
>Well, this would be the case with regular selectors as well.  C could
>reply with random crap or it could reply with a valid answer that is not
>the optimal answer (so, for example, not the right-most child or
>something).
>This is something that we can?t prevent.
>
>In the case of CCN, our fast path does not check signatures, so you
>wouldn?t be able to check the signature of the reply no matter what.  I?m
>unsure if NDN is still advocating that every node checks signatures.  If
>you are, then this approach might not work for you.
>
>Nodes that DO understand the Selector Protocol can check the signature of
>the encapsulated reply (if they wanted to).
>Nodes that DO understand the Selector Protocol can unpack the reply, and
>add the corresponding object to their cache, effectively enabling them to
>answer other Selector Protocol queries.
>
>- The reply from the Selector Protocol enabled node (C), could:
>?  include a list of all valid answers
>?  embed no objects
>?  embed more than 1 object
>?  process complex queries, regex, etc.
>
>The Selector Protocol could also:
>- include a method for authentication
>- include a cursor or some other state between queries
>
>
>I think this sort of protocol gives you everything you want while still
>maintaining an exact match protocol as the core protocol.
>
>
>What is this protocol missing to satisfy your needs?
>Can we create a protocol that will satisfy your needs on top of exact
>matching?
>
>
>Nacho
>
>
>
>--
>Nacho (Ignacio) Solis
>Protocol Architect
>Principal Scientist
>Palo Alto Research Center (PARC)
>+1(650)812-4458
>Ignacio.Solis at parc.com
>
>
>
>
>




From jburke at remap.ucla.edu  Fri Oct  3 19:23:08 2014
From: jburke at remap.ucla.edu (Burke, Jeff)
Date: Sat, 4 Oct 2014 02:23:08 +0000
Subject: [Ndn-interest] Discovery (was: any comments on naming
 convention?)
In-Reply-To: <8B644CB3-E05E-4A1E-A9D6-222B09E64B7F@parc.com>
Message-ID: <D054A71B.175BC7%jburke@remap.ucla.edu>


So far this makes sense to me.   My impression is that the current NDN
proposal is to use set reconciliation / SYNC style approaches for
'discover all' and that LPM+selectors provide an option for 'discover
greatest' or 'discover nearest'.  (An example of the latter is an interest
in a video seek operation that is designed to return the closest keyframe
to a specified time, which is then followed by exact match interests to
get subsequent segments for playout.)

Another class of 'queries' that has come up in several applications is kNN
- given some target object name, return the k nearest neighbors to that
interest.  

Jeff


On 9/22/14, 1:33 PM, "Marc.Mosko at parc.com" <Marc.Mosko at parc.com> wrote:

>I received some feedback that people would like to know more about
>discovery and what we have done in ccnx 1.0, so I am starting a new
>thread for it.
>
>As I mentioned, we are not ready to publish our discovery protocols and I
>don?t want to go off half way with something we are still working on, but
>I can talk about what I think discovery is and should do.  Discovery is a
>very important topic and I don?t think research is anywhere near done on
>the topic.
>
>First, I think we need a clear definition of what services discover
>offers.  I would say it should do, at least, ?discover greatest? and
>?discover all? with the obvious variations based on sort order and range,
>for some given prefix.  It should also support discovery scoped by one
>(or possibly more) publisher keys.
>
>What does ?discover greatest? mean?  One could do something similar to
>ccnx 0.x and ndn, where its based on a canonical sort order of name
>components and one could ask for the greatest name after a given prefix.
>Or, one could do something specific to a versioning protocol or creation
>time, etc.
>
>What does ?discover all? mean?  First, I think we should recognize that
>some data sets might be very, very large.  Like millions or billions of
>possible results (content objects).  The discovery protocol should be
>able to discover it all, efficiently (if not optimally).
>
>I also think there is no one discovery protocol.  Some applications may
>want strict ACID-style discovery (i.e. only see ?completed? or ?whole?
>results, such as where all segments are available) some might want
>eventually consistent discovery, some might take best-effort discovery.
>Some discovery protocols may require authentication and some may be open
>to the world.
>
>I think the discovery process should be separate from the retrieval
>process.  If one is publishing, say, 64KB objects or even 4GB objects (or
>even larger objects!), one does not want to have to fetch each object to
>discover it.
>
>All this leads me to think we need discovery protocols that allow us to
>talk about content without actually transferring the content.
>
>The discovery protocol should be able to handle multiple objects with the
>same name ? i.e. two publishers overwrite each other or even a single
>publisher publishes two objects with the same name.
>
>Discovery is also closely related to how routing works and forwarding
>strategy.  Does an Interest flood all replicas?  Does it only go anycast
>style?  How large can an Interest be, and what?s the performance tradeoff
>for large interests (i.e. dropping fragments)?  Is there an in-network
>control protocol to NAK or are all NAKs end-to-end?  How does a discovery
>process terminate (i.e. when do you know you?re done)?
>
>Marc
>
>
>
>On Sep 21, 2014, at 9:23 AM, Mosko, Marc <Marc.Mosko at parc.com>
><Marc.Mosko at parc.com> wrote:
>
>> No matter what the expressiveness of the predicates if the forwarder
>>can send interests different ways you don't have a consistent underlying
>>set to talk about so you would always need non-range exclusions to
>>discover every version.
>> 
>> Range exclusions only work I believe if you get an authoritative
>>answer.   If different content pieces are scattered between different
>>caches I don't see how range exclusions would work to discover every
>>version.
>> 
>> I'm sorry to be pointing out problems without offering solutions but
>>we're not ready to publish our discovery protocols.
>> 
>> Sent from my telephone
>> 
>>> On Sep 21, 2014, at 8:50, "Tai-Lin Chu" <tailinchu at gmail.com> wrote:
>>> 
>>> I see. Can you briefly describe how ccnx discovery protocol solves the
>>> all problems that you mentioned (not just exclude)? a doc will be
>>> better.
>>> 
>>> My unserious conjecture( :) ) : exclude is equal to [not]. I will soon
>>> expect [and] and [or], so boolean algebra is fully supported.  Regular
>>> language or context free language might become part of selector too.
>>> 
>>>> On Sat, Sep 20, 2014 at 11:25 PM,  <Marc.Mosko at parc.com> wrote:
>>>> That will get you one reading then you need to exclude it and ask
>>>>again.
>>>> 
>>>> Sent from my telephone
>>>> 
>>>> On Sep 21, 2014, at 8:22, "Tai-Lin Chu" <tailinchu at gmail.com> wrote:
>>>> 
>>>>>> Yes, my point was that if you cannot talk about a consistent set
>>>>>>with a particular cache, then you need to always use individual
>>>>>>excludes not range excludes if you want to discover all the versions
>>>>>>of an object.
>>>>> 
>>>>> I am very confused. For your example, if I want to get all today's
>>>>> sensor data, I just do (Any..Last second of last day)(First second of
>>>>> tomorrow..Any). That's 18 bytes.
>>>>> 
>>>>> 
>>>>> [1]http://named-data.net/doc/ndn-tlv/interest.html#exclude
>>>>> 
>>>>>> On Sat, Sep 20, 2014 at 10:55 PM,  <Marc.Mosko at parc.com> wrote:
>>>>>> 
>>>>>> On Sep 21, 2014, at 1:47 AM, Tai-Lin Chu <tailinchu at gmail.com>
>>>>>>wrote:
>>>>>> 
>>>>>>>> If you talk sometimes to A and sometimes to B, you very easily
>>>>>>>>could miss content objects you want to discovery unless you avoid
>>>>>>>>all range exclusions and only exclude explicit versions.
>>>>>>> 
>>>>>>> Could you explain why missing content object situation happens?
>>>>>>>also
>>>>>>> range exclusion is just a shorter notation for many explicit
>>>>>>>exclude;
>>>>>>> converting from explicit excludes to ranged exclude is always
>>>>>>> possible.
>>>>>> 
>>>>>> Yes, my point was that if you cannot talk about a consistent set
>>>>>>with a particular cache, then you need to always use individual
>>>>>>excludes not range excludes if you want to discover all the versions
>>>>>>of an object.  For something like a sensor reading that is updated,
>>>>>>say, once per second you will have 86,400 of them per day.  If each
>>>>>>exclusion is a timestamp (say 8 bytes), that?s 691,200 bytes of
>>>>>>exclusions (plus encoding overhead) per day.
>>>>>> 
>>>>>> yes, maybe using a more deterministic version number than a
>>>>>>timestamp makes sense here, but its just an example of needing a lot
>>>>>>of exclusions.
>>>>>> 
>>>>>>> 
>>>>>>>> You exclude through 100 then issue a new interest.  This goes to
>>>>>>>>cache B
>>>>>>> 
>>>>>>> I feel this case is invalid because cache A will also get the
>>>>>>> interest, and cache A will return v101 if it exists. Like you
>>>>>>>said, if
>>>>>>> this goes to cache B only, it means that cache A dies. How do you
>>>>>>>know
>>>>>>> that v101 even exist?
>>>>>> 
>>>>>> I guess this depends on what the forwarding strategy is.  If the
>>>>>>forwarder will always send each interest to all replicas, then yes,
>>>>>>modulo packet loss, you would discover v101 on cache A.  If the
>>>>>>forwarder is just doing ?best path? and can round-robin between
>>>>>>cache A and cache B, then your application could miss v101.
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> c,d In general I agree that LPM performance is related to the
>>>>>>>number
>>>>>>> of components. In my own thread-safe LMP implementation, I used
>>>>>>>only
>>>>>>> one RWMutex for the whole tree. I don't know whether adding lock
>>>>>>>for
>>>>>>> every node will be faster or not because of lock overhead.
>>>>>>> 
>>>>>>> However, we should compare (exact match + discovery protocol) vs
>>>>>>>(ndn
>>>>>>> lpm). Comparing performance of exact match to lpm is unfair.
>>>>>> 
>>>>>> Yes, we should compare them.  And we need to publish the ccnx 1.0
>>>>>>specs for doing the exact match discovery.  So, as I said, I?m not
>>>>>>ready to claim its better yet because we have not done that.
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sat, Sep 20, 2014 at 2:38 PM,  <Marc.Mosko at parc.com> wrote:
>>>>>>>> I would point out that using LPM on content object to Interest
>>>>>>>>matching to do discovery has its own set of problems.  Discovery
>>>>>>>>involves more than just ?latest version? discovery too.
>>>>>>>> 
>>>>>>>> This is probably getting off-topic from the original post about
>>>>>>>>naming conventions.
>>>>>>>> 
>>>>>>>> a.  If Interests can be forwarded multiple directions and two
>>>>>>>>different caches are responding, the exclusion set you build up
>>>>>>>>talking with cache A will be invalid for cache B.  If you talk
>>>>>>>>sometimes to A and sometimes to B, you very easily could miss
>>>>>>>>content objects you want to discovery unless you avoid all range
>>>>>>>>exclusions and only exclude explicit versions.  That will lead to
>>>>>>>>very large interest packets.  In ccnx 1.0, we believe that an
>>>>>>>>explicit discovery protocol that allows conversations about
>>>>>>>>consistent sets is better.
>>>>>>>> 
>>>>>>>> b. Yes, if you just want the ?latest version? discovery that
>>>>>>>>should be transitive between caches, but imagine this.  You send
>>>>>>>>Interest #1 to cache A which returns version 100.  You exclude
>>>>>>>>through 100 then issue a new interest.  This goes to cache B who
>>>>>>>>only has version 99, so the interest times out or is NACK?d.  So
>>>>>>>>you think you have it!  But, cache A already has version 101, you
>>>>>>>>just don?t know.   If you cannot have a conversation around
>>>>>>>>consistent sets, it seems like even doing latest version discovery
>>>>>>>>is difficult with selector based discovery.  From what I saw in
>>>>>>>>ccnx 0.x, one ended up getting an Interest all the way to the
>>>>>>>>authoritative source because you can never believe an intermediate
>>>>>>>>cache that there?s not something more recent.
>>>>>>>> 
>>>>>>>> I?m sure you?ve walked through cases (a) and (b) in ndn, I?d be
>>>>>>>>interest in seeing your analysis.  Case (a) is that a node can
>>>>>>>>correctly discover every version of a name prefix, and (b) is that
>>>>>>>>a node can correctly discover the latest version.  We have not
>>>>>>>>formally compared (or yet published) our discovery protocols (we
>>>>>>>>have three, 2 for content, 1 for device) compared to selector
>>>>>>>>based discovery, so I cannot yet claim they are better, but they
>>>>>>>>do not have the non-determinism sketched above.
>>>>>>>> 
>>>>>>>> c. Using LPM, there is a non-deterministic number of lookups you
>>>>>>>>must do in the PIT to match a content object.  If you have a name
>>>>>>>>tree or a threaded hash table, those don?t all need to be hash
>>>>>>>>lookups, but you need to walk up the name tree for every prefix of
>>>>>>>>the content object name and evaluate the selector predicate.
>>>>>>>>Content Based Networking (CBN) had some some methods to create
>>>>>>>>data structures based on predicates, maybe those would be better.
>>>>>>>>But in any case, you will potentially need to retrieve many PIT
>>>>>>>>entries if there is Interest traffic for many prefixes of a root.
>>>>>>>>Even on an Intel system, you?ll likely miss cache lines, so you?ll
>>>>>>>>have a lot of NUMA access for each one.  In CCNx 1.0, even a naive
>>>>>>>>implementation only requires at most 3 lookups (one by name, one
>>>>>>>>by name + keyid, one by name + content object hash), and one can
>>>>>>>>do other things to optimize lookup for an extra write.
>>>>>>>> 
>>>>>>>> d. In (c) above, if you have a threaded name tree or are just
>>>>>>>>walking parent pointers, I suspect you?ll need locking of the
>>>>>>>>ancestors in a multi-threaded system (?threaded" here meaning LWP)
>>>>>>>>and that will be expensive.  It would be interesting to see what a
>>>>>>>>cache consistent multi-threaded name tree looks like.
>>>>>>>> 
>>>>>>>> Marc
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 20, 2014, at 8:15 PM, Tai-Lin Chu <tailinchu at gmail.com>
>>>>>>>>>wrote:
>>>>>>>>> 
>>>>>>>>> I had thought about these questions, but I want to know your idea
>>>>>>>>> besides typed component:
>>>>>>>>> 1. LPM allows "data discovery". How will exact match do similar
>>>>>>>>>things?
>>>>>>>>> 2. will removing selectors improve performance? How do we use
>>>>>>>>>other
>>>>>>>>> faster technique to replace selector?
>>>>>>>>> 3. fixed byte length and type. I agree more that type can be
>>>>>>>>>fixed
>>>>>>>>> byte, but 2 bytes for length might not be enough for future.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Sat, Sep 20, 2014 at 5:36 AM, Dave Oran (oran)
>>>>>>>>>><oran at cisco.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> On Sep 18, 2014, at 9:09 PM, Tai-Lin Chu <tailinchu at gmail.com>
>>>>>>>>>>wrote:
>>>>>>>>>> 
>>>>>>>>>>>> I know how to make #2 flexible enough to do what things I can
>>>>>>>>>>>>envision we need to do, and with a few simple conventions on
>>>>>>>>>>>>how the registry of types is managed.
>>>>>>>>>>> 
>>>>>>>>>>> Could you share it with us?
>>>>>>>>>> Sure. Here?s a strawman.
>>>>>>>>>> 
>>>>>>>>>> The type space is 16 bits, so you have 65,565 types.
>>>>>>>>>> 
>>>>>>>>>> The type space is currently shared with the types used for the
>>>>>>>>>>entire protocol, that gives us two options:
>>>>>>>>>> (1) we reserve a range for name component types. Given the
>>>>>>>>>>likelihood there will be at least as much and probably more need
>>>>>>>>>>to component types than protocol extensions, we could reserve
>>>>>>>>>>1/2 of the type space, giving us 32K types for name components.
>>>>>>>>>> (2) since there is no parsing ambiguity between name components
>>>>>>>>>>and other fields of the protocol (sine they are sub-types of the
>>>>>>>>>>name type) we could reuse numbers and thereby have an entire 65K
>>>>>>>>>>name component types.
>>>>>>>>>> 
>>>>>>>>>> We divide the type space into regions, and manage it with a
>>>>>>>>>>registry. If we ever get to the point of creating an IETF
>>>>>>>>>>standard, IANA has 25 years of experience running registries and
>>>>>>>>>>there are well-understood rule sets for different kinds of
>>>>>>>>>>registries (open, requires a written spec, requires standards
>>>>>>>>>>approval).
>>>>>>>>>> 
>>>>>>>>>> - We allocate one ?default" name component type for ?generic
>>>>>>>>>>name?, which would be used on name prefixes and other common
>>>>>>>>>>cases where there are no special semantics on the name component.
>>>>>>>>>> - We allocate a range of name component types, say 1024, to
>>>>>>>>>>globally understood types that are part of the base or extension
>>>>>>>>>>NDN specifications (e.g. chunk#, version#, etc.
>>>>>>>>>> - We reserve some portion of the space for unanticipated uses
>>>>>>>>>>(say another 1024 types)
>>>>>>>>>> - We give the rest of the space to application assignment.
>>>>>>>>>> 
>>>>>>>>>> Make sense?
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>>> While I?m sympathetic to that view, there are three ways in
>>>>>>>>>>>>which Moore?s law or hardware tricks will not save us from
>>>>>>>>>>>>performance flaws in the design
>>>>>>>>>>> 
>>>>>>>>>>> we could design for performance,
>>>>>>>>>> That?s not what people are advocating. We are advocating that
>>>>>>>>>>we *not* design for known bad performance and hope serendipity
>>>>>>>>>>or Moore?s Law will come to the rescue.
>>>>>>>>>> 
>>>>>>>>>>> but I think there will be a turning
>>>>>>>>>>> point when the slower design starts to become "fast enough?.
>>>>>>>>>> Perhaps, perhaps not. Relative performance is what matters so
>>>>>>>>>>things that don?t get faster while others do tend to get dropped
>>>>>>>>>>or not used because they impose a performance penalty relative
>>>>>>>>>>to the things that go faster. There is also the ?low-end?
>>>>>>>>>>phenomenon where impovements in technology get applied to
>>>>>>>>>>lowering cost rather than improving performance. For those
>>>>>>>>>>environments bad performance just never get better.
>>>>>>>>>> 
>>>>>>>>>>> Do you
>>>>>>>>>>> think there will be some design of ndn that will *never* have
>>>>>>>>>>> performance improvement?
>>>>>>>>>> I suspect LPM on data will always be slow (relative to the
>>>>>>>>>>other functions).
>>>>>>>>>> i suspect exclusions will always be slow because they will
>>>>>>>>>>require extra memory references.
>>>>>>>>>> 
>>>>>>>>>> However I of course don?t claim to clairvoyance so this is just
>>>>>>>>>>speculation based on 35+ years of seeing performance improve by
>>>>>>>>>>4 orders of magnitude and still having to worry about counting
>>>>>>>>>>cycles and memory references?
>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, Sep 18, 2014 at 5:20 PM, Dave Oran (oran)
>>>>>>>>>>>>><oran at cisco.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sep 18, 2014, at 7:41 PM, Tai-Lin Chu
>>>>>>>>>>>>><tailinchu at gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We should not look at a certain chip nowadays and want ndn
>>>>>>>>>>>>>to perform
>>>>>>>>>>>>> well on it. It should be the other way around: once  ndn app
>>>>>>>>>>>>>becomes
>>>>>>>>>>>>> popular, a better chip will be designed for ndn.
>>>>>>>>>>>> While I?m sympathetic to that view, there are three ways in
>>>>>>>>>>>>which Moore?s law or hardware tricks will not save us from
>>>>>>>>>>>>performance flaws in the design:
>>>>>>>>>>>> a) clock rates are not getting (much) faster
>>>>>>>>>>>> b) memory accesses are getting (relatively) more expensive
>>>>>>>>>>>> c) data structures that require locks to manipulate
>>>>>>>>>>>>successfully will be relatively more expensive, even with
>>>>>>>>>>>>near-zero lock contention.
>>>>>>>>>>>> 
>>>>>>>>>>>> The fact is, IP *did* have some serious performance flaws in
>>>>>>>>>>>>its design. We just forgot those because the design elements
>>>>>>>>>>>>that depended on those mistakes have fallen into disuse. The
>>>>>>>>>>>>poster children for this are:
>>>>>>>>>>>> 1. IP options. Nobody can use them because they are too slow
>>>>>>>>>>>>on modern forwarding hardware, so they can?t be reliably used
>>>>>>>>>>>>anywhere
>>>>>>>>>>>> 2. the UDP checksum, which was a bad design when it was
>>>>>>>>>>>>specified and is now a giant PITA that still causes major pain
>>>>>>>>>>>>in working around.
>>>>>>>>>>>> 
>>>>>>>>>>>> I?m afraid students today are being taught the that designers
>>>>>>>>>>>>of IP were flawless, as opposed to very good scientists and
>>>>>>>>>>>>engineers that got most of it right.
>>>>>>>>>>>> 
>>>>>>>>>>>>> I feel the discussion today and yesterday has been
>>>>>>>>>>>>>off-topic. Now I
>>>>>>>>>>>>> see that there are 3 approaches:
>>>>>>>>>>>>> 1. we should not define a naming convention at all
>>>>>>>>>>>>> 2. typed component: use tlv type space and add a handful of
>>>>>>>>>>>>>types
>>>>>>>>>>>>> 3. marked component: introduce only one more type and add
>>>>>>>>>>>>>additional
>>>>>>>>>>>>> marker space
>>>>>>>>>>>> I know how to make #2 flexible enough to do what things I can
>>>>>>>>>>>>envision we need to do, and with a few simple conventions on
>>>>>>>>>>>>how the registry of types is managed.
>>>>>>>>>>>> 
>>>>>>>>>>>> It is just as powerful in practice as either throwing up our
>>>>>>>>>>>>hands and letting applications design their own mutually
>>>>>>>>>>>>incompatible schemes or trying to make naming conventions with
>>>>>>>>>>>>markers in a way that is fast to generate/parse and also
>>>>>>>>>>>>resilient against aliasing.
>>>>>>>>>>>> 
>>>>>>>>>>>>> Also everybody thinks that the current utf8 marker naming
>>>>>>>>>>>>>convention
>>>>>>>>>>>>> needs to be revised.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Sep 18, 2014 at 3:27 PM, Felix Rabe <felix at rabe.io>
>>>>>>>>>>>>>>wrote:
>>>>>>>>>>>>>> Would that chip be suitable, i.e. can we expect most names
>>>>>>>>>>>>>>to fit in (the
>>>>>>>>>>>>>> magnitude of) 96 bytes? What length are names usually in
>>>>>>>>>>>>>>current NDN
>>>>>>>>>>>>>> experiments?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I guess wide deployment could make for even longer names.
>>>>>>>>>>>>>>Related: Many URLs
>>>>>>>>>>>>>> I encounter nowadays easily don't fit within two 80-column
>>>>>>>>>>>>>>text lines, and
>>>>>>>>>>>>>> NDN will have to carry more information than URLs, as far
>>>>>>>>>>>>>>as I see.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 18/Sep/14 23:15, Marc.Mosko at parc.com wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In fact, the index in separate TLV will be slower on some
>>>>>>>>>>>>>>architectures,
>>>>>>>>>>>>>> like the ezChip NP4.  The NP4 can hold the fist 96 frame
>>>>>>>>>>>>>>bytes in memory,
>>>>>>>>>>>>>> then any subsequent memory is accessed only as two adjacent
>>>>>>>>>>>>>>32-byte blocks
>>>>>>>>>>>>>> (there can be at most 5 blocks available at any one time).
>>>>>>>>>>>>>>If you need to
>>>>>>>>>>>>>> switch between arrays, it would be very expensive.  If you
>>>>>>>>>>>>>>have to read past
>>>>>>>>>>>>>> the name to get to the 2nd array, then read it, then backup
>>>>>>>>>>>>>>to get to the
>>>>>>>>>>>>>> name, it will be pretty expensive too.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Marc
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sep 18, 2014, at 2:02 PM, <Ignacio.Solis at parc.com>
>>>>>>>>>>>>>> <Ignacio.Solis at parc.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Does this make that much difference?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If you want to parse the first 5 components. One way to do
>>>>>>>>>>>>>>it is:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Read the index, find entry 5, then read in that many bytes
>>>>>>>>>>>>>>from the start
>>>>>>>>>>>>>> offset of the beginning of the name.
>>>>>>>>>>>>>> OR
>>>>>>>>>>>>>> Start reading name, (find size + move ) 5 times.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> How much speed are you getting from one to the other?  You
>>>>>>>>>>>>>>seem to imply
>>>>>>>>>>>>>> that the first one is faster.  I don?t think this is the
>>>>>>>>>>>>>>case.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In the first one you?ll probably have to get the cache line
>>>>>>>>>>>>>>for the index,
>>>>>>>>>>>>>> then all the required cache lines for the first 5
>>>>>>>>>>>>>>components. For the
>>>>>>>>>>>>>> second, you?ll have to get all the cache lines for the
>>>>>>>>>>>>>>first 5 components.
>>>>>>>>>>>>>> Given an assumption that a cache miss is way more expensive
>>>>>>>>>>>>>>than
>>>>>>>>>>>>>> evaluating a number and computing an addition, you might
>>>>>>>>>>>>>>find that the
>>>>>>>>>>>>>> performance of the index is actually slower than the
>>>>>>>>>>>>>>performance of the
>>>>>>>>>>>>>> direct access.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Granted, there is a case where you don?t access the name at
>>>>>>>>>>>>>>all, for
>>>>>>>>>>>>>> example, if you just get the offsets and then send the
>>>>>>>>>>>>>>offsets as
>>>>>>>>>>>>>> parameters to another processor/GPU/NPU/etc. In this case
>>>>>>>>>>>>>>you may see a
>>>>>>>>>>>>>> gain IF there are more cache line misses in reading the
>>>>>>>>>>>>>>name than in
>>>>>>>>>>>>>> reading the index.   So, if the regular part of the name
>>>>>>>>>>>>>>that you?re
>>>>>>>>>>>>>> parsing is bigger than the cache line (64 bytes?) and the
>>>>>>>>>>>>>>name is to be
>>>>>>>>>>>>>> processed by a different processor, then your might see
>>>>>>>>>>>>>>some performance
>>>>>>>>>>>>>> gain in using the index, but in all other circumstances I
>>>>>>>>>>>>>>bet this is not
>>>>>>>>>>>>>> the case.   I may be wrong, haven?t actually tested it.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This is all to say, I don?t think we should be designing
>>>>>>>>>>>>>>the protocol with
>>>>>>>>>>>>>> only one architecture in mind. (The architecture of sending
>>>>>>>>>>>>>>the name to a
>>>>>>>>>>>>>> different processor than the index).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If you have numbers that show that the index is faster I
>>>>>>>>>>>>>>would like to see
>>>>>>>>>>>>>> under what conditions and architectural assumptions.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Nacho
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> (I may have misinterpreted your description so feel free to
>>>>>>>>>>>>>>correct me if
>>>>>>>>>>>>>> I?m wrong.)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Nacho (Ignacio) Solis
>>>>>>>>>>>>>> Protocol Architect
>>>>>>>>>>>>>> Principal Scientist
>>>>>>>>>>>>>> Palo Alto Research Center (PARC)
>>>>>>>>>>>>>> +1(650)812-4458
>>>>>>>>>>>>>> Ignacio.Solis at parc.com
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 9/18/14, 12:54 AM, "Massimo Gallo"
>>>>>>>>>>>>>><massimo.gallo at alcatel-lucent.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Indeed each components' offset must be encoded using a
>>>>>>>>>>>>>>fixed amount of
>>>>>>>>>>>>>> bytes:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> i.e.,
>>>>>>>>>>>>>> Type = Offsets
>>>>>>>>>>>>>> Length = 10 Bytes
>>>>>>>>>>>>>> Value = Offset1(1byte), Offset2(1byte), ...
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> You may also imagine to have a "Offset_2byte" type if your
>>>>>>>>>>>>>>name is too
>>>>>>>>>>>>>> long.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Max
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 18/09/2014 09:27, Tai-Lin Chu wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> if you do not need the entire hierarchal structure (suppose
>>>>>>>>>>>>>>you only
>>>>>>>>>>>>>> want the first x components) you can directly have it using
>>>>>>>>>>>>>>the
>>>>>>>>>>>>>> offsets. With the Nested TLV structure you have to
>>>>>>>>>>>>>>iteratively parse
>>>>>>>>>>>>>> the first x-1 components. With the offset structure you
>>>>>>>>>>>>>>cane directly
>>>>>>>>>>>>>> access to the firs x components.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I don't get it. What you described only works if the
>>>>>>>>>>>>>>"offset" is
>>>>>>>>>>>>>> encoded in fixed bytes. With varNum, you will still need to
>>>>>>>>>>>>>>parse x-1
>>>>>>>>>>>>>> offsets to get to the x offset.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Wed, Sep 17, 2014 at 11:57 PM, Massimo Gallo
>>>>>>>>>>>>>> <massimo.gallo at alcatel-lucent.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 17/09/2014 14:56, Mark Stapp wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ah, thanks - that's helpful. I thought you were saying "I
>>>>>>>>>>>>>>like the
>>>>>>>>>>>>>> existing NDN UTF8 'convention'." I'm still not sure I
>>>>>>>>>>>>>>understand what
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> _do_ prefer, though. it sounds like you're describing an
>>>>>>>>>>>>>>entirely
>>>>>>>>>>>>>> different
>>>>>>>>>>>>>> scheme where the info that describes the name-components is
>>>>>>>>>>>>>>...
>>>>>>>>>>>>>> someplace
>>>>>>>>>>>>>> other than _in_ the name-components. is that correct? when
>>>>>>>>>>>>>>you say
>>>>>>>>>>>>>> "field
>>>>>>>>>>>>>> separator", what do you mean (since that's not a "TL" from
>>>>>>>>>>>>>>a TLV)?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Correct.
>>>>>>>>>>>>>> In particular, with our name encoding, a TLV indicates the
>>>>>>>>>>>>>>name
>>>>>>>>>>>>>> hierarchy
>>>>>>>>>>>>>> with offsets in the name and other TLV(s) indicates the
>>>>>>>>>>>>>>offset to use
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> order to retrieve special components.
>>>>>>>>>>>>>> As for the field separator, it is something like "/".
>>>>>>>>>>>>>>Aliasing is
>>>>>>>>>>>>>> avoided as
>>>>>>>>>>>>>> you do not rely on field separators to parse the name; you
>>>>>>>>>>>>>>use the
>>>>>>>>>>>>>> "offset
>>>>>>>>>>>>>> TLV " to do that.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So now, it may be an aesthetic question but:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> if you do not need the entire hierarchal structure (suppose
>>>>>>>>>>>>>>you only
>>>>>>>>>>>>>> want
>>>>>>>>>>>>>> the first x components) you can directly have it using the
>>>>>>>>>>>>>>offsets.
>>>>>>>>>>>>>> With the
>>>>>>>>>>>>>> Nested TLV structure you have to iteratively parse the
>>>>>>>>>>>>>>first x-1
>>>>>>>>>>>>>> components.
>>>>>>>>>>>>>> With the offset structure you cane directly access to the
>>>>>>>>>>>>>>firs x
>>>>>>>>>>>>>> components.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Max
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -- Mark
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 9/17/14 6:02 AM, Massimo Gallo wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The why is simple:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> You use a lot of "generic component type" and very few
>>>>>>>>>>>>>>"specific
>>>>>>>>>>>>>> component type". You are imposing types for every component
>>>>>>>>>>>>>>in order
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> handle few exceptions (segmentation, etc..). You create a
>>>>>>>>>>>>>>rule
>>>>>>>>>>>>>> (specify
>>>>>>>>>>>>>> the component's type ) to handle exceptions!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I would prefer not to have typed components. Instead I
>>>>>>>>>>>>>>would prefer
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> have the name as simple sequence bytes with a field
>>>>>>>>>>>>>>separator. Then,
>>>>>>>>>>>>>> outside the name, if you have some components that could be
>>>>>>>>>>>>>>used at
>>>>>>>>>>>>>> network layer (e.g. a TLV field), you simply need something
>>>>>>>>>>>>>>that
>>>>>>>>>>>>>> indicates which is the offset allowing you to retrieve the
>>>>>>>>>>>>>>version,
>>>>>>>>>>>>>> segment, etc in the name...
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Max
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 16/09/2014 20:33, Mark Stapp wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 9/16/14 10:29 AM, Massimo Gallo wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I think we agree on the small number of "component types".
>>>>>>>>>>>>>> However, if you have a small number of types, you will end
>>>>>>>>>>>>>>up with
>>>>>>>>>>>>>> names
>>>>>>>>>>>>>> containing many generic components types and few specific
>>>>>>>>>>>>>> components
>>>>>>>>>>>>>> types. Due to the fact that the component type
>>>>>>>>>>>>>>specification is an
>>>>>>>>>>>>>> exception in the name, I would prefer something that specify
>>>>>>>>>>>>>> component's
>>>>>>>>>>>>>> type only when needed (something like UTF8 conventions but
>>>>>>>>>>>>>>that
>>>>>>>>>>>>>> applications MUST use).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> so ... I can't quite follow that. the thread has had some
>>>>>>>>>>>>>> explanation
>>>>>>>>>>>>>> about why the UTF8 requirement has problems (with aliasing,
>>>>>>>>>>>>>>e.g.)
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> there's been email trying to explain that applications
>>>>>>>>>>>>>>don't have to
>>>>>>>>>>>>>> use types if they don't need to. your email sounds like "I
>>>>>>>>>>>>>>prefer
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> UTF8 convention", but it doesn't say why you have that
>>>>>>>>>>>>>>preference in
>>>>>>>>>>>>>> the face of the points about the problems. can you say why
>>>>>>>>>>>>>>it is
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> you express a preference for the "convention" with problems
>>>>>>>>>>>>>>?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Mark
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> .
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Ndn-interest mailing list
>>>>>>>>>>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>>>>>>>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Ndn-interest mailing list
>>>>>>>>>>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>>>>>>>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Ndn-interest mailing list
>>>>>>>>>>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>>>>>>>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Ndn-interest mailing list
>>>>>>>>>>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>>>>>>>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Ndn-interest mailing list
>>>>>>>>>>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>>>>>>>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>>>>>>>>>> 
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Ndn-interest mailing list
>>>>>>>>>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>>>>>>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> Ndn-interest mailing list
>>>>>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>>> 
>
>_______________________________________________
>Ndn-interest mailing list
>Ndn-interest at lists.cs.ucla.edu
>http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest




From jburke at remap.ucla.edu  Fri Oct  3 19:29:43 2014
From: jburke at remap.ucla.edu (Burke, Jeff)
Date: Sat, 4 Oct 2014 02:29:43 +0000
Subject: [Ndn-interest] any comments on naming convention? [version
 marker comment]
In-Reply-To: <541DB596.5010400@illinois.edu>
Message-ID: <D054A833.175BE3%jburke@remap.ucla.edu>

Tarek,

Sorry for the delayed reply.  I am curious whether the network would need
to know the priority in the applications that you're thinking of, or
whether it would be sufficient for the application (or some part of the
stack).  If I remember the information maximization proposal correctly, it
would use data objects named (in either a hierarchical or flat structure)
in a way that reflects what they refer to... for example, their geography
coverage or a feature vector of some sort. 	Wouldn't it be sufficient for
the requesting application to issue interests according to an information
maximizing approach rather than relying on anything within the network to
handle this?  

Jeff

On 9/20/14, 9:12 PM, "Abdelzaher, Tarek" <zaher at illinois.edu> wrote:

>Jeff,
>
>> in the document I sent, there are seven specific, though not
>> equally well considered, reasons to use marker components that have
>> nothing to with the so-called type explosion.  As far as I can tell, no
>> one has addressed these from the application developer's perspective.
>> Could someone?
>
>I have a small comment on the original name conventions document sent by
>Tai-Lin. I'd like to suggest a slight semantic generalization of the
>meaning of one of the proposed markers; namely, the versioning marker
>0xFD (which identifies the version of the component).
>
>Do you see harm in overloading the semantics of the field identified by
>0xFD to refer to a general "priority" hint? As the document suggests, by
>convention, priority could be given to, say, lexicographically larger
>values in that field. The document says "[versioning] can be used by
>third-parties, e.g., to prioritize caching of the latest versions of
>data". I can see other reasons that applications or transport protocols
>might want to instruct caches to prioritize objects that have the same
>prefix. It would be convenient to be able to insert a value in the 0xFD
>field such that larger lexicographic values are prioritized (in caches,
>etc) over smaller ones. One such scenario is described in the
>"Information Funnel" paper, where the lexicographical priority "value"
>would be an entire name postfix (referring a subtree in which some
>lexicographically ordered branches are more important than others).
>Application software and/or transport layers could then properly name
>producer objects slated for sharing with subscribers such that the
>network is aware of their relative importance. Think, for example, of
>layered video encoding, where base-layer objects should be prioritized
>over enhancement-layer objects. Any thoughts/comments?
>
>Tarek
>_______________________________________________
>Ndn-interest mailing list
>Ndn-interest at lists.cs.ucla.edu
>http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest




From jburke at remap.ucla.edu  Fri Oct  3 20:02:21 2014
From: jburke at remap.ucla.edu (Burke, Jeff)
Date: Sat, 4 Oct 2014 03:02:21 +0000
Subject: [Ndn-interest] any comments on naming convention?
In-Reply-To: <34691EA2-C408-4DBB-962D-621D7C6B40C8@parc.com>
Message-ID: <D054AF1A.175C98%jburke@remap.ucla.edu>



On 9/25/14, 12:17 AM, "Marc.Mosko at parc.com" <Marc.Mosko at parc.com> wrote:

>My beating on ?discover all? is exactly because of this.  Let?s define
>discovery service.  If the service is just ?discover latest?
>(left/right), can we not simplify the current approach?  If the service
>includes more than ?latest?, then is the current approach the right
>approach?
>
>Sync has its place and is the right solution for somethings.  However, it
>should not be a a bandage over discovery.  Discovery should be its own
>valid and useful service.

I think this makes sense but want to make sure I am following the
definitions - can you then define discovery more specifically?

>
>I agree that the exclusion approach can work, and work relatively well,
>for finding the rightmost/leftmost child.  I believe this is because that
>operation is transitive through caches.  So, within whatever timeout an
>application is willing to wait to find the ?latest?, it can keep asking
>and asking.  
>
>I do think it would be best to actually try to ask an authoritative
>source first (i.e. a non-cached value), and if that fails then probe
>caches, but experimentation may show what works well.  This is based on
>my belief that in the real world in broad use, the namespace will become
>pretty polluted and probing will result in a  lot of junk, but that?s
>future prognosticating.

I'm not sure about the emphasis on an authoritative sources... What makes
something authoritative? That it assembled and signed the packet? For a
number of scenarios, from data muling in vehicular networking to video
publishing or even retrieving the "sufficiently-current" homepage of the
New York Times, getting to that source is either impossible (some V2V
scenario) or unnecessary (NYTimes) if some cache or repo in the network
can answer the interest.  Using selectors doesn't necessarily mean probing
a huge, messy namespace--I think there is agreement that is not
efficient--it can be trying to target an Interest in a well-understood
namespace without having to know the exact child name.

>
>Also, in the exact match vs. continuation match of content object to
>interest, it is pretty easy to encode that ?selector? request in a name
>component (i.e. ?exclude_before=(t=version, l=2, v=279) & sort=right?)
>and any participating cache can respond with a link (or encapsulate) a
>response in an exact match system.
>
>In the CCNx 1.0 spec, one could also encode this a different way.  One
>could use a name like ?/mail/inbox/selector_matching/<hash of payload>?
>and in the payload include "exclude_before=(t=version, l=2, v=279) &
>sort=right?.  This means that any cache that could process the ?
>selector_matching? function could look at the interest payload and
>evaluate the predicate there.  The predicate could become large and not
>pollute the PIT with all the computation state.  Including ?<hash of
>payload>? in the name means that one could get a cached response if
>someone else had asked the same exact question (subject to the content
>object?s cache lifetime) and it also servers to multiplex different
>payloads for the same function (selector_matching).
>
>Marc
>
>
>On Sep 25, 2014, at 8:18 AM, Burke, Jeff <jburke at remap.ucla.edu> wrote:
>
>> 
>> http://irl.cs.ucla.edu/~zhenkai/papers/chronosync.pdf
>> 
>> 
>>https://www.ccnx.org/releases/ccnx-0.7.0rc1/doc/technical/Synchronization
>>Pr
>> otocol.html
>> 
>> J. 
>> 
>> 
>> On 9/24/14, 11:16 PM, "Tai-Lin Chu" <tailinchu at gmail.com> wrote:
>> 
>>> However, I cannot see whether we can achieve "best-effort *all*-value"
>>> efficiently.
>>> There are still interesting topics on
>>> 1. how do we express the discovery query?
>>> 2. is selector "discovery-complete"? i. e. can we express any
>>> discovery query with current selector?
>>> 3. if so, can we re-express current selector in a more efficient way?
>>> 
>>> I personally see a named data as a set, which can then be categorized
>>> into "ordered set", and "unordered set".
>>> some questions that any discovery expression must solve:
>>> 1. is this a nil set or not? nil set means that this name is the leaf
>>> 2. set contains member X?
>>> 3. is set ordered or not
>>> 4. (ordered) first, prev, next, last
>>> 5. if we enforce component ordering, answer question 4.
>>> 6. recursively answer all questions above on any set member
>>> 
>>> 
>>> 
>>> On Wed, Sep 24, 2014 at 10:45 PM, Burke, Jeff <jburke at remap.ucla.edu>
>>> wrote:
>>>> 
>>>> 
>>>> From: <Marc.Mosko at parc.com>
>>>> Date: Wed, 24 Sep 2014 16:25:53 +0000
>>>> To: Jeff Burke <jburke at remap.ucla.edu>
>>>> Cc: <Ignacio.Solis at parc.com>, <tailinchu at gmail.com>,
>>>> <ndn-interest at lists.cs.ucla.edu>
>>>> Subject: Re: [Ndn-interest] any comments on naming convention?
>>>> 
>>>> I think Tai-Lin?s example was just fine to talk about discovery.
>>>> /blah/blah/value, how do you discover all the ?value?s?  Discovery
>>>> shouldn?t
>>>> care if its email messages or temperature readings or world cup
>>>>photos.
>>>> 
>>>> 
>>>> This is true if discovery means "finding everything" - in which case,
>>>> as you
>>>> point out, sync-style approaches may be best.  But I am not sure that
>>>> this
>>>> definition is complete.  The most pressing example that I can think of
>>>> is
>>>> best-effort latest-value, in which the consumer's goal is to get the
>>>> latest
>>>> copy the network can deliver at the moment, and may not care about
>>>> previous
>>>> values or (if freshness is used well) potential later versions.
>>>> 
>>>> Another case that seems to work well is video seeking.  Let's say I
>>>> want to
>>>> enable random access to a video by timecode. The publisher can
>>>>provide a
>>>> time-code based discovery namespace that's queried using an Interest
>>>> that
>>>> essentially says "give me the closest keyframe to 00:37:03:12", which
>>>> returns an interest that, via the name, provides the exact timecode of
>>>> the
>>>> keyframe in question and a link to a segment-based namespace for
>>>> efficient
>>>> exact match playout.  In two roundtrips and in a very lightweight way,
>>>> the
>>>> consumer has random access capability.   If the NDN is the moral
>>>> equivalent
>>>> of IP, then I am not sure we should be afraid of roundtrips that
>>>>provide
>>>> this kind of functionality, just as they are used in TCP.
>>>> 
>>>> 
>>>> I described one set of problems using the exclusion approach, and that
>>>> an
>>>> NDN paper on device discovery described a similar problem, though they
>>>> did
>>>> not go into the details of splitting interests, etc.  That all was
>>>> simple
>>>> enough to see from the example.
>>>> 
>>>> Another question is how does one do the discovery with exact match
>>>> names,
>>>> which is also conflating things.  You could do a different discovery
>>>> with
>>>> continuation names too, just not the exclude method.
>>>> 
>>>> As I alluded to, one needs a way to talk with a specific cache about
>>>>its
>>>> ?table of contents? for a prefix so one can get a consistent set of
>>>> results
>>>> without all the round-trips of exclusions.  Actually downloading the
>>>> ?headers? of the messages would be the same bytes, more or less.  In a
>>>> way,
>>>> this is a little like name enumeration from a ccnx 0.x repo, but that
>>>> protocol has its own set of problems and I?m not suggesting to use
>>>>that
>>>> directly.
>>>> 
>>>> One approach is to encode a request in a name component and a
>>>> participating
>>>> cache can reply.  It replies in such a way that one could continue
>>>> talking
>>>> with that cache to get its TOC.  One would then issue another interest
>>>> with
>>>> a request for not-that-cache.
>>>> 
>>>> 
>>>> I'm curious how the TOC approach works in a multi-publisher scenario?
>>>> 
>>>> 
>>>> Another approach is to try to ask the authoritative source for the
>>>> ?current?
>>>> manifest name, i.e. /mail/inbox/current/<nonce>, which could return
>>>>the
>>>> manifest or a link to the manifest.  Then fetching the actual manifest
>>>> from
>>>> the link could come from caches because you how have a consistent set
>>>>of
>>>> names to ask for.  If you cannot talk with an authoritative source,
>>>>you
>>>> could try again without the nonce and see if there?s a cached copy of
>>>>a
>>>> recent version around.
>>>> 
>>>> Marc
>>>> 
>>>> 
>>>> On Sep 24, 2014, at 5:46 PM, Burke, Jeff <jburke at remap.ucla.edu>
>>>>wrote:
>>>> 
>>>> 
>>>> 
>>>> On 9/24/14, 8:20 AM, "Ignacio.Solis at parc.com" <Ignacio.Solis at parc.com>
>>>> wrote:
>>>> 
>>>> On 9/24/14, 4:27 AM, "Tai-Lin Chu" <tailinchu at gmail.com> wrote:
>>>> 
>>>> For example, I see a pattern /mail/inbox/148. I, a human being, see a
>>>> pattern with static (/mail/inbox) and variable (148) components; with
>>>> proper naming convention, computers can also detect this pattern
>>>> easily. Now I want to look for all mails in my inbox. I can generate a
>>>> list of /mail/inbox/<number>. These are my guesses, and with selectors
>>>> I can further refine my guesses.
>>>> 
>>>> 
>>>> I think this is a very bad example  (or at least a very bad
>>>>application
>>>> design).  You have an app (a mail server / inbox) and you want it to
>>>> list
>>>> your emails?  An email list is an application data structure.  I don?t
>>>> think you should use the network structure to reflect this.
>>>> 
>>>> 
>>>> I think Tai-Lin is trying to sketch a small example, not propose a
>>>> full-scale approach to email. (Maybe I am misunderstanding.)
>>>> 
>>>> 
>>>> Another way to look at it is that if the network architecture is
>>>> providing
>>>> the equivalent of distributed storage to the application, perhaps the
>>>> application data structure could be adapted to match the affordances
>>>>of
>>>> the network.  Then it would not be so bad that the two structures were
>>>> aligned.
>>>> 
>>>> 
>>>> I?ll give you an example, how do you delete emails from your inbox? If
>>>> an
>>>> email was cached in the network it can never be deleted from your
>>>>inbox?
>>>> 
>>>> 
>>>> This is conflating two issues - what you are pointing out is that the
>>>> data
>>>> structure of a linear list doesn't handle common email management
>>>> operations well.  Again, I'm not sure if that's what he was getting at
>>>> here.  But deletion is not the issue - the availability of a data
>>>>object
>>>> on the network does not necessarily mean it's valid from the
>>>>perspective
>>>> of the application.
>>>> 
>>>> Or moved to another mailbox? Do you rely on the emails expiring?
>>>> 
>>>> This problem is true for most (any?) situations where you use network
>>>> name
>>>> structure to directly reflect the application data structure.
>>>> 
>>>> 
>>>> Not sure I understand how you make the leap from the example to the
>>>> general statement.
>>>> 
>>>> Jeff
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Nacho
>>>> 
>>>> 
>>>> 
>>>> On Tue, Sep 23, 2014 at 2:34 AM,  <Marc.Mosko at parc.com> wrote:
>>>> 
>>>> Ok, yes I think those would all be good things.
>>>> 
>>>> One thing to keep in mind, especially with things like time series
>>>> sensor
>>>> data, is that people see a pattern and infer a way of doing it.
>>>>That?s
>>>> easy
>>>> for a human :)  But in Discovery, one should assume that one does not
>>>> know
>>>> of patterns in the data beyond what the protocols used to publish the
>>>> data
>>>> explicitly require.  That said, I think some of the things you listed
>>>> are
>>>> good places to start: sensor data, web content, climate data or genome
>>>> data.
>>>> 
>>>> We also need to state what the forwarding strategies are and what the
>>>> cache
>>>> behavior is.
>>>> 
>>>> I outlined some of the points that I think are important in that other
>>>> posting.  While ?discover latest? is useful, ?discover all? is also
>>>> important, and that one gets complicated fast.  So points like
>>>> separating
>>>> discovery from retrieval and working with large data sets have been
>>>> important in shaping our thinking.  That all said, I?d be happy
>>>> starting
>>>> from 0 and working through the Discovery service definition from
>>>> scratch
>>>> along with data set use cases.
>>>> 
>>>> Marc
>>>> 
>>>> On Sep 23, 2014, at 12:36 AM, Burke, Jeff <jburke at remap.ucla.edu>
>>>> wrote:
>>>> 
>>>> Hi Marc,
>>>> 
>>>> Thanks ? yes, I saw that as well. I was just trying to get one step
>>>> more
>>>> specific, which was to see if we could identify a few specific use
>>>> cases
>>>> around which to have the conversation.  (e.g., time series sensor data
>>>> and
>>>> web content retrieval for "get latest"; climate data for huge data
>>>> sets;
>>>> local data in a vehicular network; etc.)  What have you been looking
>>>>at
>>>> that's driving considerations of discovery?
>>>> 
>>>> Thanks,
>>>> Jeff
>>>> 
>>>> From: <Marc.Mosko at parc.com>
>>>> Date: Mon, 22 Sep 2014 22:29:43 +0000
>>>> To: Jeff Burke <jburke at remap.ucla.edu>
>>>> Cc: <tailinchu at gmail.com>, <ndn-interest at lists.cs.ucla.edu>
>>>> Subject: Re: [Ndn-interest] any comments on naming convention?
>>>> 
>>>> Jeff,
>>>> 
>>>> Take a look at my posting (that Felix fixed) in a new thread on
>>>> Discovery.
>>>> 
>>>> 
>>>> 
>>>>http://www.lists.cs.ucla.edu/pipermail/ndn-interest/2014-September/0002
>>>>0
>>>> 0
>>>> .html
>>>> 
>>>> I think it would be very productive to talk about what Discovery
>>>>should
>>>> do,
>>>> and not focus on the how.  It is sometimes easy to get caught up in
>>>>the
>>>> how,
>>>> which I think is a less important topic than the what at this stage.
>>>> 
>>>> Marc
>>>> 
>>>> On Sep 22, 2014, at 11:04 PM, Burke, Jeff <jburke at remap.ucla.edu>
>>>> wrote:
>>>> 
>>>> Marc,
>>>> 
>>>> If you can't talk about your protocols, perhaps we can discuss this
>>>> based
>>>> on use cases.   What are the use cases you are using to evaluate
>>>> discovery?
>>>> 
>>>> Jeff
>>>> 
>>>> 
>>>> 
>>>> On 9/21/14, 11:23 AM, "Marc.Mosko at parc.com" <Marc.Mosko at parc.com>
>>>> wrote:
>>>> 
>>>> No matter what the expressiveness of the predicates if the forwarder
>>>> can
>>>> send interests different ways you don't have a consistent underlying
>>>> set
>>>> to talk about so you would always need non-range exclusions to
>>>>discover
>>>> every version.
>>>> 
>>>> Range exclusions only work I believe if you get an authoritative
>>>> answer.
>>>> If different content pieces are scattered between different caches I
>>>> don't see how range exclusions would work to discover every version.
>>>> 
>>>> I'm sorry to be pointing out problems without offering solutions but
>>>> we're not ready to publish our discovery protocols.
>>>> 
>>>> Sent from my telephone
>>>> 
>>>> On Sep 21, 2014, at 8:50, "Tai-Lin Chu" <tailinchu at gmail.com> wrote:
>>>> 
>>>> I see. Can you briefly describe how ccnx discovery protocol solves the
>>>> all problems that you mentioned (not just exclude)? a doc will be
>>>> better.
>>>> 
>>>> My unserious conjecture( :) ) : exclude is equal to [not]. I will soon
>>>> expect [and] and [or], so boolean algebra is fully supported.  Regular
>>>> language or context free language might become part of selector too.
>>>> 
>>>> On Sat, Sep 20, 2014 at 11:25 PM,  <Marc.Mosko at parc.com> wrote:
>>>> That will get you one reading then you need to exclude it and ask
>>>> again.
>>>> 
>>>> Sent from my telephone
>>>> 
>>>> On Sep 21, 2014, at 8:22, "Tai-Lin Chu" <tailinchu at gmail.com> wrote:
>>>> 
>>>> Yes, my point was that if you cannot talk about a consistent set
>>>> with a particular cache, then you need to always use individual
>>>> excludes not range excludes if you want to discover all the versions
>>>> of an object.
>>>> 
>>>> 
>>>> I am very confused. For your example, if I want to get all today's
>>>> sensor data, I just do (Any..Last second of last day)(First second of
>>>> tomorrow..Any). That's 18 bytes.
>>>> 
>>>> 
>>>> [1]http://named-data.net/doc/ndn-tlv/interest.html#exclude
>>>> 
>>>> On Sat, Sep 20, 2014 at 10:55 PM,  <Marc.Mosko at parc.com> wrote:
>>>> 
>>>> On Sep 21, 2014, at 1:47 AM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>>>> 
>>>> If you talk sometimes to A and sometimes to B, you very easily
>>>> could miss content objects you want to discovery unless you avoid
>>>> all range exclusions and only exclude explicit versions.
>>>> 
>>>> 
>>>> Could you explain why missing content object situation happens? also
>>>> range exclusion is just a shorter notation for many explicit
>>>> exclude;
>>>> converting from explicit excludes to ranged exclude is always
>>>> possible.
>>>> 
>>>> 
>>>> Yes, my point was that if you cannot talk about a consistent set
>>>> with a particular cache, then you need to always use individual
>>>> excludes not range excludes if you want to discover all the versions
>>>> of an object.  For something like a sensor reading that is updated,
>>>> say, once per second you will have 86,400 of them per day.  If each
>>>> exclusion is a timestamp (say 8 bytes), that?s 691,200 bytes of
>>>> exclusions (plus encoding overhead) per day.
>>>> 
>>>> yes, maybe using a more deterministic version number than a
>>>> timestamp makes sense here, but its just an example of needing a lot
>>>> of exclusions.
>>>> 
>>>> 
>>>> You exclude through 100 then issue a new interest.  This goes to
>>>> cache B
>>>> 
>>>> 
>>>> I feel this case is invalid because cache A will also get the
>>>> interest, and cache A will return v101 if it exists. Like you said,
>>>> if
>>>> this goes to cache B only, it means that cache A dies. How do you
>>>> know
>>>> that v101 even exist?
>>>> 
>>>> 
>>>> I guess this depends on what the forwarding strategy is.  If the
>>>> forwarder will always send each interest to all replicas, then yes,
>>>> modulo packet loss, you would discover v101 on cache A.  If the
>>>> forwarder is just doing ?best path? and can round-robin between cache
>>>> A and cache B, then your application could miss v101.
>>>> 
>>>> 
>>>> 
>>>> c,d In general I agree that LPM performance is related to the number
>>>> of components. In my own thread-safe LMP implementation, I used only
>>>> one RWMutex for the whole tree. I don't know whether adding lock for
>>>> every node will be faster or not because of lock overhead.
>>>> 
>>>> However, we should compare (exact match + discovery protocol) vs
>>>> (ndn
>>>> lpm). Comparing performance of exact match to lpm is unfair.
>>>> 
>>>> 
>>>> Yes, we should compare them.  And we need to publish the ccnx 1.0
>>>> specs for doing the exact match discovery.  So, as I said, I?m not
>>>> ready to claim its better yet because we have not done that.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Sat, Sep 20, 2014 at 2:38 PM,  <Marc.Mosko at parc.com> wrote:
>>>> I would point out that using LPM on content object to Interest
>>>> matching to do discovery has its own set of problems.  Discovery
>>>> involves more than just ?latest version? discovery too.
>>>> 
>>>> This is probably getting off-topic from the original post about
>>>> naming conventions.
>>>> 
>>>> a.  If Interests can be forwarded multiple directions and two
>>>> different caches are responding, the exclusion set you build up
>>>> talking with cache A will be invalid for cache B.  If you talk
>>>> sometimes to A and sometimes to B, you very easily could miss
>>>> content objects you want to discovery unless you avoid all range
>>>> exclusions and only exclude explicit versions.  That will lead to
>>>> very large interest packets.  In ccnx 1.0, we believe that an
>>>> explicit discovery protocol that allows conversations about
>>>> consistent sets is better.
>>>> 
>>>> b. Yes, if you just want the ?latest version? discovery that
>>>> should be transitive between caches, but imagine this.  You send
>>>> Interest #1 to cache A which returns version 100.  You exclude
>>>> through 100 then issue a new interest.  This goes to cache B who
>>>> only has version 99, so the interest times out or is NACK?d.  So
>>>> you think you have it!  But, cache A already has version 101, you
>>>> just don?t know.   If you cannot have a conversation around
>>>> consistent sets, it seems like even doing latest version discovery
>>>> is difficult with selector based discovery.  From what I saw in
>>>> ccnx 0.x, one ended up getting an Interest all the way to the
>>>> authoritative source because you can never believe an intermediate
>>>> cache that there?s not something more recent.
>>>> 
>>>> I?m sure you?ve walked through cases (a) and (b) in ndn, I?d be
>>>> interest in seeing your analysis.  Case (a) is that a node can
>>>> correctly discover every version of a name prefix, and (b) is that
>>>> a node can correctly discover the latest version.  We have not
>>>> formally compared (or yet published) our discovery protocols (we
>>>> have three, 2 for content, 1 for device) compared to selector based
>>>> discovery, so I cannot yet claim they are better, but they do not
>>>> have the non-determinism sketched above.
>>>> 
>>>> c. Using LPM, there is a non-deterministic number of lookups you
>>>> must do in the PIT to match a content object.  If you have a name
>>>> tree or a threaded hash table, those don?t all need to be hash
>>>> lookups, but you need to walk up the name tree for every prefix of
>>>> the content object name and evaluate the selector predicate.
>>>> Content Based Networking (CBN) had some some methods to create data
>>>> structures based on predicates, maybe those would be better.  But
>>>> in any case, you will potentially need to retrieve many PIT entries
>>>> if there is Interest traffic for many prefixes of a root.  Even on
>>>> an Intel system, you?ll likely miss cache lines, so you?ll have a
>>>> lot of NUMA access for each one.  In CCNx 1.0, even a naive
>>>> implementation only requires at most 3 lookups (one by name, one by
>>>> name + keyid, one by name + content object hash), and one can do
>>>> other things to optimize lookup for an extra write.
>>>> 
>>>> d. In (c) above, if you have a threaded name tree or are just
>>>> walking parent pointers, I suspect you?ll need locking of the
>>>> ancestors in a multi-threaded system (?threaded" here meaning LWP)
>>>> and that will be expensive.  It would be interesting to see what a
>>>> cache consistent multi-threaded name tree looks like.
>>>> 
>>>> Marc
>>>> 
>>>> 
>>>> On Sep 20, 2014, at 8:15 PM, Tai-Lin Chu <tailinchu at gmail.com>
>>>> wrote:
>>>> 
>>>> I had thought about these questions, but I want to know your idea
>>>> besides typed component:
>>>> 1. LPM allows "data discovery". How will exact match do similar
>>>> things?
>>>> 2. will removing selectors improve performance? How do we use
>>>> other
>>>> faster technique to replace selector?
>>>> 3. fixed byte length and type. I agree more that type can be fixed
>>>> byte, but 2 bytes for length might not be enough for future.
>>>> 
>>>> 
>>>> On Sat, Sep 20, 2014 at 5:36 AM, Dave Oran (oran)
>>>> <oran at cisco.com> wrote:
>>>> 
>>>> On Sep 18, 2014, at 9:09 PM, Tai-Lin Chu <tailinchu at gmail.com>
>>>> wrote:
>>>> 
>>>> I know how to make #2 flexible enough to do what things I can
>>>> envision we need to do, and with a few simple conventions on
>>>> how the registry of types is managed.
>>>> 
>>>> 
>>>> Could you share it with us?
>>>> 
>>>> Sure. Here?s a strawman.
>>>> 
>>>> The type space is 16 bits, so you have 65,565 types.
>>>> 
>>>> The type space is currently shared with the types used for the
>>>> entire protocol, that gives us two options:
>>>> (1) we reserve a range for name component types. Given the
>>>> likelihood there will be at least as much and probably more need
>>>> to component types than protocol extensions, we could reserve 1/2
>>>> of the type space, giving us 32K types for name components.
>>>> (2) since there is no parsing ambiguity between name components
>>>> and other fields of the protocol (sine they are sub-types of the
>>>> name type) we could reuse numbers and thereby have an entire 65K
>>>> name component types.
>>>> 
>>>> We divide the type space into regions, and manage it with a
>>>> registry. If we ever get to the point of creating an IETF
>>>> standard, IANA has 25 years of experience running registries and
>>>> there are well-understood rule sets for different kinds of
>>>> registries (open, requires a written spec, requires standards
>>>> approval).
>>>> 
>>>> - We allocate one ?default" name component type for ?generic
>>>> name?, which would be used on name prefixes and other common
>>>> cases where there are no special semantics on the name component.
>>>> - We allocate a range of name component types, say 1024, to
>>>> globally understood types that are part of the base or extension
>>>> NDN specifications (e.g. chunk#, version#, etc.
>>>> - We reserve some portion of the space for unanticipated uses
>>>> (say another 1024 types)
>>>> - We give the rest of the space to application assignment.
>>>> 
>>>> Make sense?
>>>> 
>>>> 
>>>> While I?m sympathetic to that view, there are three ways in
>>>> which Moore?s law or hardware tricks will not save us from
>>>> performance flaws in the design
>>>> 
>>>> 
>>>> we could design for performance,
>>>> 
>>>> That?s not what people are advocating. We are advocating that we
>>>> *not* design for known bad performance and hope serendipity or
>>>> Moore?s Law will come to the rescue.
>>>> 
>>>> but I think there will be a turning
>>>> point when the slower design starts to become "fast enough?.
>>>> 
>>>> Perhaps, perhaps not. Relative performance is what matters so
>>>> things that don?t get faster while others do tend to get dropped
>>>> or not used because they impose a performance penalty relative to
>>>> the things that go faster. There is also the ?low-end? phenomenon
>>>> where impovements in technology get applied to lowering cost
>>>> rather than improving performance. For those environments bad
>>>> performance just never get better.
>>>> 
>>>> Do you
>>>> think there will be some design of ndn that will *never* have
>>>> performance improvement?
>>>> 
>>>> I suspect LPM on data will always be slow (relative to the other
>>>> functions).
>>>> i suspect exclusions will always be slow because they will
>>>> require extra memory references.
>>>> 
>>>> However I of course don?t claim to clairvoyance so this is just
>>>> speculation based on 35+ years of seeing performance improve by 4
>>>> orders of magnitude and still having to worry about counting
>>>> cycles and memory references?
>>>> 
>>>> On Thu, Sep 18, 2014 at 5:20 PM, Dave Oran (oran)
>>>> <oran at cisco.com> wrote:
>>>> 
>>>> On Sep 18, 2014, at 7:41 PM, Tai-Lin Chu
>>>> <tailinchu at gmail.com> wrote:
>>>> 
>>>> We should not look at a certain chip nowadays and want ndn to
>>>> perform
>>>> well on it. It should be the other way around: once  ndn app
>>>> becomes
>>>> popular, a better chip will be designed for ndn.
>>>> 
>>>> While I?m sympathetic to that view, there are three ways in
>>>> which Moore?s law or hardware tricks will not save us from
>>>> performance flaws in the design:
>>>> a) clock rates are not getting (much) faster
>>>> b) memory accesses are getting (relatively) more expensive
>>>> c) data structures that require locks to manipulate
>>>> successfully will be relatively more expensive, even with
>>>> near-zero lock contention.
>>>> 
>>>> The fact is, IP *did* have some serious performance flaws in
>>>> its design. We just forgot those because the design elements
>>>> that depended on those mistakes have fallen into disuse. The
>>>> poster children for this are:
>>>> 1. IP options. Nobody can use them because they are too slow
>>>> on modern forwarding hardware, so they can?t be reliably used
>>>> anywhere
>>>> 2. the UDP checksum, which was a bad design when it was
>>>> specified and is now a giant PITA that still causes major pain
>>>> in working around.
>>>> 
>>>> I?m afraid students today are being taught the that designers
>>>> of IP were flawless, as opposed to very good scientists and
>>>> engineers that got most of it right.
>>>> 
>>>> I feel the discussion today and yesterday has been off-topic.
>>>> Now I
>>>> see that there are 3 approaches:
>>>> 1. we should not define a naming convention at all
>>>> 2. typed component: use tlv type space and add a handful of
>>>> types
>>>> 3. marked component: introduce only one more type and add
>>>> additional
>>>> marker space
>>>> 
>>>> I know how to make #2 flexible enough to do what things I can
>>>> envision we need to do, and with a few simple conventions on
>>>> how the registry of types is managed.
>>>> 
>>>> It is just as powerful in practice as either throwing up our
>>>> hands and letting applications design their own mutually
>>>> incompatible schemes or trying to make naming conventions with
>>>> markers in a way that is fast to generate/parse and also
>>>> resilient against aliasing.
>>>> 
>>>> Also everybody thinks that the current utf8 marker naming
>>>> convention
>>>> needs to be revised.
>>>> 
>>>> 
>>>> 
>>>> On Thu, Sep 18, 2014 at 3:27 PM, Felix Rabe <felix at rabe.io>
>>>> wrote:
>>>> Would that chip be suitable, i.e. can we expect most names
>>>> to fit in (the
>>>> magnitude of) 96 bytes? What length are names usually in
>>>> current NDN
>>>> experiments?
>>>> 
>>>> I guess wide deployment could make for even longer names.
>>>> Related: Many URLs
>>>> I encounter nowadays easily don't fit within two 80-column
>>>> text lines, and
>>>> NDN will have to carry more information than URLs, as far as
>>>> I see.
>>>> 
>>>> 
>>>> On 18/Sep/14 23:15, Marc.Mosko at parc.com wrote:
>>>> 
>>>> In fact, the index in separate TLV will be slower on some
>>>> architectures,
>>>> like the ezChip NP4.  The NP4 can hold the fist 96 frame
>>>> bytes in memory,
>>>> then any subsequent memory is accessed only as two adjacent
>>>> 32-byte blocks
>>>> (there can be at most 5 blocks available at any one time).
>>>> If you need to
>>>> switch between arrays, it would be very expensive.  If you
>>>> have to read past
>>>> the name to get to the 2nd array, then read it, then backup
>>>> to get to the
>>>> name, it will be pretty expensive too.
>>>> 
>>>> Marc
>>>> 
>>>> On Sep 18, 2014, at 2:02 PM, <Ignacio.Solis at parc.com>
>>>> <Ignacio.Solis at parc.com> wrote:
>>>> 
>>>> Does this make that much difference?
>>>> 
>>>> If you want to parse the first 5 components. One way to do
>>>> it is:
>>>> 
>>>> Read the index, find entry 5, then read in that many bytes
>>>> from the start
>>>> offset of the beginning of the name.
>>>> OR
>>>> Start reading name, (find size + move ) 5 times.
>>>> 
>>>> How much speed are you getting from one to the other?  You
>>>> seem to imply
>>>> that the first one is faster.  I don?t think this is the
>>>> case.
>>>> 
>>>> In the first one you?ll probably have to get the cache line
>>>> for the index,
>>>> then all the required cache lines for the first 5
>>>> components. For the
>>>> second, you?ll have to get all the cache lines for the first
>>>> 5 components.
>>>> Given an assumption that a cache miss is way more expensive
>>>> than
>>>> evaluating a number and computing an addition, you might
>>>> find that the
>>>> performance of the index is actually slower than the
>>>> performance of the
>>>> direct access.
>>>> 
>>>> Granted, there is a case where you don?t access the name at
>>>> all, for
>>>> example, if you just get the offsets and then send the
>>>> offsets as
>>>> parameters to another processor/GPU/NPU/etc. In this case
>>>> you may see a
>>>> gain IF there are more cache line misses in reading the name
>>>> than in
>>>> reading the index.   So, if the regular part of the name
>>>> that you?re
>>>> parsing is bigger than the cache line (64 bytes?) and the
>>>> name is to be
>>>> processed by a different processor, then your might see some
>>>> performance
>>>> gain in using the index, but in all other circumstances I
>>>> bet this is not
>>>> the case.   I may be wrong, haven?t actually tested it.
>>>> 
>>>> This is all to say, I don?t think we should be designing the
>>>> protocol with
>>>> only one architecture in mind. (The architecture of sending
>>>> the name to a
>>>> different processor than the index).
>>>> 
>>>> If you have numbers that show that the index is faster I
>>>> would like to see
>>>> under what conditions and architectural assumptions.
>>>> 
>>>> Nacho
>>>> 
>>>> (I may have misinterpreted your description so feel free to
>>>> correct me if
>>>> I?m wrong.)
>>>> 
>>>> 
>>>> --
>>>> Nacho (Ignacio) Solis
>>>> Protocol Architect
>>>> Principal Scientist
>>>> Palo Alto Research Center (PARC)
>>>> +1(650)812-4458
>>>> Ignacio.Solis at parc.com
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 9/18/14, 12:54 AM, "Massimo Gallo"
>>>> <massimo.gallo at alcatel-lucent.com>
>>>> wrote:
>>>> 
>>>> Indeed each components' offset must be encoded using a fixed
>>>> amount of
>>>> bytes:
>>>> 
>>>> i.e.,
>>>> Type = Offsets
>>>> Length = 10 Bytes
>>>> Value = Offset1(1byte), Offset2(1byte), ...
>>>> 
>>>> You may also imagine to have a "Offset_2byte" type if your
>>>> name is too
>>>> long.
>>>> 
>>>> Max
>>>> 
>>>> On 18/09/2014 09:27, Tai-Lin Chu wrote:
>>>> 
>>>> if you do not need the entire hierarchal structure (suppose
>>>> you only
>>>> want the first x components) you can directly have it using
>>>> the
>>>> offsets. With the Nested TLV structure you have to
>>>> iteratively parse
>>>> the first x-1 components. With the offset structure you cane
>>>> directly
>>>> access to the firs x components.
>>>> 
>>>> I don't get it. What you described only works if the
>>>> "offset" is
>>>> encoded in fixed bytes. With varNum, you will still need to
>>>> parse x-1
>>>> offsets to get to the x offset.
>>>> 
>>>> 
>>>> 
>>>> On Wed, Sep 17, 2014 at 11:57 PM, Massimo Gallo
>>>> <massimo.gallo at alcatel-lucent.com> wrote:
>>>> 
>>>> On 17/09/2014 14:56, Mark Stapp wrote:
>>>> 
>>>> ah, thanks - that's helpful. I thought you were saying "I
>>>> like the
>>>> existing NDN UTF8 'convention'." I'm still not sure I
>>>> understand what
>>>> you
>>>> _do_ prefer, though. it sounds like you're describing an
>>>> entirely
>>>> different
>>>> scheme where the info that describes the name-components is
>>>> ...
>>>> someplace
>>>> other than _in_ the name-components. is that correct? when
>>>> you say
>>>> "field
>>>> separator", what do you mean (since that's not a "TL" from a
>>>> TLV)?
>>>> 
>>>> Correct.
>>>> In particular, with our name encoding, a TLV indicates the
>>>> name
>>>> hierarchy
>>>> with offsets in the name and other TLV(s) indicates the
>>>> offset to use
>>>> in
>>>> order to retrieve special components.
>>>> As for the field separator, it is something like "/".
>>>> Aliasing is
>>>> avoided as
>>>> you do not rely on field separators to parse the name; you
>>>> use the
>>>> "offset
>>>> TLV " to do that.
>>>> 
>>>> So now, it may be an aesthetic question but:
>>>> 
>>>> if you do not need the entire hierarchal structure (suppose
>>>> you only
>>>> want
>>>> the first x components) you can directly have it using the
>>>> offsets.
>>>> With the
>>>> Nested TLV structure you have to iteratively parse the first
>>>> x-1
>>>> components.
>>>> With the offset structure you cane directly access to the
>>>> firs x
>>>> components.
>>>> 
>>>> Max
>>>> 
>>>> 
>>>> -- Mark
>>>> 
>>>> On 9/17/14 6:02 AM, Massimo Gallo wrote:
>>>> 
>>>> The why is simple:
>>>> 
>>>> You use a lot of "generic component type" and very few
>>>> "specific
>>>> component type". You are imposing types for every component
>>>> in order
>>>> to
>>>> handle few exceptions (segmentation, etc..). You create a
>>>> rule
>>>> (specify
>>>> the component's type ) to handle exceptions!
>>>> 
>>>> I would prefer not to have typed components. Instead I would
>>>> prefer
>>>> to
>>>> have the name as simple sequence bytes with a field
>>>> separator. Then,
>>>> outside the name, if you have some components that could be
>>>> used at
>>>> network layer (e.g. a TLV field), you simply need something
>>>> that
>>>> indicates which is the offset allowing you to retrieve the
>>>> version,
>>>> segment, etc in the name...
>>>> 
>>>> 
>>>> Max
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 16/09/2014 20:33, Mark Stapp wrote:
>>>> 
>>>> On 9/16/14 10:29 AM, Massimo Gallo wrote:
>>>> 
>>>> I think we agree on the small number of "component types".
>>>> However, if you have a small number of types, you will end
>>>> up with
>>>> names
>>>> containing many generic components types and few specific
>>>> components
>>>> types. Due to the fact that the component type specification
>>>> is an
>>>> exception in the name, I would prefer something that specify
>>>> component's
>>>> type only when needed (something like UTF8 conventions but
>>>> that
>>>> applications MUST use).
>>>> 
>>>> so ... I can't quite follow that. the thread has had some
>>>> explanation
>>>> about why the UTF8 requirement has problems (with aliasing,
>>>> e.g.)
>>>> and
>>>> there's been email trying to explain that applications don't
>>>> have to
>>>> use types if they don't need to. your email sounds like "I
>>>> prefer
>>>> the
>>>> UTF8 convention", but it doesn't say why you have that
>>>> preference in
>>>> the face of the points about the problems. can you say why
>>>> it is
>>>> that
>>>> you express a preference for the "convention" with problems ?
>>>> 
>>>> Thanks,
>>>> Mark
>>>> 
>>>> .
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> 
>> 
>> 
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>




From jburke at remap.ucla.edu  Fri Oct  3 20:37:58 2014
From: jburke at remap.ucla.edu (Burke, Jeff)
Date: Sat, 4 Oct 2014 03:37:58 +0000
Subject: [Ndn-interest] any comments on naming convention?
In-Reply-To: <7ACEC7C8-4DCB-4B22-8540-49D4C71FE116@parc.com>
Message-ID: <D054B43F.175D1A%jburke@remap.ucla.edu>



On 9/25/14, 2:09 PM, "Marc.Mosko at parc.com" <Marc.Mosko at parc.com> wrote:

>
>On Sep 25, 2014, at 10:01 PM, Lan Wang (lanwang) <lanwang at memphis.edu>
>wrote:
>
>>>> - Benefit seems apparent in multi-consumer scenarios, even without
>>>>sync.
>>>> Let's say I have 5 personal devices requesting mail. In Scheme B,
>>>>every
>>>> publisher receives and processes 5 interests per second on average.
>>>>In
>>>> Scheme A, with an upstream caching node, each receives 1 per second
>>>> maximum. The publisher  still has to throttle requests, but with no
>>>>help
>>>> or scaling support from the network.
>>> 
>>> This can be done without selectors.  As long as all the clients
>>>produce a
>>> request for the same name they can take advantage caching.
>> 
>> What Jeff responded to is that scheme B requires a freshness of 0 for
>>the initial interest to get to the producer (in order to get the latest
>>list of email names).  If freshness is 0, then there's no caching of the
>>data.  No meter how the clients name their Interests, they can't take
>>advantage of caching.
>> 
>
>How do selectors prevent you from sending an Interest to the producer, if
>it?s connected.  I send first interest ?exclude <= 100? and cache A
>responds with version 110.  Don?t you then turn around and send a second
>interest ?exclude <= 110? to see if another cache has a more recent
>version?  Won?t that interest go to the producer, if its connected?  It
>will then need to send a NACk (or you need to timeout), if there?s
>nothing more recent.
>
>Using selectors, you still never know if there?s a more recent version
>until you get to the producer or you timeout.  You always need to keep
>asking and asking.

Just musing but I'm not sure whether those are the only two cases.
Another way to think about it could be that instead of focusing on the
network as providing a path between a producer and consumer, with caches
along the way, and the 'gold standard' for latest version requests being
direct producer-consumer communication, is that the consumer faces a black
box that answers its Interests with certain (varying) timing properties
due to the mixture of multi-homed producers, caches, repos, etc.

In this alternate conceptualization, what's important is what pattern of
Interest/Data exchange seeking the "latest version" the network can
sustain--rather than communication with the producer. I.e., what's the
"latest version" the network can get to the consumer.   In live video
streaming, a respectful consumer application might not care to talk to an
authoritative producer for scaling reasons. Instead, the consumer
determines which of a set of target bitrates within some latency
constraint the "black box" of the network can sustain.  To do this on IP
requires probing and adaptation, so why not on NDN?  The probing for
"latest data" doesn't need to reach the producer - it needs to reflect
what the network can provide.  Some requests might go to the producer, but
they do not all need to.  This may open up more flexibility in how the
network can supply the data, how many producers there can be, when they
need to be online, etc.

A recent SIGCOMM paper [1] proposed that video streaming applications
could focus on simply on playout buffer fill rather than trying to
estimate network capacity, perhaps there is an an analogous idea here - we
can look for approaches that focus on what the network can deliver to the
consumer rather than trying to manage from whom the data comes.

Huang, Te-Yuan, Ramesh Johari, and Nick McKeown. "Downton abbey without
the hiccups:
Buffer-based rate adaptation for http video streaming." Proceedings of the
2013 ACM SIGCOMM workshop on Future human-centric multimedia networking.
ACM, 2013.



> Also, there?s nothing special about the content object from the
>producer, so you still don?t necessarily believe that its the most
>recent, and you?ll then ask again.  Sure, an application could just
>accept the 1st or 2nd content object it gets back, but it never really
>knows.  Sure, if the CreateTime (I think you call it Timestamp in NDN, if
>you still have it) is very recent and you assume synchronized clocks,
>then you might have some belief that it?s current.
>
>We could also talk about FreshnessSeconds and MustBeFresh, but that would
>be best to start its own thread on.
>
>Marc
>




From Marc.Mosko at parc.com  Mon Oct  6 09:15:44 2014
From: Marc.Mosko at parc.com (Marc.Mosko at parc.com)
Date: Mon, 6 Oct 2014 16:15:44 +0000
Subject: [Ndn-interest] any comments on naming convention?
In-Reply-To: <D054AF1A.175C98%jburke@remap.ucla.edu>
References: <D054AF1A.175C98%jburke@remap.ucla.edu>
Message-ID: <01EEAE37-D125-4A89-B430-A14C09DDE6D3@parc.com>


On Oct 3, 2014, at 8:02 PM, Burke, Jeff <jburke at remap.ucla.edu> wrote:

> 
> 
> On 9/25/14, 12:17 AM, "Marc.Mosko at parc.com" <Marc.Mosko at parc.com> wrote:
> 
>> My beating on ?discover all? is exactly because of this.  Let?s define
>> discovery service.  If the service is just ?discover latest?
>> (left/right), can we not simplify the current approach?  If the service
>> includes more than ?latest?, then is the current approach the right
>> approach?
>> 
>> Sync has its place and is the right solution for somethings.  However, it
>> should not be a a bandage over discovery.  Discovery should be its own
>> valid and useful service.
> 
> I think this makes sense but want to make sure I am following the
> definitions - can you then define discovery more specifically?

Discovery should offer mechanisms to enumerate content names based on 
a criteria (and other associated data like hash and create time and
publisher).  Two obvious ones are to enumerate the greatest (by some sort
criteria) or to enumerate all or to enumerate all within some bounds.
I believe it should specifically be to enumerate content names, not to
fetch content.

I do not believe there will be one discovery protocol that suites all needs.
Some applications might be okay with best-effort (similar to what we have now),
some might want eventually consistent, some might want strictly consistent based
on a master or consensus.

> 
>> 
>> I agree that the exclusion approach can work, and work relatively well,
>> for finding the rightmost/leftmost child.  I believe this is because that
>> operation is transitive through caches.  So, within whatever timeout an
>> application is willing to wait to find the ?latest?, it can keep asking
>> and asking.  
>> 
>> I do think it would be best to actually try to ask an authoritative
>> source first (i.e. a non-cached value), and if that fails then probe
>> caches, but experimentation may show what works well.  This is based on
>> my belief that in the real world in broad use, the namespace will become
>> pretty polluted and probing will result in a  lot of junk, but that?s
>> future prognosticating.
> 
> I'm not sure about the emphasis on an authoritative sources... What makes
> something authoritative? That it assembled and signed the packet? For a
> number of scenarios, from data muling in vehicular networking to video
> publishing or even retrieving the "sufficiently-current" homepage of the
> New York Times, getting to that source is either impossible (some V2V
> scenario) or unnecessary (NYTimes) if some cache or repo in the network
> can answer the interest.  Using selectors doesn't necessarily mean probing
> a huge, messy namespace--I think there is agreement that is not
> efficient--it can be trying to target an Interest in a well-understood
> namespace without having to know the exact child name.

I expect the world will be adversarial and not have clean namespaces except
in some controlled environments where one can enforce some form of access
control.  If caches do not reject content with bad signatures, then one
will have a messy namespace meaning heavy use of exclusions.

Authoritative, to me, means it is signed with the expected KeyId.

Yes, for some scenarios, best-effort discovery is ok.  But I think for many
scenarios one needs stronger consistency guarantees.  Thus my point about
needing multiple discovery protocols.  I?m also all for using protocol
building blocks. If we were to, for example, mandate a basic discovery protocol
operate on every cache and more consistent or complex discovery protocols
could be build on top of that, lets do it.  But I don?t think we?re there
yet.  I, personally, do not know such a protocol, but I?m pretty sure it
is not the selector based system the combines discovery with data retrieval
and gets, at most, one result per round trip.


> 
>> 
>> Also, in the exact match vs. continuation match of content object to
>> interest, it is pretty easy to encode that ?selector? request in a name
>> component (i.e. ?exclude_before=(t=version, l=2, v=279) & sort=right?)
>> and any participating cache can respond with a link (or encapsulate) a
>> response in an exact match system.
>> 
>> In the CCNx 1.0 spec, one could also encode this a different way.  One
>> could use a name like ?/mail/inbox/selector_matching/<hash of payload>?
>> and in the payload include "exclude_before=(t=version, l=2, v=279) &
>> sort=right?.  This means that any cache that could process the ?
>> selector_matching? function could look at the interest payload and
>> evaluate the predicate there.  The predicate could become large and not
>> pollute the PIT with all the computation state.  Including ?<hash of
>> payload>? in the name means that one could get a cached response if
>> someone else had asked the same exact question (subject to the content
>> object?s cache lifetime) and it also servers to multiplex different
>> payloads for the same function (selector_matching).
>> 
>> Marc
>> 
>> 
>> On Sep 25, 2014, at 8:18 AM, Burke, Jeff <jburke at remap.ucla.edu> wrote:
>> 
>>> 
>>> http://irl.cs.ucla.edu/~zhenkai/papers/chronosync.pdf
>>> 
>>> 
>>> https://www.ccnx.org/releases/ccnx-0.7.0rc1/doc/technical/Synchronization
>>> Pr
>>> otocol.html
>>> 
>>> J. 
>>> 
>>> 
>>> On 9/24/14, 11:16 PM, "Tai-Lin Chu" <tailinchu at gmail.com> wrote:
>>> 
>>>> However, I cannot see whether we can achieve "best-effort *all*-value"
>>>> efficiently.
>>>> There are still interesting topics on
>>>> 1. how do we express the discovery query?
>>>> 2. is selector "discovery-complete"? i. e. can we express any
>>>> discovery query with current selector?
>>>> 3. if so, can we re-express current selector in a more efficient way?
>>>> 
>>>> I personally see a named data as a set, which can then be categorized
>>>> into "ordered set", and "unordered set".
>>>> some questions that any discovery expression must solve:
>>>> 1. is this a nil set or not? nil set means that this name is the leaf
>>>> 2. set contains member X?
>>>> 3. is set ordered or not
>>>> 4. (ordered) first, prev, next, last
>>>> 5. if we enforce component ordering, answer question 4.
>>>> 6. recursively answer all questions above on any set member
>>>> 
>>>> 
>>>> 
>>>> On Wed, Sep 24, 2014 at 10:45 PM, Burke, Jeff <jburke at remap.ucla.edu>
>>>> wrote:
>>>>> 
>>>>> 
>>>>> From: <Marc.Mosko at parc.com>
>>>>> Date: Wed, 24 Sep 2014 16:25:53 +0000
>>>>> To: Jeff Burke <jburke at remap.ucla.edu>
>>>>> Cc: <Ignacio.Solis at parc.com>, <tailinchu at gmail.com>,
>>>>> <ndn-interest at lists.cs.ucla.edu>
>>>>> Subject: Re: [Ndn-interest] any comments on naming convention?
>>>>> 
>>>>> I think Tai-Lin?s example was just fine to talk about discovery.
>>>>> /blah/blah/value, how do you discover all the ?value?s?  Discovery
>>>>> shouldn?t
>>>>> care if its email messages or temperature readings or world cup
>>>>> photos.
>>>>> 
>>>>> 
>>>>> This is true if discovery means "finding everything" - in which case,
>>>>> as you
>>>>> point out, sync-style approaches may be best.  But I am not sure that
>>>>> this
>>>>> definition is complete.  The most pressing example that I can think of
>>>>> is
>>>>> best-effort latest-value, in which the consumer's goal is to get the
>>>>> latest
>>>>> copy the network can deliver at the moment, and may not care about
>>>>> previous
>>>>> values or (if freshness is used well) potential later versions.
>>>>> 
>>>>> Another case that seems to work well is video seeking.  Let's say I
>>>>> want to
>>>>> enable random access to a video by timecode. The publisher can
>>>>> provide a
>>>>> time-code based discovery namespace that's queried using an Interest
>>>>> that
>>>>> essentially says "give me the closest keyframe to 00:37:03:12", which
>>>>> returns an interest that, via the name, provides the exact timecode of
>>>>> the
>>>>> keyframe in question and a link to a segment-based namespace for
>>>>> efficient
>>>>> exact match playout.  In two roundtrips and in a very lightweight way,
>>>>> the
>>>>> consumer has random access capability.   If the NDN is the moral
>>>>> equivalent
>>>>> of IP, then I am not sure we should be afraid of roundtrips that
>>>>> provide
>>>>> this kind of functionality, just as they are used in TCP.
>>>>> 
>>>>> 
>>>>> I described one set of problems using the exclusion approach, and that
>>>>> an
>>>>> NDN paper on device discovery described a similar problem, though they
>>>>> did
>>>>> not go into the details of splitting interests, etc.  That all was
>>>>> simple
>>>>> enough to see from the example.
>>>>> 
>>>>> Another question is how does one do the discovery with exact match
>>>>> names,
>>>>> which is also conflating things.  You could do a different discovery
>>>>> with
>>>>> continuation names too, just not the exclude method.
>>>>> 
>>>>> As I alluded to, one needs a way to talk with a specific cache about
>>>>> its
>>>>> ?table of contents? for a prefix so one can get a consistent set of
>>>>> results
>>>>> without all the round-trips of exclusions.  Actually downloading the
>>>>> ?headers? of the messages would be the same bytes, more or less.  In a
>>>>> way,
>>>>> this is a little like name enumeration from a ccnx 0.x repo, but that
>>>>> protocol has its own set of problems and I?m not suggesting to use
>>>>> that
>>>>> directly.
>>>>> 
>>>>> One approach is to encode a request in a name component and a
>>>>> participating
>>>>> cache can reply.  It replies in such a way that one could continue
>>>>> talking
>>>>> with that cache to get its TOC.  One would then issue another interest
>>>>> with
>>>>> a request for not-that-cache.
>>>>> 
>>>>> 
>>>>> I'm curious how the TOC approach works in a multi-publisher scenario?
>>>>> 
>>>>> 
>>>>> Another approach is to try to ask the authoritative source for the
>>>>> ?current?
>>>>> manifest name, i.e. /mail/inbox/current/<nonce>, which could return
>>>>> the
>>>>> manifest or a link to the manifest.  Then fetching the actual manifest
>>>>> from
>>>>> the link could come from caches because you how have a consistent set
>>>>> of
>>>>> names to ask for.  If you cannot talk with an authoritative source,
>>>>> you
>>>>> could try again without the nonce and see if there?s a cached copy of
>>>>> a
>>>>> recent version around.
>>>>> 
>>>>> Marc
>>>>> 
>>>>> 
>>>>> On Sep 24, 2014, at 5:46 PM, Burke, Jeff <jburke at remap.ucla.edu>
>>>>> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> On 9/24/14, 8:20 AM, "Ignacio.Solis at parc.com" <Ignacio.Solis at parc.com>
>>>>> wrote:
>>>>> 
>>>>> On 9/24/14, 4:27 AM, "Tai-Lin Chu" <tailinchu at gmail.com> wrote:
>>>>> 
>>>>> For example, I see a pattern /mail/inbox/148. I, a human being, see a
>>>>> pattern with static (/mail/inbox) and variable (148) components; with
>>>>> proper naming convention, computers can also detect this pattern
>>>>> easily. Now I want to look for all mails in my inbox. I can generate a
>>>>> list of /mail/inbox/<number>. These are my guesses, and with selectors
>>>>> I can further refine my guesses.
>>>>> 
>>>>> 
>>>>> I think this is a very bad example  (or at least a very bad
>>>>> application
>>>>> design).  You have an app (a mail server / inbox) and you want it to
>>>>> list
>>>>> your emails?  An email list is an application data structure.  I don?t
>>>>> think you should use the network structure to reflect this.
>>>>> 
>>>>> 
>>>>> I think Tai-Lin is trying to sketch a small example, not propose a
>>>>> full-scale approach to email. (Maybe I am misunderstanding.)
>>>>> 
>>>>> 
>>>>> Another way to look at it is that if the network architecture is
>>>>> providing
>>>>> the equivalent of distributed storage to the application, perhaps the
>>>>> application data structure could be adapted to match the affordances
>>>>> of
>>>>> the network.  Then it would not be so bad that the two structures were
>>>>> aligned.
>>>>> 
>>>>> 
>>>>> I?ll give you an example, how do you delete emails from your inbox? If
>>>>> an
>>>>> email was cached in the network it can never be deleted from your
>>>>> inbox?
>>>>> 
>>>>> 
>>>>> This is conflating two issues - what you are pointing out is that the
>>>>> data
>>>>> structure of a linear list doesn't handle common email management
>>>>> operations well.  Again, I'm not sure if that's what he was getting at
>>>>> here.  But deletion is not the issue - the availability of a data
>>>>> object
>>>>> on the network does not necessarily mean it's valid from the
>>>>> perspective
>>>>> of the application.
>>>>> 
>>>>> Or moved to another mailbox? Do you rely on the emails expiring?
>>>>> 
>>>>> This problem is true for most (any?) situations where you use network
>>>>> name
>>>>> structure to directly reflect the application data structure.
>>>>> 
>>>>> 
>>>>> Not sure I understand how you make the leap from the example to the
>>>>> general statement.
>>>>> 
>>>>> Jeff
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> Nacho
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, Sep 23, 2014 at 2:34 AM,  <Marc.Mosko at parc.com> wrote:
>>>>> 
>>>>> Ok, yes I think those would all be good things.
>>>>> 
>>>>> One thing to keep in mind, especially with things like time series
>>>>> sensor
>>>>> data, is that people see a pattern and infer a way of doing it.
>>>>> That?s
>>>>> easy
>>>>> for a human :)  But in Discovery, one should assume that one does not
>>>>> know
>>>>> of patterns in the data beyond what the protocols used to publish the
>>>>> data
>>>>> explicitly require.  That said, I think some of the things you listed
>>>>> are
>>>>> good places to start: sensor data, web content, climate data or genome
>>>>> data.
>>>>> 
>>>>> We also need to state what the forwarding strategies are and what the
>>>>> cache
>>>>> behavior is.
>>>>> 
>>>>> I outlined some of the points that I think are important in that other
>>>>> posting.  While ?discover latest? is useful, ?discover all? is also
>>>>> important, and that one gets complicated fast.  So points like
>>>>> separating
>>>>> discovery from retrieval and working with large data sets have been
>>>>> important in shaping our thinking.  That all said, I?d be happy
>>>>> starting
>>>>> from 0 and working through the Discovery service definition from
>>>>> scratch
>>>>> along with data set use cases.
>>>>> 
>>>>> Marc
>>>>> 
>>>>> On Sep 23, 2014, at 12:36 AM, Burke, Jeff <jburke at remap.ucla.edu>
>>>>> wrote:
>>>>> 
>>>>> Hi Marc,
>>>>> 
>>>>> Thanks ? yes, I saw that as well. I was just trying to get one step
>>>>> more
>>>>> specific, which was to see if we could identify a few specific use
>>>>> cases
>>>>> around which to have the conversation.  (e.g., time series sensor data
>>>>> and
>>>>> web content retrieval for "get latest"; climate data for huge data
>>>>> sets;
>>>>> local data in a vehicular network; etc.)  What have you been looking
>>>>> at
>>>>> that's driving considerations of discovery?
>>>>> 
>>>>> Thanks,
>>>>> Jeff
>>>>> 
>>>>> From: <Marc.Mosko at parc.com>
>>>>> Date: Mon, 22 Sep 2014 22:29:43 +0000
>>>>> To: Jeff Burke <jburke at remap.ucla.edu>
>>>>> Cc: <tailinchu at gmail.com>, <ndn-interest at lists.cs.ucla.edu>
>>>>> Subject: Re: [Ndn-interest] any comments on naming convention?
>>>>> 
>>>>> Jeff,
>>>>> 
>>>>> Take a look at my posting (that Felix fixed) in a new thread on
>>>>> Discovery.
>>>>> 
>>>>> 
>>>>> 
>>>>> http://www.lists.cs.ucla.edu/pipermail/ndn-interest/2014-September/0002
>>>>> 0
>>>>> 0
>>>>> .html
>>>>> 
>>>>> I think it would be very productive to talk about what Discovery
>>>>> should
>>>>> do,
>>>>> and not focus on the how.  It is sometimes easy to get caught up in
>>>>> the
>>>>> how,
>>>>> which I think is a less important topic than the what at this stage.
>>>>> 
>>>>> Marc
>>>>> 
>>>>> On Sep 22, 2014, at 11:04 PM, Burke, Jeff <jburke at remap.ucla.edu>
>>>>> wrote:
>>>>> 
>>>>> Marc,
>>>>> 
>>>>> If you can't talk about your protocols, perhaps we can discuss this
>>>>> based
>>>>> on use cases.   What are the use cases you are using to evaluate
>>>>> discovery?
>>>>> 
>>>>> Jeff
>>>>> 
>>>>> 
>>>>> 
>>>>> On 9/21/14, 11:23 AM, "Marc.Mosko at parc.com" <Marc.Mosko at parc.com>
>>>>> wrote:
>>>>> 
>>>>> No matter what the expressiveness of the predicates if the forwarder
>>>>> can
>>>>> send interests different ways you don't have a consistent underlying
>>>>> set
>>>>> to talk about so you would always need non-range exclusions to
>>>>> discover
>>>>> every version.
>>>>> 
>>>>> Range exclusions only work I believe if you get an authoritative
>>>>> answer.
>>>>> If different content pieces are scattered between different caches I
>>>>> don't see how range exclusions would work to discover every version.
>>>>> 
>>>>> I'm sorry to be pointing out problems without offering solutions but
>>>>> we're not ready to publish our discovery protocols.
>>>>> 
>>>>> Sent from my telephone
>>>>> 
>>>>> On Sep 21, 2014, at 8:50, "Tai-Lin Chu" <tailinchu at gmail.com> wrote:
>>>>> 
>>>>> I see. Can you briefly describe how ccnx discovery protocol solves the
>>>>> all problems that you mentioned (not just exclude)? a doc will be
>>>>> better.
>>>>> 
>>>>> My unserious conjecture( :) ) : exclude is equal to [not]. I will soon
>>>>> expect [and] and [or], so boolean algebra is fully supported.  Regular
>>>>> language or context free language might become part of selector too.
>>>>> 
>>>>> On Sat, Sep 20, 2014 at 11:25 PM,  <Marc.Mosko at parc.com> wrote:
>>>>> That will get you one reading then you need to exclude it and ask
>>>>> again.
>>>>> 
>>>>> Sent from my telephone
>>>>> 
>>>>> On Sep 21, 2014, at 8:22, "Tai-Lin Chu" <tailinchu at gmail.com> wrote:
>>>>> 
>>>>> Yes, my point was that if you cannot talk about a consistent set
>>>>> with a particular cache, then you need to always use individual
>>>>> excludes not range excludes if you want to discover all the versions
>>>>> of an object.
>>>>> 
>>>>> 
>>>>> I am very confused. For your example, if I want to get all today's
>>>>> sensor data, I just do (Any..Last second of last day)(First second of
>>>>> tomorrow..Any). That's 18 bytes.
>>>>> 
>>>>> 
>>>>> [1]http://named-data.net/doc/ndn-tlv/interest.html#exclude
>>>>> 
>>>>> On Sat, Sep 20, 2014 at 10:55 PM,  <Marc.Mosko at parc.com> wrote:
>>>>> 
>>>>> On Sep 21, 2014, at 1:47 AM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>>>>> 
>>>>> If you talk sometimes to A and sometimes to B, you very easily
>>>>> could miss content objects you want to discovery unless you avoid
>>>>> all range exclusions and only exclude explicit versions.
>>>>> 
>>>>> 
>>>>> Could you explain why missing content object situation happens? also
>>>>> range exclusion is just a shorter notation for many explicit
>>>>> exclude;
>>>>> converting from explicit excludes to ranged exclude is always
>>>>> possible.
>>>>> 
>>>>> 
>>>>> Yes, my point was that if you cannot talk about a consistent set
>>>>> with a particular cache, then you need to always use individual
>>>>> excludes not range excludes if you want to discover all the versions
>>>>> of an object.  For something like a sensor reading that is updated,
>>>>> say, once per second you will have 86,400 of them per day.  If each
>>>>> exclusion is a timestamp (say 8 bytes), that?s 691,200 bytes of
>>>>> exclusions (plus encoding overhead) per day.
>>>>> 
>>>>> yes, maybe using a more deterministic version number than a
>>>>> timestamp makes sense here, but its just an example of needing a lot
>>>>> of exclusions.
>>>>> 
>>>>> 
>>>>> You exclude through 100 then issue a new interest.  This goes to
>>>>> cache B
>>>>> 
>>>>> 
>>>>> I feel this case is invalid because cache A will also get the
>>>>> interest, and cache A will return v101 if it exists. Like you said,
>>>>> if
>>>>> this goes to cache B only, it means that cache A dies. How do you
>>>>> know
>>>>> that v101 even exist?
>>>>> 
>>>>> 
>>>>> I guess this depends on what the forwarding strategy is.  If the
>>>>> forwarder will always send each interest to all replicas, then yes,
>>>>> modulo packet loss, you would discover v101 on cache A.  If the
>>>>> forwarder is just doing ?best path? and can round-robin between cache
>>>>> A and cache B, then your application could miss v101.
>>>>> 
>>>>> 
>>>>> 
>>>>> c,d In general I agree that LPM performance is related to the number
>>>>> of components. In my own thread-safe LMP implementation, I used only
>>>>> one RWMutex for the whole tree. I don't know whether adding lock for
>>>>> every node will be faster or not because of lock overhead.
>>>>> 
>>>>> However, we should compare (exact match + discovery protocol) vs
>>>>> (ndn
>>>>> lpm). Comparing performance of exact match to lpm is unfair.
>>>>> 
>>>>> 
>>>>> Yes, we should compare them.  And we need to publish the ccnx 1.0
>>>>> specs for doing the exact match discovery.  So, as I said, I?m not
>>>>> ready to claim its better yet because we have not done that.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Sat, Sep 20, 2014 at 2:38 PM,  <Marc.Mosko at parc.com> wrote:
>>>>> I would point out that using LPM on content object to Interest
>>>>> matching to do discovery has its own set of problems.  Discovery
>>>>> involves more than just ?latest version? discovery too.
>>>>> 
>>>>> This is probably getting off-topic from the original post about
>>>>> naming conventions.
>>>>> 
>>>>> a.  If Interests can be forwarded multiple directions and two
>>>>> different caches are responding, the exclusion set you build up
>>>>> talking with cache A will be invalid for cache B.  If you talk
>>>>> sometimes to A and sometimes to B, you very easily could miss
>>>>> content objects you want to discovery unless you avoid all range
>>>>> exclusions and only exclude explicit versions.  That will lead to
>>>>> very large interest packets.  In ccnx 1.0, we believe that an
>>>>> explicit discovery protocol that allows conversations about
>>>>> consistent sets is better.
>>>>> 
>>>>> b. Yes, if you just want the ?latest version? discovery that
>>>>> should be transitive between caches, but imagine this.  You send
>>>>> Interest #1 to cache A which returns version 100.  You exclude
>>>>> through 100 then issue a new interest.  This goes to cache B who
>>>>> only has version 99, so the interest times out or is NACK?d.  So
>>>>> you think you have it!  But, cache A already has version 101, you
>>>>> just don?t know.   If you cannot have a conversation around
>>>>> consistent sets, it seems like even doing latest version discovery
>>>>> is difficult with selector based discovery.  From what I saw in
>>>>> ccnx 0.x, one ended up getting an Interest all the way to the
>>>>> authoritative source because you can never believe an intermediate
>>>>> cache that there?s not something more recent.
>>>>> 
>>>>> I?m sure you?ve walked through cases (a) and (b) in ndn, I?d be
>>>>> interest in seeing your analysis.  Case (a) is that a node can
>>>>> correctly discover every version of a name prefix, and (b) is that
>>>>> a node can correctly discover the latest version.  We have not
>>>>> formally compared (or yet published) our discovery protocols (we
>>>>> have three, 2 for content, 1 for device) compared to selector based
>>>>> discovery, so I cannot yet claim they are better, but they do not
>>>>> have the non-determinism sketched above.
>>>>> 
>>>>> c. Using LPM, there is a non-deterministic number of lookups you
>>>>> must do in the PIT to match a content object.  If you have a name
>>>>> tree or a threaded hash table, those don?t all need to be hash
>>>>> lookups, but you need to walk up the name tree for every prefix of
>>>>> the content object name and evaluate the selector predicate.
>>>>> Content Based Networking (CBN) had some some methods to create data
>>>>> structures based on predicates, maybe those would be better.  But
>>>>> in any case, you will potentially need to retrieve many PIT entries
>>>>> if there is Interest traffic for many prefixes of a root.  Even on
>>>>> an Intel system, you?ll likely miss cache lines, so you?ll have a
>>>>> lot of NUMA access for each one.  In CCNx 1.0, even a naive
>>>>> implementation only requires at most 3 lookups (one by name, one by
>>>>> name + keyid, one by name + content object hash), and one can do
>>>>> other things to optimize lookup for an extra write.
>>>>> 
>>>>> d. In (c) above, if you have a threaded name tree or are just
>>>>> walking parent pointers, I suspect you?ll need locking of the
>>>>> ancestors in a multi-threaded system (?threaded" here meaning LWP)
>>>>> and that will be expensive.  It would be interesting to see what a
>>>>> cache consistent multi-threaded name tree looks like.
>>>>> 
>>>>> Marc
>>>>> 
>>>>> 
>>>>> On Sep 20, 2014, at 8:15 PM, Tai-Lin Chu <tailinchu at gmail.com>
>>>>> wrote:
>>>>> 
>>>>> I had thought about these questions, but I want to know your idea
>>>>> besides typed component:
>>>>> 1. LPM allows "data discovery". How will exact match do similar
>>>>> things?
>>>>> 2. will removing selectors improve performance? How do we use
>>>>> other
>>>>> faster technique to replace selector?
>>>>> 3. fixed byte length and type. I agree more that type can be fixed
>>>>> byte, but 2 bytes for length might not be enough for future.
>>>>> 
>>>>> 
>>>>> On Sat, Sep 20, 2014 at 5:36 AM, Dave Oran (oran)
>>>>> <oran at cisco.com> wrote:
>>>>> 
>>>>> On Sep 18, 2014, at 9:09 PM, Tai-Lin Chu <tailinchu at gmail.com>
>>>>> wrote:
>>>>> 
>>>>> I know how to make #2 flexible enough to do what things I can
>>>>> envision we need to do, and with a few simple conventions on
>>>>> how the registry of types is managed.
>>>>> 
>>>>> 
>>>>> Could you share it with us?
>>>>> 
>>>>> Sure. Here?s a strawman.
>>>>> 
>>>>> The type space is 16 bits, so you have 65,565 types.
>>>>> 
>>>>> The type space is currently shared with the types used for the
>>>>> entire protocol, that gives us two options:
>>>>> (1) we reserve a range for name component types. Given the
>>>>> likelihood there will be at least as much and probably more need
>>>>> to component types than protocol extensions, we could reserve 1/2
>>>>> of the type space, giving us 32K types for name components.
>>>>> (2) since there is no parsing ambiguity between name components
>>>>> and other fields of the protocol (sine they are sub-types of the
>>>>> name type) we could reuse numbers and thereby have an entire 65K
>>>>> name component types.
>>>>> 
>>>>> We divide the type space into regions, and manage it with a
>>>>> registry. If we ever get to the point of creating an IETF
>>>>> standard, IANA has 25 years of experience running registries and
>>>>> there are well-understood rule sets for different kinds of
>>>>> registries (open, requires a written spec, requires standards
>>>>> approval).
>>>>> 
>>>>> - We allocate one ?default" name component type for ?generic
>>>>> name?, which would be used on name prefixes and other common
>>>>> cases where there are no special semantics on the name component.
>>>>> - We allocate a range of name component types, say 1024, to
>>>>> globally understood types that are part of the base or extension
>>>>> NDN specifications (e.g. chunk#, version#, etc.
>>>>> - We reserve some portion of the space for unanticipated uses
>>>>> (say another 1024 types)
>>>>> - We give the rest of the space to application assignment.
>>>>> 
>>>>> Make sense?
>>>>> 
>>>>> 
>>>>> While I?m sympathetic to that view, there are three ways in
>>>>> which Moore?s law or hardware tricks will not save us from
>>>>> performance flaws in the design
>>>>> 
>>>>> 
>>>>> we could design for performance,
>>>>> 
>>>>> That?s not what people are advocating. We are advocating that we
>>>>> *not* design for known bad performance and hope serendipity or
>>>>> Moore?s Law will come to the rescue.
>>>>> 
>>>>> but I think there will be a turning
>>>>> point when the slower design starts to become "fast enough?.
>>>>> 
>>>>> Perhaps, perhaps not. Relative performance is what matters so
>>>>> things that don?t get faster while others do tend to get dropped
>>>>> or not used because they impose a performance penalty relative to
>>>>> the things that go faster. There is also the ?low-end? phenomenon
>>>>> where impovements in technology get applied to lowering cost
>>>>> rather than improving performance. For those environments bad
>>>>> performance just never get better.
>>>>> 
>>>>> Do you
>>>>> think there will be some design of ndn that will *never* have
>>>>> performance improvement?
>>>>> 
>>>>> I suspect LPM on data will always be slow (relative to the other
>>>>> functions).
>>>>> i suspect exclusions will always be slow because they will
>>>>> require extra memory references.
>>>>> 
>>>>> However I of course don?t claim to clairvoyance so this is just
>>>>> speculation based on 35+ years of seeing performance improve by 4
>>>>> orders of magnitude and still having to worry about counting
>>>>> cycles and memory references?
>>>>> 
>>>>> On Thu, Sep 18, 2014 at 5:20 PM, Dave Oran (oran)
>>>>> <oran at cisco.com> wrote:
>>>>> 
>>>>> On Sep 18, 2014, at 7:41 PM, Tai-Lin Chu
>>>>> <tailinchu at gmail.com> wrote:
>>>>> 
>>>>> We should not look at a certain chip nowadays and want ndn to
>>>>> perform
>>>>> well on it. It should be the other way around: once  ndn app
>>>>> becomes
>>>>> popular, a better chip will be designed for ndn.
>>>>> 
>>>>> While I?m sympathetic to that view, there are three ways in
>>>>> which Moore?s law or hardware tricks will not save us from
>>>>> performance flaws in the design:
>>>>> a) clock rates are not getting (much) faster
>>>>> b) memory accesses are getting (relatively) more expensive
>>>>> c) data structures that require locks to manipulate
>>>>> successfully will be relatively more expensive, even with
>>>>> near-zero lock contention.
>>>>> 
>>>>> The fact is, IP *did* have some serious performance flaws in
>>>>> its design. We just forgot those because the design elements
>>>>> that depended on those mistakes have fallen into disuse. The
>>>>> poster children for this are:
>>>>> 1. IP options. Nobody can use them because they are too slow
>>>>> on modern forwarding hardware, so they can?t be reliably used
>>>>> anywhere
>>>>> 2. the UDP checksum, which was a bad design when it was
>>>>> specified and is now a giant PITA that still causes major pain
>>>>> in working around.
>>>>> 
>>>>> I?m afraid students today are being taught the that designers
>>>>> of IP were flawless, as opposed to very good scientists and
>>>>> engineers that got most of it right.
>>>>> 
>>>>> I feel the discussion today and yesterday has been off-topic.
>>>>> Now I
>>>>> see that there are 3 approaches:
>>>>> 1. we should not define a naming convention at all
>>>>> 2. typed component: use tlv type space and add a handful of
>>>>> types
>>>>> 3. marked component: introduce only one more type and add
>>>>> additional
>>>>> marker space
>>>>> 
>>>>> I know how to make #2 flexible enough to do what things I can
>>>>> envision we need to do, and with a few simple conventions on
>>>>> how the registry of types is managed.
>>>>> 
>>>>> It is just as powerful in practice as either throwing up our
>>>>> hands and letting applications design their own mutually
>>>>> incompatible schemes or trying to make naming conventions with
>>>>> markers in a way that is fast to generate/parse and also
>>>>> resilient against aliasing.
>>>>> 
>>>>> Also everybody thinks that the current utf8 marker naming
>>>>> convention
>>>>> needs to be revised.
>>>>> 
>>>>> 
>>>>> 
>>>>> On Thu, Sep 18, 2014 at 3:27 PM, Felix Rabe <felix at rabe.io>
>>>>> wrote:
>>>>> Would that chip be suitable, i.e. can we expect most names
>>>>> to fit in (the
>>>>> magnitude of) 96 bytes? What length are names usually in
>>>>> current NDN
>>>>> experiments?
>>>>> 
>>>>> I guess wide deployment could make for even longer names.
>>>>> Related: Many URLs
>>>>> I encounter nowadays easily don't fit within two 80-column
>>>>> text lines, and
>>>>> NDN will have to carry more information than URLs, as far as
>>>>> I see.
>>>>> 
>>>>> 
>>>>> On 18/Sep/14 23:15, Marc.Mosko at parc.com wrote:
>>>>> 
>>>>> In fact, the index in separate TLV will be slower on some
>>>>> architectures,
>>>>> like the ezChip NP4.  The NP4 can hold the fist 96 frame
>>>>> bytes in memory,
>>>>> then any subsequent memory is accessed only as two adjacent
>>>>> 32-byte blocks
>>>>> (there can be at most 5 blocks available at any one time).
>>>>> If you need to
>>>>> switch between arrays, it would be very expensive.  If you
>>>>> have to read past
>>>>> the name to get to the 2nd array, then read it, then backup
>>>>> to get to the
>>>>> name, it will be pretty expensive too.
>>>>> 
>>>>> Marc
>>>>> 
>>>>> On Sep 18, 2014, at 2:02 PM, <Ignacio.Solis at parc.com>
>>>>> <Ignacio.Solis at parc.com> wrote:
>>>>> 
>>>>> Does this make that much difference?
>>>>> 
>>>>> If you want to parse the first 5 components. One way to do
>>>>> it is:
>>>>> 
>>>>> Read the index, find entry 5, then read in that many bytes
>>>>> from the start
>>>>> offset of the beginning of the name.
>>>>> OR
>>>>> Start reading name, (find size + move ) 5 times.
>>>>> 
>>>>> How much speed are you getting from one to the other?  You
>>>>> seem to imply
>>>>> that the first one is faster.  I don?t think this is the
>>>>> case.
>>>>> 
>>>>> In the first one you?ll probably have to get the cache line
>>>>> for the index,
>>>>> then all the required cache lines for the first 5
>>>>> components. For the
>>>>> second, you?ll have to get all the cache lines for the first
>>>>> 5 components.
>>>>> Given an assumption that a cache miss is way more expensive
>>>>> than
>>>>> evaluating a number and computing an addition, you might
>>>>> find that the
>>>>> performance of the index is actually slower than the
>>>>> performance of the
>>>>> direct access.
>>>>> 
>>>>> Granted, there is a case where you don?t access the name at
>>>>> all, for
>>>>> example, if you just get the offsets and then send the
>>>>> offsets as
>>>>> parameters to another processor/GPU/NPU/etc. In this case
>>>>> you may see a
>>>>> gain IF there are more cache line misses in reading the name
>>>>> than in
>>>>> reading the index.   So, if the regular part of the name
>>>>> that you?re
>>>>> parsing is bigger than the cache line (64 bytes?) and the
>>>>> name is to be
>>>>> processed by a different processor, then your might see some
>>>>> performance
>>>>> gain in using the index, but in all other circumstances I
>>>>> bet this is not
>>>>> the case.   I may be wrong, haven?t actually tested it.
>>>>> 
>>>>> This is all to say, I don?t think we should be designing the
>>>>> protocol with
>>>>> only one architecture in mind. (The architecture of sending
>>>>> the name to a
>>>>> different processor than the index).
>>>>> 
>>>>> If you have numbers that show that the index is faster I
>>>>> would like to see
>>>>> under what conditions and architectural assumptions.
>>>>> 
>>>>> Nacho
>>>>> 
>>>>> (I may have misinterpreted your description so feel free to
>>>>> correct me if
>>>>> I?m wrong.)
>>>>> 
>>>>> 
>>>>> --
>>>>> Nacho (Ignacio) Solis
>>>>> Protocol Architect
>>>>> Principal Scientist
>>>>> Palo Alto Research Center (PARC)
>>>>> +1(650)812-4458
>>>>> Ignacio.Solis at parc.com
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 9/18/14, 12:54 AM, "Massimo Gallo"
>>>>> <massimo.gallo at alcatel-lucent.com>
>>>>> wrote:
>>>>> 
>>>>> Indeed each components' offset must be encoded using a fixed
>>>>> amount of
>>>>> bytes:
>>>>> 
>>>>> i.e.,
>>>>> Type = Offsets
>>>>> Length = 10 Bytes
>>>>> Value = Offset1(1byte), Offset2(1byte), ...
>>>>> 
>>>>> You may also imagine to have a "Offset_2byte" type if your
>>>>> name is too
>>>>> long.
>>>>> 
>>>>> Max
>>>>> 
>>>>> On 18/09/2014 09:27, Tai-Lin Chu wrote:
>>>>> 
>>>>> if you do not need the entire hierarchal structure (suppose
>>>>> you only
>>>>> want the first x components) you can directly have it using
>>>>> the
>>>>> offsets. With the Nested TLV structure you have to
>>>>> iteratively parse
>>>>> the first x-1 components. With the offset structure you cane
>>>>> directly
>>>>> access to the firs x components.
>>>>> 
>>>>> I don't get it. What you described only works if the
>>>>> "offset" is
>>>>> encoded in fixed bytes. With varNum, you will still need to
>>>>> parse x-1
>>>>> offsets to get to the x offset.
>>>>> 
>>>>> 
>>>>> 
>>>>> On Wed, Sep 17, 2014 at 11:57 PM, Massimo Gallo
>>>>> <massimo.gallo at alcatel-lucent.com> wrote:
>>>>> 
>>>>> On 17/09/2014 14:56, Mark Stapp wrote:
>>>>> 
>>>>> ah, thanks - that's helpful. I thought you were saying "I
>>>>> like the
>>>>> existing NDN UTF8 'convention'." I'm still not sure I
>>>>> understand what
>>>>> you
>>>>> _do_ prefer, though. it sounds like you're describing an
>>>>> entirely
>>>>> different
>>>>> scheme where the info that describes the name-components is
>>>>> ...
>>>>> someplace
>>>>> other than _in_ the name-components. is that correct? when
>>>>> you say
>>>>> "field
>>>>> separator", what do you mean (since that's not a "TL" from a
>>>>> TLV)?
>>>>> 
>>>>> Correct.
>>>>> In particular, with our name encoding, a TLV indicates the
>>>>> name
>>>>> hierarchy
>>>>> with offsets in the name and other TLV(s) indicates the
>>>>> offset to use
>>>>> in
>>>>> order to retrieve special components.
>>>>> As for the field separator, it is something like "/".
>>>>> Aliasing is
>>>>> avoided as
>>>>> you do not rely on field separators to parse the name; you
>>>>> use the
>>>>> "offset
>>>>> TLV " to do that.
>>>>> 
>>>>> So now, it may be an aesthetic question but:
>>>>> 
>>>>> if you do not need the entire hierarchal structure (suppose
>>>>> you only
>>>>> want
>>>>> the first x components) you can directly have it using the
>>>>> offsets.
>>>>> With the
>>>>> Nested TLV structure you have to iteratively parse the first
>>>>> x-1
>>>>> components.
>>>>> With the offset structure you cane directly access to the
>>>>> firs x
>>>>> components.
>>>>> 
>>>>> Max
>>>>> 
>>>>> 
>>>>> -- Mark
>>>>> 
>>>>> On 9/17/14 6:02 AM, Massimo Gallo wrote:
>>>>> 
>>>>> The why is simple:
>>>>> 
>>>>> You use a lot of "generic component type" and very few
>>>>> "specific
>>>>> component type". You are imposing types for every component
>>>>> in order
>>>>> to
>>>>> handle few exceptions (segmentation, etc..). You create a
>>>>> rule
>>>>> (specify
>>>>> the component's type ) to handle exceptions!
>>>>> 
>>>>> I would prefer not to have typed components. Instead I would
>>>>> prefer
>>>>> to
>>>>> have the name as simple sequence bytes with a field
>>>>> separator. Then,
>>>>> outside the name, if you have some components that could be
>>>>> used at
>>>>> network layer (e.g. a TLV field), you simply need something
>>>>> that
>>>>> indicates which is the offset allowing you to retrieve the
>>>>> version,
>>>>> segment, etc in the name...
>>>>> 
>>>>> 
>>>>> Max
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 16/09/2014 20:33, Mark Stapp wrote:
>>>>> 
>>>>> On 9/16/14 10:29 AM, Massimo Gallo wrote:
>>>>> 
>>>>> I think we agree on the small number of "component types".
>>>>> However, if you have a small number of types, you will end
>>>>> up with
>>>>> names
>>>>> containing many generic components types and few specific
>>>>> components
>>>>> types. Due to the fact that the component type specification
>>>>> is an
>>>>> exception in the name, I would prefer something that specify
>>>>> component's
>>>>> type only when needed (something like UTF8 conventions but
>>>>> that
>>>>> applications MUST use).
>>>>> 
>>>>> so ... I can't quite follow that. the thread has had some
>>>>> explanation
>>>>> about why the UTF8 requirement has problems (with aliasing,
>>>>> e.g.)
>>>>> and
>>>>> there's been email trying to explain that applications don't
>>>>> have to
>>>>> use types if they don't need to. your email sounds like "I
>>>>> prefer
>>>>> the
>>>>> UTF8 convention", but it doesn't say why you have that
>>>>> preference in
>>>>> the face of the points about the problems. can you say why
>>>>> it is
>>>>> that
>>>>> you express a preference for the "convention" with problems ?
>>>>> 
>>>>> Thanks,
>>>>> Mark
>>>>> 
>>>>> .
>>>>> 
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>> 
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>> 
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>> 
>>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Ndn-interest mailing list
>>> Ndn-interest at lists.cs.ucla.edu
>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> 
> 




From Ignacio.Solis at parc.com  Mon Oct  6 10:13:19 2014
From: Ignacio.Solis at parc.com (Ignacio.Solis at parc.com)
Date: Mon, 6 Oct 2014 17:13:19 +0000
Subject: [Ndn-interest] Selector Protocol over exact matching
In-Reply-To: <D054A247.175B49%jburke@remap.ucla.edu>
References: <D049953D.33672%Ignacio.Solis@parc.com>
	<D054A247.175B49%jburke@remap.ucla.edu>
Message-ID: <D0581483.33DCB%Ignacio.Solis@parc.com>

On 10/3/14, 7:17 PM, "Burke, Jeff" <jburke at remap.ucla.edu> wrote:

>- Data names become about the query rather than about the data; this seems
>to be a fundamental design difference.  If the response to each interest
>includes a hash of the query, it can't be used to fulfill different
>queries that would have the same result. So some caching benefit is lost.
>(This seems like the basic tradeoff of exact match vs lpm+selectors.)

This is true only at the nodes that don?t speak the Selector Protocol.
The ones that do speak the protocol are free to decapsulate and use that
answer for other queries.


>- In the case where there are multiple publishers with different answers
>to the same selector protocol query, they would return different data for
>the same request.   In the current NDN case, this is disambiguated because
>the response includes the name of the data itself.  But in the selector
>protocol case, couldn't you have many answers to the question all sharing
>the same name?  

Yes.

However, in current NDN there is no requirement front he network for
responses to be unique.  A node can produce data D1, name it /a/b and sign
with key K1 and another node can produce data D2, name it /a/b and sign it
with key K2.  There is no way to enforce that they are the same.   So, for
all intents and purposes the network must be ready for same name,
different data.   This is no different.

To reflect your question, would it be possible in NDN for many answers to
have the same name?  If I issue an interest for /a in the example above,
both /a/b?s would be valid answers.  The network has no way to force
unique (assigned) naming.

Again. For every node that speaks the Selector Protocol they are free to
decapsulate answers and do something to them.

>- Further, without exclusions and without more specific names, how do you
>retrieve multiple answers to the same query?  Let's assume that in some
>cases this is a desired feature.

You?re free to use exclusions in the Selector Protocol.  Any node that
runs the protocol can give you more answers. The Selector Protocol works
effective exactly like NDN for any node that runs the protocol.

However, we obviously don?t recommend this as a solution, otherwise we
would keep selectors. What we do is answer queries with manifests, and
these in turn point you to specific responses.

Also, for the same query, we would argue that the same response is the
correct thing. If you ask: "Give me a document older than Monday? and you
get Tuesday.  That is a correct answer for any time you have the same
question.  If you ask: ?Give me a document older than Monday that is not
Tuesday? that is effectively a different query.

The system does not forbid different answers for the same interest though.
 So if things go on different paths, or if the object availability
(caches) change, or if the lifetime of the object is 0 or something then
you may get a different answer.

>- How do you envision applications interacting with the selector protocol?
> Would they implement it directly or publish "normally named" content to a
>stack that then implements the protocol?

We envision applications eventually using an API. The API abstracts how
things are done underneath. There is no need for applications to
understand wire formats or any of that. Applications shouldn?t care how
integers are encoded. The API can expose the same feature set that you
want out of Selectors but it?s up to the network to implement those
features.  

Obviously the idea is that the network agrees with how things are
implemented.  So, for example, if you had an NDN app that used selectors
(and a reasonable API) we could easily add such functionality to our stack
so you could run over CCN. As a matter of fact, we could also add a few
modules (forwarder module, encoder module, etc) and run our stack (and
your app) on top of NFD if needed.   The networks wouldn?t be compatible,
but the apps wouldn?t know the difference in terms of basic functionality.


Nacho


>On 9/25/14, 1:10 AM, "Ignacio.Solis at parc.com" <Ignacio.Solis at parc.com>
>wrote:
>
>>On 9/25/14, 9:17 AM, "Marc.Mosko at parc.com" <Marc.Mosko at parc.com> wrote:
>>>In the CCNx 1.0 spec, one could also encode this a different way.  One
>>>could use a name like ?/mail/inbox/selector_matching/<hash of payload>?
>>>and in the payload include "exclude_before=(t=version, l=2, v=279) &
>>>sort=right?.
>>
>>
>>I want to highlight this.
>>
>>There is a role that selectors can play in a network.  However, our
>>biggest issue with selectors is that they are mandated at the forwarder
>>level.  This means that every node must support selectors.
>>
>>We want to make sure that the core protocol is simple and efficient.
>>Exact matching gives us that.  If you?re interested in selector matching
>>and searching, then create that protocol over exact matching.
>>
>>Marc just described a simple ?Selector Protocol", basically:
>>- Encode selectors (or any query you want) in the interest payload.
>>- Add a name segment to indicate that this is a selector based query
>>- Add a name segment to uniquely identify the query (a hash of the
>>payload
>>for example)
>>
>>Example:
>>name    = /mail/inbox/list/selector_matching/<hash of interest payload>
>>
>>payload = version > 100
>>
>>Topology:
>>
>>A ?? B ?? C
>>
>>A and C run the Selector Protocol
>>B does not run the Selector Protocol
>>
>>Now: 
>>- Any node that does not understand the Selector Protocol (B) forwards
>>normally and does exact matching.
>>- Any node that understands the Selector Protocol (C) can parse the
>>payload to find a match.
>>
>>If no match is found, forward the interest.
>>If a match is found, create a reply.
>>
>>The reply can contain 2 types of data:
>>- Structured data with links to the actual content objects
>>- Encapsulated content objects
>>
>>So, in our example, the Selector Protocol reply could be:
>>
>>name = /mail/inbox/list/selector_matching/<hash of interest payload>
>>payload =
>>  [  matching name = /mail/inbox/list/v101 ]
>>  [  embedded object < name = /mail/inbox/list/v101, payload = list,
>>signature = mail server > ]
>>signature = responding cache
>>
>>
>>
>>A few notes:
>>- Malicious nodes could inject false replies.  So, if C is malicious, it
>>can insert a reply linking to some random object or just return junk.
>>Well, this would be the case with regular selectors as well.  C could
>>reply with random crap or it could reply with a valid answer that is not
>>the optimal answer (so, for example, not the right-most child or
>>something).
>>This is something that we can?t prevent.
>>
>>In the case of CCN, our fast path does not check signatures, so you
>>wouldn?t be able to check the signature of the reply no matter what.  I?m
>>unsure if NDN is still advocating that every node checks signatures.  If
>>you are, then this approach might not work for you.
>>
>>Nodes that DO understand the Selector Protocol can check the signature of
>>the encapsulated reply (if they wanted to).
>>Nodes that DO understand the Selector Protocol can unpack the reply, and
>>add the corresponding object to their cache, effectively enabling them to
>>answer other Selector Protocol queries.
>>
>>- The reply from the Selector Protocol enabled node (C), could:
>>?  include a list of all valid answers
>>?  embed no objects
>>?  embed more than 1 object
>>?  process complex queries, regex, etc.
>>
>>The Selector Protocol could also:
>>- include a method for authentication
>>- include a cursor or some other state between queries
>>
>>
>>I think this sort of protocol gives you everything you want while still
>>maintaining an exact match protocol as the core protocol.
>>
>>
>>What is this protocol missing to satisfy your needs?
>>Can we create a protocol that will satisfy your needs on top of exact
>>matching?
>>
>>
>>Nacho
>>
>>
>>
>>--
>>Nacho (Ignacio) Solis
>>Protocol Architect
>>Principal Scientist
>>Palo Alto Research Center (PARC)
>>+1(650)812-4458
>>Ignacio.Solis at parc.com
>>
>>
>>
>>
>>
>




From emerson.barea at gmail.com  Mon Oct  6 07:27:04 2014
From: emerson.barea at gmail.com (Emerson Barea)
Date: Mon, 6 Oct 2014 11:27:04 -0300
Subject: [Ndn-interest] NDN link protocol
Message-ID: <CAO1UhdXd7WBAnbs6nOrc+s5HU52PsG536BbVruF+JVpdBbbvkw@mail.gmail.com>

Hi, I'm newbie in NDN and I have some doubts about how NDN uses data link
protocolos. NDN is not dependent of IP packets, so, NDN Interests and Data
packets traffic directly on data link protocols? Can I use ethernet, PPP,
HDLC and so other data link protocols to carrie NDN traffic?

If yes, I just need to create routes in NDN (i.e, with NLSR routing
protocols) in two computer connected in a ethernet switch running NDN and
all will work?

Thank you.

Tocha.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141006/d9f72ac0/attachment.html>

From iliamo at UCLA.EDU  Mon Oct  6 13:21:24 2014
From: iliamo at UCLA.EDU (Ilya Moiseenko)
Date: Mon, 6 Oct 2014 13:21:24 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
Message-ID: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>

Hi all,
while working on some of the research projects / demos / apps, many people noticed that there is a potential or real benefit of using information objects that do not carry any content, but instead carry useful meta-information about some other Data packets. Right now, these ?supportive? objects are usually called Manifests.

One particular use case of manifests is speeding up signing and verification of multiple Data packets. The speed-up comes from the fact that instead of signing each Data packet using public key crypto methods, the Data packets can be secured by much faster hash computation. The name of each Data packet along with a corresponding hash is placed in the Manifest packet, which is signed with public key crypto.  

This scheme makes packet verification faster, because only a manifest packet must be verified using public key crypto methods, and all other data packets listed in the manifest can be verified by recomputing the digest and comparing to the digest specified in the manifest.

One disadvantage of this scheme, is that in general case there is an additional round-trip delay, because manifest needs to be fetched first. (More info in the paper ?Self verifying names for read-only named data?)

I?ve put online a technical report that describes a protocol eliminating round-trip delay for fetching multi-segment content with embedded manifests: http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/

I expect that this approach can be useful for passing around multi-segment content (made dynamically) with roughly 5-10 times faster signing and verification execution over the traditional approach. 

If you have comments or counterarguments, let?s have a discussion in this thread.

Thanks
Ilya


From tailinchu at gmail.com  Mon Oct  6 19:47:17 2014
From: tailinchu at gmail.com (Tai-Lin Chu)
Date: Mon, 6 Oct 2014 19:47:17 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
In-Reply-To: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
References: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
Message-ID: <CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>

1. This reminds me of the discovery protocol that parc talks about.
metadata for data discovery, maybe?
2. how do you differentiate metadata from data? contentType?
3. can we leave /a/b/<id> just for data segments for orthogonality?

On Mon, Oct 6, 2014 at 1:21 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
> Hi all,
> while working on some of the research projects / demos / apps, many people noticed that there is a potential or real benefit of using information objects that do not carry any content, but instead carry useful meta-information about some other Data packets. Right now, these ?supportive? objects are usually called Manifests.
>
> One particular use case of manifests is speeding up signing and verification of multiple Data packets. The speed-up comes from the fact that instead of signing each Data packet using public key crypto methods, the Data packets can be secured by much faster hash computation. The name of each Data packet along with a corresponding hash is placed in the Manifest packet, which is signed with public key crypto.
>
> This scheme makes packet verification faster, because only a manifest packet must be verified using public key crypto methods, and all other data packets listed in the manifest can be verified by recomputing the digest and comparing to the digest specified in the manifest.
>
> One disadvantage of this scheme, is that in general case there is an additional round-trip delay, because manifest needs to be fetched first. (More info in the paper ?Self verifying names for read-only named data?)
>
> I?ve put online a technical report that describes a protocol eliminating round-trip delay for fetching multi-segment content with embedded manifests: http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/
>
> I expect that this approach can be useful for passing around multi-segment content (made dynamically) with roughly 5-10 times faster signing and verification execution over the traditional approach.
>
> If you have comments or counterarguments, let?s have a discussion in this thread.
>
> Thanks
> Ilya
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest



From shijunxiao at email.arizona.edu  Tue Oct  7 01:36:05 2014
From: shijunxiao at email.arizona.edu (Junxiao Shi)
Date: Tue, 7 Oct 2014 01:36:05 -0700
Subject: [Ndn-interest] NDN link protocol
In-Reply-To: <CAO1UhdXd7WBAnbs6nOrc+s5HU52PsG536BbVruF+JVpdBbbvkw@mail.gmail.com>
References: <CAO1UhdXd7WBAnbs6nOrc+s5HU52PsG536BbVruF+JVpdBbbvkw@mail.gmail.com>
Message-ID: <CAOFH+ObxM37nmmB2FY2RoXDO_P1R0LDDD2twSg1m-huKoocY8w@mail.gmail.com>

Hi Emerson

NDNLP is a link protocol for NDN. Its implementation is incomplete in NFD.

NLSR doesn't currently support Ethernet faces due to its use of
`faces/create` API. Only UDP unicast and TCP faces are supported.

Yours, Junxiao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141007/770bd2d4/attachment.html>

From emerson.barea at gmail.com  Tue Oct  7 05:02:43 2014
From: emerson.barea at gmail.com (Emerson Barea)
Date: Tue, 7 Oct 2014 09:02:43 -0300
Subject: [Ndn-interest] NDN link protocol
In-Reply-To: <CAOFH+ObxM37nmmB2FY2RoXDO_P1R0LDDD2twSg1m-huKoocY8w@mail.gmail.com>
References: <CAO1UhdXd7WBAnbs6nOrc+s5HU52PsG536BbVruF+JVpdBbbvkw@mail.gmail.com>
	<CAOFH+ObxM37nmmB2FY2RoXDO_P1R0LDDD2twSg1m-huKoocY8w@mail.gmail.com>
Message-ID: <CAO1UhdUqw7e1YZFFNhFWsCCBjZVzYxfJvg0=Ksi+zxPeAGfUHg@mail.gmail.com>

So, what's the best way to create and administer a NDN topology with
hundreds of routers and content servers? Should I use NFD? What routing
protocolo to use? Use IP under or other L2 or L3 protocol? Should I use a
NDNS server to concentrate all names?

Please, help me with these doubts.

Regards.

Tocha.

2014-10-07 5:36 GMT-03:00 Junxiao Shi <shijunxiao at email.arizona.edu>:

> Hi Emerson
>
> NDNLP is a link protocol for NDN. Its implementation is incomplete in NFD.
>
> NLSR doesn't currently support Ethernet faces due to its use of
> `faces/create` API. Only UDP unicast and TCP faces are supported.
>
> Yours, Junxiao
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141007/6090f022/attachment.html>

From iliamo at ucla.edu  Tue Oct  7 09:55:34 2014
From: iliamo at ucla.edu (Ilya Moiseenko)
Date: Tue, 7 Oct 2014 09:55:34 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
In-Reply-To: <CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>
References: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
	<CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>
Message-ID: <C6DACF77-BE7A-4EF8-A6F2-261E4B9FCADE@ucla.edu>


> 1. This reminds me of the discovery protocol that parc talks about.
> metadata for data discovery, maybe?

There is no discovery phase. Name is known a priori, only segment number component is computed dynamically. 

> 2. how do you differentiate metadata from data? contentType?

Yes, I defined a new contentType.

> 3. can we leave /a/b/<id> just for data segments for orthogonality?

Why do you need this orthogonality ?

Ilya

> 
> On Mon, Oct 6, 2014 at 1:21 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>> Hi all,
>> while working on some of the research projects / demos / apps, many people noticed that there is a potential or real benefit of using information objects that do not carry any content, but instead carry useful meta-information about some other Data packets. Right now, these ?supportive? objects are usually called Manifests.
>> 
>> One particular use case of manifests is speeding up signing and verification of multiple Data packets. The speed-up comes from the fact that instead of signing each Data packet using public key crypto methods, the Data packets can be secured by much faster hash computation. The name of each Data packet along with a corresponding hash is placed in the Manifest packet, which is signed with public key crypto.
>> 
>> This scheme makes packet verification faster, because only a manifest packet must be verified using public key crypto methods, and all other data packets listed in the manifest can be verified by recomputing the digest and comparing to the digest specified in the manifest.
>> 
>> One disadvantage of this scheme, is that in general case there is an additional round-trip delay, because manifest needs to be fetched first. (More info in the paper ?Self verifying names for read-only named data?)
>> 
>> I?ve put online a technical report that describes a protocol eliminating round-trip delay for fetching multi-segment content with embedded manifests: http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/
>> 
>> I expect that this approach can be useful for passing around multi-segment content (made dynamically) with roughly 5-10 times faster signing and verification execution over the traditional approach.
>> 
>> If you have comments or counterarguments, let?s have a discussion in this thread.
>> 
>> Thanks
>> Ilya
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest




From tailinchu at gmail.com  Tue Oct  7 15:40:24 2014
From: tailinchu at gmail.com (Tai-Lin Chu)
Date: Tue, 7 Oct 2014 15:40:24 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
In-Reply-To: <C6DACF77-BE7A-4EF8-A6F2-261E4B9FCADE@ucla.edu>
References: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
	<CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>
	<C6DACF77-BE7A-4EF8-A6F2-261E4B9FCADE@ucla.edu>
Message-ID: <CABf5LwB=aZi2ZaJpVA2M1aSPGJ0dw2C0_+O5RqkqeUv7nwg=mg@mail.gmail.com>

sorry for the confusion. metadata can also be used to discover name.
Imagine metadata as a pointer to a list of data.

>Why do you need this orthogonality ?

Orthogonality is always nice to have, and it also makes name matches
what it actually is.

My proposal is based on exact matching but converting to lpm is just
relaxing the constraints. To me, if we add metadata, we don't need
lpm(also selectors because it is highly coupled with lpm) and
FinalBlockId. (exact matching + metadata) not only improves
performance of verification, but also allows us to remove many things.

1. every interest will return one of 3 types of packets: data,
metadata and nack. (still 1-1 relation)
data is used if there is only 1 data packet that matches. metadata is
used when there are more than 1: for either segmentation or children.
nack is for data not found.

2. the metadata contains a list of pair(suffix, digest) for quicker
verification. Note it is just suffix not the full name. Because we
also enforce 1-1 relation and we might have too many suffixes to
segment, this suggests that "metadata about metadata" should be
considered valid. I do not see this in the tr.

3. so what do I envision here? metadata tells consumer a data-fetching
tree, which has internal node as meta data and leaves as data. In most
cases this tree is very shallow but wide. We only need to verify
pubkey once at the root. Because we will know all names in metadata,
this allows parallel fetching. Therefore, finalblockId is then
obsolete.








On Tue, Oct 7, 2014 at 9:55 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>
>> 1. This reminds me of the discovery protocol that parc talks about.
>> metadata for data discovery, maybe?
>
> There is no discovery phase. Name is known a priori, only segment number component is computed dynamically.
>
>> 2. how do you differentiate metadata from data? contentType?
>
> Yes, I defined a new contentType.
>
>> 3. can we leave /a/b/<id> just for data segments for orthogonality?
>
> Why do you need this orthogonality ?
>
> Ilya
>
>>
>> On Mon, Oct 6, 2014 at 1:21 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>> Hi all,
>>> while working on some of the research projects / demos / apps, many people noticed that there is a potential or real benefit of using information objects that do not carry any content, but instead carry useful meta-information about some other Data packets. Right now, these ?supportive? objects are usually called Manifests.
>>>
>>> One particular use case of manifests is speeding up signing and verification of multiple Data packets. The speed-up comes from the fact that instead of signing each Data packet using public key crypto methods, the Data packets can be secured by much faster hash computation. The name of each Data packet along with a corresponding hash is placed in the Manifest packet, which is signed with public key crypto.
>>>
>>> This scheme makes packet verification faster, because only a manifest packet must be verified using public key crypto methods, and all other data packets listed in the manifest can be verified by recomputing the digest and comparing to the digest specified in the manifest.
>>>
>>> One disadvantage of this scheme, is that in general case there is an additional round-trip delay, because manifest needs to be fetched first. (More info in the paper ?Self verifying names for read-only named data?)
>>>
>>> I?ve put online a technical report that describes a protocol eliminating round-trip delay for fetching multi-segment content with embedded manifests: http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/
>>>
>>> I expect that this approach can be useful for passing around multi-segment content (made dynamically) with roughly 5-10 times faster signing and verification execution over the traditional approach.
>>>
>>> If you have comments or counterarguments, let?s have a discussion in this thread.
>>>
>>> Thanks
>>> Ilya
>>> _______________________________________________
>>> Ndn-interest mailing list
>>> Ndn-interest at lists.cs.ucla.edu
>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>



From abannis at ucla.edu  Tue Oct  7 16:27:08 2014
From: abannis at ucla.edu (Adeola Bannis)
Date: Tue, 7 Oct 2014 16:27:08 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
In-Reply-To: <CABf5LwB=aZi2ZaJpVA2M1aSPGJ0dw2C0_+O5RqkqeUv7nwg=mg@mail.gmail.com>
References: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
	<CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>
	<C6DACF77-BE7A-4EF8-A6F2-261E4B9FCADE@ucla.edu>
	<CABf5LwB=aZi2ZaJpVA2M1aSPGJ0dw2C0_+O5RqkqeUv7nwg=mg@mail.gmail.com>
Message-ID: <CAPZn7wgPoHaQDiYrOa4_ds6ng-Nwed2OtDv=5QLfewWgEabwnA@mail.gmail.com>

On Tue, Oct 7, 2014 at 3:40 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:

> sorry for the confusion. metadata can also be used to discover name.
> Imagine metadata as a pointer to a list of data.
>
> >Why do you need this orthogonality ?
>
> Orthogonality is always nice to have, and it also makes name matches
> what it actually is.
>
> My proposal is based on exact matching but converting to lpm is just
> relaxing the constraints. To me, if we add metadata, we don't need
> lpm(also selectors because it is highly coupled with lpm) and
> FinalBlockId. (exact matching + metadata) not only improves
> performance of verification, but also allows us to remove many things.
>
> 1. every interest will return one of 3 types of packets: data,
> metadata and nack. (still 1-1 relation)
> data is used if there is only 1 data packet that matches. metadata is
> used when there are more than 1: for either segmentation or children.
> nack is for data not found.
>

I think manifests are great for segmentation. However, I'm not sure it's
appropriate to use them to list child suffixes.

What if /..../mypage is the name of some data, and /.../mypage/image1 is
also valid data? Does this mean that publishers using metadata can never
have data with names that are prefixes of other data names?


>
> 2. the metadata contains a list of pair(suffix, digest) for quicker
> verification. Note it is just suffix not the full name. Because we
> also enforce 1-1 relation and we might have too many suffixes to
> segment, this suggests that "metadata about metadata" should be
> considered valid. I do not see this in the tr.
>
> 3. so what do I envision here? metadata tells consumer a data-fetching
> tree, which has internal node as meta data and leaves as data. In most
> cases this tree is very shallow but wide. We only need to verify
> pubkey once at the root. Because we will know all names in metadata,
> this allows parallel fetching. Therefore, finalblockId is then
> obsolete.
>
>
If the manifest includes all the segments, then we can get rid of
finalBlockId. However, as I understand Ilya's suggestion for embedded
manifests, a sequence of interests will receive some segments and some
manifests for upcoming segments. Without finalBlockId, a consumer will not
be able to tell a network failure just before the next manifest from the
end of the data.


>
>
>
>
>
>
>
> On Tue, Oct 7, 2014 at 9:55 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
> >
> >> 1. This reminds me of the discovery protocol that parc talks about.
> >> metadata for data discovery, maybe?
> >
> > There is no discovery phase. Name is known a priori, only segment number
> component is computed dynamically.
> >
> >> 2. how do you differentiate metadata from data? contentType?
> >
> > Yes, I defined a new contentType.
> >
> >> 3. can we leave /a/b/<id> just for data segments for orthogonality?
> >
> > Why do you need this orthogonality ?
> >
> > Ilya
> >
> >>
> >> On Mon, Oct 6, 2014 at 1:21 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
> >>> Hi all,
> >>> while working on some of the research projects / demos / apps, many
> people noticed that there is a potential or real benefit of using
> information objects that do not carry any content, but instead carry useful
> meta-information about some other Data packets. Right now, these
> ?supportive? objects are usually called Manifests.
> >>>
> >>> One particular use case of manifests is speeding up signing and
> verification of multiple Data packets. The speed-up comes from the fact
> that instead of signing each Data packet using public key crypto methods,
> the Data packets can be secured by much faster hash computation. The name
> of each Data packet along with a corresponding hash is placed in the
> Manifest packet, which is signed with public key crypto.
> >>>
> >>> This scheme makes packet verification faster, because only a manifest
> packet must be verified using public key crypto methods, and all other data
> packets listed in the manifest can be verified by recomputing the digest
> and comparing to the digest specified in the manifest.
> >>>
> >>> One disadvantage of this scheme, is that in general case there is an
> additional round-trip delay, because manifest needs to be fetched first.
> (More info in the paper ?Self verifying names for read-only named data?)
> >>>
> >>> I?ve put online a technical report that describes a protocol
> eliminating round-trip delay for fetching multi-segment content with
> embedded manifests:
> http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/
> >>>
> >>> I expect that this approach can be useful for passing around
> multi-segment content (made dynamically) with roughly 5-10 times faster
> signing and verification execution over the traditional approach.
> >>>
> >>> If you have comments or counterarguments, let?s have a discussion in
> this thread.
> >>>
> >>> Thanks
> >>> Ilya
> >>> _______________________________________________
> >>> Ndn-interest mailing list
> >>> Ndn-interest at lists.cs.ucla.edu
> >>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
> >
>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141007/fe7c9970/attachment.html>

From tailinchu at gmail.com  Tue Oct  7 17:12:41 2014
From: tailinchu at gmail.com (Tai-Lin Chu)
Date: Tue, 7 Oct 2014 17:12:41 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
In-Reply-To: <CAPZn7wgPoHaQDiYrOa4_ds6ng-Nwed2OtDv=5QLfewWgEabwnA@mail.gmail.com>
References: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
	<CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>
	<C6DACF77-BE7A-4EF8-A6F2-261E4B9FCADE@ucla.edu>
	<CABf5LwB=aZi2ZaJpVA2M1aSPGJ0dw2C0_+O5RqkqeUv7nwg=mg@mail.gmail.com>
	<CAPZn7wgPoHaQDiYrOa4_ds6ng-Nwed2OtDv=5QLfewWgEabwnA@mail.gmail.com>
Message-ID: <CABf5LwBSW+SNfUaE3QPp-Eh1E6ApoWs6ouD_acarbFnj9=08bA@mail.gmail.com>

> What if /..../mypage is the name of some data, and /.../mypage/image1 is also valid data? Does this mean that publishers using metadata can never have data with names that are prefixes of other data names?

1. my proposal is not compatible with current ndn.
2. sorry I did not say precisely. whether mypage is a data or a
metadata (dataset) is decided by producer. It is not "list all
children", so image1 might or might not be in mypage. "For now" (I am
thinking whether it is useful to put data with other prefix into
metadata), every item in metadata must be a suffix.

> If the manifest includes all the segments, then we can get rid of finalBlockId. However, as I understand Ilya's suggestion for embedded manifests, a sequence of interests will receive some segments and some manifests for upcoming segments. Without finalBlockId, a consumer will not be able to tell a network failure just before the next manifest from the end of the data.

Yes. This is what I mean. FinalBlockId only allows sequential
fetching. My proposal allows you to fetch in any order you like
because you know the complete set.

On Tue, Oct 7, 2014 at 4:27 PM, Adeola Bannis <abannis at ucla.edu> wrote:
>
>
> On Tue, Oct 7, 2014 at 3:40 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>>
>> sorry for the confusion. metadata can also be used to discover name.
>> Imagine metadata as a pointer to a list of data.
>>
>> >Why do you need this orthogonality ?
>>
>> Orthogonality is always nice to have, and it also makes name matches
>> what it actually is.
>>
>> My proposal is based on exact matching but converting to lpm is just
>> relaxing the constraints. To me, if we add metadata, we don't need
>> lpm(also selectors because it is highly coupled with lpm) and
>> FinalBlockId. (exact matching + metadata) not only improves
>> performance of verification, but also allows us to remove many things.
>>
>> 1. every interest will return one of 3 types of packets: data,
>> metadata and nack. (still 1-1 relation)
>> data is used if there is only 1 data packet that matches. metadata is
>> used when there are more than 1: for either segmentation or children.
>> nack is for data not found.
>
>
> I think manifests are great for segmentation. However, I'm not sure it's
> appropriate to use them to list child suffixes.
>
> What if /..../mypage is the name of some data, and /.../mypage/image1 is
> also valid data? Does this mean that publishers using metadata can never
> have data with names that are prefixes of other data names?
>
>>
>>
>> 2. the metadata contains a list of pair(suffix, digest) for quicker
>> verification. Note it is just suffix not the full name. Because we
>> also enforce 1-1 relation and we might have too many suffixes to
>> segment, this suggests that "metadata about metadata" should be
>> considered valid. I do not see this in the tr.
>>
>> 3. so what do I envision here? metadata tells consumer a data-fetching
>> tree, which has internal node as meta data and leaves as data. In most
>> cases this tree is very shallow but wide. We only need to verify
>> pubkey once at the root. Because we will know all names in metadata,
>> this allows parallel fetching. Therefore, finalblockId is then
>> obsolete.
>>
>
> If the manifest includes all the segments, then we can get rid of
> finalBlockId. However, as I understand Ilya's suggestion for embedded
> manifests, a sequence of interests will receive some segments and some
> manifests for upcoming segments. Without finalBlockId, a consumer will not
> be able to tell a network failure just before the next manifest from the end
> of the data.
>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Oct 7, 2014 at 9:55 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>> >
>> >> 1. This reminds me of the discovery protocol that parc talks about.
>> >> metadata for data discovery, maybe?
>> >
>> > There is no discovery phase. Name is known a priori, only segment number
>> > component is computed dynamically.
>> >
>> >> 2. how do you differentiate metadata from data? contentType?
>> >
>> > Yes, I defined a new contentType.
>> >
>> >> 3. can we leave /a/b/<id> just for data segments for orthogonality?
>> >
>> > Why do you need this orthogonality ?
>> >
>> > Ilya
>> >
>> >>
>> >> On Mon, Oct 6, 2014 at 1:21 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>> >>> Hi all,
>> >>> while working on some of the research projects / demos / apps, many
>> >>> people noticed that there is a potential or real benefit of using
>> >>> information objects that do not carry any content, but instead carry useful
>> >>> meta-information about some other Data packets. Right now, these
>> >>> ?supportive? objects are usually called Manifests.
>> >>>
>> >>> One particular use case of manifests is speeding up signing and
>> >>> verification of multiple Data packets. The speed-up comes from the fact that
>> >>> instead of signing each Data packet using public key crypto methods, the
>> >>> Data packets can be secured by much faster hash computation. The name of
>> >>> each Data packet along with a corresponding hash is placed in the Manifest
>> >>> packet, which is signed with public key crypto.
>> >>>
>> >>> This scheme makes packet verification faster, because only a manifest
>> >>> packet must be verified using public key crypto methods, and all other data
>> >>> packets listed in the manifest can be verified by recomputing the digest and
>> >>> comparing to the digest specified in the manifest.
>> >>>
>> >>> One disadvantage of this scheme, is that in general case there is an
>> >>> additional round-trip delay, because manifest needs to be fetched first.
>> >>> (More info in the paper ?Self verifying names for read-only named data?)
>> >>>
>> >>> I?ve put online a technical report that describes a protocol
>> >>> eliminating round-trip delay for fetching multi-segment content with
>> >>> embedded manifests:
>> >>> http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/
>> >>>
>> >>> I expect that this approach can be useful for passing around
>> >>> multi-segment content (made dynamically) with roughly 5-10 times faster
>> >>> signing and verification execution over the traditional approach.
>> >>>
>> >>> If you have comments or counterarguments, let?s have a discussion in
>> >>> this thread.
>> >>>
>> >>> Thanks
>> >>> Ilya
>> >>> _______________________________________________
>> >>> Ndn-interest mailing list
>> >>> Ndn-interest at lists.cs.ucla.edu
>> >>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> >
>>
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
>



From iliamo at ucla.edu  Wed Oct  8 10:45:00 2014
From: iliamo at ucla.edu (Ilya Moiseenko)
Date: Wed, 8 Oct 2014 10:45:00 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
In-Reply-To: <CABf5LwB=aZi2ZaJpVA2M1aSPGJ0dw2C0_+O5RqkqeUv7nwg=mg@mail.gmail.com>
References: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
	<CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>
	<C6DACF77-BE7A-4EF8-A6F2-261E4B9FCADE@ucla.edu>
	<CABf5LwB=aZi2ZaJpVA2M1aSPGJ0dw2C0_+O5RqkqeUv7nwg=mg@mail.gmail.com>
Message-ID: <498E129F-3E71-45D4-B761-53EDABF7127B@ucla.edu>


On Oct 7, 2014, at 3:40 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:

> sorry for the confusion. metadata can also be used to discover name.
> Imagine metadata as a pointer to a list of data.
> 
>> Why do you need this orthogonality ?
> 
> Orthogonality is always nice to have,
Why is it nice to have?

> and it also makes name matches
> what it actually is.

In my opinion, it makes packet reassembly much more difficult.
When manifests are embedded in the sequence, and a normal Data packet is received, reassembly knows where to look for the corresponding manifest (just look for the first preceding manifest). 

And when you have a separate enumeration for Data packets and manifests, it?s impossible to know which manifest corresponds to a particular Data packet. (I?m not going to check every manifest, of course). Or you need some backlink from the Data packet towards the corresponding manifest. I don?t think that it is a good design.

> 
> My proposal is based on exact matching but converting to lpm is just
> relaxing the constraints. To me, if we add metadata, we don't need
> lpm(also selectors because it is highly coupled with lpm) and
> FinalBlockId. (exact matching + metadata) not only improves
> performance of verification, but also allows us to remove many things.

FinalBlockId is a great thing, don?t remove it :) When I perform segmentation, it?s often not possible to know what the last sequence number is going to be until near the end of segmentation. So when I finally know this number, I put it in the FinalBlockID. Consumer will stop fetching when this number is read from any Data packet.

And I use implicit digest selector for interest retransmission when received Data packet fails verification (with manifest).

> 
> 1. every interest will return one of 3 types of packets: data,
> metadata and nack. (still 1-1 relation)
> data is used if there is only 1 data packet that matches. metadata is
> used when there are more than 1: for either segmentation or children.
> nack is for data not found.
> 
> 2. the metadata contains a list of pair(suffix, digest) for quicker
> verification. Note it is just suffix not the full name. Because we
> also enforce 1-1 relation and we might have too many suffixes to
> segment, this suggests that "metadata about metadata" should be
> considered valid. I do not see this in the tr.

In general, I think there should be only full names in the manifest.
Because if we use suffixes, there could be some nasty security attacks via suffix manipulation.
Especially when we have longest prefix match.

> 
> 3. so what do I envision here? metadata tells consumer a data-fetching
> tree, which has internal node as meta data and leaves as data. In most
> cases this tree is very shallow but wide. We only need to verify
> pubkey once at the root. Because we will know all names in metadata,
> this allows parallel fetching. Therefore, finalblockId is then
> obsolete.
> 
> 
> 
> 
> 
> 
> 
> 
> On Tue, Oct 7, 2014 at 9:55 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>> 
>>> 1. This reminds me of the discovery protocol that parc talks about.
>>> metadata for data discovery, maybe?
>> 
>> There is no discovery phase. Name is known a priori, only segment number component is computed dynamically.
>> 
>>> 2. how do you differentiate metadata from data? contentType?
>> 
>> Yes, I defined a new contentType.
>> 
>>> 3. can we leave /a/b/<id> just for data segments for orthogonality?
>> 
>> Why do you need this orthogonality ?
>> 
>> Ilya
>> 
>>> 
>>> On Mon, Oct 6, 2014 at 1:21 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>>> Hi all,
>>>> while working on some of the research projects / demos / apps, many people noticed that there is a potential or real benefit of using information objects that do not carry any content, but instead carry useful meta-information about some other Data packets. Right now, these ?supportive? objects are usually called Manifests.
>>>> 
>>>> One particular use case of manifests is speeding up signing and verification of multiple Data packets. The speed-up comes from the fact that instead of signing each Data packet using public key crypto methods, the Data packets can be secured by much faster hash computation. The name of each Data packet along with a corresponding hash is placed in the Manifest packet, which is signed with public key crypto.
>>>> 
>>>> This scheme makes packet verification faster, because only a manifest packet must be verified using public key crypto methods, and all other data packets listed in the manifest can be verified by recomputing the digest and comparing to the digest specified in the manifest.
>>>> 
>>>> One disadvantage of this scheme, is that in general case there is an additional round-trip delay, because manifest needs to be fetched first. (More info in the paper ?Self verifying names for read-only named data?)
>>>> 
>>>> I?ve put online a technical report that describes a protocol eliminating round-trip delay for fetching multi-segment content with embedded manifests: http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/
>>>> 
>>>> I expect that this approach can be useful for passing around multi-segment content (made dynamically) with roughly 5-10 times faster signing and verification execution over the traditional approach.
>>>> 
>>>> If you have comments or counterarguments, let?s have a discussion in this thread.
>>>> 
>>>> Thanks
>>>> Ilya
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> 




From nicholas.h.briggs at gmail.com  Wed Oct  8 11:12:18 2014
From: nicholas.h.briggs at gmail.com (Nick Briggs)
Date: Wed, 8 Oct 2014 11:12:18 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
In-Reply-To: <498E129F-3E71-45D4-B761-53EDABF7127B@ucla.edu>
References: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
	<CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>
	<C6DACF77-BE7A-4EF8-A6F2-261E4B9FCADE@ucla.edu>
	<CABf5LwB=aZi2ZaJpVA2M1aSPGJ0dw2C0_+O5RqkqeUv7nwg=mg@mail.gmail.com>
	<498E129F-3E71-45D4-B761-53EDABF7127B@ucla.edu>
Message-ID: <D619A197-80DE-419B-A640-4A48325A8AD0@gmail.com>


On Oct 8, 2014, at 10:45 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:

> 
> On Oct 7, 2014, at 3:40 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
> 
>> sorry for the confusion. metadata can also be used to discover name.
>> Imagine metadata as a pointer to a list of data.
>> 
>>> Why do you need this orthogonality ?
>> 
>> Orthogonality is always nice to have,
> Why is it nice to have?
> 
>> and it also makes name matches
>> what it actually is.
> 
> In my opinion, it makes packet reassembly much more difficult.
> When manifests are embedded in the sequence, and a normal Data packet is received, reassembly knows where to look for the corresponding manifest (just look for the first preceding manifest). 
> 
> And when you have a separate enumeration for Data packets and manifests, it?s impossible to know which manifest corresponds to a particular Data packet. (I?m not going to check every manifest, of course). Or you need some backlink from the Data packet towards the corresponding manifest. I don?t think that it is a good design.
> 
>> 
>> My proposal is based on exact matching but converting to lpm is just
>> relaxing the constraints. To me, if we add metadata, we don't need
>> lpm(also selectors because it is highly coupled with lpm) and
>> FinalBlockId. (exact matching + metadata) not only improves
>> performance of verification, but also allows us to remove many things.
> 
> FinalBlockId is a great thing, don?t remove it :) When I perform segmentation, it?s often not possible to know what the last sequence number is going to be until near the end of segmentation. So when I finally know this number, I put it in the FinalBlockID. Consumer will stop fetching when this number is read from any Data packet.

The original definition of FinalBlockID was that the consumer should take the FinalBlockID segment number as a hint and therefore may choose not to send Interests for segments beyond that, but if it reaches the FinalBlockID segment previously identified and it does NOT have the FinalBlockID pointing to itself (either it has none, or it refers to some future segment) then it should continue fetching because it has not reached the end of the stream.

> 
> And I use implicit digest selector for interest retransmission when received Data packet fails verification (with manifest).
> 
>> 
>> 1. every interest will return one of 3 types of packets: data,
>> metadata and nack. (still 1-1 relation)
>> data is used if there is only 1 data packet that matches. metadata is
>> used when there are more than 1: for either segmentation or children.
>> nack is for data not found.
>> 
>> 2. the metadata contains a list of pair(suffix, digest) for quicker
>> verification. Note it is just suffix not the full name. Because we
>> also enforce 1-1 relation and we might have too many suffixes to
>> segment, this suggests that "metadata about metadata" should be
>> considered valid. I do not see this in the tr.
> 
> In general, I think there should be only full names in the manifest.
> Because if we use suffixes, there could be some nasty security attacks via suffix manipulation.
> Especially when we have longest prefix match.
> 
>> 
>> 3. so what do I envision here? metadata tells consumer a data-fetching
>> tree, which has internal node as meta data and leaves as data. In most
>> cases this tree is very shallow but wide. We only need to verify
>> pubkey once at the root. Because we will know all names in metadata,
>> this allows parallel fetching. Therefore, finalblockId is then
>> obsolete.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Tue, Oct 7, 2014 at 9:55 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>> 
>>>> 1. This reminds me of the discovery protocol that parc talks about.
>>>> metadata for data discovery, maybe?
>>> 
>>> There is no discovery phase. Name is known a priori, only segment number component is computed dynamically.
>>> 
>>>> 2. how do you differentiate metadata from data? contentType?
>>> 
>>> Yes, I defined a new contentType.
>>> 
>>>> 3. can we leave /a/b/<id> just for data segments for orthogonality?
>>> 
>>> Why do you need this orthogonality ?
>>> 
>>> Ilya
>>> 
>>>> 
>>>> On Mon, Oct 6, 2014 at 1:21 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>>>> Hi all,
>>>>> while working on some of the research projects / demos / apps, many people noticed that there is a potential or real benefit of using information objects that do not carry any content, but instead carry useful meta-information about some other Data packets. Right now, these ?supportive? objects are usually called Manifests.
>>>>> 
>>>>> One particular use case of manifests is speeding up signing and verification of multiple Data packets. The speed-up comes from the fact that instead of signing each Data packet using public key crypto methods, the Data packets can be secured by much faster hash computation. The name of each Data packet along with a corresponding hash is placed in the Manifest packet, which is signed with public key crypto.
>>>>> 
>>>>> This scheme makes packet verification faster, because only a manifest packet must be verified using public key crypto methods, and all other data packets listed in the manifest can be verified by recomputing the digest and comparing to the digest specified in the manifest.
>>>>> 
>>>>> One disadvantage of this scheme, is that in general case there is an additional round-trip delay, because manifest needs to be fetched first. (More info in the paper ?Self verifying names for read-only named data?)
>>>>> 
>>>>> I?ve put online a technical report that describes a protocol eliminating round-trip delay for fetching multi-segment content with embedded manifests: http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/
>>>>> 
>>>>> I expect that this approach can be useful for passing around multi-segment content (made dynamically) with roughly 5-10 times faster signing and verification execution over the traditional approach.
>>>>> 
>>>>> If you have comments or counterarguments, let?s have a discussion in this thread.
>>>>> 
>>>>> Thanks
>>>>> Ilya
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>> 
> 
> 
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest




From tailinchu at gmail.com  Wed Oct  8 14:07:07 2014
From: tailinchu at gmail.com (Tai-Lin Chu)
Date: Wed, 8 Oct 2014 14:07:07 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
In-Reply-To: <498E129F-3E71-45D4-B761-53EDABF7127B@ucla.edu>
References: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
	<CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>
	<C6DACF77-BE7A-4EF8-A6F2-261E4B9FCADE@ucla.edu>
	<CABf5LwB=aZi2ZaJpVA2M1aSPGJ0dw2C0_+O5RqkqeUv7nwg=mg@mail.gmail.com>
	<498E129F-3E71-45D4-B761-53EDABF7127B@ucla.edu>
Message-ID: <CABf5LwB6j80DhSa56dx51NDwFBwNEKP+oOcE2Hht_xpeU6pLoQ@mail.gmail.com>

> In my opinion, it makes packet reassembly much more difficult.

I think it is very simple to implement in recursion: for every node,
fetch all children in parallel, and then check digest if data returns.


>When I perform segmentation, it?s often not possible to know what the last sequence number is going to be until near the end of segmentation.
yes, but my version of metadata does not prevent you from doing
"deferred" final block.
Lets see an example.

metadata
/id/1 (data)
/id/2 (data)
...
/<uuid> (deferred metadata, which is also similar to this one; verify
rsa if you don't provide digest)


> suffix manipulation
could you explain more?

On Wed, Oct 8, 2014 at 10:45 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>
> On Oct 7, 2014, at 3:40 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>
>> sorry for the confusion. metadata can also be used to discover name.
>> Imagine metadata as a pointer to a list of data.
>>
>>> Why do you need this orthogonality ?
>>
>> Orthogonality is always nice to have,
> Why is it nice to have?
>
>> and it also makes name matches
>> what it actually is.
>
> In my opinion, it makes packet reassembly much more difficult.
> When manifests are embedded in the sequence, and a normal Data packet is received, reassembly knows where to look for the corresponding manifest (just look for the first preceding manifest).
>
> And when you have a separate enumeration for Data packets and manifests, it?s impossible to know which manifest corresponds to a particular Data packet. (I?m not going to check every manifest, of course). Or you need some backlink from the Data packet towards the corresponding manifest. I don?t think that it is a good design.
>
>>
>> My proposal is based on exact matching but converting to lpm is just
>> relaxing the constraints. To me, if we add metadata, we don't need
>> lpm(also selectors because it is highly coupled with lpm) and
>> FinalBlockId. (exact matching + metadata) not only improves
>> performance of verification, but also allows us to remove many things.
>
> FinalBlockId is a great thing, don?t remove it :) When I perform segmentation, it?s often not possible to know what the last sequence number is going to be until near the end of segmentation. So when I finally know this number, I put it in the FinalBlockID. Consumer will stop fetching when this number is read from any Data packet.
>
> And I use implicit digest selector for interest retransmission when received Data packet fails verification (with manifest).
>
>>
>> 1. every interest will return one of 3 types of packets: data,
>> metadata and nack. (still 1-1 relation)
>> data is used if there is only 1 data packet that matches. metadata is
>> used when there are more than 1: for either segmentation or children.
>> nack is for data not found.
>>
>> 2. the metadata contains a list of pair(suffix, digest) for quicker
>> verification. Note it is just suffix not the full name. Because we
>> also enforce 1-1 relation and we might have too many suffixes to
>> segment, this suggests that "metadata about metadata" should be
>> considered valid. I do not see this in the tr.
>
> In general, I think there should be only full names in the manifest.
> Because if we use suffixes, there could be some nasty security attacks via suffix manipulation.
> Especially when we have longest prefix match.
>
>>
>> 3. so what do I envision here? metadata tells consumer a data-fetching
>> tree, which has internal node as meta data and leaves as data. In most
>> cases this tree is very shallow but wide. We only need to verify
>> pubkey once at the root. Because we will know all names in metadata,
>> this allows parallel fetching. Therefore, finalblockId is then
>> obsolete.
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Oct 7, 2014 at 9:55 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>>
>>>> 1. This reminds me of the discovery protocol that parc talks about.
>>>> metadata for data discovery, maybe?
>>>
>>> There is no discovery phase. Name is known a priori, only segment number component is computed dynamically.
>>>
>>>> 2. how do you differentiate metadata from data? contentType?
>>>
>>> Yes, I defined a new contentType.
>>>
>>>> 3. can we leave /a/b/<id> just for data segments for orthogonality?
>>>
>>> Why do you need this orthogonality ?
>>>
>>> Ilya
>>>
>>>>
>>>> On Mon, Oct 6, 2014 at 1:21 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>>>> Hi all,
>>>>> while working on some of the research projects / demos / apps, many people noticed that there is a potential or real benefit of using information objects that do not carry any content, but instead carry useful meta-information about some other Data packets. Right now, these ?supportive? objects are usually called Manifests.
>>>>>
>>>>> One particular use case of manifests is speeding up signing and verification of multiple Data packets. The speed-up comes from the fact that instead of signing each Data packet using public key crypto methods, the Data packets can be secured by much faster hash computation. The name of each Data packet along with a corresponding hash is placed in the Manifest packet, which is signed with public key crypto.
>>>>>
>>>>> This scheme makes packet verification faster, because only a manifest packet must be verified using public key crypto methods, and all other data packets listed in the manifest can be verified by recomputing the digest and comparing to the digest specified in the manifest.
>>>>>
>>>>> One disadvantage of this scheme, is that in general case there is an additional round-trip delay, because manifest needs to be fetched first. (More info in the paper ?Self verifying names for read-only named data?)
>>>>>
>>>>> I?ve put online a technical report that describes a protocol eliminating round-trip delay for fetching multi-segment content with embedded manifests: http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/
>>>>>
>>>>> I expect that this approach can be useful for passing around multi-segment content (made dynamically) with roughly 5-10 times faster signing and verification execution over the traditional approach.
>>>>>
>>>>> If you have comments or counterarguments, let?s have a discussion in this thread.
>>>>>
>>>>> Thanks
>>>>> Ilya
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>
>



From iliamo at UCLA.EDU  Wed Oct  8 14:29:11 2014
From: iliamo at UCLA.EDU (Ilya Moiseenko)
Date: Wed, 8 Oct 2014 14:29:11 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
In-Reply-To: <CABf5LwB6j80DhSa56dx51NDwFBwNEKP+oOcE2Hht_xpeU6pLoQ@mail.gmail.com>
References: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
	<CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>
	<C6DACF77-BE7A-4EF8-A6F2-261E4B9FCADE@ucla.edu>
	<CABf5LwB=aZi2ZaJpVA2M1aSPGJ0dw2C0_+O5RqkqeUv7nwg=mg@mail.gmail.com>
	<498E129F-3E71-45D4-B761-53EDABF7127B@ucla.edu>
	<CABf5LwB6j80DhSa56dx51NDwFBwNEKP+oOcE2Hht_xpeU6pLoQ@mail.gmail.com>
Message-ID: <F8FAF521-2D24-4894-973E-8CD0A0FA33B3@ucla.edu>


>> In my opinion, it makes packet reassembly much more difficult.
> 
> I think it is very simple to implement in recursion: for every node,
> fetch all children in parallel, and then check digest if data returns.

So this introduces head of line blocking. In other words, you have to receive a manifest before you can fetch any of the referenced Data packets (children).
Modern protocols try to avoid this.

> 
> 
>> When I perform segmentation, it?s often not possible to know what the last sequence number is going to be until near the end of segmentation.
> yes, but my version of metadata does not prevent you from doing
> "deferred" final block.
> Lets see an example.
> 
> metadata
> /id/1 (data)
> /id/2 (data)
> ...
> /<uuid> (deferred metadata, which is also similar to this one; verify
> rsa if you don't provide digest)

Many things are possible, I just think it?s not a good design to look inside the payload of manifest packet to infer the final segment number. 
It?s faster and easier to get it from the packet header.

> 
> 
>> suffix manipulation
> could you explain more?

Sort of problems similar to relative URLs. 
I think it?s possible to block the data transfer by manipulating with names and digests, so that protocol will give up on retransmissions.

There should be a strict full name to digest binding, until somebody does a security analysis of possibility of this optimization (with suffixes)

 
> 
> On Wed, Oct 8, 2014 at 10:45 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>> 
>> On Oct 7, 2014, at 3:40 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>> 
>>> sorry for the confusion. metadata can also be used to discover name.
>>> Imagine metadata as a pointer to a list of data.
>>> 
>>>> Why do you need this orthogonality ?
>>> 
>>> Orthogonality is always nice to have,
>> Why is it nice to have?
>> 
>>> and it also makes name matches
>>> what it actually is.
>> 
>> In my opinion, it makes packet reassembly much more difficult.
>> When manifests are embedded in the sequence, and a normal Data packet is received, reassembly knows where to look for the corresponding manifest (just look for the first preceding manifest).
>> 
>> And when you have a separate enumeration for Data packets and manifests, it?s impossible to know which manifest corresponds to a particular Data packet. (I?m not going to check every manifest, of course). Or you need some backlink from the Data packet towards the corresponding manifest. I don?t think that it is a good design.
>> 
>>> 
>>> My proposal is based on exact matching but converting to lpm is just
>>> relaxing the constraints. To me, if we add metadata, we don't need
>>> lpm(also selectors because it is highly coupled with lpm) and
>>> FinalBlockId. (exact matching + metadata) not only improves
>>> performance of verification, but also allows us to remove many things.
>> 
>> FinalBlockId is a great thing, don?t remove it :) When I perform segmentation, it?s often not possible to know what the last sequence number is going to be until near the end of segmentation. So when I finally know this number, I put it in the FinalBlockID. Consumer will stop fetching when this number is read from any Data packet.
>> 
>> And I use implicit digest selector for interest retransmission when received Data packet fails verification (with manifest).
>> 
>>> 
>>> 1. every interest will return one of 3 types of packets: data,
>>> metadata and nack. (still 1-1 relation)
>>> data is used if there is only 1 data packet that matches. metadata is
>>> used when there are more than 1: for either segmentation or children.
>>> nack is for data not found.
>>> 
>>> 2. the metadata contains a list of pair(suffix, digest) for quicker
>>> verification. Note it is just suffix not the full name. Because we
>>> also enforce 1-1 relation and we might have too many suffixes to
>>> segment, this suggests that "metadata about metadata" should be
>>> considered valid. I do not see this in the tr.
>> 
>> In general, I think there should be only full names in the manifest.
>> Because if we use suffixes, there could be some nasty security attacks via suffix manipulation.
>> Especially when we have longest prefix match.
>> 
>>> 
>>> 3. so what do I envision here? metadata tells consumer a data-fetching
>>> tree, which has internal node as meta data and leaves as data. In most
>>> cases this tree is very shallow but wide. We only need to verify
>>> pubkey once at the root. Because we will know all names in metadata,
>>> this allows parallel fetching. Therefore, finalblockId is then
>>> obsolete.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Tue, Oct 7, 2014 at 9:55 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>>> 
>>>>> 1. This reminds me of the discovery protocol that parc talks about.
>>>>> metadata for data discovery, maybe?
>>>> 
>>>> There is no discovery phase. Name is known a priori, only segment number component is computed dynamically.
>>>> 
>>>>> 2. how do you differentiate metadata from data? contentType?
>>>> 
>>>> Yes, I defined a new contentType.
>>>> 
>>>>> 3. can we leave /a/b/<id> just for data segments for orthogonality?
>>>> 
>>>> Why do you need this orthogonality ?
>>>> 
>>>> Ilya
>>>> 
>>>>> 
>>>>> On Mon, Oct 6, 2014 at 1:21 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>>>>> Hi all,
>>>>>> while working on some of the research projects / demos / apps, many people noticed that there is a potential or real benefit of using information objects that do not carry any content, but instead carry useful meta-information about some other Data packets. Right now, these ?supportive? objects are usually called Manifests.
>>>>>> 
>>>>>> One particular use case of manifests is speeding up signing and verification of multiple Data packets. The speed-up comes from the fact that instead of signing each Data packet using public key crypto methods, the Data packets can be secured by much faster hash computation. The name of each Data packet along with a corresponding hash is placed in the Manifest packet, which is signed with public key crypto.
>>>>>> 
>>>>>> This scheme makes packet verification faster, because only a manifest packet must be verified using public key crypto methods, and all other data packets listed in the manifest can be verified by recomputing the digest and comparing to the digest specified in the manifest.
>>>>>> 
>>>>>> One disadvantage of this scheme, is that in general case there is an additional round-trip delay, because manifest needs to be fetched first. (More info in the paper ?Self verifying names for read-only named data?)
>>>>>> 
>>>>>> I?ve put online a technical report that describes a protocol eliminating round-trip delay for fetching multi-segment content with embedded manifests: http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/
>>>>>> 
>>>>>> I expect that this approach can be useful for passing around multi-segment content (made dynamically) with roughly 5-10 times faster signing and verification execution over the traditional approach.
>>>>>> 
>>>>>> If you have comments or counterarguments, let?s have a discussion in this thread.
>>>>>> 
>>>>>> Thanks
>>>>>> Ilya
>>>>>> _______________________________________________
>>>>>> Ndn-interest mailing list
>>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>> 




From tailinchu at gmail.com  Wed Oct  8 20:56:36 2014
From: tailinchu at gmail.com (Tai-Lin Chu)
Date: Wed, 8 Oct 2014 20:56:36 -0700
Subject: [Ndn-interest] use cases of manifest objects in NDN
In-Reply-To: <F8FAF521-2D24-4894-973E-8CD0A0FA33B3@ucla.edu>
References: <FF91E55B-E1E8-4590-A8C4-3D96909674C1@ucla.edu>
	<CABf5LwBa23Tx1heUPj3-O6eP_Qi+ip9zkkrtJLEM29s4__L9yw@mail.gmail.com>
	<C6DACF77-BE7A-4EF8-A6F2-261E4B9FCADE@ucla.edu>
	<CABf5LwB=aZi2ZaJpVA2M1aSPGJ0dw2C0_+O5RqkqeUv7nwg=mg@mail.gmail.com>
	<498E129F-3E71-45D4-B761-53EDABF7127B@ucla.edu>
	<CABf5LwB6j80DhSa56dx51NDwFBwNEKP+oOcE2Hht_xpeU6pLoQ@mail.gmail.com>
	<F8FAF521-2D24-4894-973E-8CD0A0FA33B3@ucla.edu>
Message-ID: <CABf5LwCueg2ATTefy2WTSnUHm30SxHpM6H9wK11p6ufnOAxa3g@mail.gmail.com>

> So this introduces head of line blocking.

I don't think this is "HOL blocking". Anyway, metadata gives both name
and digest so that you can retrieve them in any order. This actually
prevents blocking. If we use finalBlockId, we will need to fetch the
current packet to see whether it is indeed the final block; this
forces us to fetch them sequentially.

> look inside the payload of manifest packet to infer the final segment number
Actually we don't have the idea of "final" anymore; we just fetch all
items in the set in any order.

> It?s faster and easier to get it from the packet header.
header?!

> I think it?s possible to block the data transfer by manipulating with names and digests, so that protocol will give up on retransmissions.
could you elaborate?

>There should be a strict full name to digest binding, until somebody does a security analysis of possibility of this optimization (with suffixes)

assuming that digest is generated with one way function, authenticity
of ANY packet can be verified. Suffix and full name have no
differences under this assumption. I propose suffix because I actually
cannot see the use case where we will put a completely different
prefix in the same metadata.


On Wed, Oct 8, 2014 at 2:29 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>
>>> In my opinion, it makes packet reassembly much more difficult.
>>
>> I think it is very simple to implement in recursion: for every node,
>> fetch all children in parallel, and then check digest if data returns.
>
> So this introduces head of line blocking. In other words, you have to receive a manifest before you can fetch any of the referenced Data packets (children).
> Modern protocols try to avoid this.
>
>>
>>
>>> When I perform segmentation, it?s often not possible to know what the last sequence number is going to be until near the end of segmentation.
>> yes, but my version of metadata does not prevent you from doing
>> "deferred" final block.
>> Lets see an example.
>>
>> metadata
>> /id/1 (data)
>> /id/2 (data)
>> ...
>> /<uuid> (deferred metadata, which is also similar to this one; verify
>> rsa if you don't provide digest)
>
> Many things are possible, I just think it?s not a good design to look inside the payload of manifest packet to infer the final segment number.
> It?s faster and easier to get it from the packet header.
>
>>
>>
>>> suffix manipulation
>> could you explain more?
>
> Sort of problems similar to relative URLs.
> I think it?s possible to block the data transfer by manipulating with names and digests, so that protocol will give up on retransmissions.
>
> There should be a strict full name to digest binding, until somebody does a security analysis of possibility of this optimization (with suffixes)
>
>
>>
>> On Wed, Oct 8, 2014 at 10:45 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>>
>>> On Oct 7, 2014, at 3:40 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>>>
>>>> sorry for the confusion. metadata can also be used to discover name.
>>>> Imagine metadata as a pointer to a list of data.
>>>>
>>>>> Why do you need this orthogonality ?
>>>>
>>>> Orthogonality is always nice to have,
>>> Why is it nice to have?
>>>
>>>> and it also makes name matches
>>>> what it actually is.
>>>
>>> In my opinion, it makes packet reassembly much more difficult.
>>> When manifests are embedded in the sequence, and a normal Data packet is received, reassembly knows where to look for the corresponding manifest (just look for the first preceding manifest).
>>>
>>> And when you have a separate enumeration for Data packets and manifests, it?s impossible to know which manifest corresponds to a particular Data packet. (I?m not going to check every manifest, of course). Or you need some backlink from the Data packet towards the corresponding manifest. I don?t think that it is a good design.
>>>
>>>>
>>>> My proposal is based on exact matching but converting to lpm is just
>>>> relaxing the constraints. To me, if we add metadata, we don't need
>>>> lpm(also selectors because it is highly coupled with lpm) and
>>>> FinalBlockId. (exact matching + metadata) not only improves
>>>> performance of verification, but also allows us to remove many things.
>>>
>>> FinalBlockId is a great thing, don?t remove it :) When I perform segmentation, it?s often not possible to know what the last sequence number is going to be until near the end of segmentation. So when I finally know this number, I put it in the FinalBlockID. Consumer will stop fetching when this number is read from any Data packet.
>>>
>>> And I use implicit digest selector for interest retransmission when received Data packet fails verification (with manifest).
>>>
>>>>
>>>> 1. every interest will return one of 3 types of packets: data,
>>>> metadata and nack. (still 1-1 relation)
>>>> data is used if there is only 1 data packet that matches. metadata is
>>>> used when there are more than 1: for either segmentation or children.
>>>> nack is for data not found.
>>>>
>>>> 2. the metadata contains a list of pair(suffix, digest) for quicker
>>>> verification. Note it is just suffix not the full name. Because we
>>>> also enforce 1-1 relation and we might have too many suffixes to
>>>> segment, this suggests that "metadata about metadata" should be
>>>> considered valid. I do not see this in the tr.
>>>
>>> In general, I think there should be only full names in the manifest.
>>> Because if we use suffixes, there could be some nasty security attacks via suffix manipulation.
>>> Especially when we have longest prefix match.
>>>
>>>>
>>>> 3. so what do I envision here? metadata tells consumer a data-fetching
>>>> tree, which has internal node as meta data and leaves as data. In most
>>>> cases this tree is very shallow but wide. We only need to verify
>>>> pubkey once at the root. Because we will know all names in metadata,
>>>> this allows parallel fetching. Therefore, finalblockId is then
>>>> obsolete.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Oct 7, 2014 at 9:55 AM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>>>>
>>>>>> 1. This reminds me of the discovery protocol that parc talks about.
>>>>>> metadata for data discovery, maybe?
>>>>>
>>>>> There is no discovery phase. Name is known a priori, only segment number component is computed dynamically.
>>>>>
>>>>>> 2. how do you differentiate metadata from data? contentType?
>>>>>
>>>>> Yes, I defined a new contentType.
>>>>>
>>>>>> 3. can we leave /a/b/<id> just for data segments for orthogonality?
>>>>>
>>>>> Why do you need this orthogonality ?
>>>>>
>>>>> Ilya
>>>>>
>>>>>>
>>>>>> On Mon, Oct 6, 2014 at 1:21 PM, Ilya Moiseenko <iliamo at ucla.edu> wrote:
>>>>>>> Hi all,
>>>>>>> while working on some of the research projects / demos / apps, many people noticed that there is a potential or real benefit of using information objects that do not carry any content, but instead carry useful meta-information about some other Data packets. Right now, these ?supportive? objects are usually called Manifests.
>>>>>>>
>>>>>>> One particular use case of manifests is speeding up signing and verification of multiple Data packets. The speed-up comes from the fact that instead of signing each Data packet using public key crypto methods, the Data packets can be secured by much faster hash computation. The name of each Data packet along with a corresponding hash is placed in the Manifest packet, which is signed with public key crypto.
>>>>>>>
>>>>>>> This scheme makes packet verification faster, because only a manifest packet must be verified using public key crypto methods, and all other data packets listed in the manifest can be verified by recomputing the digest and comparing to the digest specified in the manifest.
>>>>>>>
>>>>>>> One disadvantage of this scheme, is that in general case there is an additional round-trip delay, because manifest needs to be fetched first. (More info in the paper ?Self verifying names for read-only named data?)
>>>>>>>
>>>>>>> I?ve put online a technical report that describes a protocol eliminating round-trip delay for fetching multi-segment content with embedded manifests: http://named-data.net/publications/techreports/ndn-tr-25-manifest-embedding/
>>>>>>>
>>>>>>> I expect that this approach can be useful for passing around multi-segment content (made dynamically) with roughly 5-10 times faster signing and verification execution over the traditional approach.
>>>>>>>
>>>>>>> If you have comments or counterarguments, let?s have a discussion in this thread.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Ilya
>>>>>>> _______________________________________________
>>>>>>> Ndn-interest mailing list
>>>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>>
>>>
>



From marco.ulge at gmail.com  Wed Oct 15 08:01:53 2014
From: marco.ulge at gmail.com (Marco Ulgelmo)
Date: Wed, 15 Oct 2014 17:01:53 +0200
Subject: [Ndn-interest] Question about unsatisfied Interests
Message-ID: <CA+=+rrACDbQuMFZbSVexMb3sAHJy7yfzyvJA=yDCbZUynJCbpQ@mail.gmail.com>

Hello,

I've recently began studying how NDN works and i have a question about
unsatisfied Interests.

Lets say that an attacker generates a fake Interest packet that surely
doesn't exist. Let's assume that this packet is identified by the following
name: /google/randomtext

As far as i know i shuold be able to reach /google but i won't be able to
retrieve the content as it doesn't exist.

Considering the following scenario:

Interest -> Router1 -> Router2 -> Content Provider.

Is Router2 able to determine that the content doesn't exist? Can this
information be sent back to Router2 from the Content Provider (which surely
knows that he doesn't have that resource)?

Thanks,

Marco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/160f881e/attachment.html>

From shijunxiao at email.arizona.edu  Wed Oct 15 08:13:57 2014
From: shijunxiao at email.arizona.edu (Junxiao Shi)
Date: Wed, 15 Oct 2014 08:13:57 -0700
Subject: [Ndn-interest] Question about unsatisfied Interests
In-Reply-To: <CA+=+rrACDbQuMFZbSVexMb3sAHJy7yfzyvJA=yDCbZUynJCbpQ@mail.gmail.com>
References: <CA+=+rrACDbQuMFZbSVexMb3sAHJy7yfzyvJA=yDCbZUynJCbpQ@mail.gmail.com>
Message-ID: <CAOFH+OYh-HcDzyfX4KCh+tCrkfTg_pF9J8LmGXeYJV+LE+NM4w@mail.gmail.com>

Hi Marco

If Router2 has a Route toward the ContentProvider for this Name, it cannot
detect the content doesn't exist.

ContentProvider should reply with a provider-generated NACK, which is a
type of Data, that indicates the non-existence of this content. A
provider-generated NACK is cachable on routers.
A future Interest toward this Name would be satisifed by this
provider-generated NACK from the ContentStore, and doesn't have to go to
the ContentProvider.

Yours, Junxiao

On Wed, Oct 15, 2014 at 8:01 AM, Marco Ulgelmo <marco.ulge at gmail.com> wrote:

> Hello,
>
> I've recently began studying how NDN works and i have a question about
> unsatisfied Interests.
>
> Lets say that an attacker generates a fake Interest packet that surely
> doesn't exist. Let's assume that this packet is identified by the following
> name: /google/randomtext
>
> As far as i know i shuold be able to reach /google but i won't be able to
> retrieve the content as it doesn't exist.
>
> Considering the following scenario:
>
> Interest -> Router1 -> Router2 -> Content Provider.
>
> Is Router2 able to determine that the content doesn't exist? Can this
> information be sent back to Router2 from the Content Provider (which surely
> knows that he doesn't have that resource)?
>
> Thanks,
>
> Marco
>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/e845929b/attachment.html>

From shock.jiang at gmail.com  Wed Oct 15 08:20:15 2014
From: shock.jiang at gmail.com (Xiaoke Jiang)
Date: Wed, 15 Oct 2014 08:20:15 -0700
Subject: [Ndn-interest] Question about unsatisfied Interests
In-Reply-To: <CAOFH+OYh-HcDzyfX4KCh+tCrkfTg_pF9J8LmGXeYJV+LE+NM4w@mail.gmail.com>
References: <CA+=+rrACDbQuMFZbSVexMb3sAHJy7yfzyvJA=yDCbZUynJCbpQ@mail.gmail.com>
	<CAOFH+OYh-HcDzyfX4KCh+tCrkfTg_pF9J8LmGXeYJV+LE+NM4w@mail.gmail.com>
Message-ID: <703DB44C99104A2F9F5D80B63AFD1A0A@gmail.com>

Hi Marco,
   This paper   http://named-data.net/wp-content/uploads/comcom-stateful-forwarding.pdf give more details on how stateful data plane helps to get rid of some kinds of attacher.





On Wednesday, 15 October, 2014 at 8:13 am, Junxiao Shi wrote:

> Hi Marco
> 
> If Router2 has a Route toward the ContentProvider for this Name, it cannot detect the content doesn't exist.
> 
> ContentProvider should reply with a provider-generated NACK, which is a type of Data, that indicates the non-existence of this content. A provider-generated NACK is cachable on routers.
> A future Interest toward this Name would be satisifed by this provider-generated NACK from the ContentStore, and doesn't have to go to the ContentProvider.
> 
> Yours, Junxiao
> 
> On Wed, Oct 15, 2014 at 8:01 AM, Marco Ulgelmo <marco.ulge at gmail.com (mailto:marco.ulge at gmail.com)> wrote:
> > Hello,
> > 
> > I've recently began studying how NDN works and i have a question about unsatisfied Interests.
> > 
> > Lets say that an attacker generates a fake Interest packet that surely doesn't exist. Let's assume that this packet is identified by the following name: /google/randomtext
> > 
> > As far as i know i shuold be able to reach /google but i won't be able to retrieve the content as it doesn't exist.
> > 
> > Considering the following scenario:
> > 
> > Interest -> Router1 -> Router2 -> Content Provider.
> > 
> > Is Router2 able to determine that the content doesn't exist? Can this information be sent back to Router2 from the Content Provider (which surely knows that he doesn't have that resource)?
> > 
> > Thanks,
> > 
> > Marco
> > 
> > _______________________________________________
> > Ndn-interest mailing list
> > Ndn-interest at lists.cs.ucla.edu (mailto:Ndn-interest at lists.cs.ucla.edu)
> > http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
> > 
> 
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu (mailto:Ndn-interest at lists.cs.ucla.edu)
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/77c00804/attachment.html>

From Marc.Mosko at parc.com  Wed Oct 15 08:24:22 2014
From: Marc.Mosko at parc.com (Marc.Mosko at parc.com)
Date: Wed, 15 Oct 2014 15:24:22 +0000
Subject: [Ndn-interest] Question about unsatisfied Interests
In-Reply-To: <CAOFH+OYh-HcDzyfX4KCh+tCrkfTg_pF9J8LmGXeYJV+LE+NM4w@mail.gmail.com>
References: <CA+=+rrACDbQuMFZbSVexMb3sAHJy7yfzyvJA=yDCbZUynJCbpQ@mail.gmail.com>
	<CAOFH+OYh-HcDzyfX4KCh+tCrkfTg_pF9J8LmGXeYJV+LE+NM4w@mail.gmail.com>
Message-ID: <4F0AEAC4-7D53-4D67-887D-D33598C4F832@parc.com>

Shouldn?t the NAK have a limited lifetime?

If I ask for /news/version=100 an the provider only has version=98, it would NACK 100.  But, the provider will eventually want to publish version 100.

Marc

On Oct 15, 2014, at 8:13 AM, Junxiao Shi <shijunxiao at email.arizona.edu<mailto:shijunxiao at email.arizona.edu>> wrote:

Hi Marco

If Router2 has a Route toward the ContentProvider for this Name, it cannot detect the content doesn't exist.

ContentProvider should reply with a provider-generated NACK, which is a type of Data, that indicates the non-existence of this content. A provider-generated NACK is cachable on routers.
A future Interest toward this Name would be satisifed by this provider-generated NACK from the ContentStore, and doesn't have to go to the ContentProvider.

Yours, Junxiao

On Wed, Oct 15, 2014 at 8:01 AM, Marco Ulgelmo <marco.ulge at gmail.com<mailto:marco.ulge at gmail.com>> wrote:
Hello,

I've recently began studying how NDN works and i have a question about unsatisfied Interests.

Lets say that an attacker generates a fake Interest packet that surely doesn't exist. Let's assume that this packet is identified by the following name: /google/randomtext

As far as i know i shuold be able to reach /google but i won't be able to retrieve the content as it doesn't exist.

Considering the following scenario:

Interest -> Router1 -> Router2 -> Content Provider.

Is Router2 able to determine that the content doesn't exist? Can this information be sent back to Router2 from the Content Provider (which surely knows that he doesn't have that resource)?

Thanks,

Marco

_______________________________________________
Ndn-interest mailing list
Ndn-interest at lists.cs.ucla.edu<mailto:Ndn-interest at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest


_______________________________________________
Ndn-interest mailing list
Ndn-interest at lists.cs.ucla.edu<mailto:Ndn-interest at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/ce3a8410/attachment.html>

From shijunxiao at email.arizona.edu  Wed Oct 15 08:26:52 2014
From: shijunxiao at email.arizona.edu (Junxiao Shi)
Date: Wed, 15 Oct 2014 08:26:52 -0700
Subject: [Ndn-interest] Question about unsatisfied Interests
In-Reply-To: <4F0AEAC4-7D53-4D67-887D-D33598C4F832@parc.com>
References: <CA+=+rrACDbQuMFZbSVexMb3sAHJy7yfzyvJA=yDCbZUynJCbpQ@mail.gmail.com>
	<CAOFH+OYh-HcDzyfX4KCh+tCrkfTg_pF9J8LmGXeYJV+LE+NM4w@mail.gmail.com>
	<4F0AEAC4-7D53-4D67-887D-D33598C4F832@parc.com>
Message-ID: <CAOFH+OaEOHvWnzjpdezwvXcGwBg5jY-TJ2Wovks+J+U65MeFYA@mail.gmail.com>

Hi Marc

Producer-generated NACK is a type of Data. It has the FreshnessPeriod field
which determines how long it could be used to satisfy an Interest with
Selector MustBeFresh=yes.
ContentProvider should set FreshnessPeriod to the duration in which it
believes that the Data won't be generated.

Yours, Junxiao

On Wed, Oct 15, 2014 at 8:24 AM, <Marc.Mosko at parc.com> wrote:

>  Shouldn?t the NAK have a limited lifetime?
>
>  If I ask for /news/version=100 an the provider only has version=98, it
> would NACK 100.  But, the provider will eventually want to publish version
> 100.
>
>  Marc
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/a1c09aef/attachment.html>

From Marc.Mosko at parc.com  Wed Oct 15 08:33:42 2014
From: Marc.Mosko at parc.com (Marc.Mosko at parc.com)
Date: Wed, 15 Oct 2014 15:33:42 +0000
Subject: [Ndn-interest] Question about unsatisfied Interests
In-Reply-To: <CAOFH+OaEOHvWnzjpdezwvXcGwBg5jY-TJ2Wovks+J+U65MeFYA@mail.gmail.com>
References: <CA+=+rrACDbQuMFZbSVexMb3sAHJy7yfzyvJA=yDCbZUynJCbpQ@mail.gmail.com>
	<CAOFH+OYh-HcDzyfX4KCh+tCrkfTg_pF9J8LmGXeYJV+LE+NM4w@mail.gmail.com>
	<4F0AEAC4-7D53-4D67-887D-D33598C4F832@parc.com>
	<CAOFH+OaEOHvWnzjpdezwvXcGwBg5jY-TJ2Wovks+J+U65MeFYA@mail.gmail.com>
Message-ID: <E7FD6C46-8A35-4E8E-A813-457D0DECCC09@parc.com>

So there is a field in the content object that distinguishes if it is fresh or not?  I thought there was only the FreshnessPeriod, which is a relative number from the time received and a remote node cannot determine if a content object was fresh or not when it came out of the content store.

Example:

A ? B ? C
D -/

Node A sends a normal (staled allowed) interest to B, who forwards it to C.  Just a bit later, node D issues a ?MustBeFresh? interest for the same name.  Node B sees that they are different requests, so forwards the second interest to C also.

Node C replies to the first interest with a stale content object.  How can node B know  that the content object should only go to A and not to D?

Marc

On Oct 15, 2014, at 8:26 AM, Junxiao Shi <shijunxiao at email.arizona.edu<mailto:shijunxiao at email.arizona.edu>> wrote:

Hi Marc

Producer-generated NACK is a type of Data. It has the FreshnessPeriod field which determines how long it could be used to satisfy an Interest with Selector MustBeFresh=yes.
ContentProvider should set FreshnessPeriod to the duration in which it believes that the Data won't be generated.

Yours, Junxiao

On Wed, Oct 15, 2014 at 8:24 AM, <Marc.Mosko at parc.com<mailto:Marc.Mosko at parc.com>> wrote:
Shouldn?t the NAK have a limited lifetime?

If I ask for /news/version=100 an the provider only has version=98, it would NACK 100.  But, the provider will eventually want to publish version 100.

Marc


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/1eb8f6f4/attachment.html>

From shijunxiao at email.arizona.edu  Wed Oct 15 08:37:43 2014
From: shijunxiao at email.arizona.edu (Junxiao Shi)
Date: Wed, 15 Oct 2014 08:37:43 -0700
Subject: [Ndn-interest] Question about unsatisfied Interests
In-Reply-To: <E7FD6C46-8A35-4E8E-A813-457D0DECCC09@parc.com>
References: <CA+=+rrACDbQuMFZbSVexMb3sAHJy7yfzyvJA=yDCbZUynJCbpQ@mail.gmail.com>
	<CAOFH+OYh-HcDzyfX4KCh+tCrkfTg_pF9J8LmGXeYJV+LE+NM4w@mail.gmail.com>
	<4F0AEAC4-7D53-4D67-887D-D33598C4F832@parc.com>
	<CAOFH+OaEOHvWnzjpdezwvXcGwBg5jY-TJ2Wovks+J+U65MeFYA@mail.gmail.com>
	<E7FD6C46-8A35-4E8E-A813-457D0DECCC09@parc.com>
Message-ID: <CAOFH+OZawhmj2X3FgbiU4gdfhpmhG9Yc6vfx2b0fK9JApqih6w@mail.gmail.com>

Hi Marc

There's no such field. We only have FreshnessPeriod.
By staleness definition, when the Data arrives at B, it's fresh until
FreshnessPeriod has elapsed, regardless of how it's requested.

Yours, Junxiao

On Wed, Oct 15, 2014 at 8:33 AM, <Marc.Mosko at parc.com> wrote:

>  So there is a field in the content object that distinguishes if it is
> fresh or not?  I thought there was only the FreshnessPeriod, which is a
> relative number from the time received and a remote node cannot determine
> if a content object was fresh or not when it came out of the content store.
>
>  Example:
>
>  A ? B ? C
> D -/
>
>  Node A sends a normal (staled allowed) interest to B, who forwards it to
> C.  Just a bit later, node D issues a ?MustBeFresh? interest for the same
> name.  Node B sees that they are different requests, so forwards the second
> interest to C also.
>
>  Node C replies to the first interest with a stale content object.  How
> can node B know  that the content object should only go to A and not to D?
>
>  Marc
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/3497a429/attachment.html>

From marco.ulge at gmail.com  Wed Oct 15 08:42:16 2014
From: marco.ulge at gmail.com (Marco Ulgelmo)
Date: Wed, 15 Oct 2014 17:42:16 +0200
Subject: [Ndn-interest] R: Re:  Question about unsatisfied Interests
Message-ID: <gr05kej19vrjcib82pbr4e8g.1413387736828@email.android.com>

Thanks for the replies. Actually I found all I needed in the link provided by Xiaoke. I read that this kind of interest neck is what they propose in the research. What I'd like to know now is: has this been implemented in NDN? I mean, can I rely in this for a future work??

Marco


Inviato da Samsung Mobile.

<div>-------- Messaggio originale --------</div><div>Da: Xiaoke Jiang <shock.jiang at gmail.com> </div><div>Data:15/10/2014  17:20  (GMT+01:00) </div><div>A: Marco Ulgelmo <marco.ulge at gmail.com> </div><div>Cc: ndn-interest at lists.cs.ucla.edu </div><div>Oggetto: Re: [Ndn-interest] Question about unsatisfied Interests </div><div>
</div>Hi Marco,
   This paper   http://named-data.net/wp-content/uploads/comcom-stateful-forwarding.pdf give more details on how stateful data plane helps to get rid of some kinds of attacher.


On Wednesday, 15 October, 2014 at 8:13 am, Junxiao Shi wrote:

Hi Marco

If Router2 has a Route toward the ContentProvider for this Name, it cannot detect the content doesn't exist.

ContentProvider should reply with a provider-generated NACK, which is a type of Data, that indicates the non-existence of this content. A provider-generated NACK is cachable on routers.
A future Interest toward this Name would be satisifed by this provider-generated NACK from the ContentStore, and doesn't have to go to the ContentProvider.

Yours, Junxiao

On Wed, Oct 15, 2014 at 8:01 AM, Marco Ulgelmo <marco.ulge at gmail.com> wrote:
Hello,

I've recently began studying how NDN works and i have a question about unsatisfied Interests.

Lets say that an attacker generates a fake Interest packet that surely doesn't exist. Let's assume that this packet is identified by the following name: /google/randomtext

As far as i know i shuold be able to reach /google but i won't be able to retrieve the content as it doesn't exist.

Considering the following scenario:

Interest -> Router1 -> Router2 -> Content Provider.

Is Router2 able to determine that the content doesn't exist? Can this information be sent back to Router2 from the Content Provider (which surely knows that he doesn't have that resource)?

Thanks,

Marco

_______________________________________________
Ndn-interest mailing list
Ndn-interest at lists.cs.ucla.edu
http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest


_______________________________________________
Ndn-interest mailing list
Ndn-interest at lists.cs.ucla.edu
http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/dfaebaf1/attachment.html>

From Marc.Mosko at parc.com  Wed Oct 15 08:47:19 2014
From: Marc.Mosko at parc.com (Marc.Mosko at parc.com)
Date: Wed, 15 Oct 2014 15:47:19 +0000
Subject: [Ndn-interest] R: Re:  Question about unsatisfied Interests
In-Reply-To: <gr05kej19vrjcib82pbr4e8g.1413387736828@email.android.com>
References: <gr05kej19vrjcib82pbr4e8g.1413387736828@email.android.com>
Message-ID: <47A160FC-5204-4181-96CD-F8198E995D11@parc.com>

I thought the Interest NACK is a type of control message that is not cached in the content store.

That is different than the publisher issuing a Data object of ContentType = NACK.  Or even issuing a Data object with type = Data, but a body like a 403.  I thought that the ContentType = NACK was removed from the NDN spec.

So, Junxiao answer that the NACk could stay in the content store was surprising to me.

Marc

On Oct 15, 2014, at 8:42 AM, Marco Ulgelmo <marco.ulge at gmail.com<mailto:marco.ulge at gmail.com>> wrote:

Thanks for the replies. Actually I found all I needed in the link provided by Xiaoke. I read that this kind of interest neck is what they propose in the research. What I'd like to know now is: has this been implemented in NDN? I mean, can I rely in this for a future work?

Marco


Inviato da Samsung Mobile.


-------- Messaggio originale --------
Da: Xiaoke Jiang
Data:15/10/2014 17:20 (GMT+01:00)
A: Marco Ulgelmo
Cc: ndn-interest at lists.cs.ucla.edu<mailto:ndn-interest at lists.cs.ucla.edu>
Oggetto: Re: [Ndn-interest] Question about unsatisfied Interests

Hi Marco,
   This paper   http://named-data.net/wp-content/uploads/comcom-stateful-forwarding.pdf give more details on how stateful data plane helps to get rid of some kinds of attacher.


On Wednesday, 15 October, 2014 at 8:13 am, Junxiao Shi wrote:

Hi Marco

If Router2 has a Route toward the ContentProvider for this Name, it cannot detect the content doesn't exist.

ContentProvider should reply with a provider-generated NACK, which is a type of Data, that indicates the non-existence of this content. A provider-generated NACK is cachable on routers.
A future Interest toward this Name would be satisifed by this provider-generated NACK from the ContentStore, and doesn't have to go to the ContentProvider.

Yours, Junxiao

On Wed, Oct 15, 2014 at 8:01 AM, Marco Ulgelmo <marco.ulge at gmail.com<mailto:marco.ulge at gmail.com>> wrote:
Hello,

I've recently began studying how NDN works and i have a question about unsatisfied Interests.

Lets say that an attacker generates a fake Interest packet that surely doesn't exist. Let's assume that this packet is identified by the following name: /google/randomtext

As far as i know i shuold be able to reach /google but i won't be able to retrieve the content as it doesn't exist.

Considering the following scenario:

Interest -> Router1 -> Router2 -> Content Provider.

Is Router2 able to determine that the content doesn't exist? Can this information be sent back to Router2 from the Content Provider (which surely knows that he doesn't have that resource)?

Thanks,

Marco

_______________________________________________
Ndn-interest mailing list
Ndn-interest at lists.cs.ucla.edu<mailto:Ndn-interest at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest


_______________________________________________
Ndn-interest mailing list
Ndn-interest at lists.cs.ucla.edu<mailto:Ndn-interest at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest

_______________________________________________
Ndn-interest mailing list
Ndn-interest at lists.cs.ucla.edu<mailto:Ndn-interest at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/0077588c/attachment.html>

From shijunxiao at email.arizona.edu  Wed Oct 15 08:49:08 2014
From: shijunxiao at email.arizona.edu (Junxiao Shi)
Date: Wed, 15 Oct 2014 08:49:08 -0700
Subject: [Ndn-interest] R: Re: Question about unsatisfied Interests
In-Reply-To: <47A160FC-5204-4181-96CD-F8198E995D11@parc.com>
References: <gr05kej19vrjcib82pbr4e8g.1413387736828@email.android.com>
	<47A160FC-5204-4181-96CD-F8198E995D11@parc.com>
Message-ID: <CAOFH+OZ+QEAE6E+t3aUGOdO8Mx83tuimqKWr2ZEXtaXKkS7p8w@mail.gmail.com>

Hi Marc

I'm explicitly referring to producer-generated NACK, a type of Data
(ContentType=NACK).

Yours, Junxiao

On Wed, Oct 15, 2014 at 8:47 AM, <Marc.Mosko at parc.com> wrote:

>  I thought the Interest NACK is a type of control message that is not
> cached in the content store.
>
>  That is different than the publisher issuing a Data object of
> ContentType = NACK.  Or even issuing a Data object with type = Data, but a
> body like a 403.  I thought that the ContentType = NACK was removed from
> the NDN spec.
>
>  So, Junxiao answer that the NACk could stay in the content store was
> surprising to me.
>
>  Marc
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/71e821c6/attachment.html>

From Marc.Mosko at parc.com  Wed Oct 15 08:56:18 2014
From: Marc.Mosko at parc.com (Marc.Mosko at parc.com)
Date: Wed, 15 Oct 2014 15:56:18 +0000
Subject: [Ndn-interest] R: Re: Question about unsatisfied Interests
In-Reply-To: <CAOFH+OZ+QEAE6E+t3aUGOdO8Mx83tuimqKWr2ZEXtaXKkS7p8w@mail.gmail.com>
References: <gr05kej19vrjcib82pbr4e8g.1413387736828@email.android.com>
	<47A160FC-5204-4181-96CD-F8198E995D11@parc.com>
	<CAOFH+OZ+QEAE6E+t3aUGOdO8Mx83tuimqKWr2ZEXtaXKkS7p8w@mail.gmail.com>
Message-ID: <B5553871-E8CB-4949-97A4-7061EDF8E0F9@parc.com>

http://named-data.net/doc/ndn-tlv/data.html#contenttype

says that NACK was removed.  I think NACK has problems when combined with MustBeFresh and FreshnessPeriod.

A Data object with a FreshnessPeriod of, say 1s, can still circle around in the network for much longer than 1s, especially if stale responses are allowed.  Someone trying to discover the latest version, for example, can be mislead by the NACK.

Another example:

Alice: get latest version for /news
Cache: /news/v=100

Alice: get latest version for /news greater than 100
Publisher: /news/v=110

Eve: Fetch /news/v=1000
Publisher: NACK /news/v=1000

Alice: Get latest version greater than 110
Cache: /news/v=1000

1000 now becomes the latest version for /news.  An attacker could force the publisher ? if they want to use ContentType=NACK response ? to increase their version number to arbitrary lengths.  And FreshnessPeriod does not provide a deterministic way to expire that content or prevent it from being cached.

I don?t think using a ContentType=NACK is a good idea with these cache control directives.

Marc

On Oct 15, 2014, at 8:49 AM, Junxiao Shi <shijunxiao at email.arizona.edu> wrote:

Hi Marc

I'm explicitly referring to producer-generated NACK, a type of Data (ContentType=NACK).

Yours, Junxiao

On Wed, Oct 15, 2014 at 8:47 AM, <Marc.Mosko at parc.com<mailto:Marc.Mosko at parc.com>> wrote:
I thought the Interest NACK is a type of control message that is not cached in the content store.

That is different than the publisher issuing a Data object of ContentType = NACK.  Or even issuing a Data object with type = Data, but a body like a 403.  I thought that the ContentType = NACK was removed from the NDN spec.

So, Junxiao answer that the NACk could stay in the content store was surprising to me.

Marc


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141015/ad5ba586/attachment.html>

From David at OrandomNAS2.FamilyDS.net  Wed Oct 15 08:58:59 2014
From: David at OrandomNAS2.FamilyDS.net (David R Oran)
Date: Wed, 15 Oct 2014 11:58:59 -0400
Subject: [Ndn-interest] Question about unsatisfied Interests
In-Reply-To: <CAOFH+OZawhmj2X3FgbiU4gdfhpmhG9Yc6vfx2b0fK9JApqih6w@mail.gmail.com>
References: <CA+=+rrACDbQuMFZbSVexMb3sAHJy7yfzyvJA=yDCbZUynJCbpQ@mail.gmail.com>
	<CAOFH+OYh-HcDzyfX4KCh+tCrkfTg_pF9J8LmGXeYJV+LE+NM4w@mail.gmail.com>
	<4F0AEAC4-7D53-4D67-887D-D33598C4F832@parc.com>
	<CAOFH+OaEOHvWnzjpdezwvXcGwBg5jY-TJ2Wovks+J+U65MeFYA@mail.gmail.com>
	<E7FD6C46-8A35-4E8E-A813-457D0DECCC09@parc.com>
	<CAOFH+OZawhmj2X3FgbiU4gdfhpmhG9Yc6vfx2b0fK9JApqih6w@mail.gmail.com>
Message-ID: <0893CBB4-298D-4B56-9ADA-BA4563FBA2E8@Orandomnas2.familyds.net>


On Oct 15, 2014, at 11:37 AM, Junxiao Shi <shijunxiao at email.arizona.edu> wrote:

> Hi Marc
> 
> There's no such field. We only have FreshnessPeriod.
Right. And as I?ve pointed out in the past, this is in my opinion this is a bug/missing feature.

We need to separate data validity period from cache retention period. The former has to be baked into the data object for security reasons, the latter probably does not.


> By staleness definition, when the Data arrives at B, it's fresh until FreshnessPeriod has elapsed, regardless of how it's requested.
> 
> Yours, Junxiao
> 
> On Wed, Oct 15, 2014 at 8:33 AM, <Marc.Mosko at parc.com> wrote:
> So there is a field in the content object that distinguishes if it is fresh or not?  I thought there was only the FreshnessPeriod, which is a relative number from the time received and a remote node cannot determine if a content object was fresh or not when it came out of the content store. 
> 
> Example:
> 
> A ? B ? C
> D -/
> 
> Node A sends a normal (staled allowed) interest to B, who forwards it to C.  Just a bit later, node D issues a ?MustBeFresh? interest for the same name.  Node B sees that they are different requests, so forwards the second interest to C also.  
> 
> Node C replies to the first interest with a stale content object.  How can node B know  that the content object should only go to A and not to D?
> 
> Marc
> 
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest




From ishita25.mittal at gmail.com  Thu Oct 16 03:19:22 2014
From: ishita25.mittal at gmail.com (Ishita Mittal)
Date: Thu, 16 Oct 2014 18:19:22 +0800
Subject: [Ndn-interest] Help on cache policy code
Message-ID: <CAHHwLDpEXX6SYYsuk82U9to6gnSjZYhg-t5STzUZ8kAHw_-Ytw@mail.gmail.com>

Hi

I am working on Named Data Networking as my Final Year Project in Computer
Engineering. I am using NFD as the platform. My project relates to
improving the cache efficiency of the routers. So, basically. I will start
with introducing the hop count as one of the attributes of both the
interest and data packets and the processing that at the routers. May I
know how to go about it - the code I should look at, the documentation
related to it, if any and anything anyone can suggest?

Thanks and Regards
Ishita
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141016/bca6e354/attachment.html>

From lanwang at memphis.edu  Tue Oct 21 13:35:34 2014
From: lanwang at memphis.edu (Lan Wang (lanwang))
Date: Tue, 21 Oct 2014 20:35:34 +0000
Subject: [Ndn-interest] application scenario
Message-ID: <F0792922-A263-4ED8-9D3A-14F7C2CE370F@memphis.edu>

Hi there,

Suppose I have a node A that has limited memory.  It wants to synchronize partial data with other nodes (B and C).    The other nodes are fully synchronized with each other (they use ChronoSync).   Node A wants to get updates whenever the other nodes have more recent versions of the data it has, but the other nodes do not know what A has.  What's the best way to design this application?  Ideally we don't want A to specifically sync with one particular node (B or C).

Here's an illustration:

* Node A has:

- /a/v1
- /b/v1

* Node B and C have:

- /a/v1
- /b/v2
- /c/v3

Ideally A should get update about /b/v2, but not /c/v3 as it's interested in /b, not interested in /c.

We don't want to run ChronoSync on A, as it doesn't have the space for all the data and it is not interested in all the data either.

Lan









From christos at cs.colostate.edu  Tue Oct 21 14:51:56 2014
From: christos at cs.colostate.edu (Christos Papadopoulos)
Date: Tue, 21 Oct 2014 15:51:56 -0600
Subject: [Ndn-interest] application scenario
In-Reply-To: <F0792922-A263-4ED8-9D3A-14F7C2CE370F@memphis.edu>
References: <F0792922-A263-4ED8-9D3A-14F7C2CE370F@memphis.edu>
Message-ID: <5446D57C.1010302@cs.colostate.edu>

Sounds like you need the ability to synchronize prefixes, not the entire 
root.
So you would like to synchronize /a and /b.

Christos.

On 10/21/2014 02:35 PM, Lan Wang (lanwang) wrote:
> Hi there,
>
> Suppose I have a node A that has limited memory.  It wants to synchronize partial data with other nodes (B and C).    The other nodes are fully synchronized with each other (they use ChronoSync).   Node A wants to get updates whenever the other nodes have more recent versions of the data it has, but the other nodes do not know what A has.  What's the best way to design this application?  Ideally we don't want A to specifically sync with one particular node (B or C).
>
> Here's an illustration:
>
> * Node A has:
>
> - /a/v1
> - /b/v1
>
> * Node B and C have:
>
> - /a/v1
> - /b/v2
> - /c/v3
>
> Ideally A should get update about /b/v2, but not /c/v3 as it's interested in /b, not interested in /c.
>
> We don't want to run ChronoSync on A, as it doesn't have the space for all the data and it is not interested in all the data either.
>
> Lan
>
>
>
>
>
>
>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>



From lanwang at memphis.edu  Thu Oct 23 08:40:29 2014
From: lanwang at memphis.edu (Lan Wang (lanwang))
Date: Thu, 23 Oct 2014 15:40:29 +0000
Subject: [Ndn-interest] application scenario
In-Reply-To: <5446D57C.1010302@cs.colostate.edu>
References: <F0792922-A263-4ED8-9D3A-14F7C2CE370F@memphis.edu>
	<5446D57C.1010302@cs.colostate.edu>
Message-ID: <602E6E8E-8F08-4F3F-9C8C-AC8EE20A2F83@memphis.edu>

Yes, I want to synchronize only part of the data.

I can still use ChronoSync to get updates for all the prefixes and only retrieve the ones I'm interested in.  This will work, but it would be nice if I don't have to maintain state for all the name prefixes. 

Lan

On Oct 21, 2014, at 4:51 PM, Christos Papadopoulos <christos at CS.ColoState.EDU> wrote:

> Sounds like you need the ability to synchronize prefixes, not the entire root.
> So you would like to synchronize /a and /b.
> 
> Christos.
> 
> On 10/21/2014 02:35 PM, Lan Wang (lanwang) wrote:
>> Hi there,
>> 
>> Suppose I have a node A that has limited memory.  It wants to synchronize partial data with other nodes (B and C).    The other nodes are fully synchronized with each other (they use ChronoSync).   Node A wants to get updates whenever the other nodes have more recent versions of the data it has, but the other nodes do not know what A has.  What's the best way to design this application?  Ideally we don't want A to specifically sync with one particular node (B or C).
>> 
>> Here's an illustration:
>> 
>> * Node A has:
>> 
>> - /a/v1
>> - /b/v1
>> 
>> * Node B and C have:
>> 
>> - /a/v1
>> - /b/v2
>> - /c/v3
>> 
>> Ideally A should get update about /b/v2, but not /c/v3 as it's interested in /b, not interested in /c.
>> 
>> We don't want to run ChronoSync on A, as it doesn't have the space for all the data and it is not interested in all the data either.
>> 
>> Lan
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> 
> 
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest




From bassem.labib at gmail.com  Thu Oct 23 09:15:42 2014
From: bassem.labib at gmail.com (Bassem Labib)
Date: Thu, 23 Oct 2014 18:15:42 +0200
Subject: [Ndn-interest] Send and Receive Video using jNDN Client
Message-ID: <CAEakb=HYMp+H3uQ0pnu0eCtcVPSuVbhkejZWnVCTojhVUnKJmg@mail.gmail.com>

Hi,

I am trying to send and receive a video file using jNDN Client.
There is a limitation of the ByteBuffer with max 8K bytes only!!
How can I extend it without receiving it in the onTimeout state.

 Best Regards,
Bassem Labib
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141023/d2e8e8d7/attachment.html>

From jefft0 at remap.ucla.edu  Thu Oct 23 10:27:11 2014
From: jefft0 at remap.ucla.edu (Thompson, Jeff)
Date: Thu, 23 Oct 2014 17:27:11 +0000
Subject: [Ndn-interest] Send and Receive Video using jNDN Client
In-Reply-To: <CAEakb=HYMp+H3uQ0pnu0eCtcVPSuVbhkejZWnVCTojhVUnKJmg@mail.gmail.com>
References: <CAEakb=HYMp+H3uQ0pnu0eCtcVPSuVbhkejZWnVCTojhVUnKJmg@mail.gmail.com>
Message-ID: <D06E8524.12540%jefft0@remap.ucla.edu>

Hello,

I'm a little confused. The Java ByteBuffer doesn't have a limit of 8K bytes.  Do you mean a limit on the size of an NDN data packet?  The NDN data packet has a max size of 8800 (including content, signature, etc.)  This is due to network MTU size.  NDN handles this by dividing a file into segments.

Before I explain more, are you familiar with NDN segments and how NDN names segments?  These are described in section 3.2 of "NDN Technical Memo: Naming Conventions":
http://named-data.net/publications/techreports/ndn-tr-22-ndn-memo-naming-conventions/

Adding and parsing segment markers in names is supported in jNDN with Name.appendSegment and Name.Component.toSegment.
http://named-data.net/doc/ndn-ccl-api/name.html#name-appendsegment-method
http://named-data.net/doc/ndn-ccl-api/name-component.html#name-component-tosegment-method

Other than supporing segments in names, jNDN does not yet have a utility to automatically send a file in segments or fetch a file with segments. This is a planned future higher-level API.  For now, the application must divide the file into segments and create names that have a segment marker.  In practice, each application may want to handle this differently. For example a real-time streaming video application may want to drop segments if the network does not deliver them.  But with a file transfer application, the appilcation may want to repeatedly try to fetch a segment if the fetch times out.

Thank you,
- Jeff T

From: Bassem Labib <bassem.labib at gmail.com<mailto:bassem.labib at gmail.com>>
Date: Thursday, October 23, 2014 at 9:15 AM
To: "ndn-interest at lists.cs.ucla.edu<mailto:ndn-interest at lists.cs.ucla.edu>" <ndn-interest at lists.cs.ucla.edu<mailto:ndn-interest at lists.cs.ucla.edu>>
Subject: [Ndn-interest] Send and Receive Video using jNDN Client

Hi,

I am trying to send and receive a video file using jNDN Client.
There is a limitation of the ByteBuffer with max 8K bytes only!!
How can I extend it without receiving it in the onTimeout state.

 Best Regards,
Bassem Labib
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141023/ac01900f/attachment.html>

From dibenede at cs.colostate.edu  Thu Oct 23 10:30:45 2014
From: dibenede at cs.colostate.edu (Steve DiBenedetto)
Date: Thu, 23 Oct 2014 13:30:45 -0400
Subject: [Ndn-interest] Send and Receive Video using jNDN Client
In-Reply-To: <CAEakb=HYMp+H3uQ0pnu0eCtcVPSuVbhkejZWnVCTojhVUnKJmg@mail.gmail.com>
References: <CAEakb=HYMp+H3uQ0pnu0eCtcVPSuVbhkejZWnVCTojhVUnKJmg@mail.gmail.com>
Message-ID: <CAG95Nwt6fwvhJDe4w2pf5aZ2f=XT-D7j_LqMeNo1N7smEQpygA@mail.gmail.com>

Hi,

There is a hard limit of 8800 bytes (payload + packet header) throughout
the NDN code base. For example, I believe NFD will drop anything over that
size.

Are you trying to chunk your video file into Data packets larger than 8K or
attempting to send the entire video file as a single packet?

-Steve

On Thursday, October 23, 2014, Bassem Labib <bassem.labib at gmail.com> wrote:

> Hi,
>
> I am trying to send and receive a video file using jNDN Client.
> There is a limitation of the ByteBuffer with max 8K bytes only!!
> How can I extend it without receiving it in the onTimeout state.
>
>  Best Regards,
> Bassem Labib
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141023/c9db4f28/attachment.html>

From Cedric.Westphal at huawei.com  Thu Oct 23 13:54:53 2014
From: Cedric.Westphal at huawei.com (Cedric Westphal)
Date: Thu, 23 Oct 2014 20:54:53 +0000
Subject: [Ndn-interest] Immediate opening for intern for CCN/NDN project
In-Reply-To: <D06E8524.12540%jefft0@remap.ucla.edu>
References: <CAEakb=HYMp+H3uQ0pnu0eCtcVPSuVbhkejZWnVCTojhVUnKJmg@mail.gmail.com>
	<D06E8524.12540%jefft0@remap.ucla.edu>
Message-ID: <369480A01F73974DAC423D05A977B4F214920823@SJCEML701-CHM.china.huawei.com>

Internship available immediately with Huawei Technology in Santa Clara, CA. We are looking for a part-time intern for a one or two month project that involves:  adding a new functionality to an existing CCN IoT platform (the new functionality is similar to the existing functionalities) and writing a simple GUI to display the results. Code is in C/C++/Java. This is a relatively simple project that should last one or two month part time. I am looking for someone to start relatively soon. If interested, please email cedric.westphal at huawei.com<mailto:cedric.westphal at huawei.com> with contact info and availability.
Thanks,
Cedric Westphal cedric.westphal at huawei.com<mailto:cedric.westphal at huawei.com>



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141023/5faf88dd/attachment.html>

From gq.wang at huawei.com  Thu Oct 23 14:28:58 2014
From: gq.wang at huawei.com (GQ Wang)
Date: Thu, 23 Oct 2014 21:28:58 +0000
Subject: [Ndn-interest]  application scenario
In-Reply-To: <mailman.1705.1414097711.22598.ndn-interest@lists.cs.ucla.edu>
References: <mailman.1705.1414097711.22598.ndn-interest@lists.cs.ucla.edu>
Message-ID: <6759D6D370C7024D9BB22CD5F92D2D331857CA01@SJCEML701-CHM.china.huawei.com>

It seems to me the ideal approach for these syncs is a pub-sub model that A should subscribe certain "interests" at B and C, and
get some events back (if any). Once received these events (saying some prefixes), A will determine what it wants to sync.
It can be done effectively in a well-organized network (e.g., a cluster-head), or somewhat awkward in a totally ad-hoc environment.
G.Q 

-----Original Message-----
From: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] On Behalf Of ndn-interest-request at lists.cs.ucla.edu
Sent: Thursday, October 23, 2014 1:55 PM
To: ndn-interest at lists.cs.ucla.edu
Subject: Ndn-interest Digest, Vol 8, Issue 10

Send Ndn-interest mailing list submissions to
	ndn-interest at lists.cs.ucla.edu

To subscribe or unsubscribe via the World Wide Web, visit
	http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
or, via email, send a message with subject or body 'help' to
	ndn-interest-request at lists.cs.ucla.edu

You can reach the person managing the list at
	ndn-interest-owner at lists.cs.ucla.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Ndn-interest digest..."


Today's Topics:

   1. application scenario (Lan Wang (lanwang))
   2. Re: application scenario (Christos Papadopoulos)
   3. Re: application scenario (Lan Wang (lanwang))
   4. Send and Receive Video using jNDN Client (Bassem Labib)
   5. Re: Send and Receive Video using jNDN Client (Thompson, Jeff)
   6. Re: Send and Receive Video using jNDN Client (Steve DiBenedetto)
   7. Immediate opening for intern for CCN/NDN project (Cedric Westphal)


----------------------------------------------------------------------

Message: 1
Date: Tue, 21 Oct 2014 20:35:34 +0000
From: "Lan Wang (lanwang)" <lanwang at memphis.edu>
To: "<ndn-interest at lists.cs.ucla.edu>"
	<ndn-interest at lists.cs.ucla.edu>
Subject: [Ndn-interest] application scenario
Message-ID: <F0792922-A263-4ED8-9D3A-14F7C2CE370F at memphis.edu>
Content-Type: text/plain; charset="us-ascii"

Hi there,

Suppose I have a node A that has limited memory.  It wants to synchronize partial data with other nodes (B and C).    The other nodes are fully synchronized with each other (they use ChronoSync).   Node A wants to get updates whenever the other nodes have more recent versions of the data it has, but the other nodes do not know what A has.  What's the best way to design this application?  Ideally we don't want A to specifically sync with one particular node (B or C).

Here's an illustration:

* Node A has:

- /a/v1
- /b/v1

* Node B and C have:

- /a/v1
- /b/v2
- /c/v3

Ideally A should get update about /b/v2, but not /c/v3 as it's interested in /b, not interested in /c.

We don't want to run ChronoSync on A, as it doesn't have the space for all the data and it is not interested in all the data either.

Lan













From gq.wang at huawei.com  Thu Oct 23 16:28:42 2014
From: gq.wang at huawei.com (GQ Wang)
Date: Thu, 23 Oct 2014 23:28:42 +0000
Subject: [Ndn-interest] application scenario
In-Reply-To: <mailman.1705.1414097711.22598.ndn-interest@lists.cs.ucla.edu>
References: <mailman.1705.1414097711.22598.ndn-interest@lists.cs.ucla.edu>
Message-ID: <6759D6D370C7024D9BB22CD5F92D2D331857CAD7@SJCEML701-CHM.china.huawei.com>

It seems to me the ideal approach for these syncs is a pub-sub model that A should subscribe certain "interests" at B and C, and get some events back (if any). Once received these events (saying some prefixes), A will determine what it wants to sync.
It can be done effectively in a well-organized network (e.g., a cluster-head), or somewhat awkward in a totally ad-hoc environment.
G.Q


Date: Tue, 21 Oct 2014 20:35:34 +0000
From: "Lan Wang (lanwang)" <lanwang at memphis.edu>
To: "<ndn-interest at lists.cs.ucla.edu>"
	<ndn-interest at lists.cs.ucla.edu>
Subject: [Ndn-interest] application scenario
Message-ID: <F0792922-A263-4ED8-9D3A-14F7C2CE370F at memphis.edu>
Content-Type: text/plain; charset="us-ascii"

Hi there,

Suppose I have a node A that has limited memory.  It wants to synchronize partial data with other nodes (B and C).    The other nodes are fully synchronized with each other (they use ChronoSync).   Node A wants to get updates whenever the other nodes have more recent versions of the data it has, but the other nodes do not know what A has.  What's the best way to design this application?  Ideally we don't want A to specifically sync with one particular node (B or C).

Here's an illustration:

* Node A has:

- /a/v1
- /b/v1

* Node B and C have:

- /a/v1
- /b/v2
- /c/v3

Ideally A should get update about /b/v2, but not /c/v3 as it's interested in /b, not interested in /c.

We don't want to run ChronoSync on A, as it doesn't have the space for all the data and it is not interested in all the data either.

Lan





From bassem.labib at gmail.com  Sat Oct 25 07:00:26 2014
From: bassem.labib at gmail.com (Bassem Labib)
Date: Sat, 25 Oct 2014 16:00:26 +0200
Subject: [Ndn-interest] Send and Receive Video using jNDN Client
In-Reply-To: <CAEakb=H8LffRaVORtBr_vQBKAYfhEF0QY_EquX18DhVRSwh7DQ@mail.gmail.com>
References: <CAEakb=HYMp+H3uQ0pnu0eCtcVPSuVbhkejZWnVCTojhVUnKJmg@mail.gmail.com>
	<D06E8524.12540%jefft0@remap.ucla.edu>
	<CAEakb=H8LffRaVORtBr_vQBKAYfhEF0QY_EquX18DhVRSwh7DQ@mail.gmail.com>
Message-ID: <CAEakb=EDdYq68Po-wWOzE_Jhj=GGgBp_rf0ucP=GSd6kQ2iH4g@mail.gmail.com>

Hi Jeff,

I got confused regarding segmenting the chunk file into the Name, and how
to use the Data object to send its content, and receive it from the other
side to be display.

how can I use a Data.setContent(Blob)?
As the below code:

public void onInterest(Name prefix, Interest interest, Transport
transport,long registeredPrefixId)
    {
        Name intrstName = interest.getName();
        // Read the Video file and conver it to ByteBuffer.
        ByteBuffer[] contentArr = (new FileHandler()).readArray();

        for (int i = 0; i < contentArr.length; i++)
        {
            ByteBuffer contentBuffer = contentArr[i];
            contentBuffer.flip();

            Blob contentBlob = new Blob(contentBuffer.array());
            Name.Component nComp = new Name.Component(contentBlob);

            long segment = -1;
            try {
                segment = nComp.toSegment();
            } catch (EncodingException e) {
                e.printStackTrace();
            }
            intrstName = intrstName.appendSegment(segment);
        }

        Data data =  new Data(intrstName);
        //data = data.setContent(contentBlob);   <==== How To Use This

 data.getMetaInfo().setTimestampMilliseconds(System.currentTimeMillis());

        try {
            keyChain_.sign(data, certificateName_);
            Blob encodedData = data.wireEncode();
            transport.send(encodedData.buf());
        }
        catch (IOException | SecurityException ex) {
            System.out.println("Echo: Exception: "+ ex.getMessage());
        }
    }



public void onData(Interest interest, Data data)
    {
        ++callbackCount_;

        //ByteBuffer contentBuffer = data.getContent().buf();

        Name intrstName = interest.getName();
        ByteBuffer[] contentBufferArr = new ByteBuffer[intrstName.size()];
        for(int i=0; i < intrstName.size(); i++)
        {
            contentBufferArr[i] = intrstName.get(i).getValue().buf();
        }

        // Write the recieved data into a file.
        (new FileHandler()).writeArray(contentBufferArr);

    }


This code is not work properly, Please advise if there is other
implementation for that issue.

Thanks

On Fri, Oct 24, 2014 at 9:15 AM, Bassem Labib <bassem.labib at gmail.com>
wrote:

> Hi,
>
> Thanks, for your quick response, I will go through this document and try
> to implement segments solution.
>
>
> On Thu, Oct 23, 2014 at 7:27 PM, Thompson, Jeff <jefft0 at remap.ucla.edu>
> wrote:
>
>>  Hello,
>>
>>  I'm a little confused. The Java ByteBuffer doesn't have a limit of 8K
>> bytes.  Do you mean a limit on the size of an NDN data packet?  The NDN
>> data packet has a max size of 8800 (including content, signature, etc.)
>>  This is due to network MTU size.  NDN handles this by dividing a file into
>> segments.
>>
>>  Before I explain more, are you familiar with NDN segments and how NDN
>> names segments?  These are described in section 3.2 of "NDN Technical Memo:
>> Naming Conventions":
>>
>> http://named-data.net/publications/techreports/ndn-tr-22-ndn-memo-naming-conventions
>> /
>>
>>  Adding and parsing segment markers in names is supported in jNDN
>> with Name.appendSegment and Name.Component.toSegment.
>> http://named-data.net/doc/ndn-ccl-api/name.html#name-appendsegment-method
>>
>> http://named-data.net/doc/ndn-ccl-api/name-component.html#name-component-tosegment-method
>>
>>  Other than supporing segments in names, jNDN does not yet have a
>> utility to automatically send a file in segments or fetch a file with
>> segments. This is a planned future higher-level API.  For now, the
>> application must divide the file into segments and create names that have a
>> segment marker.  In practice, each application may want to handle this
>> differently. For example a real-time streaming video application may want
>> to drop segments if the network does not deliver them.  But with a file
>> transfer application, the appilcation may want to repeatedly try to fetch a
>> segment if the fetch times out.
>>
>>  Thank you,
>> - Jeff T
>>
>>   From: Bassem Labib <bassem.labib at gmail.com>
>> Date: Thursday, October 23, 2014 at 9:15 AM
>> To: "ndn-interest at lists.cs.ucla.edu" <ndn-interest at lists.cs.ucla.edu>
>> Subject: [Ndn-interest] Send and Receive Video using jNDN Client
>>
>>     Hi,
>>
>>  I am trying to send and receive a video file using jNDN Client.
>>  There is a limitation of the ByteBuffer with max 8K bytes only!!
>> How can I extend it without receiving it in the onTimeout state.
>>
>>   Best Regards,
>>  Bassem Labib
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141025/66516e04/attachment.html>

From jefft0 at remap.ucla.edu  Mon Oct 27 11:54:58 2014
From: jefft0 at remap.ucla.edu (Thompson, Jeff)
Date: Mon, 27 Oct 2014 18:54:58 +0000
Subject: [Ndn-interest] Send and Receive Video using jNDN Client
In-Reply-To: <CAEakb=EDdYq68Po-wWOzE_Jhj=GGgBp_rf0ucP=GSd6kQ2iH4g@mail.gmail.com>
References: <CAEakb=HYMp+H3uQ0pnu0eCtcVPSuVbhkejZWnVCTojhVUnKJmg@mail.gmail.com>
	<D06E8524.12540%jefft0@remap.ucla.edu>
	<CAEakb=H8LffRaVORtBr_vQBKAYfhEF0QY_EquX18DhVRSwh7DQ@mail.gmail.com>
	<CAEakb=EDdYq68Po-wWOzE_Jhj=GGgBp_rf0ucP=GSd6kQ2iH4g@mail.gmail.com>
Message-ID: <D073E1B0.12623%jefft0@remap.ucla.edu>

Hello,

           ByteBuffer contentBuffer = contentArr[i];
            contentBuffer.flip();
            Blob contentBlob = new Blob(contentBuffer.array());
            Name.Component nComp = new Name.Component(contentBlob);

What is in contentArr?  The "segment" that goes in a Name component is just an integer for the number of the segment, starting from 0. (It is not the segment data itself.) In the name component, these are encoded as a short byte array. As described in
http://named-data.net/publications/techreports/ndn-tr-22-ndn-memo-naming-conventions/

The name has the segment number. This is the name of a data packet, where the content is the actual bytes for the segment of the file.

Thanks,
- Jeff T

From: Bassem Labib <bassem.labib at gmail.com<mailto:bassem.labib at gmail.com>>
Date: Saturday, October 25, 2014 at 7:00 AM
To: "ndn-interest at lists.cs.ucla.edu<mailto:ndn-interest at lists.cs.ucla.edu>" <ndn-interest at lists.cs.ucla.edu<mailto:ndn-interest at lists.cs.ucla.edu>>, Jeff Thompson <jefft0 at remap.ucla.edu<mailto:jefft0 at remap.ucla.edu>>
Cc: Steve DiBenedetto <dibenede at cs.colostate.edu<mailto:dibenede at cs.colostate.edu>>
Subject: Re: [Ndn-interest] Send and Receive Video using jNDN Client

Hi Jeff,

I got confused regarding segmenting the chunk file into the Name, and how to use the Data object to send its content, and receive it from the other side to be display.

how can I use a Data.setContent(Blob)?
As the below code:

public void onInterest(Name prefix, Interest interest, Transport transport,long registeredPrefixId)
    {
        Name intrstName = interest.getName();
        // Read the Video file and conver it to ByteBuffer.
        ByteBuffer[] contentArr = (new FileHandler()).readArray();

        for (int i = 0; i < contentArr.length; i++)
        {
            ByteBuffer contentBuffer = contentArr[i];
            contentBuffer.flip();

            Blob contentBlob = new Blob(contentBuffer.array());
            Name.Component nComp = new Name.Component(contentBlob);

            long segment = -1;
            try {
                segment = nComp.toSegment();
            } catch (EncodingException e) {
                e.printStackTrace();
            }
            intrstName = intrstName.appendSegment(segment);
        }

        Data data =  new Data(intrstName);
        //data = data.setContent(contentBlob);   <==== How To Use This
        data.getMetaInfo().setTimestampMilliseconds(System.currentTimeMillis());

        try {
            keyChain_.sign(data, certificateName_);
            Blob encodedData = data.wireEncode();
            transport.send(encodedData.buf());
        }
        catch (IOException | SecurityException ex) {
            System.out.println("Echo: Exception: "+ ex.getMessage());
        }
    }



public void onData(Interest interest, Data data)
    {
        ++callbackCount_;

        //ByteBuffer contentBuffer = data.getContent().buf();

        Name intrstName = interest.getName();
        ByteBuffer[] contentBufferArr = new ByteBuffer[intrstName.size()];
        for(int i=0; i < intrstName.size(); i++)
        {
            contentBufferArr[i] = intrstName.get(i).getValue().buf();
        }

        // Write the recieved data into a file.
        (new FileHandler()).writeArray(contentBufferArr);

    }


This code is not work properly, Please advise if there is other implementation for that issue.

Thanks

On Fri, Oct 24, 2014 at 9:15 AM, Bassem Labib <bassem.labib at gmail.com<mailto:bassem.labib at gmail.com>> wrote:
Hi,

Thanks, for your quick response, I will go through this document and try to implement segments solution.


On Thu, Oct 23, 2014 at 7:27 PM, Thompson, Jeff <jefft0 at remap.ucla.edu<mailto:jefft0 at remap.ucla.edu>> wrote:
Hello,

I'm a little confused. The Java ByteBuffer doesn't have a limit of 8K bytes.  Do you mean a limit on the size of an NDN data packet?  The NDN data packet has a max size of 8800 (including content, signature, etc.)  This is due to network MTU size.  NDN handles this by dividing a file into segments.

Before I explain more, are you familiar with NDN segments and how NDN names segments?  These are described in section 3.2 of "NDN Technical Memo: Naming Conventions":
http://named-data.net/publications/techreports/ndn-tr-22-ndn-memo-naming-conventions/

Adding and parsing segment markers in names is supported in jNDN with Name.appendSegment and Name.Component.toSegment.
http://named-data.net/doc/ndn-ccl-api/name.html#name-appendsegment-method
http://named-data.net/doc/ndn-ccl-api/name-component.html#name-component-tosegment-method

Other than supporing segments in names, jNDN does not yet have a utility to automatically send a file in segments or fetch a file with segments. This is a planned future higher-level API.  For now, the application must divide the file into segments and create names that have a segment marker.  In practice, each application may want to handle this differently. For example a real-time streaming video application may want to drop segments if the network does not deliver them.  But with a file transfer application, the appilcation may want to repeatedly try to fetch a segment if the fetch times out.

Thank you,
- Jeff T

From: Bassem Labib <bassem.labib at gmail.com<mailto:bassem.labib at gmail.com>>
Date: Thursday, October 23, 2014 at 9:15 AM
To: "ndn-interest at lists.cs.ucla.edu<mailto:ndn-interest at lists.cs.ucla.edu>" <ndn-interest at lists.cs.ucla.edu<mailto:ndn-interest at lists.cs.ucla.edu>>
Subject: [Ndn-interest] Send and Receive Video using jNDN Client

Hi,

I am trying to send and receive a video file using jNDN Client.
There is a limitation of the ByteBuffer with max 8K bytes only!!
How can I extend it without receiving it in the onTimeout state.

 Best Regards,
Bassem Labib


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20141027/4f5b76ec/attachment.html>