[Ndn-interest] FinalBlockId in non-final segment

Wed Jun 27 12:45:49 PDT 2018

Here is a bit more context around the CCNx evolution to build on what Nick wrote.  It also brings up another corner case that likely still applies to NDN too, as the spec is the same as CCNx 0.8.2.

Michael Plass had also told me at one time that it was legal, in the original CCNx, for multiple segments to evaluate IsFinalBlock() as True.  If you look at the 0.8.2 docs [1], you see this is correct.  There is nothing that says “at most one Content Object whose name varies only in the final name component may be the FinalBlock.”  That would not make sense, as there’s not strong control over multi-user publication.  Nor is there a restriction on a single publisher or app behavior.  One could argue that “final” means “final” and there’s nothing after “final,” for which I have some sympathy.  As an example, I stream something under a prefix, then get to what I think is the end and set FinalBlockId such that IsFinalBlock(content) is true.  Then I later stream some more and make new segments and some later (time wise) segment also evaluates to True.  It really depends on the app.  I could be using final name components like ‘alice’ and ‘zack’ and ‘ming’, who was the original FinalBlockId.  Then later ‘lars’ and ‘elizabeth’ show up, so now ‘elizabeth’ is the FinalBlockId.  The spec only specified the condition under which IsFinalBlock() evaluates to True.  It is a very general system.

In the original CCNx (and similarly in NDN), there was an additional convention around segmentation [2] (command marker %00), and the segment number was, by convention, the last explicit name component [3].  So, by convention, CCNx used a sequence number for segments and the FinalBlockId would be the %00 command marker segment.  It is important to note that FinalBlockId is independent of segmentation, so there is no guarantee about this.  Some apps could use a different convention like timestamps (the %FB non-consecutive segments) or some other non-standard mechanism.  Note that NDN changed %FB to be byte-offset segmentation in specific, whereas in 0.8.2 it was just non-sequential and byte offsets were one example.

In CCNx 1.0, we changed it.  At the network level, there is no longer a specification for FinalBlockId or the name component type (command marker) for segments or anything special about the final explicit name component.  It is not a layer-3 concept [4].   We thought that because segmentation is independent from the FinalBlockId concept and that it is really transport or app-dependent what the final block really means, that it did not belong in the layer-3 spec.

We did write a separate specification for the transport/application layer, which we called chunking (as opposed to the original “segmentation”) [5].  The chunking spec is very specific about the binary form of the name component, its meaning, and the EndChunkNumber ContentObject field — and they are tied together.  Because all the fields are defined outside of the layer-3 protocol, it is possible for there to be other chunking/segmentation specs, as there are different specs like UDP/TCP/RTP/SCTP, etc.

Personally, I think the approach we took in CCNx 1.0 is the right one.  The meaning of FinalBlockId is too general for layer-3.  There should be specific ways of detecting end with clearer semantics so a pre-encoded movie could do one thing, a live stream could do another, and a digital fountain multi-layer encoding could do another.  The meaning of FinalBlockId is only significant if you understand the segmentation/chunking protocol, and if you want to allow multiple specifications they need to be outside layer-3.  I think trying to have a general layer-3 FinalBlockId independent of its use in segmentation (which is not layer-3) will lead to confusion above layer-3.

[1] https://github.com/ProjectCCNx/ccnx/blob/master/doc/technical/ContentObject.txt
[2] https://github.com/ProjectCCNx/ccnx/blob/master/doc/technical/NameConventions.txt
[3] This is not an exact statement, as one could have multiple segment numbers, for example ccnx:/foo/%FD01/%0001/bar/%FD02/%0005, where we have chunk 1 of version 1 of ‘foo' containing chunk 5 of version 2 of ‘bar'.  In this case, FinalBlockId is matched against %0005.
[4] By layer-3, I mean the lowest-level packet format spec that must be implemented by all nodes.
[5] https://www.ietf.org/archive/id/draft-mosko-icnrg-ccnxchunking-02.txt

Marc

On Jun 27, 2018, at 11:34 AM, Nick Briggs <nicholas.h.briggs at gmail.com<mailto:nicholas.h.briggs at gmail.com>> wrote:

I agree that NDN should clarify that.

In CCNx the intent was that the FinalBlockId in the packet provided a means to implement "boolean IsFinalBlock(content)", and that it could be implemented without any interpretation of the value of the FinalBlockId.

There was no restriction whatsoever on FinalBlockId values in packets where IsFinalBlock() was false -- any interpretation of FinalBlockId in non-final blocks was above the basic protocol level (e.g., in the generic stream profile, or in a specialized application profile).  The "advanced warning" was one interpretation, others are possible.  If the value of the FinalBlockId field is interpreted care should be given to avoiding failures when unexpected values are presented.

-- Nick

On Jun 27, 2018, at 10:45 AM, Junxiao Shi <shijunxiao at email.arizona.edu<mailto:shijunxiao at email.arizona.edu>> wrote:

Dear folks

I have a question regarding the use of FinalBlockId element in a Data packet that is not the final segment itself.
Note that "segment" in this message could be replaced with "timestamp", "sequence number", etc, because FinalBlockId is not required to be a segment number.

NDN Packet Spec defines<https://named-data.net/doc/NDN-packet-spec/0.3/data.html#finalblockid> FinalBlockId as follows:
The optional FinalBlockId identifies the final block in a sequence of fragments. It should be present in the final block itself, and may also be present in other fragments to provide advanced warning of the end to consumers. The value here should be equal to the last explicit name component of the final block.
CCNx has the same definition<https://github.com/ProjectCCNx/ccnx/blob/5be2b715d0cc6bb4b7c6bb7459d311bddff46834/doc/technical/ContentObject.txt#L82-L88>.

As I remember (but couldn't find records):
In the final segment, FinalBlockId must equal the last explicit name component.
In earlier segments, producer may include FinalBlockId as an "advanced warning" that a stream is ending. In that case, the segment number in FinalBlockId represents an estimation, and is not required to be exact.
A use case is that, a video stream is going to end within 5 seconds but the producer does not know the exact segment number of the last segment, so it provides an estimate so that consumers can stop sending Interests much more beyond that segment number.

Change 4651 patchset 26 treats a different FinalBlockId in a stream as an error: https://gerrit.named-data.net/#/c/ndn-cxx/+/4651/26/src/util/segment-fetcher.hpp@138
Although the later merged code has reverted to my view above, I think the NDN protocol should clarify whether FinalBlockId in non-final segment must be exact, or may be an estimation and therefore could change in later segments.

Yours, Junxiao

_______________________________________________
Ndn-interest mailing list
Ndn-interest at lists.cs.ucla.edu<mailto:Ndn-interest at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest

_______________________________________________
Ndn-interest mailing list
Ndn-interest at lists.cs.ucla.edu<mailto:Ndn-interest at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20180627/36b32548/attachment.html>