[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

Ilya Moiseenko iliamo at CS.UCLA.EDU
Wed Apr 8 14:09:20 PDT 2015


On Apr 8, 2015, at 1:52 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:

> I see, so the validation is done at the consumer side, not on the repo.
> 
> BTW, could you also tell me what the RDR, SDR, and UDR stand for?

RDR - reliable retrieval of multi-segment content (multiple data packets). It persistently retransmits interests for lost or unverified data packets, and performs automatic reassembly of segments (Data packets)
UDR - unreliable retrieval of multi-segment content (multiple data packets). Doesn’t do retransmissions or packet reassembly, but sends multiple Interest packets until it reaches finalBlockId.
SDR - unreliable retrieval of a single-segment content — pretty much the same as sending out one Interest packet.

Ilya

> 
> On Wed, Apr 8, 2015 at 2:24 PM, Ilya Moiseenko <iliamo at cs.ucla.edu> wrote:
> 
> On Apr 8, 2015, at 1:17 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
> 
>> Thanks Dave and Ilya.
>> 
>> I was just about to ask how the FAST_SIGNING works ...
>> 
>> It's sure that just signing manifests would improve the performance, but not sure if we still can utilize the repo-ng's validator to verify the content?
> 
> We have repo-ng with disabled validation. Packets are correctly validated at the time of their retrieval if you use RDR protocol in the consumer API.
> 
> Ilya
> 
>> 
>> On Wed, Apr 8, 2015 at 2:07 PM, Ilya Moiseenko <iliamo at cs.ucla.edu> wrote:
>> On Apr 8, 2015, at 1:03 PM, Dave Oran (oran) <oran at cisco.com> wrote:
>> 
>> > Change the architecture - only sign manifests, not individual packets.
>> Hello Dave, this is how the API provides the speed increase.
>> 
>> Ilya
>> 
>> >
>> >> On Apr 8, 2015, at 12:55 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>> >>
>> >> I think it is due to the packet signing.
>> >> ndnputfile signs each segment, not the whole data. Therefore, if the segment size is small, ndnputfile has to sign more segments.
>> >>
>> >> Wonder if there are efficient signing methods for big data provided in the ndn-cxx library?
>> >>
>> >> On Wed, Apr 8, 2015 at 1:24 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>> >> Is this due to packet signing? or simply segmentation itself?
>> >>
>> >> Thanks.
>> >>
>> >> On Wed, Apr 8, 2015 at 10:26 AM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>> >>>
>> >>>
>> >>> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger
>> >>> <urs.schnurrenberger at unibas.ch> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>>
>> >>>>
>> >>>> why not using ndnputfile? Encoding is done automatically.
>> >>>>
>> >>>>
>> >>>>
>> >>>> ndnputfile reponame ndnname filename:
>> >>>>
>> >>>> e.g.
>> >>>>
>> >>>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>> >>>
>> >>>
>> >>> Yes, we tried ndnputfile. It is a little bit slow.
>> >>> However, the segment size is 1000 by default, and takes about 30 minutes to
>> >>> insert a 1.2GB file.
>> >>> I changed the segment size to 8000 (the max NDN packet size is 8800), then
>> >>> the time reduced to 4 minutes.
>> >>>
>> >>> I wonder if the Tcp Bulk Insert protocol can improve this more ...
>> >>>
>> >>> BTW, such large file will generate a lots of segments, I have to set the
>> >>> max-packets to a larger number in repo-ng.conf.
>> >>> However if the number is too large, repo-ng would complain "ERROR:
>> >>> conversion of data to type "i" failed". I think it is a bug.
>> >>>
>> >>>>
>> >>>>
>> >>>> Interesting as well would be how to delete files from the repo? I still do
>> >>>> it by directly manipulating the database. Seems that there is no tool
>> >>>> around…?
>> >>>
>> >>>
>> >>> I didn't see any tool for data deletion either, but the protocol
>> >>> specification is given, so it is possible to take some time to  implement
>> >>> it.
>> >>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Chengyu, don't you have problems to retrieve so large files from the repo?
>> >>>> I have already problems with files around 50MB. The developers told me that
>> >>>> it might be a memory issue.
>> >>>
>> >>>
>> >>> I don't have this problem.
>> >>>
>> >>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Best,
>> >>>>
>> >>>> Urs
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] Im
>> >>>> Auftrag von Alex Afanasyev
>> >>>> Gesendet: Mittwoch, 8. April 2015 02:03
>> >>>> An: Chengyu Fan
>> >>>> Cc: ndn-interest at lists.cs.ucla.edu
>> >>>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
>> >>>> NDN-repo-ng
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data
>> >>>> packets into the repo.
>> >>>>
>> >>>> If need to insert the raw data, I must convert them into NDN packet format
>> >>>> encoding in advance. Correct?
>> >>>>
>> >>>>
>> >>>>
>> >>>> Yes.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev
>> >>>> <alexander.afanasyev at ucla.edu> wrote:
>> >>>>
>> >>>> TcpBulkInsert is not really a protocol, rather a socket that accepts a
>> >>>> stream of data packets.
>> >>>>
>> >>>>
>> >>>>
>> >>>> Assuming file bulk-data.tlv contains a collection of data packets in NDN
>> >>>> packet format encoding (just piled up, one after another one), you can use
>> >>>> ‘nc’ command to push them into the repo
>> >>>>
>> >>>>
>> >>>>
>> >>>> nc localhost 7376 < bulk-data.tlv
>> >>>>
>> >>>>
>> >>>>
>> >>>> —
>> >>>>
>> >>>> Alex
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>>
>> >>>>
>> >>>> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in
>> >>>> NDN-repo-ng?
>> >>>>
>> >>>>
>> >>>>
>> >>>> We have lots of data need to insert into the repository, and each one
>> >>>> typically contains about 1.2GB.
>> >>>>
>> >>>>
>> >>>>
>> >>>> According to the wiki
>> >>>> (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp Bulk Insert
>> >>>> Repo Insertion Protocol" might be the right option.
>> >>>>
>> >>>>
>> >>>>
>> >>>> However, the wiki doesn't contain the protocol specification, and there is
>> >>>> no tool for this protocol either ...
>> >>>>
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Chengyu
>> >>>>
>> >>>> _______________________________________________
>> >>>> Ndn-interest mailing list
>> >>>> Ndn-interest at lists.cs.ucla.edu
>> >>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Ndn-interest mailing list
>> >>> Ndn-interest at lists.cs.ucla.edu
>> >>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> >>>
>> >>
>> >> _______________________________________________
>> >> Ndn-interest mailing list
>> >> Ndn-interest at lists.cs.ucla.edu
>> >> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> >
>> >
>> > _______________________________________________
>> > Ndn-interest mailing list
>> > Ndn-interest at lists.cs.ucla.edu
>> > http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20150408/e327326b/attachment.html>


More information about the Ndn-interest mailing list