[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

Peter Lee peter at corenova.com
Thu Apr 9 10:05:25 PDT 2015


I'm new to the list/project and curious to understand how client initiated upload of content is facilitated in ndn protocol. 

Does Interest packet carry data to named destination?  Or does the client "register" a new named data source and the destination reciprocates Interest back to the client for streaming Data packets the other direction?

Peter Lee


> On Apr 9, 2015, at 9:45 AM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
> 
> 
>> On Thu, Apr 9, 2015 at 2:44 AM, Urs Schnurrenberger <urs.schnurrenberger at unibas.ch> wrote:
>> Thanks Chengyu!
>> 
>>  
>> 
>> Yes, I also had to increase the max-packets value. How do you increase the segment size?
>> 
> 
> I modified the ndnputfile code.
>  
>>  
>> 
>> The strange thing is that I can retrieve (a segment of) the 50 MB file right after insertion into the repo, but one day later (at the latest) the retrieval fails. I get neither an error nor data. Smaller files always work.
>> 
> 
> I need to try this ...
> Could you tell me how do you run the repo-ng? Do you keep it running, or relaunch it one day later?
>  
>>  
>> 
>> Urs
>> 
>>  
>> 
>> Von: chengy.fan at gmail.com [mailto:chengy.fan at gmail.com] Im Auftrag von Chengyu Fan
>> Gesendet: Mittwoch, 8. April 2015 19:27
>> An: Urs Schnurrenberger
>> Cc: Alex Afanasyev; ndn-interest at lists.cs.ucla.edu
>> 
>> 
>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng
>>  
>> 
>>  
>> 
>>  
>> 
>> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger <urs.schnurrenberger at unibas.ch> wrote:
>> 
>> Hi,
>> 
>>  
>> 
>> why not using ndnputfile? Encoding is done automatically.
>> 
>>  
>> 
>> ndnputfile reponame ndnname filename:
>> 
>> e.g.
>> 
>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>> 
>>  
>> 
>> Yes, we tried ndnputfile. It is a little bit slow.
>> 
>> However, the segment size is 1000 by default, and takes about 30 minutes to insert a 1.2GB file.
>> 
>> I changed the segment size to 8000 (the max NDN packet size is 8800), then the time reduced to 4 minutes.
>> 
>>  
>> 
>> I wonder if the Tcp Bulk Insert protocol can improve this more ...
>> 
>> 
>> BTW, such large file will generate a lots of segments, I have to set the max-packets to a larger number in repo-ng.conf.
>> 
>> However if the number is too large, repo-ng would complain "ERROR: conversion of data to type "i" failed". I think it is a bug.
>> 
>>  
>> 
>>  
>> 
>> Interesting as well would be how to delete files from the repo? I still do it by directly manipulating the database. Seems that there is no tool around…?
>> 
>>  
>> 
>> I didn't see any tool for data deletion either, but the protocol specification is given, so it is possible to take some time to  implement it.
>> 
>>  
>> 
>>  
>> 
>> Chengyu, don't you have problems to retrieve so large files from the repo? I have already problems with files around 50MB. The developers told me that it might be a memory issue.
>> 
>>  
>> 
>> I don't have this problem. 
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> Best,
>> 
>> Urs
>> 
>>  
>> 
>>  
>> 
>> Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] Im Auftrag von Alex Afanasyev
>> Gesendet: Mittwoch, 8. April 2015 02:03
>> An: Chengyu Fan
>> Cc: ndn-interest at lists.cs.ucla.edu
>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng
>> 
>>  
>> 
>>  
>> 
>> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>> 
>>  
>> 
>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data packets into the repo.
>> 
>> If need to insert the raw data, I must convert them into NDN packet format encoding in advance. Correct?
>> 
>>  
>> 
>> Yes.
>> 
>>  
>> 
>>  
>> 
>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev <alexander.afanasyev at ucla.edu> wrote:
>> 
>> TcpBulkInsert is not really a protocol, rather a socket that accepts a stream of data packets.
>> 
>>  
>> 
>> Assuming file bulk-data.tlv contains a collection of data packets in NDN packet format encoding (just piled up, one after another one), you can use ‘nc’ command to push them into the repo
>> 
>>  
>> 
>> nc localhost 7376 < bulk-data.tlv
>> 
>>  
>> 
>>>> 
>> Alex
>> 
>>  
>> 
>>  
>> 
>> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>> 
>>  
>> 
>> Hi,
>> 
>>  
>> 
>> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in NDN-repo-ng?
>> 
>>  
>> 
>> We have lots of data need to insert into the repository, and each one typically contains about 1.2GB.
>> 
>>  
>> 
>> According to the wiki (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp Bulk Insert Repo Insertion Protocol" might be the right option.
>> 
>>  
>> 
>> However, the wiki doesn't contain the protocol specification, and there is no tool for this protocol either ...
>> 
>>  
>> 
>> Thanks,
>> 
>> Chengyu
>> 
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
> 
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20150409/7eed8c0d/attachment.html>


More information about the Ndn-interest mailing list