[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

[Ilya Moiseenko]

On Apr 8, 2015, at 1:03 PM, Dave Oran (oran) <oran at cisco.com> wrote:

> Change the architecture - only sign manifests, not individual packets.
Hello Dave, this is how the API provides the speed increase.

Ilya 

> 
>> On Apr 8, 2015, at 12:55 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>> 
>> I think it is due to the packet signing.
>> ndnputfile signs each segment, not the whole data. Therefore, if the segment size is small, ndnputfile has to sign more segments.
>> 
>> Wonder if there are efficient signing methods for big data provided in the ndn-cxx library?
>> 
>> On Wed, Apr 8, 2015 at 1:24 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>> Is this due to packet signing? or simply segmentation itself?
>> 
>> Thanks.
>> 
>> On Wed, Apr 8, 2015 at 10:26 AM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>> 
>>> 
>>> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger
>>> <urs.schnurrenberger at unibas.ch> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> 
>>>> 
>>>> why not using ndnputfile? Encoding is done automatically.
>>>> 
>>>> 
>>>> 
>>>> ndnputfile reponame ndnname filename:
>>>> 
>>>> e.g.
>>>> 
>>>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>>> 
>>> 
>>> Yes, we tried ndnputfile. It is a little bit slow.
>>> However, the segment size is 1000 by default, and takes about 30 minutes to
>>> insert a 1.2GB file.
>>> I changed the segment size to 8000 (the max NDN packet size is 8800), then
>>> the time reduced to 4 minutes.
>>> 
>>> I wonder if the Tcp Bulk Insert protocol can improve this more ...
>>> 
>>> BTW, such large file will generate a lots of segments, I have to set the
>>> max-packets to a larger number in repo-ng.conf.
>>> However if the number is too large, repo-ng would complain "ERROR:
>>> conversion of data to type "i" failed". I think it is a bug.
>>> 
>>>> 
>>>> 
>>>> Interesting as well would be how to delete files from the repo? I still do
>>>> it by directly manipulating the database. Seems that there is no tool
>>>> around…?
>>> 
>>> 
>>> I didn't see any tool for data deletion either, but the protocol
>>> specification is given, so it is possible to take some time to  implement
>>> it.
>>> 
>>>> 
>>>> 
>>>> 
>>>> Chengyu, don't you have problems to retrieve so large files from the repo?
>>>> I have already problems with files around 50MB. The developers told me that
>>>> it might be a memory issue.
>>> 
>>> 
>>> I don't have this problem.
>>> 
>>> 
>>>> 
>>>> 
>>>> 
>>>> Best,
>>>> 
>>>> Urs
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] Im
>>>> Auftrag von Alex Afanasyev
>>>> Gesendet: Mittwoch, 8. April 2015 02:03
>>>> An: Chengyu Fan
>>>> Cc: ndn-interest at lists.cs.ucla.edu
>>>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
>>>> NDN-repo-ng
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>>> 
>>>> 
>>>> 
>>>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data
>>>> packets into the repo.
>>>> 
>>>> If need to insert the raw data, I must convert them into NDN packet format
>>>> encoding in advance. Correct?
>>>> 
>>>> 
>>>> 
>>>> Yes.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev
>>>> <alexander.afanasyev at ucla.edu> wrote:
>>>> 
>>>> TcpBulkInsert is not really a protocol, rather a socket that accepts a
>>>> stream of data packets.
>>>> 
>>>> 
>>>> 
>>>> Assuming file bulk-data.tlv contains a collection of data packets in NDN
>>>> packet format encoding (just piled up, one after another one), you can use
>>>> ‘nc’ command to push them into the repo
>>>> 
>>>> 
>>>> 
>>>> nc localhost 7376 < bulk-data.tlv
>>>> 
>>>> 
>>>> 
>>>> —
>>>> 
>>>> Alex
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>>> 
>>>> 
>>>> 
>>>> Hi,
>>>> 
>>>> 
>>>> 
>>>> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in
>>>> NDN-repo-ng?
>>>> 
>>>> 
>>>> 
>>>> We have lots of data need to insert into the repository, and each one
>>>> typically contains about 1.2GB.
>>>> 
>>>> 
>>>> 
>>>> According to the wiki
>>>> (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp Bulk Insert
>>>> Repo Insertion Protocol" might be the right option.
>>>> 
>>>> 
>>>> 
>>>> However, the wiki doesn't contain the protocol specification, and there is
>>>> no tool for this protocol either ...
>>>> 
>>>> 
>>>> 
>>>> Thanks,
>>>> 
>>>> Chengyu
>>>> 
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Ndn-interest mailing list
>>> Ndn-interest at lists.cs.ucla.edu
>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>> 
>> 
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
> 
> 
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest