[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng
Peter Lee
peter at corenova.com
Thu Apr 9 10:05:25 PDT 2015
I'm new to the list/project and curious to understand how client initiated upload of content is facilitated in ndn protocol.
Does Interest packet carry data to named destination? Or does the client "register" a new named data source and the destination reciprocates Interest back to the client for streaming Data packets the other direction?
Peter Lee
> On Apr 9, 2015, at 9:45 AM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>
>
>> On Thu, Apr 9, 2015 at 2:44 AM, Urs Schnurrenberger <urs.schnurrenberger at unibas.ch> wrote:
>> Thanks Chengyu!
>>
>>
>>
>> Yes, I also had to increase the max-packets value. How do you increase the segment size?
>>
>
> I modified the ndnputfile code.
>
>>
>>
>> The strange thing is that I can retrieve (a segment of) the 50 MB file right after insertion into the repo, but one day later (at the latest) the retrieval fails. I get neither an error nor data. Smaller files always work.
>>
>
> I need to try this ...
> Could you tell me how do you run the repo-ng? Do you keep it running, or relaunch it one day later?
>
>>
>>
>> Urs
>>
>>
>>
>> Von: chengy.fan at gmail.com [mailto:chengy.fan at gmail.com] Im Auftrag von Chengyu Fan
>> Gesendet: Mittwoch, 8. April 2015 19:27
>> An: Urs Schnurrenberger
>> Cc: Alex Afanasyev; ndn-interest at lists.cs.ucla.edu
>>
>>
>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng
>>
>>
>>
>>
>>
>>
>> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger <urs.schnurrenberger at unibas.ch> wrote:
>>
>> Hi,
>>
>>
>>
>> why not using ndnputfile? Encoding is done automatically.
>>
>>
>>
>> ndnputfile reponame ndnname filename:
>>
>> e.g.
>>
>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>>
>>
>>
>> Yes, we tried ndnputfile. It is a little bit slow.
>>
>> However, the segment size is 1000 by default, and takes about 30 minutes to insert a 1.2GB file.
>>
>> I changed the segment size to 8000 (the max NDN packet size is 8800), then the time reduced to 4 minutes.
>>
>>
>>
>> I wonder if the Tcp Bulk Insert protocol can improve this more ...
>>
>>
>> BTW, such large file will generate a lots of segments, I have to set the max-packets to a larger number in repo-ng.conf.
>>
>> However if the number is too large, repo-ng would complain "ERROR: conversion of data to type "i" failed". I think it is a bug.
>>
>>
>>
>>
>>
>> Interesting as well would be how to delete files from the repo? I still do it by directly manipulating the database. Seems that there is no tool around…?
>>
>>
>>
>> I didn't see any tool for data deletion either, but the protocol specification is given, so it is possible to take some time to implement it.
>>
>>
>>
>>
>>
>> Chengyu, don't you have problems to retrieve so large files from the repo? I have already problems with files around 50MB. The developers told me that it might be a memory issue.
>>
>>
>>
>> I don't have this problem.
>>
>>
>>
>>
>>
>>
>>
>> Best,
>>
>> Urs
>>
>>
>>
>>
>>
>> Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] Im Auftrag von Alex Afanasyev
>> Gesendet: Mittwoch, 8. April 2015 02:03
>> An: Chengyu Fan
>> Cc: ndn-interest at lists.cs.ucla.edu
>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng
>>
>>
>>
>>
>>
>> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>
>>
>>
>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data packets into the repo.
>>
>> If need to insert the raw data, I must convert them into NDN packet format encoding in advance. Correct?
>>
>>
>>
>> Yes.
>>
>>
>>
>>
>>
>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev <alexander.afanasyev at ucla.edu> wrote:
>>
>> TcpBulkInsert is not really a protocol, rather a socket that accepts a stream of data packets.
>>
>>
>>
>> Assuming file bulk-data.tlv contains a collection of data packets in NDN packet format encoding (just piled up, one after another one), you can use ‘nc’ command to push them into the repo
>>
>>
>>
>> nc localhost 7376 < bulk-data.tlv
>>
>>
>>
>> —
>>
>> Alex
>>
>>
>>
>>
>>
>> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>
>>
>>
>> Hi,
>>
>>
>>
>> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in NDN-repo-ng?
>>
>>
>>
>> We have lots of data need to insert into the repository, and each one typically contains about 1.2GB.
>>
>>
>>
>> According to the wiki (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp Bulk Insert Repo Insertion Protocol" might be the right option.
>>
>>
>>
>> However, the wiki doesn't contain the protocol specification, and there is no tool for this protocol either ...
>>
>>
>>
>> Thanks,
>>
>> Chengyu
>>
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20150409/7eed8c0d/attachment.html>
More information about the Ndn-interest
mailing list