[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

[Tai-Lin Chu]

Is this due to packet signing? or simply segmentation itself?

Thanks.

On Wed, Apr 8, 2015 at 10:26 AM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>
>
> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger
> <urs.schnurrenberger at unibas.ch> wrote:
>>
>> Hi,
>>
>>
>>
>> why not using ndnputfile? Encoding is done automatically.
>>
>>
>>
>> ndnputfile reponame ndnname filename:
>>
>> e.g.
>>
>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>
>
> Yes, we tried ndnputfile. It is a little bit slow.
> However, the segment size is 1000 by default, and takes about 30 minutes to
> insert a 1.2GB file.
> I changed the segment size to 8000 (the max NDN packet size is 8800), then
> the time reduced to 4 minutes.
>
> I wonder if the Tcp Bulk Insert protocol can improve this more ...
>
> BTW, such large file will generate a lots of segments, I have to set the
> max-packets to a larger number in repo-ng.conf.
> However if the number is too large, repo-ng would complain "ERROR:
> conversion of data to type "i" failed". I think it is a bug.
>
>>
>>
>> Interesting as well would be how to delete files from the repo? I still do
>> it by directly manipulating the database. Seems that there is no tool
>> around…?
>
>
> I didn't see any tool for data deletion either, but the protocol
> specification is given, so it is possible to take some time to  implement
> it.
>
>>
>>
>>
>> Chengyu, don't you have problems to retrieve so large files from the repo?
>> I have already problems with files around 50MB. The developers told me that
>> it might be a memory issue.
>
>
> I don't have this problem.
>
>
>>
>>
>>
>> Best,
>>
>> Urs
>>
>>
>>
>>
>>
>> Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] Im
>> Auftrag von Alex Afanasyev
>> Gesendet: Mittwoch, 8. April 2015 02:03
>> An: Chengyu Fan
>> Cc: ndn-interest at lists.cs.ucla.edu
>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
>> NDN-repo-ng
>>
>>
>>
>>
>>
>> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>
>>
>>
>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data
>> packets into the repo.
>>
>> If need to insert the raw data, I must convert them into NDN packet format
>> encoding in advance. Correct?
>>
>>
>>
>> Yes.
>>
>>
>>
>>
>>
>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev
>> <alexander.afanasyev at ucla.edu> wrote:
>>
>> TcpBulkInsert is not really a protocol, rather a socket that accepts a
>> stream of data packets.
>>
>>
>>
>> Assuming file bulk-data.tlv contains a collection of data packets in NDN
>> packet format encoding (just piled up, one after another one), you can use
>> ‘nc’ command to push them into the repo
>>
>>
>>
>> nc localhost 7376 < bulk-data.tlv
>>
>>
>>
>> —
>>
>> Alex
>>
>>
>>
>>
>>
>> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>
>>
>>
>> Hi,
>>
>>
>>
>> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in
>> NDN-repo-ng?
>>
>>
>>
>> We have lots of data need to insert into the repository, and each one
>> typically contains about 1.2GB.
>>
>>
>>
>> According to the wiki
>> (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp Bulk Insert
>> Repo Insertion Protocol" might be the right option.
>>
>>
>>
>> However, the wiki doesn't contain the protocol specification, and there is
>> no tool for this protocol either ...
>>
>>
>>
>> Thanks,
>>
>> Chengyu
>>
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>
>>
>>
>>
>>
>>
>
>
>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>