[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

Sat May 2 10:29:45 PDT 2015

I do not restart the repo manually, but I think it automatically restarts every day, independently of NFD.

Cheers,
Urs

Von: chengy.fan at gmail.com [mailto:chengy.fan at gmail.com] Im Auftrag von Chengyu Fan
Gesendet: Donnerstag, 9. April 2015 18:45
An: Urs Schnurrenberger
Cc: Alex Afanasyev; ndn-interest at lists.cs.ucla.edu
Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

On Thu, Apr 9, 2015 at 2:44 AM, Urs Schnurrenberger <urs.schnurrenberger at unibas.ch<mailto:urs.schnurrenberger at unibas.ch>> wrote:
Thanks Chengyu!

Yes, I also had to increase the max-packets value. How do you increase the segment size?

I modified the ndnputfile code.

The strange thing is that I can retrieve (a segment of) the 50 MB file right after insertion into the repo, but one day later (at the latest) the retrieval fails. I get neither an error nor data. Smaller files always work.

I need to try this ...
Could you tell me how do you run the repo-ng? Do you keep it running, or relaunch it one day later?

Urs

Von: chengy.fan at gmail.com<mailto:chengy.fan at gmail.com> [mailto:chengy.fan at gmail.com<mailto:chengy.fan at gmail.com>] Im Auftrag von Chengyu Fan
Gesendet: Mittwoch, 8. April 2015 19:27
An: Urs Schnurrenberger
Cc: Alex Afanasyev; ndn-interest at lists.cs.ucla.edu<mailto:ndn-interest at lists.cs.ucla.edu>

Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger <urs.schnurrenberger at unibas.ch<mailto:urs.schnurrenberger at unibas.ch>> wrote:
Hi,

why not using ndnputfile? Encoding is done automatically.

ndnputfile reponame ndnname filename:
e.g.
ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt

Yes, we tried ndnputfile. It is a little bit slow.
However, the segment size is 1000 by default, and takes about 30 minutes to insert a 1.2GB file.
I changed the segment size to 8000 (the max NDN packet size is 8800), then the time reduced to 4 minutes.

I wonder if the Tcp Bulk Insert protocol can improve this more ...

BTW, such large file will generate a lots of segments, I have to set the max-packets to a larger number in repo-ng.conf.
However if the number is too large, repo-ng would complain "ERROR: conversion of data to type "i" failed". I think it is a bug.

Interesting as well would be how to delete files from the repo? I still do it by directly manipulating the database. Seems that there is no tool around…?

I didn't see any tool for data deletion either, but the protocol specification is given, so it is possible to take some time to  implement it.

Chengyu, don't you have problems to retrieve so large files from the repo? I have already problems with files around 50MB. The developers told me that it might be a memory issue.

I don't have this problem.

Best,
Urs

Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu<mailto:ndn-interest-bounces at lists.cs.ucla.edu>] Im Auftrag von Alex Afanasyev
Gesendet: Mittwoch, 8. April 2015 02:03
An: Chengyu Fan
Cc: ndn-interest at lists.cs.ucla.edu<mailto:ndn-interest at lists.cs.ucla.edu>
Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu<mailto:chengyu at cs.colostate.edu>> wrote:

Okay, so the TcpBulkInsert is to push the NDN packet format encoding data packets into the repo.
If need to insert the raw data, I must convert them into NDN packet format encoding in advance. Correct?

Yes.

On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev <alexander.afanasyev at ucla.edu<mailto:alexander.afanasyev at ucla.edu>> wrote:
TcpBulkInsert is not really a protocol, rather a socket that accepts a stream of data packets.

Assuming file bulk-data.tlv contains a collection of data packets in NDN packet format encoding (just piled up, one after another one), you can use ‘nc’ command to push them into the repo

nc localhost 7376 < bulk-data.tlv

—
Alex

On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu<mailto:chengyu at cs.colostate.edu>> wrote:

Hi,

Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in NDN-repo-ng?

We have lots of data need to insert into the repository, and each one typically contains about 1.2GB.

According to the wiki (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp Bulk Insert Repo Insertion Protocol" might be the right option.

However, the wiki doesn't contain the protocol specification, and there is no tool for this protocol either ...

Thanks,
Chengyu
_______________________________________________
Ndn-interest mailing list
Ndn-interest at lists.cs.ucla.edu<mailto:Ndn-interest at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20150502/e45da06f/attachment.html>