[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

Chengyu Fan chengyu at cs.colostate.edu
Wed Apr 8 10:26:46 PDT 2015


On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger <
urs.schnurrenberger at unibas.ch> wrote:

>  Hi,
>
>
>
> why not using ndnputfile? Encoding is done automatically.
>

>
> ndnputfile reponame ndnname filename:
>
> e.g.
>
> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>

Yes, we tried ndnputfile. It is a little bit slow.
However, the segment size is 1000 by default, and takes about 30 minutes to
insert a 1.2GB file.
I changed the segment size to 8000 (the max NDN packet size is 8800), then
the time reduced to 4 minutes.

I wonder if the Tcp Bulk Insert protocol can improve this more ...

BTW, such large file will generate a lots of segments, I have to set the
max-packets to a larger number in repo-ng.conf.
However if the number is too large, repo-ng would complain "ERROR:
conversion of data to type "i" failed". I think it is a bug.


>
> Interesting as well would be how to delete files from the repo? I still do
> it by directly manipulating the database. Seems that there is no tool
> around…?
>

I didn't see any tool for data deletion either, but the protocol
specification is given, so it is possible to take some time to  implement
it.


>
>
> Chengyu, don't you have problems to retrieve so large files from the repo?
> I have already problems with files around 50MB. The developers told me that
> it might be a memory issue.
>

I don't have this problem.



>
>
> Best,
>
> Urs
>
>
>
>
>
> *Von:* Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] *Im
> Auftrag von *Alex Afanasyev
> *Gesendet:* Mittwoch, 8. April 2015 02:03
> *An:* Chengyu Fan
> *Cc:* ndn-interest at lists.cs.ucla.edu
> *Betreff:* Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
> NDN-repo-ng
>
>
>
>
>
>  On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>
>
>
> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data
> packets into the repo.
>
> If need to insert the raw data, I must convert them into NDN packet format
> encoding in advance. Correct?
>
>
>
> Yes.
>
>
>
>
>
> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev <
> alexander.afanasyev at ucla.edu> wrote:
>
>  TcpBulkInsert is not really a protocol, rather a socket that accepts a
> stream of data packets.
>
>
>
> Assuming file bulk-data.tlv contains a collection of data packets in NDN
> packet format encoding (just piled up, one after another one), you can use
> ‘nc’ command to push them into the repo
>
>
>
> nc localhost 7376 < bulk-data.tlv
>
>
>
>>
> Alex
>
>
>
>
>
>  On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>
>
>
> Hi,
>
>
>
> Do anyone know the "*Tcp Bulk Insert Repo Insertion Protocol*" in
> NDN-repo-ng?
>
>
>
> We have lots of data need to insert into the repository, and each one
> typically contains about 1.2GB.
>
>
>
> According to the wiki (http://redmine.named-data.net/projects/repo-ng/wiki),
> the "*Tcp Bulk Insert Repo Insertion Protocol*" might be the right option.
>
>
>
> However, the wiki doesn't contain the protocol specification, and there is
> no tool for this protocol either ...
>
>
>
> Thanks,
>
> Chengyu
>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20150408/cadb7096/attachment.html>


More information about the Ndn-interest mailing list