[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

Chengyu Fan chengyu at cs.colostate.edu
Thu Apr 9 09:45:01 PDT 2015


On Thu, Apr 9, 2015 at 2:44 AM, Urs Schnurrenberger <
urs.schnurrenberger at unibas.ch> wrote:

>  Thanks Chengyu!
>
>
>
> Yes, I also had to increase the max-packets value. How do you increase the
> segment size?
>

I modified the ndnputfile code.


>
>
> The strange thing is that I can retrieve (a segment of) the 50 MB file
> right after insertion into the repo, but one day later (at the latest) the
> retrieval fails. I get neither an error nor data. Smaller files always work.
>

I need to try this ...
Could you tell me how do you run the repo-ng? Do you keep it running, or
relaunch it one day later?


>
>
> Urs
>
>
>
> *Von:* chengy.fan at gmail.com [mailto:chengy.fan at gmail.com] *Im Auftrag von
> *Chengyu Fan
> *Gesendet:* Mittwoch, 8. April 2015 19:27
> *An:* Urs Schnurrenberger
> *Cc:* Alex Afanasyev; ndn-interest at lists.cs.ucla.edu
>
> *Betreff:* Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
> NDN-repo-ng
>
>
>
>
>
>
>
> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger <
> urs.schnurrenberger at unibas.ch> wrote:
>
>  Hi,
>
>
>
> why not using ndnputfile? Encoding is done automatically.
>
>
>
> ndnputfile reponame ndnname filename:
>
> e.g.
>
> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>
>
>
> Yes, we tried ndnputfile. It is a little bit slow.
>
> However, the segment size is 1000 by default, and takes about 30 minutes
> to insert a 1.2GB file.
>
> I changed the segment size to 8000 (the max NDN packet size is 8800), then
> the time reduced to 4 minutes.
>
>
>
> I wonder if the Tcp Bulk Insert protocol can improve this more ...
>
>
> BTW, such large file will generate a lots of segments, I have to set the
> max-packets to a larger number in repo-ng.conf.
>
> However if the number is too large, repo-ng would complain "ERROR:
> conversion of data to type "i" failed". I think it is a bug.
>
>
>
>
>
> Interesting as well would be how to delete files from the repo? I still do
> it by directly manipulating the database. Seems that there is no tool around
> …?
>
>
>
> I didn't see any tool for data deletion either, but the protocol
> specification is given, so it is possible to take some time to  implement
> it.
>
>
>
>
>
> Chengyu, don't you have problems to retrieve so large files from the repo?
> I have already problems with files around 50MB. The developers told me that
> it might be a memory issue.
>
>
>
> I don't have this problem.
>
>
>
>
>
>
>
> Best,
>
> Urs
>
>
>
>
>
> *Von:* Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] *Im
> Auftrag von *Alex Afanasyev
> *Gesendet:* Mittwoch, 8. April 2015 02:03
> *An:* Chengyu Fan
> *Cc:* ndn-interest at lists.cs.ucla.edu
> *Betreff:* Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
> NDN-repo-ng
>
>
>
>
>
>  On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>
>
>
> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data
> packets into the repo.
>
> If need to insert the raw data, I must convert them into NDN packet format
> encoding in advance. Correct?
>
>
>
> Yes.
>
>
>
>
>
> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev <
> alexander.afanasyev at ucla.edu> wrote:
>
>  TcpBulkInsert is not really a protocol, rather a socket that accepts a
> stream of data packets.
>
>
>
> Assuming file bulk-data.tlv contains a collection of data packets in NDN
> packet format encoding (just piled up, one after another one), you can use
> ‘nc’ command to push them into the repo
>
>
>
> nc localhost 7376 < bulk-data.tlv
>
>
>
>>
> Alex
>
>
>
>
>
>  On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>
>
>
> Hi,
>
>
>
> Do anyone know the "*Tcp Bulk Insert Repo Insertion Protocol*" in
> NDN-repo-ng?
>
>
>
> We have lots of data need to insert into the repository, and each one
> typically contains about 1.2GB.
>
>
>
> According to the wiki (http://redmine.named-data.net/projects/repo-ng/wiki),
> the "*Tcp Bulk Insert Repo Insertion Protocol*" might be the right option.
>
>
>
> However, the wiki doesn't contain the protocol specification, and there is
> no tool for this protocol either ...
>
>
>
> Thanks,
>
> Chengyu
>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20150409/6965d9f3/attachment.html>


More information about the Ndn-interest mailing list