<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger <span dir="ltr"><<a href="mailto:urs.schnurrenberger@unibas.ch" target="_blank">urs.schnurrenberger@unibas.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div lang="DE-CH" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif">why not using ndnputfile? Encoding is done automatically.</span></p></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="DE-CH" link="blue" vlink="purple"><div><p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif">ndnputfile reponame ndnname filename:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif">e.g.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif">ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt</span></p></div></div></blockquote><div><br></div><div>Yes, we tried ndnputfile. It is a little bit slow.</div><div>However, the segment size is 1000 by default, and takes about 30 minutes to insert a 1.2GB file.</div><div>I changed the segment size to 8000 (the max NDN packet size is 8800), then the time reduced to 4 minutes.</div><div> </div><div>I wonder if the Tcp Bulk Insert protocol can improve this more ...</div><div><br class="">BTW, such large file will generate a lots of segments, I have to set the max-packets to a larger number in repo-ng.conf.<br></div><div>However if the number is too large, repo-ng would complain "ERROR: conversion of data to type "i" failed". I think it is a bug.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="DE-CH" link="blue" vlink="purple"><div><p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif">Interesting as well would be how to delete files from the repo? I still do it by directly manipulating the database. Seems that there is no tool
around…?</span></p></div></div></blockquote><div><br></div><div>I didn't see any tool for data deletion either, but the protocol specification is given, so it is possible to take some time to implement it.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="DE-CH" link="blue" vlink="purple"><div><p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif">Chengyu, don't you have problems to retrieve so large files from the repo? I have already problems with files around 50MB. The developers told
me that it might be a memory issue.</span></p></div></div></blockquote><div><br></div><div>I don't have this problem. </div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="DE-CH" link="blue" vlink="purple"><div><p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:'Malgun Gothic',sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:'Malgun Gothic',sans-serif">Best,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:'Malgun Gothic',sans-serif">Urs<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:'Malgun Gothic',sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:'Malgun Gothic',sans-serif"><u></u> <u></u></span></p>
<div>
<div style="border-style:solid none none;border-top-color:rgb(225,225,225);border-top-width:1pt;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="DE" style="font-size:11pt;font-family:Calibri,sans-serif">Von:</span></b><span lang="DE" style="font-size:11pt;font-family:Calibri,sans-serif"> Ndn-interest [mailto:<a href="mailto:ndn-interest-bounces@lists.cs.ucla.edu" target="_blank">ndn-interest-bounces@lists.cs.ucla.edu</a>]
<b>Im Auftrag von </b>Alex Afanasyev<br>
<b>Gesendet:</b> Mittwoch, 8. April 2015 02:03<br>
<b>An:</b> Chengyu Fan<br>
<b>Cc:</b> <a href="mailto:ndn-interest@lists.cs.ucla.edu" target="_blank">ndn-interest@lists.cs.ucla.edu</a><br>
<b>Betreff:</b> Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng<u></u><u></u></span></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal"><span lang="EN-US">On Apr 7, 2015, at 5:01 PM, Chengyu Fan <</span><a href="mailto:chengyu@cs.colostate.edu" target="_blank"><span lang="EN-US">chengyu@cs.colostate.edu</span></a><span lang="EN-US">> wrote:<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Okay, so the TcpBulkInsert is to push the </span><span lang="EN-US" style="font-size:9.5pt">NDN packet format encoding data packets into the repo.</span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.5pt">If need to insert the raw data, I must convert them into NDN packet format encoding in advance. Correct?</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Yes.<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><br>
<br>
<u></u><u></u></span></p>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev <</span><a href="mailto:alexander.afanasyev@ucla.edu" target="_blank"><span lang="EN-US">alexander.afanasyev@ucla.edu</span></a><span lang="EN-US">> wrote:<u></u><u></u></span></p>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">TcpBulkInsert is not really a protocol, rather a socket that accepts a stream of data packets.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Assuming file bulk-data.tlv contains a collection of data packets in NDN packet format encoding (just piled up, one after another one), you can use ‘nc’ command to push them into the repo<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">nc localhost 7376 < bulk-data.tlv<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">—<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="color:rgb(136,136,136)">Alex<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="color:rgb(136,136,136)"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal"><span lang="EN-US">On Apr 7, 2015, at 3:28 PM, Chengyu Fan <</span><a href="mailto:chengyu@cs.colostate.edu" target="_blank"><span lang="EN-US">chengyu@cs.colostate.edu</span></a><span lang="EN-US">> wrote:<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi,<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Do anyone know the "</span><strong><span lang="EN-US" style="font-size:9pt;font-family:Verdana,sans-serif;color:rgb(72,72,72)">Tcp Bulk Insert Repo Insertion Protocol</span></strong><span lang="EN-US">" in NDN-repo-ng?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">We have lots of data need to insert into the repository, and each one typically contains about 1.2GB.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">According to the wiki (</span><a href="http://redmine.named-data.net/projects/repo-ng/wiki" target="_blank"><span lang="EN-US">http://redmine.named-data.net/projects/repo-ng/wiki</span></a><span lang="EN-US">), the "</span><strong><span lang="EN-US" style="font-size:9pt;font-family:Verdana,sans-serif;color:rgb(72,72,72)">Tcp
Bulk Insert Repo Insertion Protocol</span></strong><span lang="EN-US">" might be the right option.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">However, the wiki doesn't contain the protocol specification, and there is no tool for this protocol either ...<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Chengyu<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US">_______________________________________________<br>
Ndn-interest mailing list<br>
</span><a href="mailto:Ndn-interest@lists.cs.ucla.edu" target="_blank"><span lang="EN-US">Ndn-interest@lists.cs.ucla.edu</span></a><span lang="EN-US"><br>
</span><a href="http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest" target="_blank"><span lang="EN-US">http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest</span></a><span lang="EN-US"><u></u><u></u></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div></div></div>
</div>
</blockquote></div><br></div></div>