<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">If you use Consumer / Producer API, it has an option FAST_SIGNING that speeds up signing by ~32 times.<div><br></div><div>Ilya</div><div><br><div><div>On Apr 8, 2015, at 12:55 PM, Chengyu Fan <<a href="mailto:chengyu@cs.colostate.edu">chengyu@cs.colostate.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">I think it is due to the packet signing.<div>ndnputfile signs each segment, not the whole data. Therefore, if the segment size is small, ndnputfile has to sign more segments.</div><div><br></div><div>Wonder if there are efficient signing methods for big data provided in the ndn-cxx library?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 8, 2015 at 1:24 PM, Tai-Lin Chu <span dir="ltr"><<a href="mailto:tailinchu@gmail.com" target="_blank">tailinchu@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Is this due to packet signing? or simply segmentation itself?<br>
<br>
Thanks.<br>
<div class="HOEnZb"><div class="h5"><br>
On Wed, Apr 8, 2015 at 10:26 AM, Chengyu Fan <<a href="mailto:chengyu@cs.colostate.edu">chengyu@cs.colostate.edu</a>> wrote:<br>
><br>
><br>
> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger<br>
> <<a href="mailto:urs.schnurrenberger@unibas.ch">urs.schnurrenberger@unibas.ch</a>> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>><br>
>><br>
>> why not using ndnputfile? Encoding is done automatically.<br>
>><br>
>><br>
>><br>
>> ndnputfile reponame ndnname filename:<br>
>><br>
>> e.g.<br>
>><br>
>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt<br>
><br>
><br>
> Yes, we tried ndnputfile. It is a little bit slow.<br>
> However, the segment size is 1000 by default, and takes about 30 minutes to<br>
> insert a 1.2GB file.<br>
> I changed the segment size to 8000 (the max NDN packet size is 8800), then<br>
> the time reduced to 4 minutes.<br>
><br>
> I wonder if the Tcp Bulk Insert protocol can improve this more ...<br>
><br>
> BTW, such large file will generate a lots of segments, I have to set the<br>
> max-packets to a larger number in repo-ng.conf.<br>
> However if the number is too large, repo-ng would complain "ERROR:<br>
> conversion of data to type "i" failed". I think it is a bug.<br>
><br>
>><br>
>><br>
>> Interesting as well would be how to delete files from the repo? I still do<br>
>> it by directly manipulating the database. Seems that there is no tool<br>
>> around…?<br>
><br>
><br>
> I didn't see any tool for data deletion either, but the protocol<br>
> specification is given, so it is possible to take some time to implement<br>
> it.<br>
><br>
>><br>
>><br>
>><br>
>> Chengyu, don't you have problems to retrieve so large files from the repo?<br>
>> I have already problems with files around 50MB. The developers told me that<br>
>> it might be a memory issue.<br>
><br>
><br>
> I don't have this problem.<br>
><br>
><br>
>><br>
>><br>
>><br>
>> Best,<br>
>><br>
>> Urs<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> Von: Ndn-interest [mailto:<a href="mailto:ndn-interest-bounces@lists.cs.ucla.edu">ndn-interest-bounces@lists.cs.ucla.edu</a>] Im<br>
>> Auftrag von Alex Afanasyev<br>
>> Gesendet: Mittwoch, 8. April 2015 02:03<br>
>> An: Chengyu Fan<br>
>> Cc: <a href="mailto:ndn-interest@lists.cs.ucla.edu">ndn-interest@lists.cs.ucla.edu</a><br>
>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in<br>
>> NDN-repo-ng<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <<a href="mailto:chengyu@cs.colostate.edu">chengyu@cs.colostate.edu</a>> wrote:<br>
>><br>
>><br>
>><br>
>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data<br>
>> packets into the repo.<br>
>><br>
>> If need to insert the raw data, I must convert them into NDN packet format<br>
>> encoding in advance. Correct?<br>
>><br>
>><br>
>><br>
>> Yes.<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev<br>
>> <<a href="mailto:alexander.afanasyev@ucla.edu">alexander.afanasyev@ucla.edu</a>> wrote:<br>
>><br>
>> TcpBulkInsert is not really a protocol, rather a socket that accepts a<br>
>> stream of data packets.<br>
>><br>
>><br>
>><br>
>> Assuming file bulk-data.tlv contains a collection of data packets in NDN<br>
>> packet format encoding (just piled up, one after another one), you can use<br>
>> ‘nc’ command to push them into the repo<br>
>><br>
>><br>
>><br>
>> nc localhost 7376 < bulk-data.tlv<br>
>><br>
>><br>
>><br>
>> —<br>
>><br>
>> Alex<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <<a href="mailto:chengyu@cs.colostate.edu">chengyu@cs.colostate.edu</a>> wrote:<br>
>><br>
>><br>
>><br>
>> Hi,<br>
>><br>
>><br>
>><br>
>> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in<br>
>> NDN-repo-ng?<br>
>><br>
>><br>
>><br>
>> We have lots of data need to insert into the repository, and each one<br>
>> typically contains about 1.2GB.<br>
>><br>
>><br>
>><br>
>> According to the wiki<br>
>> (<a href="http://redmine.named-data.net/projects/repo-ng/wiki" target="_blank">http://redmine.named-data.net/projects/repo-ng/wiki</a>), the "Tcp Bulk Insert<br>
>> Repo Insertion Protocol" might be the right option.<br>
>><br>
>><br>
>><br>
>> However, the wiki doesn't contain the protocol specification, and there is<br>
>> no tool for this protocol either ...<br>
>><br>
>><br>
>><br>
>> Thanks,<br>
>><br>
>> Chengyu<br>
>><br>
>> _______________________________________________<br>
>> Ndn-interest mailing list<br>
>> <a href="mailto:Ndn-interest@lists.cs.ucla.edu">Ndn-interest@lists.cs.ucla.edu</a><br>
>> <a href="http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest" target="_blank">http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest</a><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Ndn-interest mailing list<br>
> <a href="mailto:Ndn-interest@lists.cs.ucla.edu">Ndn-interest@lists.cs.ucla.edu</a><br>
> <a href="http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest" target="_blank">http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest</a><br>
><br>
</div></div></blockquote></div><br></div>
_______________________________________________<br>Ndn-interest mailing list<br><a href="mailto:Ndn-interest@lists.cs.ucla.edu">Ndn-interest@lists.cs.ucla.edu</a><br>http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest<br></blockquote></div><br></div></body></html>