[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

[GTS]

I second Dave's sentiments.
Cheers,
Gene


On 4/8/15, 10:03 PM, Dave Oran (oran) wrote:
> Change the architecture - only sign manifests, not individual packets.
>






>> On Apr 8, 2015, at 12:55 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>
>> I think it is due to the packet signing.
>> ndnputfile signs each segment, not the whole data. Therefore, if the segment size is small, ndnputfile has to sign more segments.
>>
>> Wonder if there are efficient signing methods for big data provided in the ndn-cxx library?
>>
>> On Wed, Apr 8, 2015 at 1:24 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
>> Is this due to packet signing? or simply segmentation itself?
>>
>> Thanks.
>>
>> On Wed, Apr 8, 2015 at 10:26 AM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>>
>>> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger
>>> <urs.schnurrenberger at unibas.ch> wrote:
>>>> Hi,
>>>>
>>>>
>>>>
>>>> why not using ndnputfile? Encoding is done automatically.
>>>>
>>>>
>>>>
>>>> ndnputfile reponame ndnname filename:
>>>>
>>>> e.g.
>>>>
>>>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>>>
>>> Yes, we tried ndnputfile. It is a little bit slow.
>>> However, the segment size is 1000 by default, and takes about 30 minutes to
>>> insert a 1.2GB file.
>>> I changed the segment size to 8000 (the max NDN packet size is 8800), then
>>> the time reduced to 4 minutes.
>>>
>>> I wonder if the Tcp Bulk Insert protocol can improve this more ...
>>>
>>> BTW, such large file will generate a lots of segments, I have to set the
>>> max-packets to a larger number in repo-ng.conf.
>>> However if the number is too large, repo-ng would complain "ERROR:
>>> conversion of data to type "i" failed". I think it is a bug.
>>>
>>>>
>>>> Interesting as well would be how to delete files from the repo? I still do
>>>> it by directly manipulating the database. Seems that there is no tool
>>>> around…?
>>>
>>> I didn't see any tool for data deletion either, but the protocol
>>> specification is given, so it is possible to take some time to  implement
>>> it.
>>>
>>>>
>>>>
>>>> Chengyu, don't you have problems to retrieve so large files from the repo?
>>>> I have already problems with files around 50MB. The developers told me that
>>>> it might be a memory issue.
>>>
>>> I don't have this problem.
>>>
>>>
>>>>
>>>>
>>>> Best,
>>>>
>>>> Urs
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] Im
>>>> Auftrag von Alex Afanasyev
>>>> Gesendet: Mittwoch, 8. April 2015 02:03
>>>> An: Chengyu Fan
>>>> Cc: ndn-interest at lists.cs.ucla.edu
>>>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
>>>> NDN-repo-ng
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>>>
>>>>
>>>>
>>>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data
>>>> packets into the repo.
>>>>
>>>> If need to insert the raw data, I must convert them into NDN packet format
>>>> encoding in advance. Correct?
>>>>
>>>>
>>>>
>>>> Yes.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev
>>>> <alexander.afanasyev at ucla.edu> wrote:
>>>>
>>>> TcpBulkInsert is not really a protocol, rather a socket that accepts a
>>>> stream of data packets.
>>>>
>>>>
>>>>
>>>> Assuming file bulk-data.tlv contains a collection of data packets in NDN
>>>> packet format encoding (just piled up, one after another one), you can use
>>>> ‘nc’ command to push them into the repo
>>>>
>>>>
>>>>
>>>> nc localhost 7376 < bulk-data.tlv
>>>>
>>>>
>>>>
>>>> —
>>>>
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in
>>>> NDN-repo-ng?
>>>>
>>>>
>>>>
>>>> We have lots of data need to insert into the repository, and each one
>>>> typically contains about 1.2GB.
>>>>
>>>>
>>>>
>>>> According to the wiki
>>>> (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp Bulk Insert
>>>> Repo Insertion Protocol" might be the right option.
>>>>
>>>>
>>>>
>>>> However, the wiki doesn't contain the protocol specification, and there is
>>>> no tool for this protocol either ...
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Chengyu
>>>>
>>>> _______________________________________________
>>>> Ndn-interest mailing list
>>>> Ndn-interest at lists.cs.ucla.edu
>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Ndn-interest mailing list
>>> Ndn-interest at lists.cs.ucla.edu
>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest