[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

Wed Apr 8 13:52:24 PDT 2015

I see, so the validation is done at the consumer side, not on the repo.

BTW, could you also tell me what the RDR, SDR, and UDR stand for?

On Wed, Apr 8, 2015 at 2:24 PM, Ilya Moiseenko <iliamo at cs.ucla.edu> wrote:

>
> On Apr 8, 2015, at 1:17 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>
> Thanks Dave and Ilya.
>
> I was just about to ask how the FAST_SIGNING works ...
>
> It's sure that just signing manifests would improve the performance, but
> not sure if we still can utilize the repo-ng's validator to verify the
> content?
>
>
> We have repo-ng with disabled validation. Packets are correctly validated
> at the time of their retrieval if you use RDR protocol in the consumer API.
>
> Ilya
>
>
> On Wed, Apr 8, 2015 at 2:07 PM, Ilya Moiseenko <iliamo at cs.ucla.edu> wrote:
>
>> On Apr 8, 2015, at 1:03 PM, Dave Oran (oran) <oran at cisco.com> wrote:
>>
>> > Change the architecture - only sign manifests, not individual packets.
>> Hello Dave, this is how the API provides the speed increase.
>>
>> Ilya
>>
>> >
>> >> On Apr 8, 2015, at 12:55 PM, Chengyu Fan <chengyu at cs.colostate.edu>
>> wrote:
>> >>
>> >> I think it is due to the packet signing.
>> >> ndnputfile signs each segment, not the whole data. Therefore, if the
>> segment size is small, ndnputfile has to sign more segments.
>> >>
>> >> Wonder if there are efficient signing methods for big data provided in
>> the ndn-cxx library?
>> >>
>> >> On Wed, Apr 8, 2015 at 1:24 PM, Tai-Lin Chu <tailinchu at gmail.com>
>> wrote:
>> >> Is this due to packet signing? or simply segmentation itself?
>> >>
>> >> Thanks.
>> >>
>> >> On Wed, Apr 8, 2015 at 10:26 AM, Chengyu Fan <chengyu at cs.colostate.edu>
>> wrote:
>> >>>
>> >>>
>> >>> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger
>> >>> <urs.schnurrenberger at unibas.ch> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>>
>> >>>>
>> >>>> why not using ndnputfile? Encoding is done automatically.
>> >>>>
>> >>>>
>> >>>>
>> >>>> ndnputfile reponame ndnname filename:
>> >>>>
>> >>>> e.g.
>> >>>>
>> >>>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>> >>>
>> >>>
>> >>> Yes, we tried ndnputfile. It is a little bit slow.
>> >>> However, the segment size is 1000 by default, and takes about 30
>> minutes to
>> >>> insert a 1.2GB file.
>> >>> I changed the segment size to 8000 (the max NDN packet size is 8800),
>> then
>> >>> the time reduced to 4 minutes.
>> >>>
>> >>> I wonder if the Tcp Bulk Insert protocol can improve this more ...
>> >>>
>> >>> BTW, such large file will generate a lots of segments, I have to set
>> the
>> >>> max-packets to a larger number in repo-ng.conf.
>> >>> However if the number is too large, repo-ng would complain "ERROR:
>> >>> conversion of data to type "i" failed". I think it is a bug.
>> >>>
>> >>>>
>> >>>>
>> >>>> Interesting as well would be how to delete files from the repo? I
>> still do
>> >>>> it by directly manipulating the database. Seems that there is no tool
>> >>>> around…?
>> >>>
>> >>>
>> >>> I didn't see any tool for data deletion either, but the protocol
>> >>> specification is given, so it is possible to take some time to
>> implement
>> >>> it.
>> >>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Chengyu, don't you have problems to retrieve so large files from the
>> repo?
>> >>>> I have already problems with files around 50MB. The developers told
>> me that
>> >>>> it might be a memory issue.
>> >>>
>> >>>
>> >>> I don't have this problem.
>> >>>
>> >>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Best,
>> >>>>
>> >>>> Urs
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] Im
>> >>>> Auftrag von Alex Afanasyev
>> >>>> Gesendet: Mittwoch, 8. April 2015 02:03
>> >>>> An: Chengyu Fan
>> >>>> Cc: ndn-interest at lists.cs.ucla.edu
>> >>>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol
>> in
>> >>>> NDN-repo-ng
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu>
>> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding
>> data
>> >>>> packets into the repo.
>> >>>>
>> >>>> If need to insert the raw data, I must convert them into NDN packet
>> format
>> >>>> encoding in advance. Correct?
>> >>>>
>> >>>>
>> >>>>
>> >>>> Yes.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev
>> >>>> <alexander.afanasyev at ucla.edu> wrote:
>> >>>>
>> >>>> TcpBulkInsert is not really a protocol, rather a socket that accepts
>> a
>> >>>> stream of data packets.
>> >>>>
>> >>>>
>> >>>>
>> >>>> Assuming file bulk-data.tlv contains a collection of data packets in
>> NDN
>> >>>> packet format encoding (just piled up, one after another one), you
>> can use
>> >>>> ‘nc’ command to push them into the repo
>> >>>>
>> >>>>
>> >>>>
>> >>>> nc localhost 7376 < bulk-data.tlv
>> >>>>
>> >>>>
>> >>>>
>> >>>> —
>> >>>>
>> >>>> Alex
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu>
>> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>>
>> >>>>
>> >>>> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in
>> >>>> NDN-repo-ng?
>> >>>>
>> >>>>
>> >>>>
>> >>>> We have lots of data need to insert into the repository, and each one
>> >>>> typically contains about 1.2GB.
>> >>>>
>> >>>>
>> >>>>
>> >>>> According to the wiki
>> >>>> (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp
>> Bulk Insert
>> >>>> Repo Insertion Protocol" might be the right option.
>> >>>>
>> >>>>
>> >>>>
>> >>>> However, the wiki doesn't contain the protocol specification, and
>> there is
>> >>>> no tool for this protocol either ...
>> >>>>
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Chengyu
>> >>>>
>> >>>> _______________________________________________
>> >>>> Ndn-interest mailing list
>> >>>> Ndn-interest at lists.cs.ucla.edu
>> >>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Ndn-interest mailing list
>> >>> Ndn-interest at lists.cs.ucla.edu
>> >>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> >>>
>> >>
>> >> _______________________________________________
>> >> Ndn-interest mailing list
>> >> Ndn-interest at lists.cs.ucla.edu
>> >> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> >
>> >
>> > _______________________________________________
>> > Ndn-interest mailing list
>> > Ndn-interest at lists.cs.ucla.edu
>> > http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20150408/2cb4a133/attachment.html>