[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

[Chengyu Fan]

Thanks Dave and Ilya.

I was just about to ask how the FAST_SIGNING works ...

It's sure that just signing manifests would improve the performance, but
not sure if we still can utilize the repo-ng's validator to verify the
content?

On Wed, Apr 8, 2015 at 2:07 PM, Ilya Moiseenko <iliamo at cs.ucla.edu> wrote:

> On Apr 8, 2015, at 1:03 PM, Dave Oran (oran) <oran at cisco.com> wrote:
>
> > Change the architecture - only sign manifests, not individual packets.
> Hello Dave, this is how the API provides the speed increase.
>
> Ilya
>
> >
> >> On Apr 8, 2015, at 12:55 PM, Chengyu Fan <chengyu at cs.colostate.edu>
> wrote:
> >>
> >> I think it is due to the packet signing.
> >> ndnputfile signs each segment, not the whole data. Therefore, if the
> segment size is small, ndnputfile has to sign more segments.
> >>
> >> Wonder if there are efficient signing methods for big data provided in
> the ndn-cxx library?
> >>
> >> On Wed, Apr 8, 2015 at 1:24 PM, Tai-Lin Chu <tailinchu at gmail.com>
> wrote:
> >> Is this due to packet signing? or simply segmentation itself?
> >>
> >> Thanks.
> >>
> >> On Wed, Apr 8, 2015 at 10:26 AM, Chengyu Fan <chengyu at cs.colostate.edu>
> wrote:
> >>>
> >>>
> >>> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger
> >>> <urs.schnurrenberger at unibas.ch> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>>
> >>>>
> >>>> why not using ndnputfile? Encoding is done automatically.
> >>>>
> >>>>
> >>>>
> >>>> ndnputfile reponame ndnname filename:
> >>>>
> >>>> e.g.
> >>>>
> >>>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
> >>>
> >>>
> >>> Yes, we tried ndnputfile. It is a little bit slow.
> >>> However, the segment size is 1000 by default, and takes about 30
> minutes to
> >>> insert a 1.2GB file.
> >>> I changed the segment size to 8000 (the max NDN packet size is 8800),
> then
> >>> the time reduced to 4 minutes.
> >>>
> >>> I wonder if the Tcp Bulk Insert protocol can improve this more ...
> >>>
> >>> BTW, such large file will generate a lots of segments, I have to set
> the
> >>> max-packets to a larger number in repo-ng.conf.
> >>> However if the number is too large, repo-ng would complain "ERROR:
> >>> conversion of data to type "i" failed". I think it is a bug.
> >>>
> >>>>
> >>>>
> >>>> Interesting as well would be how to delete files from the repo? I
> still do
> >>>> it by directly manipulating the database. Seems that there is no tool
> >>>> around…?
> >>>
> >>>
> >>> I didn't see any tool for data deletion either, but the protocol
> >>> specification is given, so it is possible to take some time to
> implement
> >>> it.
> >>>
> >>>>
> >>>>
> >>>>
> >>>> Chengyu, don't you have problems to retrieve so large files from the
> repo?
> >>>> I have already problems with files around 50MB. The developers told
> me that
> >>>> it might be a memory issue.
> >>>
> >>>
> >>> I don't have this problem.
> >>>
> >>>
> >>>>
> >>>>
> >>>>
> >>>> Best,
> >>>>
> >>>> Urs
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] Im
> >>>> Auftrag von Alex Afanasyev
> >>>> Gesendet: Mittwoch, 8. April 2015 02:03
> >>>> An: Chengyu Fan
> >>>> Cc: ndn-interest at lists.cs.ucla.edu
> >>>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
> >>>> NDN-repo-ng
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu>
> wrote:
> >>>>
> >>>>
> >>>>
> >>>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding
> data
> >>>> packets into the repo.
> >>>>
> >>>> If need to insert the raw data, I must convert them into NDN packet
> format
> >>>> encoding in advance. Correct?
> >>>>
> >>>>
> >>>>
> >>>> Yes.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev
> >>>> <alexander.afanasyev at ucla.edu> wrote:
> >>>>
> >>>> TcpBulkInsert is not really a protocol, rather a socket that accepts a
> >>>> stream of data packets.
> >>>>
> >>>>
> >>>>
> >>>> Assuming file bulk-data.tlv contains a collection of data packets in
> NDN
> >>>> packet format encoding (just piled up, one after another one), you
> can use
> >>>> ‘nc’ command to push them into the repo
> >>>>
> >>>>
> >>>>
> >>>> nc localhost 7376 < bulk-data.tlv
> >>>>
> >>>>
> >>>>
> >>>> —
> >>>>
> >>>> Alex
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu>
> wrote:
> >>>>
> >>>>
> >>>>
> >>>> Hi,
> >>>>
> >>>>
> >>>>
> >>>> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in
> >>>> NDN-repo-ng?
> >>>>
> >>>>
> >>>>
> >>>> We have lots of data need to insert into the repository, and each one
> >>>> typically contains about 1.2GB.
> >>>>
> >>>>
> >>>>
> >>>> According to the wiki
> >>>> (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp Bulk
> Insert
> >>>> Repo Insertion Protocol" might be the right option.
> >>>>
> >>>>
> >>>>
> >>>> However, the wiki doesn't contain the protocol specification, and
> there is
> >>>> no tool for this protocol either ...
> >>>>
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Chengyu
> >>>>
> >>>> _______________________________________________
> >>>> Ndn-interest mailing list
> >>>> Ndn-interest at lists.cs.ucla.edu
> >>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Ndn-interest mailing list
> >>> Ndn-interest at lists.cs.ucla.edu
> >>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
> >>>
> >>
> >> _______________________________________________
> >> Ndn-interest mailing list
> >> Ndn-interest at lists.cs.ucla.edu
> >> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
> >
> >
> > _______________________________________________
> > Ndn-interest mailing list
> > Ndn-interest at lists.cs.ucla.edu
> > http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20150408/2fe2a654/attachment.html>