[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

Dave Oran (oran) oran at cisco.com
Wed Apr 8 13:03:36 PDT 2015


Change the architecture - only sign manifests, not individual packets.

> On Apr 8, 2015, at 12:55 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
> 
> I think it is due to the packet signing.
> ndnputfile signs each segment, not the whole data. Therefore, if the segment size is small, ndnputfile has to sign more segments.
> 
> Wonder if there are efficient signing methods for big data provided in the ndn-cxx library?
> 
> On Wed, Apr 8, 2015 at 1:24 PM, Tai-Lin Chu <tailinchu at gmail.com> wrote:
> Is this due to packet signing? or simply segmentation itself?
> 
> Thanks.
> 
> On Wed, Apr 8, 2015 at 10:26 AM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
> >
> >
> > On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger
> > <urs.schnurrenberger at unibas.ch> wrote:
> >>
> >> Hi,
> >>
> >>
> >>
> >> why not using ndnputfile? Encoding is done automatically.
> >>
> >>
> >>
> >> ndnputfile reponame ndnname filename:
> >>
> >> e.g.
> >>
> >> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
> >
> >
> > Yes, we tried ndnputfile. It is a little bit slow.
> > However, the segment size is 1000 by default, and takes about 30 minutes to
> > insert a 1.2GB file.
> > I changed the segment size to 8000 (the max NDN packet size is 8800), then
> > the time reduced to 4 minutes.
> >
> > I wonder if the Tcp Bulk Insert protocol can improve this more ...
> >
> > BTW, such large file will generate a lots of segments, I have to set the
> > max-packets to a larger number in repo-ng.conf.
> > However if the number is too large, repo-ng would complain "ERROR:
> > conversion of data to type "i" failed". I think it is a bug.
> >
> >>
> >>
> >> Interesting as well would be how to delete files from the repo? I still do
> >> it by directly manipulating the database. Seems that there is no tool
> >> around…?
> >
> >
> > I didn't see any tool for data deletion either, but the protocol
> > specification is given, so it is possible to take some time to  implement
> > it.
> >
> >>
> >>
> >>
> >> Chengyu, don't you have problems to retrieve so large files from the repo?
> >> I have already problems with files around 50MB. The developers told me that
> >> it might be a memory issue.
> >
> >
> > I don't have this problem.
> >
> >
> >>
> >>
> >>
> >> Best,
> >>
> >> Urs
> >>
> >>
> >>
> >>
> >>
> >> Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] Im
> >> Auftrag von Alex Afanasyev
> >> Gesendet: Mittwoch, 8. April 2015 02:03
> >> An: Chengyu Fan
> >> Cc: ndn-interest at lists.cs.ucla.edu
> >> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
> >> NDN-repo-ng
> >>
> >>
> >>
> >>
> >>
> >> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
> >>
> >>
> >>
> >> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data
> >> packets into the repo.
> >>
> >> If need to insert the raw data, I must convert them into NDN packet format
> >> encoding in advance. Correct?
> >>
> >>
> >>
> >> Yes.
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev
> >> <alexander.afanasyev at ucla.edu> wrote:
> >>
> >> TcpBulkInsert is not really a protocol, rather a socket that accepts a
> >> stream of data packets.
> >>
> >>
> >>
> >> Assuming file bulk-data.tlv contains a collection of data packets in NDN
> >> packet format encoding (just piled up, one after another one), you can use
> >> ‘nc’ command to push them into the repo
> >>
> >>
> >>
> >> nc localhost 7376 < bulk-data.tlv
> >>
> >>
> >>
> >> —
> >>
> >> Alex
> >>
> >>
> >>
> >>
> >>
> >> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
> >>
> >>
> >>
> >> Hi,
> >>
> >>
> >>
> >> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in
> >> NDN-repo-ng?
> >>
> >>
> >>
> >> We have lots of data need to insert into the repository, and each one
> >> typically contains about 1.2GB.
> >>
> >>
> >>
> >> According to the wiki
> >> (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp Bulk Insert
> >> Repo Insertion Protocol" might be the right option.
> >>
> >>
> >>
> >> However, the wiki doesn't contain the protocol specification, and there is
> >> no tool for this protocol either ...
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Chengyu
> >>
> >> _______________________________________________
> >> Ndn-interest mailing list
> >> Ndn-interest at lists.cs.ucla.edu
> >> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
> > _______________________________________________
> > Ndn-interest mailing list
> > Ndn-interest at lists.cs.ucla.edu
> > http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
> >
> 
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest





More information about the Ndn-interest mailing list