[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng

[Chengyu Fan]

On Thu, Apr 9, 2015 at 11:05 AM, Peter Lee <peter at corenova.com> wrote:

> I'm new to the list/project and curious to understand how client initiated
> upload of content is facilitated in ndn protocol.
>
> Does Interest packet carry data to named destination?  Or does the client
> "register" a new named data source and the destination reciprocates
> Interest back to the client for streaming Data packets the other direction?
>

It is the repository to pull data out.
You may refer to the repo-ng wiki (
http://redmine.named-data.net/projects/repo-ng/wiki) for more details.


>
> Peter Lee
>
>
> On Apr 9, 2015, at 9:45 AM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>
>
> On Thu, Apr 9, 2015 at 2:44 AM, Urs Schnurrenberger <
> urs.schnurrenberger at unibas.ch> wrote:
>
>>  Thanks Chengyu!
>>
>>
>>
>> Yes, I also had to increase the max-packets value. How do you increase
>> the segment size?
>>
>
> I modified the ndnputfile code.
>
>
>>
>>
>> The strange thing is that I can retrieve (a segment of) the 50 MB file
>> right after insertion into the repo, but one day later (at the latest) the
>> retrieval fails. I get neither an error nor data. Smaller files always work.
>>
>
> I need to try this ...
> Could you tell me how do you run the repo-ng? Do you keep it running, or
> relaunch it one day later?
>
>
>>
>>
>> Urs
>>
>>
>>
>> *Von:* chengy.fan at gmail.com [mailto:chengy.fan at gmail.com] *Im Auftrag
>> von *Chengyu Fan
>> *Gesendet:* Mittwoch, 8. April 2015 19:27
>> *An:* Urs Schnurrenberger
>> *Cc:* Alex Afanasyev; ndn-interest at lists.cs.ucla.edu
>>
>> *Betreff:* Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
>> NDN-repo-ng
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger <
>> urs.schnurrenberger at unibas.ch> wrote:
>>
>>  Hi,
>>
>>
>>
>> why not using ndnputfile? Encoding is done automatically.
>>
>>
>>
>> ndnputfile reponame ndnname filename:
>>
>> e.g.
>>
>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>>
>>
>>
>> Yes, we tried ndnputfile. It is a little bit slow.
>>
>> However, the segment size is 1000 by default, and takes about 30 minutes
>> to insert a 1.2GB file.
>>
>> I changed the segment size to 8000 (the max NDN packet size is 8800),
>> then the time reduced to 4 minutes.
>>
>>
>>
>> I wonder if the Tcp Bulk Insert protocol can improve this more ...
>>
>>
>> BTW, such large file will generate a lots of segments, I have to set the
>> max-packets to a larger number in repo-ng.conf.
>>
>> However if the number is too large, repo-ng would complain "ERROR:
>> conversion of data to type "i" failed". I think it is a bug.
>>
>>
>>
>>
>>
>> Interesting as well would be how to delete files from the repo? I still
>> do it by directly manipulating the database. Seems that there is no tool
>> around…?
>>
>>
>>
>> I didn't see any tool for data deletion either, but the protocol
>> specification is given, so it is possible to take some time to  implement
>> it.
>>
>>
>>
>>
>>
>> Chengyu, don't you have problems to retrieve so large files from the
>> repo? I have already problems with files around 50MB. The developers told
>> me that it might be a memory issue.
>>
>>
>>
>> I don't have this problem.
>>
>>
>>
>>
>>
>>
>>
>> Best,
>>
>> Urs
>>
>>
>>
>>
>>
>> *Von:* Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] *Im
>> Auftrag von *Alex Afanasyev
>> *Gesendet:* Mittwoch, 8. April 2015 02:03
>> *An:* Chengyu Fan
>> *Cc:* ndn-interest at lists.cs.ucla.edu
>> *Betreff:* Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in
>> NDN-repo-ng
>>
>>
>>
>>
>>
>>  On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu>
>> wrote:
>>
>>
>>
>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding
>> data packets into the repo.
>>
>> If need to insert the raw data, I must convert them into NDN packet
>> format encoding in advance. Correct?
>>
>>
>>
>> Yes.
>>
>>
>>
>>
>>
>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev <
>> alexander.afanasyev at ucla.edu> wrote:
>>
>>  TcpBulkInsert is not really a protocol, rather a socket that accepts a
>> stream of data packets.
>>
>>
>>
>> Assuming file bulk-data.tlv contains a collection of data packets in NDN
>> packet format encoding (just piled up, one after another one), you can use
>> ‘nc’ command to push them into the repo
>>
>>
>>
>> nc localhost 7376 < bulk-data.tlv
>>
>>
>>
>> —
>>
>> Alex
>>
>>
>>
>>
>>
>>  On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu>
>> wrote:
>>
>>
>>
>> Hi,
>>
>>
>>
>> Do anyone know the "*Tcp Bulk Insert Repo Insertion Protocol*" in
>> NDN-repo-ng?
>>
>>
>>
>> We have lots of data need to insert into the repository, and each one
>> typically contains about 1.2GB.
>>
>>
>>
>> According to the wiki (
>> http://redmine.named-data.net/projects/repo-ng/wiki), the "*Tcp Bulk
>> Insert Repo Insertion Protocol*" might be the right option.
>>
>>
>>
>> However, the wiki doesn't contain the protocol specification, and there
>> is no tool for this protocol either ...
>>
>>
>>
>> Thanks,
>>
>> Chengyu
>>
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20150409/6882bf0a/attachment.html>