[Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng
Peter Lee
peter at corenova.com
Thu Apr 9 16:38:37 PDT 2015
Hi Ilya, my comments inline.
Peter Lee
> On Apr 9, 2015, at 12:24 PM, Ilya Moiseenko <iliamo at cs.ucla.edu> wrote:
>
>
>> On Apr 9, 2015, at 11:06 AM, Peter Lee <peter at corenova.com> wrote:
>>
>> Also another question...
>>
>> I'm wondering whether a Data packet can encapsulate an Interest packet (or even a collection of these) in the event where the Requestor is also the data producer?
> You can put anything in the Data packet payload. It’s up to the application.
PL - wouldn't the application normally be abstracted from the underlay protocol such as NDN or TCP/IP? I was thinking more as a separate section in the Data Packet instead of inside the payload for the network layer processing the actual NDN protocol transactions.
>> The initial Interest can stay resident in the PIT (assuming the reply Data packet from the repo has a proper flag set) and it can be used for bi-directional forwarding across the forwarding hops - until a final Data Packet denoting Interest fulfillment is observed.
>>
>> This would reduce the extra Interest chatter and reduce the per packet insert/delete ops in the PIT?
> I’m not quite following your writing here. I just suggest to check Repo-NG protocol. It has some form of Interest-Interest exchange
PL - sorry I wasn't quite explicit with the scenario.
Currently, the sequence of transaction in repo-NG is as follows:
1. Requestor -> Interest -> Repo
2. Repo -> Data -> Requestor
(Here the PIT is cleared for the Interest above)
3. Repo -> Interest -> Data Provider
4. Data Provider -> Data -> Repo
(Series of step 3 and 4)
And another async transaction:
5. Requestor -> Interest (status) -> Repo
6. Repo -> Data -> Requestor
(Series of step 5 and 6)
What I was wondering is if the Requestor and the Data Provider is one and the same (which I would consider a typical scenario), we can collapse as following:
1. Requestor -> Interest -> Repo
2. Repo -> Data + Interest(s) -> Requestor
(PIT is NOT cleared for initial Interest)
3. Requestor -> Data -> Repo
(Series of step 2 and 3)
(PIT will be cleared once Data Packet is observed from Repo without any additional reply Interest being sent to Requestor).
And Requestor wouldn't need to send status Interest since it would know when transaction is completed upon no further Interest from Repo.
Also since additional Interests from Repo would be encapsulated in Data packet reply to the Requestor, the PIT would not need to maintain a separate entry per each segment being transacted.
>>
>> While I can see the value in being able to have repo directly fetch content from alternative data provider, I'm not sure how authorization can be transacted between the repo and the data provider if only the requestor has the proper access rights to the data provider source.
> Repo-NG can accept data based on publisher’s identity (public key).
PL - true, that would ensure data integrity but it doesn't mean that the repo has the necessary authorization to fetch the content from the publisher on behalf of the Requestor.
>> In such a case, the Interest from repo should come back to the requestor and the requestor should establish a pipe to stream data from the data provider back to the repo as a conduit. It also has the benefit that content may come from a non-NDN producer (such as Dropbox, local file system, etc)
> No, it cannot come from non-NDN producer. All content must be in a form of Data packets.
PL - if the Requestor has ability to directly access non-NDN data producer (local filesystem or has dual network stack TCP/IP and NDN), then it can stream that data retrieved from publisher back to the Repo in the form of Data Packets.
>> By encapsulating interest as data packet reply from the repo, the repo would not need to know the named destination of the data source (it can be anything arbitrary the requestor designates).
>>
>> Just wondering if such encapsulation scheme has been under consideration.
>
> I don’t quite understand what you mean by your scheme, but the repo protocol works in a different way. I think other approaches have been considered.
PL - hopefully the workflow described above provides a bit more clarity on the interest encapsulation inside Data packets.
>
> Ilya
>
>>
>> Thanks,
>>
>> Peter Lee
>>
>>
>>> On Apr 9, 2015, at 10:13 AM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>>
>>>
>>>
>>>> On Thu, Apr 9, 2015 at 11:05 AM, Peter Lee <peter at corenova.com> wrote:
>>>> I'm new to the list/project and curious to understand how client initiated upload of content is facilitated in ndn protocol.
>>>>
>>>> Does Interest packet carry data to named destination? Or does the client "register" a new named data source and the destination reciprocates Interest back to the client for streaming Data packets the other direction?
>>>
>>> It is the repository to pull data out.
>>> You may refer to the repo-ng wiki (http://redmine.named-data.net/projects/repo-ng/wiki) for more details.
>>>
>>>>
>>>> Peter Lee
>>>>
>>>>
>>>>> On Apr 9, 2015, at 9:45 AM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>>>>
>>>>>
>>>>>> On Thu, Apr 9, 2015 at 2:44 AM, Urs Schnurrenberger <urs.schnurrenberger at unibas.ch> wrote:
>>>>>> Thanks Chengyu!
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yes, I also had to increase the max-packets value. How do you increase the segment size?
>>>>>>
>>>>>
>>>>> I modified the ndnputfile code.
>>>>>
>>>>>>
>>>>>>
>>>>>> The strange thing is that I can retrieve (a segment of) the 50 MB file right after insertion into the repo, but one day later (at the latest) the retrieval fails. I get neither an error nor data. Smaller files always work.
>>>>>>
>>>>>
>>>>> I need to try this ...
>>>>> Could you tell me how do you run the repo-ng? Do you keep it running, or relaunch it one day later?
>>>>>
>>>>>>
>>>>>>
>>>>>> Urs
>>>>>>
>>>>>>
>>>>>>
>>>>>> Von: chengy.fan at gmail.com [mailto:chengy.fan at gmail.com] Im Auftrag von Chengyu Fan
>>>>>> Gesendet: Mittwoch, 8. April 2015 19:27
>>>>>> An: Urs Schnurrenberger
>>>>>> Cc: Alex Afanasyev; ndn-interest at lists.cs.ucla.edu
>>>>>>
>>>>>>
>>>>>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 8, 2015 at 5:08 AM, Urs Schnurrenberger <urs.schnurrenberger at unibas.ch> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> why not using ndnputfile? Encoding is done automatically.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ndnputfile reponame ndnname filename:
>>>>>>
>>>>>> e.g.
>>>>>>
>>>>>> ndnputfile /localhost/repo-ng /ndn/ch/unibas/hello hello.txt
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yes, we tried ndnputfile. It is a little bit slow.
>>>>>>
>>>>>> However, the segment size is 1000 by default, and takes about 30 minutes to insert a 1.2GB file.
>>>>>>
>>>>>> I changed the segment size to 8000 (the max NDN packet size is 8800), then the time reduced to 4 minutes.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I wonder if the Tcp Bulk Insert protocol can improve this more ...
>>>>>>
>>>>>>
>>>>>> BTW, such large file will generate a lots of segments, I have to set the max-packets to a larger number in repo-ng.conf.
>>>>>>
>>>>>> However if the number is too large, repo-ng would complain "ERROR: conversion of data to type "i" failed". I think it is a bug.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Interesting as well would be how to delete files from the repo? I still do it by directly manipulating the database. Seems that there is no tool around…?
>>>>>>
>>>>>>
>>>>>>
>>>>>> I didn't see any tool for data deletion either, but the protocol specification is given, so it is possible to take some time to implement it.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Chengyu, don't you have problems to retrieve so large files from the repo? I have already problems with files around 50MB. The developers told me that it might be a memory issue.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I don't have this problem.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Urs
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Von: Ndn-interest [mailto:ndn-interest-bounces at lists.cs.ucla.edu] Im Auftrag von Alex Afanasyev
>>>>>> Gesendet: Mittwoch, 8. April 2015 02:03
>>>>>> An: Chengyu Fan
>>>>>> Cc: ndn-interest at lists.cs.ucla.edu
>>>>>> Betreff: Re: [Ndn-interest] Tcp Bulk Insert Repo Insertion Protocol in NDN-repo-ng
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Apr 7, 2015, at 5:01 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Okay, so the TcpBulkInsert is to push the NDN packet format encoding data packets into the repo.
>>>>>>
>>>>>> If need to insert the raw data, I must convert them into NDN packet format encoding in advance. Correct?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 7, 2015 at 5:38 PM, Alex Afanasyev <alexander.afanasyev at ucla.edu> wrote:
>>>>>>
>>>>>> TcpBulkInsert is not really a protocol, rather a socket that accepts a stream of data packets.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Assuming file bulk-data.tlv contains a collection of data packets in NDN packet format encoding (just piled up, one after another one), you can use ‘nc’ command to push them into the repo
>>>>>>
>>>>>>
>>>>>>
>>>>>> nc localhost 7376 < bulk-data.tlv
>>>>>>
>>>>>>
>>>>>>
>>>>>> —
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Apr 7, 2015, at 3:28 PM, Chengyu Fan <chengyu at cs.colostate.edu> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Do anyone know the "Tcp Bulk Insert Repo Insertion Protocol" in NDN-repo-ng?
>>>>>>
>>>>>>
>>>>>>
>>>>>> We have lots of data need to insert into the repository, and each one typically contains about 1.2GB.
>>>>>>
>>>>>>
>>>>>>
>>>>>> According to the wiki (http://redmine.named-data.net/projects/repo-ng/wiki), the "Tcp Bulk Insert Repo Insertion Protocol" might be the right option.
>>>>>>
>>>>>>
>>>>>>
>>>>>> However, the wiki doesn't contain the protocol specification, and there is no tool for this protocol either ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Chengyu
>>>>>>
>>>>>> _______________________________________________
>>>>>> Ndn-interest mailing list
>>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>>>>>
>>>>> _______________________________________________
>>>>> Ndn-interest mailing list
>>>>> Ndn-interest at lists.cs.ucla.edu
>>>>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>> _______________________________________________
>> Ndn-interest mailing list
>> Ndn-interest at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20150409/d6b484ac/attachment.html>
More information about the Ndn-interest
mailing list