[Ndn-interest] download a file using PyNDN
Ishita Dasgupta
ishita.dasgupta at gmail.com
Tue Jul 25 18:18:45 PDT 2017
Hi All,
I am trying to study how PyNDN segment fetcher would work if multiple
interests of a fixed window size were sent out at once. But before that, I
need to confirm its current behaviour details. Could any of you please
confirm me on these points?
1.
Each data packet goes through signature validation, segment
verification and expected data segment verification before next
data packet
is processed.
2.
It sends out interest request(one by one) for all the data packets
that the file has been segmented into. And later concatenates all the
retrieved data packets when all the data packets have been received. Each
interest have an interest timeout, data_segment_verification (whether
a segment of the data is whar we are looking for, and whether a
verified segment).
3.
If signature validation or verification of any segment fails, fetch
process is aborted. (This can be turned off)
4.
Fetch function calls the first data packet which in return calls the
next data packets. Once each interest packet is both validated and
verified, AND is the expected data packet, the content is saved and
concatenated to the final blob. Retries once, if the expected data packet
does not arrive. Each data packet, when received successfully, calls the
next data packet(_fetchNextSegment).
5.
Process a-d go on until it is the last data packet.
6.
Thus, the packets are fetched and stored one-by-one. If any packet
retrieval fails, the fetch process is aborted. (Maybe if this can be
modified to fetch a chunk of segments, the file download time will be
faster?)
Regards,
Ishita
On Thu, Jun 1, 2017 at 12:09 PM, <Marc.Mosko at parc.com> wrote:
> Just to flesh this out a little, and I agree with Nick’s suggestion, it
> does use the “right-most-child” selector. By specifying the %C1.FS.file
> name component one will get the greatest version in the next name
> component, but you will also get the greatest segment too. So, if you have
> metadata information, or key locators, or signing information, it would
> work best, I think, to put all that in the last segment which would be the
> first segment retrieved by this process.
>
> Marc
>
> On May 31, 2017, at 8:00 PM, Nick Briggs <nicholas.h.briggs at gmail.com>
> wrote:
>
> I think we misunderstand each other -- I don't think I was suggesting
> something that depended on the "greater" name component -- I'm suggesting
> that for whatever file name, in this case
> /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09.mpd
> his first interest should be for
> /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09.mpd/%C1.FS.file
> explicitly including the %C1.FS.file name component, which we know will be
> available, and which should get him a versioned response such as
> /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09.mpd/%C1.FS.file/
> %FDY.%DA%3D
> (he couldn't get just a version number in place of the %C1.FS.file
> component... it wouldn't match)
> and then he can extract the version number from that content and proceed
> to ask for the segments of the original name+newly discovered version.
> It's the same two-step retrieval that was proposed, but if you add the
> %C1.FS.file component explicitly then you don't have to deal with the case
> you were warning about.
>
> -- Nick
>
>
>
> On May 31, 2017, at 11:44 AM, Thompson, Jeff <jefft0 at remap.ucla.edu>
> wrote:
>
> In NDN we have to be careful with getting the “greater” name component
> because a longer component always comes after a shorter component in the
> ordering, independent of the bytes. And in this case, it looks like the
> version component has fewer bytes than the %C1.FS.file component which
> would be “greater”. (It doesn’t matter that it starts with the byte %C1
> which is “less” than %FD.)
>
> - Jeff T
>
> From: Nick Briggs <nicholas.h.briggs at gmail.com>
> Date: Wednesday, May 31, 2017 at 11:35:00
> To: Jeff Thompson <jefft0 at remap.ucla.edu>
> Cc: Ishita Dasgupta <ishita.dasgupta at gmail.com>, "
> ndn-interest at lists.cs.ucla.edu" <ndn-interest at lists.cs.ucla.edu>
> Subject: Re: [Ndn-interest] download a file using PyNDN
>
> And if he knows he's looking for something of a file flavor then he can
> ask explicitly for (the highest version of) the name including the
> %C1.FS.file and proceed from there.
>
> --- Nick Briggs
>
>
> On May 31, 2017, at 11:30 AM, Thompson, Jeff <jefft0 at remap.ucla.edu>
> wrote:
>
> Yes, that will work. But it’s better to keep the name components in their
> native form as binary Blobs. The string representation is really only for
> display, or creating the initial name from a known string representation.
> You can simply save version as
> data.getName().get(-1) . Then, instead of
>
> name = Name(url+version)
>
> use
>
> name = Name(url).append(version)
>
> Something else to watch out for: There are data packets with names with
> and without the %C1.FS.file component. For example:
> /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09.mpd/*%C1.FS.file*
> /%FDY.%DA%3D
> /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_
> 05_09.mpd/%FDY.%DA%3D/%00%01
>
> So far, you’ve been lucky to get the data packet with %C1.FS.file where
> the version component is at data.getName().get(-1). But this is not
> guaranteed. It is valid for the NDN server to return one of the “segment”
> data packets with the second type of name, where the version component is
> at data.getName().get(-2). But you can easily check with
> data.getName().get(-1).isVersion() .
> http://named-data.net/doc/ndn-ccl-api/name-component.html#
> name-component-isversion-method
>
> - Jeff T
>
> From: Ishita Dasgupta <ishita.dasgupta at gmail.com>
> Date: Wednesday, May 31, 2017 at 11:13:00
> To: Jeff Thompson <jefft0 at remap.ucla.edu>
> Cc: "ndn-interest at lists.cs.ucla.edu" <ndn-interest at lists.cs.ucla.edu>
> Subject: Re: [Ndn-interest] download a file using PyNDN
>
> Hi Jeff,
>
> Yes, I had gotten it working by removing the shutdown command.
> Also, this is how I extract the version:
> From the first face expressInterest, I get the version from the _onData
> method I defined as follows:
>
> defonData(self, interest, data):
>> self._callbackCount += 1
>> dump("Got data packet with name", data.getName().toUri())
>> global version
>> version = '/'+str(data.getName().get(-1))
>> dump("Version extracted",version)
>
>
> which gives me the output:
>
>> Got data packet with name /ndn/broadcast/ndnfs/
>> BigBuckBunny_4s_simple_2014_05_09.mpd/%C1.FS.file/%FDY.%DA%3D
>>
>
>
> Version extracted /%FDY.%DA%3D
>
>
> Regards,
> Ishita Dasgupta
>
> *Graduate Student **College of Information and Computer Sciences*
> *, UMass Amherst*
> *Email*: ishitadg at cs.umass.edu
>
> On Wed, May 31, 2017 at 1:32 PM, Thompson, Jeff <jefft0 at remap.ucla.edu>
> wrote:
>
>> Yes. This is called “name discovery”.
>>
>> A question: You have "Name(url+version)”. I don’t see where you get
>> “version”.
>>
>> A hint: Maybe you shouldn’t call face.shutdown() the first time. Only
>> call it at the end of your program.
>>
>> - Jeff T
>>
>> From: Ishita Dasgupta <ishita.dasgupta at gmail.com>
>> Date: Wednesday, May 31, 2017 at 8:14:00
>>
>> To: Jeff Thompson <jefft0 at remap.ucla.edu>
>> Cc: "ndn-interest at lists.cs.ucla.edu" <ndn-interest at lists.cs.ucla.edu>
>> Subject: Re: [Ndn-interest] download a file using PyNDN
>>
>> Hi Jeff,
>>
>> I have got it working! Let me know if my procedure to extract the version
>> number by sending a sample interest packet and receiving a sample data
>> packet is correct or is there a simpler method to do it?
>>
>> Regards,
>> Ishita Dasgupta
>>
>> *Graduate Student **College of Information and Computer Sciences*
>> *, UMass Amherst*
>> *Email*: ishitadg at cs.umass.edu
>>
>> On Wed, May 31, 2017 at 10:46 AM, Ishita Dasgupta <
>> ishita.dasgupta at gmail.com> wrote:
>>
>>> Hi Jeff,
>>>
>>> To extract the current version, I express an interest on the face and
>>> extract the version from the data packet received. Then use this version to
>>> extract our data in segments using segment fetcher.
>>> The main code lines look something like this
>>>
>>>
>>>> name = Name(url)
>>>> face.expressInterest(name, counter.onData, counter.onTimeout)
>>>>
>>>>
>>>> while counter._callbackCount < 1:
>>>> face.processEvents()
>>>> face.shutdown()
>>>> name = Name(url+version)
>>>> dump("Express name ", name.toUri())
>>>> interest = Interest(name)
>>>> interest.setInterestLifetimeMilliseconds(1000)
>>>> SegmentFetcher.fetch(face, interest, None, counter.onComplete,
>>>> counter.onError)
>>>> while counter._callbackCount < 2:
>>>> face.processEvents()
>>>>
>>>> face.shutdown()
>>>
>>>
>>> But for the first interest, i receive a "broken" version number
>>> (
>>> %FDY.%CF-
>>> )
>>> ? which is why the segment fetcher doesn't work (?)
>>>
>>> 1496241583.724584 From: 192.1.242.150, To: 192.1.242.151, Tunnel Type:
>>>> TCP, INTEREST: /ndn/broadcast/ndnfs/BigBuckBu
>>>> nny_4s_simple_2014_05_09.mpd?ndn.MustBeFresh=1&ndn.Nonce=1500003601
>>>> 1496241583.734759 From: 192.1.242.151, To: 192.1.242.150, Tunnel Type:
>>>> TCP, DATA: /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09.mpd/%
>>>> C1.FS.file/%FDY.%CF-
>>>
>>>
>>> Is this how I should go about to fetch the version number before
>>> expressing the interest or is there a simpler method?
>>>
>>> Regards,
>>> Ishita Dasgupta
>>>
>>> *Graduate Student **College of Information and Computer Sciences*
>>> *, UMass Amherst*
>>> *Email*: ishitadg at cs.umass.edu
>>>
>>> On Fri, May 26, 2017 at 2:39 PM, Thompson, Jeff <jefft0 at remap.ucla.edu>
>>> wrote:
>>>
>>>> That’s right. The names of the actual segment Data packets should be
>>>>
>>>> /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09.mpd/%FDY%
>>>> 28Q8/%00%00
>>>> /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09.mpd/%FDY%
>>>> 28Q8/%00%01
>>>> /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09.mpd/%FDY%
>>>> 28Q8/%00%02
>>>>
>>>> etc., up to the maximum segment number. You give SegmentFetcher the
>>>> name without the final segment number component, and it adds the segment
>>>> number component as it fetches each one.
>>>>
>>>> - Jeff T
>>>>
>>>> From: Ishita Dasgupta <ishita.dasgupta at gmail.com>
>>>> Date: Friday, May 26, 2017 at 11:35:00
>>>>
>>>> To: Jeff Thompson <jefft0 at remap.ucla.edu>
>>>> Cc: "ndn-interest at lists.cs.ucla.edu" <ndn-interest at lists.cs.ucla.edu>
>>>> Subject: Re: [Ndn-interest] download a file using PyNDN
>>>>
>>>> So, just so that I understand this correctly: " %FDY%28Q8 " is the
>>>> version component of the all data packet segments representing the file.
>>>> And then the Segment Fetcher program sequentially downloads the segments
>>>> individually (and I don't need to worry about the segment numbers?)
>>>>
>>>> Regards,
>>>> Ishita Dasgupta
>>>>
>>>> *Graduate Student **College of Information and Computer Sciences*
>>>> *, UMass Amherst*
>>>> *Email*: ishitadg at cs.umass.edu
>>>>
>>>> On Fri, May 26, 2017 at 2:31 PM, Thompson, Jeff <jefft0 at remap.ucla.edu>
>>>> wrote:
>>>>
>>>>> Please try the following: Don’t add the line to set the exclude
>>>>> filter, and change the name to include the version number (without
>>>>> “%C1.FS.file”):
>>>>>
>>>>> name3 = Name("/ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09
>>>>> .mpd/%FDY%28Q8”)
>>>>>
>>>>> The reason is that the segment component comes after the version
>>>>> component “/%FDY%28Q8”.
>>>>>
>>>>> - Jeff T
>>>>>
>>>>> From: Ishita Dasgupta <ishita.dasgupta at gmail.com>
>>>>> Date: Friday, May 26, 2017 at 11:22:00
>>>>>
>>>>> To: Jeff Thompson <jefft0 at remap.ucla.edu>
>>>>> Cc: "ndn-interest at lists.cs.ucla.edu" <ndn-interest at lists.cs.ucla.edu>
>>>>> Subject: Re: [Ndn-interest] download a file using PyNDN
>>>>>
>>>>> Hi Jeff,
>>>>>
>>>>> Now it would not return any Data packet at all.
>>>>> At consumer(ndndump):
>>>>>
>>>>>> 1495818158.862071 From: 192.1.242.150, To: 192.1.242.151, Tunnel
>>>>>> Type: TCP, INTEREST: /ndn/broadcast/ndnfs/BigBuckBu
>>>>>> nny_4s_simple_2014_05_09.mpd?ndn.ChildSelector=1&ndn.MustBeF
>>>>>> resh=1&ndn.InterestLifetime=1000&ndn.Nonce=1259570516&ndn.Ex
>>>>>> clude=%C1.FS.file
>>>>>
>>>>> <NO DATA PACKET IN RESPONSE>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Ishita Dasgupta
>>>>>
>>>>> *Graduate Student **College of Information and Computer Sciences*
>>>>> *, UMass Amherst*
>>>>> *Email*: ishitadg at cs.umass.edu
>>>>>
>>>>> On Fri, May 26, 2017 at 12:54 PM, Thompson, Jeff <
>>>>> jefft0 at remap.ucla.edu> wrote:
>>>>>
>>>>>> After
>>>>>>
>>>>>> interest = Interest(name3)
>>>>>>
>>>>>> try adding
>>>>>>
>>>>>> interest.getExclude().appendComponent(Name("%C1.FS.file")[0])
>>>>>>
>>>>>> This is because NDNFS not only serves the Data packet segments of the
>>>>>> file, but a “meta info” packet where the name has the component
>>>>>> “"%C1.FS.file”. The code above should exclude it.
>>>>>>
>>>>>> Let us know,
>>>>>> - Jeff T
>>>>>>
>>>>>> From: Ishita Dasgupta <ishita.dasgupta at gmail.com>
>>>>>> Date: Friday, May 26, 2017 at 9:38:00
>>>>>> To: Jeff Thompson <jefft0 at remap.ucla.edu>
>>>>>> Cc: "ndn-interest at lists.cs.ucla.edu" <ndn-interest at lists.cs.ucla.edu>
>>>>>>
>>>>>>
>>>>>> Subject: Re: [Ndn-interest] download a file using PyNDN
>>>>>>
>>>>>> Hi Jeff,
>>>>>>
>>>>>> I have a file test.txt being served on my ndnfs server. Prefix:
>>>>>> /ndn/broadcast/ndnfs
>>>>>> I implement SegmentFetcher in the following way and print content and
>>>>>> error message
>>>>>>
>>>>>> name3 = Name(url)
>>>>>>> dump("Express name ", name3.toUri())
>>>>>>> interest = Interest(name3)
>>>>>>> dump(interest.toUri())
>>>>>>> interest.setInterestLifetimeMilliseconds(1000)
>>>>>>> SegmentFetcher.fetch(face, interest, None, counter.onComplete,
>>>>>>> counter.onError)
>>>>>>
>>>>>>
>>>>>> This is what i get:
>>>>>>
>>>>>>> /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09.mpd?n
>>>>>>> dn.MustBeFresh=true
>>>>>>> onError
>>>>>>> 2
>>>>>>> Got an unexpected packet without a segment number:
>>>>>>> /ndn/broadcast/ndnfs/BigBuckBunny_4s_simple_2014_05_09.mpd/%
>>>>>>> C1.FS.file/%FDY%28Q8
>>>>>>
>>>>>>
>>>>>> When I do an ndndump on the consumer, I see:
>>>>>>
>>>>>>> 1495816236.074843 From: 192.1.242.150, To: 192.1.242.151, Tunnel
>>>>>>> Type: TCP, INTEREST: /ndn/broadcast/ndnfs/BigBuckBu
>>>>>>> nny_4s_simple_2014_05_09.mpd?ndn.ChildSelector=1&ndn.MustBeF
>>>>>>> resh=1&ndn.InterestLifetime=1000&ndn.Nonce=3797640908
>>>>>>
>>>>>>
>>>>>>
>>>>>> 1495816236.085709 From: 192.1.242.151, To: 192.1.242.150, Tunnel
>>>>>>> Type: TCP, DATA: /ndn/broadcast/ndnfs/BigBuckBu
>>>>>>> nny_4s_simple_2014_05_09.mpd/%C1.FS.file/%FDY%28Q8
>>>>>>
>>>>>>
>>>>>> I thought fetching segment number was internally done in your Segment
>>>>>> Fetcher util. Do I need to implement it some way on my end as well?
>>>>>>
>>>>>> Regards,
>>>>>> Ishita Dasgupta
>>>>>>
>>>>>> *Graduate Student **College of Information and Computer Sciences*
>>>>>> *, UMass Amherst*
>>>>>> *Email*: ishitadg at cs.umass.edu
>>>>>>
>>>>>> On Thu, May 25, 2017 at 2:15 PM, Thompson, Jeff <
>>>>>> jefft0 at remap.ucla.edu> wrote:
>>>>>>
>>>>>>> Hi Ishita,
>>>>>>>
>>>>>>> We have NDNFS producing sample data at memoria.ndn.ucla.edu. The
>>>>>>> PyNDN code below uses the SegmentFetcher to fetch the segments of a file (a
>>>>>>> JavaScript file). NDNFS has split the file into 12 segments, which
>>>>>>> SegmentFetcher retrieves to supply the content to the callback.
>>>>>>> SegmentFetcher documentation is here:
>>>>>>> https://github.com/named-data/PyNDN2/blob/master/python/pynd
>>>>>>> n/util/segment_fetcher.py#L142
>>>>>>>
>>>>>>> - Jeff T
>>>>>>>
>>>>>>> import time, sys
>>>>>>> from pyndn import Name, Interest, Face
>>>>>>> from pyndn.util import SegmentFetcher
>>>>>>>
>>>>>>> face = Face("memoria.ndn.ucla.edu")
>>>>>>> SegmentFetcher.fetch(face, Interest(Name(
>>>>>>> "/ndn/edu/ucla/remap/demo/ndn-js-test/named-data.net/wp-incl
>>>>>>> udes/js/jquery/jquery.js/%FDX%DC5B")),
>>>>>>> None, lambda content: sys.stdout.write(content.toRawStr()),
>>>>>>> lambda errorCode, message: sys.stdout.write("Got error " +
>>>>>>> message))
>>>>>>>
>>>>>>> while True:
>>>>>>> face.processEvents()
>>>>>>> time.sleep(0.01)
>>>>>>>
>>>>>>>
>>>>>>> From: Ishita Dasgupta <ishita.dasgupta at gmail.com>
>>>>>>> Date: Thursday, May 25, 2017 at 6:00:00
>>>>>>> To: Jeff Thompson <jefft0 at remap.ucla.edu>, "
>>>>>>> ndn-interest at lists.cs.ucla.edu" <ndn-interest at lists.cs.ucla.edu>
>>>>>>> Subject: Re: [Ndn-interest] download a file using PyNDN
>>>>>>>
>>>>>>> Hi Jeff,
>>>>>>>
>>>>>>> Yes
>>>>>>> , the producer is responding to Interests with Data packets
>>>>>>> representing the file. I run NDNFS on the producer.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 24, 2017 at 4:35 PM Thompson, Jeff <
>>>>>>> jefft0 at remap.ucla.edu> wrote:
>>>>>>>
>>>>>>>> Hi Ishta,
>>>>>>>>
>>>>>>>> In NDN, there is a producer and consumer. The consumer sends an
>>>>>>>> Interest and the producer (or a cache in the network) responds with a Data
>>>>>>>> packet. When you say “an entire file from an NDN server” I assume you mean
>>>>>>>> a producer that is responding to Interests with Data packets representing
>>>>>>>> the file. What producer software is running on the server?
>>>>>>>>
>>>>>>>> - Jeff T
>>>>>>>>
>>>>>>>> From: Ndn-interest <ndn-interest-bounces at lists.cs.ucla.edu> on
>>>>>>>> behalf of Ishita Dasgupta <ishita.dasgupta at gmail.com>
>>>>>>>> Date: Wednesday, May 24, 2017 at 8:29:00
>>>>>>>> To: "ndn-interest at lists.cs.ucla.edu" <ndn-interest at lists.cs.ucla.ed
>>>>>>>> u>
>>>>>>>> Subject: [Ndn-interest] download a file using PyNDN
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> Has anybody used PyNDN to download an entire file from an ndn
>>>>>>>> server?
>>>>>>>> I see or receive jibberish data in form of one data packet but
>>>>>>>> don't seem to copying the exact content of the file(no matter the size, i.e
>>>>>>>> irrespective of the fact that the file gets divided to multiple segments or
>>>>>>>> just one segment).
>>>>>>>> Any guidance with this would be helpful.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ishita
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
>
>
> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20170725/8a878a94/attachment-0001.html>
More information about the Ndn-interest
mailing list