[Ndn-interest] trouble with downloading File size > 1MB

Wed Jul 26 14:27:44 PDT 2017

I agree with you Nick, which is why I thought its a problem on the
ndnfs-end. I should have been more clearer, but the "File is empty" error
appears on the ndnfs-server log, when trying to download either
via ndnfs-client or PyNDN client.

I am writing down the exact outputs that might help us towards a better
conclusion.

*When I use ndnfs client
<https://github.com/remap/ndnfs-port/blob/master/test/client.cc> or my
PyNDN client to download the 10MB file (exact multiple of 8KB):*
Step 1: Last few lines of ndnfs log while moving the file to mount point
(before trying downloading it from the client) :

> - 1501103069 DEBUG: sign_segment: path=/oneMB.txt, ver=1501103054,
> seg=127, len=8192
> - 1501103069 DEBUG: sign_segment: segment name is
> /ndn/broadcast/ndnfs/oneMB.txt/%FDYy%03%CE/%00%7F
> seg:127size:8192
> - 1501103069 DEBUG: sign_segment: path=/oneMB.txt, ver=1501103054,
> seg=128, len=0
> - 1501103069 DEBUG: sign_segment: segment name is
> /ndn/broadcast/ndnfs/oneMB.txt/%FDYy%03%CE/%00%80
> seg:128size:0

Step 2: Now, when trying to downloading with ndnfs remote client, the
application times out.
Step 3: Last few lines of ndnfs-server log when using ndnfs client:

> - 1501103131 DEBUG: sendFileContent: Data returned with name:
> /ndn/broadcast/ndnfs/oneMB.txt/%FDYy%03%CE/%00%7F
> - 1501103131 DEBUG: sendFileContent: File is empty. Name:
> /ndn/broadcast/ndnfs/oneMB.txt/%FDYy%03%CE/%00%80

 Last few lines of ndnfs-server log when using PyNDN client:

> - 1501103887 DEBUG: sendFileMeta: Data returned with name:
> /ndn/broadcast/ndnfs/oneMB.txt/%C1.FS.file/%FDYy%03%CE
> - 1501103887 ERROR: onInterest: child selectors, min/maxSuffixComponents
> or excludes are not supported in current implementation.
> - 1501103887 DEBUG: sendFileContent: Data returned with name:
> /ndn/broadcast/ndnfs/oneMB.txt/%FDYy%03%CE/%00%00
> - 1501103887 ERROR: onInterest: child selectors, min/maxSuffixComponents
> or excludes are not supported in current implementation.
> - 1501103887 DEBUG: sendFileContent: File is empty. Name:
> /ndn/broadcast/ndnfs/oneMB.txt/%FDYy%03%CE/%00%80

Regards,
Ishita

On Wed, Jul 26, 2017 at 2:52 PM, Nick Briggs <nicholas.h.briggs at gmail.com>
 wrote:

> Private reply:
>
> Changing the segment size in the ndnfs code would not be a good idea.
> It's already having to use IP fragmentation to get the 8K+ byte packets
> transferred -- if you increase the segment size too much (and you're not
> running on a perfectly reliable medium) the chances of losing a fragment
> and requiring retransmission will get unmanageable.  8K is a compromise to
> reduce the signing/verification overhead down, but lets it run in jumbo
> ethernet frames when available, and matches the physical disk I/O size
> reasonably.
>
> I just looked at the segment fetcher code, and at least the description of
> the termination condition appears correct --
>
>
> 5. Keep sending Interests for the next segment while the retrieved Data
> does
>         not have a FinalBlockId or the FinalBlockId !=
> Data.getName().get(-1).
>
> (or, put more logically, ... until the retrieved Data has a FinalBlockId
> == Data.getName().get(-1))
>
> There's nothing obvious in the code that should make it fail if it gets a
> zero length final block.   You'll notice that the fetcher code appears to
> keep all the content objects received and then when it gets to the final
> one  (lines 286-291) figure out the total size, construct a byte array of
> the appropriate size, and then stuff the payload into the block.
>
> You mentioned that there was an exception when the file size was a
> multiple of 8192 bytes -- I don't see why that would be the case for this
> code, and the onComplete handler that you passed in would only get the
> complete blob of data...
>
> -- Nick
>
>
> On Jul 26, 2017, at 11:14 AM, Junxiao Shi <shijunxiao at email.arizona.edu>
> wrote:
>
> Hi Ishita
>
> As indicated on http://www.lists.cs.ucla.edu/pipermail/ndn-interest/2017-
> July/001852.html , the consumer app is at fault. If you refuse to provide
> the source code of the consumer app, I can't help you.
>
> My deployment publishes files over 30MB. The client side is ndn protocol
> add-on for Firefox. I never attempt to change any setting in ndnfs-port
> because it works out of box.
>
> Yours, Junxiao
>
> On Wed, Jul 26, 2017 at 09:13 Ishita Dasgupta <ishita.dasgupta at gmail.com>
> wrote:
>
>> Hi Junxiao,
>>
>> I understand that directly sharing my app would be helpful, but including
>> the code to my app would not be any more relevant, also I am not sure of my
>> sharing permissions. The app simply uses this PyNDN segment fetcher to
>> download a file, the link to which I provided above (along with a lot other
>> completely unrelated tasks that do not affect the file download).
>>
>> Also, you once mentioned having deployed ndnfs to download a 30MB file.
>> Did you ever work with changing the segment length in ndnfs server from 8KB
>> to something else ?
>>
>> Regards,
>> Ishita
>>
>> On Wed, Jul 26, 2017 at 12:03 PM, Junxiao Shi <
>> shijunxiao at email.arizona.edu> wrote:
>>
>>> Hi Ishita
>>>
>>> The link doesn't contain code of your app.
>>>
>>> Yours, Junxiao
>>>
>>> On Jul 26, 2017 08:45, "Ishita Dasgupta" <ishita.dasgupta at gmail.com>
>>> wrote:
>>>
>>> Hi Junxiao,
>>>
>>> Sure.
>>>
>>> My application imports PyNDN libraries. I am utilising this segment
>>> fetcher PyNDN library(https://github.com/nam
>>> ed-data/PyNDN2/blob/a4df2d207c1b76434a400f54fcaf5d45eea8b844
>>> /python/pyndn/util/segment_fetcher.py) to download the file segments
>>> that ndnfs offers.
>>>
>>> Regards,
>>> Ishita
>>>
>>>
>> _______________________________________________
> Ndn-interest mailing list
> Ndn-interest at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/ndn-interest
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/ndn-interest/attachments/20170726/401d6ec0/attachment.html>