[Nfd-dev] ndnputchunks memory usage increases linearly with MAX_NDN_PACKET_SIZE

[Davide Pesavento]

On Fri, Nov 17, 2017 at 1:03 AM, Susmit <susmit at cs.colostate.edu> wrote:
> Hi Davide,
>
>> Are you saying that the memory usage increases by 1GB for each data
>> chunk? Or is it fixed at 1GB after ndnputchunks starts?
>
> Increases by 1GB with each data chunk.
> It exhausted about 100GB memory when I tried to run it.
>
>> Keep in mind that the face on the application side has an internal
>> receive buffer of size MAX_NDN_PACKET_SIZE bytes (see
>> src/transport/stream-transport-impl.hpp in ndn-cxx), because it must
>> be prepared to receive a packet that large.
>
> Ah, that would explain it.
> However, looking at the code, it does not seem to be an array of
> buffers, just a single buffer.
> So memory usage due to that buffer should not be  MAX_SiZE*number of
> packets.  Am I reading the code correctly?

Yep, it's a single buffer of size MAX_NDN_PACKET_SIZE, so it occupies
1GB in total in your case.
So that doesn't explain what you're seeing.

My next suspect would be ndn-cxx's Encoder, which can preallocate up
to MAX_NDN_PACKET_SIZE bytes. However, that shouldn't normally happen,
because before encoding we estimate the final packet size and then ask
the Encoder to allocate only what's needed.

Davide