<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Can you get on the NFD call to discuss the issues?<br class="">
<div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class="">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class="">
<div class=""><br class="Apple-interchange-newline">
Lan</div>
</div>
</span></div>
</div>
</div>
<br class="">
<div style="">
<blockquote type="cite" class="">
<div class="">On May 9, 2016, at 1:47 PM, Anil Jangam <<a href="mailto:anilj.mailing@gmail.com" class="">anilj.mailing@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Done: <a href="http://redmine.named-data.net/issues/3618" class="">
http://redmine.named-data.net/issues/3618</a><br class="">
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Mon, May 9, 2016 at 6:58 AM, Beichuan Zhang <span dir="ltr" class="">
<<a href="mailto:bzhang@cs.arizona.edu" target="_blank" class="">bzhang@cs.arizona.edu</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Hi Anil,
<div class=""><br class="">
</div>
<div class="">Can you create a redmine issue (<a href="http://redmine.named-data.net/" target="_blank" class="">http://redmine.named-data.net</a>) to document all the information and discussion in one place? Most of us are working on paper deadline this week,
so response may be slow.</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
<div class="">Beichuan</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div class="h5">
<div class="">On May 9, 2016, at 1:39 AM, Anil Jangam <<a href="mailto:anilj.mailing@gmail.com" target="_blank" class="">anilj.mailing@gmail.com</a>> wrote:</div>
<br class="">
</div>
</div>
<div class="">
<div class="">
<div class="h5">
<div dir="ltr" class="">
<div class="">I tried to investigate using the stack tracer got from Valgrind report. Can you please analyze these further and check if there is an issue here? </div>
<div class=""><br class="">
</div>
<div class="">
<div class="">=================================</div>
<div class="">./NFD/rib/rib-manager.cpp</div>
</div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="">
<div class="">
<div class="">188 m_keyChain.sign(*responseData);</div>
</div>
<div class="">
<div class="">189 m_face.put(*responseData);</div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">./NFD/daemon/mgmt/manager-base.cpp</div>
</div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="">
<div class="">
<div class="">98 m_keyChain.sign(*responseData);</div>
</div>
<div class="">
<div class="">99 m_face->put(*responseData);</div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class=""><font color="#e06666" class="">Here each time it is allocating ~9K block of memory. I am not sure when this is released, but this is the topmost contributor for memory build up. </font></div>
<div class="">./ndn-cxx/src/encoding/encoder.cpp</div>
</div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="">
<div class="">
<div class="">27 Encoder::Encoder(size_t totalReserve/* = 8800*/, size_t reserveFromBack/* = 400*/)</div>
</div>
<div class="">
<div class="">28 : m_buffer(new Buffer(totalReserve))</div>
</div>
<div class="">
<div class="">29 {</div>
</div>
<div class="">
<div class="">30 m_begin = m_end = m_buffer->end() - (reserveFromBack < totalReserve ? reserveFromBack : 0);</div>
</div>
<div class="">
<div class="">31 }</div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
</div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="">
<div class="">
<div class="">83 Buffer* buf = new Buffer(size);</div>
</div>
<div class="">
<div class="">84 std::copy_backward(m_buffer->begin(), m_buffer->end(), buf->end());</div>
</div>
<div class="">
<div class="">85 </div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">=================================</div>
<div class="">The other possibility is that dead-nonce-list is not getting cleared after the *loop detection duration*. Or perhaps the ndn::Block() is not released after the ndn::Name wireEncode() is done for the first time.
<font color="#e06666" class="">This is the second most reason for memory build up.</font> </div>
<div class="">Ref: <a href="https://github.com/named-data/NFD/blob/master/daemon/table/dead-nonce-list.hpp#L39" target="_blank" class="">https://github.com/named-data/NFD/blob/master/daemon/table/dead-nonce-list.hpp#L39</a></div>
<div class=""><br class="">
</div>
<div class="">./NFD/daemon/table/dead-nonce-list.cpp</div>
</div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="">
<div class="">
<div class="">105 DeadNonceList::Entry</div>
</div>
<div class="">
<div class="">106 DeadNonceList::makeEntry(const Name& name, uint32_t nonce)</div>
</div>
<div class="">
<div class="">107 {</div>
</div>
<div class="">
<div class="">108 Block nameWire = name.wireEncode();</div>
</div>
</blockquote>
<div class="">
<div class=""><br class="">
</div>
<div class="">./ndn-cxx/src/encoding/block.cpp</div>
</div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class="">
<div class="">
<div class="">344 m_subBlocks.push_back(Block(m_buffer,</div>
</div>
<div class="">
<div class="">345 type,</div>
</div>
<div class="">
<div class="">346 element_begin, element_end,</div>
</div>
<div class="">
<div class="">347 begin, element_end));</div>
</div>
<div class="">
<div class="">348 </div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Sat, May 7, 2016 at 12:24 AM, Anil Jangam <span dir="ltr" class="">
<<a href="mailto:anilj.mailing@gmail.com" target="_blank" class="">anilj.mailing@gmail.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">Hello All,</div>
<div class=""><br class="">
</div>
<div class="">We debugged this issue further and below are out findings.</div>
<div class=""><br class="">
</div>
<div class="">- The issue is reproducible also on standalone NFD. Vince tried about 100 registration requests and there is a consistent increase in memory. This increase is present even if RibManager::sendSuccessResponse is not called. </div>
<div class=""><br class="">
</div>
<div class="">- The memory grows even if we bypass the RibManager completely by using "nfdc add-nexthop" and this problem is present in the latest code of NFD since Vince tested this with the most up-to-date version. </div>
<div class=""><br class="">
</div>
<div class="">- Another possibility we thought about was response messages getting cached in CS leading to increase in memory consumption by NFD. To rule this out, we set the CS size to 1 by calling 'ndnHelper.setCsSize(1);' before installing NDN L3 Stack
on nodes, but yet we see memory growth. </div>
<div class=""><br class="">
</div>
<div class="">- I also checked that default CS size is 100 packets. So even with this size, CS should not grow beyond 100 packets. So we do not think CS is causing this growth. </div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px" class="">
<div class=""> 42 StackHelper::StackHelper()</div>
<div class=""> 43 : m_needSetDefaultRoutes(false)</div>
<div class=""><span style="background-color:rgb(0,255,0)" class=""> 44 , m_maxCsSize(100)</span></div>
<div class=""> 45 , m_isRibManagerDisabled(false)</div>
<div class=""> 46 , m_isFaceManagerDisabled(false)</div>
<div class=""> 47 , m_isStatusServerDisabled(false)</div>
<div class=""> 48 , m_isStrategyChoiceManagerDisabled(false)</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">- It seems to be some internal pipeline issue, because when we performed either 1000 add-nexthop commands or 1000 registration commands for the same prefix, the memory increase was observed.</div>
<div class=""><br class="">
</div>
<div class="">As mentioned above, we believe this issue also present in standalone NFD, it is not yet reported perhaps because of the scale. Since I am running with 100+ nodes, each having its own NFD instance on my laptop (8G RAM), the growth is very quick. </div>
<div class=""><br class="">
</div>
<div class="">We need your inputs to debug this issue further. </div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">/anil</div>
</div>
<div class="">
<div class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Wed, May 4, 2016 at 1:47 PM, Anil Jangam <span dir="ltr" class="">
<<a href="mailto:anilj.mailing@gmail.com" target="_blank" class="">anilj.mailing@gmail.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">Here are some more data points from Valgrind Massiff analysis. I have ran it for 25 and 50 nodes.
<br class="">
<br class="">
</div>
/anil.<br class="">
<br class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Wed, May 4, 2016 at 2:26 AM, Anil Jangam <span dir="ltr" class="">
<<a href="mailto:anilj.mailing@gmail.com" target="_blank" class="">anilj.mailing@gmail.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">
<div class="">
<div class="">
<div class="">Hi Junxiao,<br class="">
<br class="">
</div>
The memory leak is now closed by back porting the fix you referred to. However, the growth in memory consumption is still evident. This time, I believe it is a bloating of the process size. Can you please comment looking at the attached Valgrind logs if this
is a legitimate requirement of NFD or it is just holding up the resources without really needing it? I see the allocation emanating from RibManager and on receiving Interest as some of the major contributors.
<br class="">
<br class="">
</div>
<div class="">Likewise you said earlier, these are perhaps fixed into main branch of NFD but not ported yet into the NFD of ndnSIM. Please check, reports are attached.<br class="">
<br class="">
</div>
50 node simulation valgrind summary:<br class="">
-------------------------------------------------------<br class="">
<div style="margin-left:40px" class="">==9587== LEAK SUMMARY:<br class="">
==9587== definitely lost: 0 bytes in 0 blocks<br class="">
==9587== indirectly lost: 0 bytes in 0 blocks<br class="">
==9587== possibly lost: 2,263,514 bytes in 67,928 blocks<br class="">
<span style="background-color:rgb(255,255,0)" class="">==9587== still reachable: 1,474,943,776 bytes in 3,910,237 blocks</span><br class="">
==9587== suppressed: 0 bytes in 0 blocks<br class="">
==9587== <br class="">
==9587== For counts of detected and suppressed errors, rerun with: -v<br class="">
==9587== ERROR SUMMARY: 37 errors from 37 contexts (suppressed: 0 from 0)<br class="">
</div>
<br class="">
</div>
25 node simulation valgrind summary:<br class="">
-------------------------------------------------------<br class="">
<div style="margin-left:40px" class="">==9287== LEAK SUMMARY:<br class="">
==9287== definitely lost: 0 bytes in 0 blocks<br class="">
==9287== indirectly lost: 0 bytes in 0 blocks<br class="">
==9287== possibly lost: 400,259 bytes in 11,100 blocks<br class="">
<span style="background-color:rgb(255,255,0)" class="">==9287== still reachable: 437,147,930 bytes in 1,132,024 blocks</span><br class="">
==9287== suppressed: 0 bytes in 0 blocks<br class="">
==9287== <br class="">
==9287== For counts of detected and suppressed errors, rerun with: -v<br class="">
==9287== ERROR SUMMARY: 31 errors from 31 contexts (suppressed: 0 from 0)<span class=""><font color="#888888" class=""><br class="">
</font></span></div>
<span class=""><font color="#888888" class=""><br class="">
</font></span></div>
<span class=""><font color="#888888" class="">/anil.<br class="">
<br class="">
<div class="">
<div class=""><br class="">
<br class="">
</div>
</div>
</font></span></div>
<div class="">
<div class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Tue, May 3, 2016 at 7:42 AM, Junxiao Shi <span dir="ltr" class="">
<<a href="mailto:shijunxiao@email.arizona.edu" target="_blank" class="">shijunxiao@email.arizona.edu</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">Hi Anil
<div class=""><br class="">
</div>
<div class="">The call stack in the Valgrind report indicates that you are running NFD within ndnSIM.</div>
<div class="">#3236 is fixed in NFD commit 9c903e063ea8bdb324a421458eed4f51990ccd2c on Oct 04, 2015. However, ndnSIM's NFD fork is dated back on Aug 21, 2015, and doesn't contain the fix.</div>
<div class="">You may try to backport that commit to ndnSIM's NFD fork, or ask ndnSIM developers to upgrade their fork.</div>
<div class=""><br class="">
</div>
<div class="">Yours, Junxiao
<div class="">
<div class=""><br class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Mon, May 2, 2016 at 5:23 PM, Anil Jangam <span dir="ltr" class="">
<<a href="mailto:anilj.mailing@gmail.com" target="_blank" class="">anilj.mailing@gmail.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">Hi Junxiao,<br class="">
<br class="">
</div>
I am observing a memory leak with NFD and to verify the same I did couple of Valgrind enabled simulation runs with 25 and 50 nodes. Based on the Valgrind report, and output of 'top' command, I see that RAM consumption grows consistently and rapidly. My scaling
test is affected that I am not able to run the simulation for longer time and/or with high number of nodes. Also, I see a very high number of timeouts<br class="">
</div>
</div>
<br class="">
</div>
I see a NFD leak issue in closed state, which confirms this leak however closed owing to its small size. Perhaps this is showing up a high scale?<br class="">
<a href="http://redmine.named-data.net/issues/3236/" target="_blank" class="">http://redmine.named-data.net/issues/3236/</a><br class="">
<br class="">
</div>
<div class="">Please check the attached Valgrind report. Let me know what other data you may need to debug this further. Also, please suggest a solution or workaround to this?<span class=""><font color="#888888" class=""><br class="">
<br class="">
</font></span></div>
<span class=""><font color="#888888" class="">
<div class="">/anil.<br class="">
<br class="">
</div>
</font></span></div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
_______________________________________________<br class="">
Nfd-dev mailing list<br class="">
<a href="mailto:Nfd-dev@lists.cs.ucla.edu" target="_blank" class="">Nfd-dev@lists.cs.ucla.edu</a><br class="">
<a href="http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev" target="_blank" class="">http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
Nfd-dev mailing list<br class="">
<a href="mailto:Nfd-dev@lists.cs.ucla.edu" class="">Nfd-dev@lists.cs.ucla.edu</a><br class="">
http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev<br class="">
</div>
</blockquote>
</div>
<br class="">
</body>
</html>