<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
OK. I have a clue.
<div class=""><br class="">
</div>
<div class="">I am also collecting face information every minute (nfdc face list) and storing</div>
<div class="">that in a log file so I can see what faces existed around the time of the crash.</div>
<div class="">My thinking was that when we see the giant interest in the nfd.log I would be able</div>
<div class="">to tell where it was coming from and track down what was sending it.</div>
<div class=""><br class="">
</div>
<div class="">In this case, during the time that nfd is timing out Fib Updates, I see in my</div>
<div class="">nfdc log file that nfdc no longer gets any results. I neglected to redirect stderr</div>
<div class="">to the log file but I suspect that nfdc is failing to connect to get its results.</div>
<div class="">But just before it starts failing it showed 501 TCP faces.</div>
<div class="">I have a small number of nodes in the Testbed that have a problem with</div>
<div class="">fragmentation and for them I run their faces as TCP. That seems to be a problem.</div>
<div class=""><br class="">
</div>
<div class="">For this node, SRRU, it has faces to 6 other nodes. But just before it crashed</div>
<div class="">it had 70-100 faces to each of those 6 nodes.</div>
<div class="">Why would the TCP faces not go away when a new one was needed?</div>
<div class=""><br class="">
</div>
<div class="">John</div>
<div class=""><br class="">
</div>
<div class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 7, 2017, at 9:07 AM, Dehart, John <<a href="mailto:jdd@wustl.edu" class="">jdd@wustl.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
All:
<div class=""><br class="">
</div>
<div class="">We had another FATAL error overnight.</div>
<div class="">I have a DEBUG nfd log file that caught it but I don’t see any enormous interests.</div>
<div class="">I’m still going through the log file to see what might have happened.</div>
<div class="">In case anyone else wants to look at it, it is here:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://www.arl.wustl.edu/~jdd/nfd.log.SRRU.FATAL.gz" class="">https://www.arl.wustl.edu/~jdd/nfd.log.SRRU.FATAL.gz</a></div>
<div class=""><br class="">
</div>
<div class="">I’m also very curious about what Eric said in the nfd conference call yesterday.</div>
<div class="">I believe he said that in integration testing he has seen an instance of a long interest</div>
<div class="">with empty components. Is that actually something built into the tests or is this something</div>
<div class="">we should be concerned about?</div>
<div class=""><br class="">
</div>
<div class="">John</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Sep 6, 2017, at 5:49 AM, Dehart, John <<a href="mailto:jdd@wustl.edu" class="">jdd@wustl.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" class="">
<div class=""><br class="Apple-interchange-newline">
On Sep 6, 2017, at 3:49 AM, Davide Pesavento <<a href="mailto:davide.pesavento@lip6.fr" class="">davide.pesavento@lip6.fr</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">On
Tue, Sep 5, 2017 at 7:57 PM, Dehart, John <</span><a href="mailto:jdd@wustl.edu" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">jdd@wustl.edu</a><span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">>
wrote:</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<br class="">
<blockquote type="cite" class="">On Sep 5, 2017, at 11:36 AM, Davide Pesavento <<a href="mailto:davide.pesavento@lip6.fr" class="">davide.pesavento@lip6.fr</a>> wrote:<br class="">
<br class="">
On Tue, Sep 5, 2017 at 5:23 PM, Dehart, John <<a href="mailto:jdd@wustl.edu" class="">jdd@wustl.edu</a>> wrote:<br class="">
<blockquote type="cite" class=""><br class="">
nfd.log file has not been touched in 11 hours:<br class="">
</blockquote>
<br class="">
So nfd is up but not doing anything meaningful? Can you run strace -p<br class="">
on the stuck nfd?<br class="">
</blockquote>
<br class="">
Too late. I restarted with my updates to change to DEBUG and capture log files.<br class="">
Yes, it was stuck and not responding to anything. Did attach to it with gdb<br class="">
to see if that would tell me anything but not sure this really tells us anything:<br class="">
<br class="">
(gdb) bt<br class="">
#0 malloc_consolidate (av=av@entry=0x7f6a6de53760 <main_arena>) at malloc.c:4165<br class="">
#1 0x00007f6a6db1072d in _int_free (av=0x7f6a6de53760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:4059<br class="">
#2 0x00007f6a6f50baf7 in std::_Sp_counted_ptr<ndn::Buffer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /usr/lib/x86_64-linux-gnu/libndn-cxx.so.0.5.1<br class="">
#3 0x000000000045b639 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()<br class="">
#4 0x0000000000472e44 in ?? ()<br class="">
#5 0x000000000047c267 in ?? ()<br class="">
#6 0x000000000047c393 in ?? ()<br class="">
#7 0x000000000047a567 in ?? ()<br class="">
#8 0x000000000044eff4 in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()<br class="">
#9 0x0000000000450576 in ?? ()<br class="">
#10 0x00000000004363ff in ?? ()<br class="">
#11 0x00007f6a6dab2f45 in __libc_start_main (main=0x436060, argc=3, argv=0x7ffff81bf138, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffff81bf128) at libc-start.c:287<br class="">
#12 0x0000000000449399 in ?? ()<br class="">
(gdb)<br class="">
<br class="">
When I see it again, I will try strace.<br class="">
</blockquote>
<br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Thanks.
A proper gdb backtrace could be useful too, but the above is</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">not.
To make it more useful you need to install the debug symbols for</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">ndn-cxx
and nfd.</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">What is the penalty for running with the versions with debug symbols?</div>
<div class="">Should we be doing that at all times or is it a performance killer?</div>
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div class=""><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">When
nfd was stuck, do you know if CPU consumption was at or close to 100%?</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
</div>
</blockquote>
<div class=""><br class="">
</div>
No. The CPU usage was close to 0.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
John</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
<blockquote type="cite" class="">
<div class=""><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Thanks,</span><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<span class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Davide</span></div>
</blockquote>
</div>
<br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Nfd-dev
mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="mailto:Nfd-dev@lists.cs.ucla.edu" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Nfd-dev@lists.cs.ucla.edu</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
Nfd-dev mailing list<br class="">
<a href="mailto:Nfd-dev@lists.cs.ucla.edu" class="">Nfd-dev@lists.cs.ucla.edu</a><br class="">
http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>