<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 7, 2017, at 12:12 PM, Junxiao Shi <<a href="mailto:shijunxiao@email.arizona.edu" class="">shijunxiao@email.arizona.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="auto" class="">Hi John</div>
<div class="gmail_quote">
<div dir="auto" class=""><br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<div class=""><br class="">
</div>
<div class="">I am also collecting face information every minute (nfdc face list) and storing</div>
<div class="">that in a log file so I can see what faces existed around the time of the crash.</div>
<div class="">My thinking was that when we see the giant interest in the nfd.log I would be able</div>
<div class="">to tell where it was coming from and track down what was sending it.</div>
</div>
</blockquote>
<div dir="auto" class="">You can do so without logging `nfdc face list`. Forwarder DEBUG log tells you the FaceId. FaceTable::add log tells you local and remote FaceUri associated with this FaceId.</div>
<div dir="auto" class=""><br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<div class=""></div>
<div class=""><br class="">
</div>
<div class="">In this case, during the time that nfd is timing out Fib Updates, I see in my</div>
<div class="">nfdc log file that nfdc no longer gets any results. I neglected to redirect stderr</div>
<div class="">to the log file but I suspect that nfdc is failing to connect to get its results.</div>
<div class="">But just before it starts failing it showed 501 TCP faces.</div>
<div class="">I have a small number of nodes in the Testbed that have a problem with</div>
<div class="">fragmentation and for them I run their faces as TCP. That seems to be a problem.</div>
</div>
</blockquote>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">I used to observed similar behavior for unix faces on several nodes' HTTP status page. Occasionally, there were more than 50 unix faces. I believed it's some large monitoring script starting up. I cannot tell how often this occurs.</div>
<div dir="auto" class=""><br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<div class=""></div>
<div class=""><br class="">
</div>
<div class="">For this node, SRRU, it has faces to 6 other nodes. But just before it crashed</div>
<div class="">it had 70-100 faces to each of those 6 nodes.</div>
</div>
</blockquote>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Are these faces "on-demand" (created because remote node connecting in) or "persistent" (created by a local node)? Do these faces have the same local/remote port number?</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
There are 6 persistent and the rest are on-demand</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="gmail_quote">
<div dir="auto" class=""><br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<div class=""></div>
<div class="">Why would the TCP faces not go away when a new one was needed?</div>
</div>
</blockquote>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">If the faces were "persistent" and have same remote port but different local port, this suggests a bug in NLSR. Deployed NLSR creates faces if one does not exist, but it might repeat face creation if face dataset does not reflect new
faces.</div>
<div dir="auto" class="">However, I'm not concerned by this potential bug because this feature is no longer present in NLSR source code. It instead relies on "permanent" faces already created in NFD.</div>
<div dir="auto" class=""><br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div style="word-wrap:break-word" class="">
<div class="">We had another FATAL error overnight.</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div style="word-wrap:break-word" class="">
<div class=""><a href="https://www.arl.wustl.edu/~jdd/nfd.log.SRRU.FATAL.gz" target="_blank" class="">https://www.arl.wustl.edu/~jdd/nfd.log.SRRU.FATAL.gz</a></div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<div dir="auto" class="">This log does not end with FATAL.</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
It doesn’t end with the FATAL, but the FATAL is on line 85181.</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div>
<blockquote type="cite" class="">
<div class="">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div style="word-wrap:break-word" class="">
<div class=""><a href="https://www.arl.wustl.edu/~jdd/nfd.log.SRRU.FATAL.gz" target="_blank" class=""></a></div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">
<blockquote type="cite" class="">
<div style="word-wrap:break-word" dir="auto" class="">
<div class=""><br class="">
</div>
<div class="">I’m also very curious about what Eric said in the nfd conference call yesterday.</div>
<div class="">I believe he said that in integration testing he has seen an instance of a long interest</div>
<div class="">with empty components. Is that actually something built into the tests or is this something</div>
<div class="">we should be concerned about?</div>
</div>
</blockquote>
</div>
</blockquote>
<div dir="auto" class="">No, there isn't an integration test scenario that uses a long name.</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
OK. I’ll see if I can get some more information from Eric on what he saw. Seems strange</div>
<div>that he saw in his tests what we saw on the Testbed.</div>
<div><br class="">
</div>
<div>Thanks,</div>
<div>John</div>
<div><br class="">
<div><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div class="gmail_quote">
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Yours, Junxiao</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>