<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
All:
<div class=""><br class="">
</div>
<div class="">I have not been able to recreate the problem we had during the SIGCOMM NDN Tutorial.</div>
<div class=""><br class="">
</div>
<div class="">There were three things going on that might have had some impact.</div>
<div class="">1. I had all my AWS instances up for a long time (> 12 hours) waiting for the demonstrations.</div>
<div class="">2. Haitao was doing some work with the namespace-tree app.</div>
<div class="">3. Junxiao was doing some work with his tunneling app.</div>
<div class="">But so far none of those seems to cause any problems now.</div>
<div class="">It is possible that is some combination of those three, but that will take some time to re-create.</div>
<div class=""><br class="">
</div>
<div class="">So, I would like to see if anyone has any insight into what may have happened.</div>
<div class="">Here are some details from the log files I saved from the WU gateway node:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">ndnops@wundngw:/var/log/ndn/SIGCOMM_SNAFU$ grep -B 1 FATAL *</div>
<div class="">nfd.log.2-1503351121.547045 INFO: [RibManager] Adding route /ndn/edu/uiuc nexthop=275 origin=nlsr cost=42464</div>
<div class="">nfd.log.2:1503351231.628809 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.2-1503352381.309113 INFO: [RibManager] Removing route /ndn/edu/ucla/%C1.Router/cs/spurs nexthop=275 origin=nlsr</div>
<div class="">nfd.log.2:1503352491.336944 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.2-1503352888.689848 INFO: [FibManager] add-nexthop result: FAIL reason: unknown-faceid: 334</div>
<div class="">nfd.log.2:1503352998.742155 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.3-1503300702.536604 INFO: [RibManager] Adding route /ndn/br/ufpa/%C1.Router/hostname nexthop=277 origin=nlsr cost=41860</div>
<div class="">nfd.log.3:1503300711.619102 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.4-1503257126.710786 INFO: [RibManager] Adding route /ndn/edu/memphis nexthop=8699 origin=nlsr cost=21465</div>
<div class="">nfd.log.4:1503257278.816893 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.4-1503257664.621400 INFO: [RibManager] Adding route /ndn/multicast nexthop=277 origin=autoreg cost=255</div>
<div class="">nfd.log.4:1503257774.617839 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.4-1503259091.220589 INFO: [AutoPrefixPropagator] no signing identity available for: /ndn/edu/arizona</div>
<div class="">nfd.log.4:1503259201.264904 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.4-1503260778.081639 INFO: [RibManager] Removing route /ndn/de/uni-goettingen/%C1.Router/ndntestbed nexthop=276 origin=nlsr</div>
<div class="">nfd.log.4:1503260888.015256 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.4-1503268049.795983 INFO: [FaceTable] Removed face id=350 remote=<a href="fd://39" class="">fd://39</a> local=<a href="unix:///run/nfd.sock" class="">unix:///run/nfd.sock</a></div>
<div class="">nfd.log.4:1503268216.088842 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.4-1503269941.426706 INFO: [RibManager] Removing route /ndn/edu/memphis nexthop=275 origin=nlsr</div>
<div class="">nfd.log.4:1503270051.260809 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.4.snip:1503257278.816893 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">--</div>
<div class="">nfd.log.4.snip-1503257664.621400 INFO: [RibManager] Adding route /ndn/multicast nexthop=277 origin=autoreg cost=255</div>
<div class="">nfd.log.4.snip:1503257774.617839 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">ndnops@wundngw:/var/log/ndn/SIGCOMM_SNAFU$</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">And let me highlight one of those which seems typical:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">--</div>
<div class="">nfd.log.4-1503257664.621400 INFO: [RibManager] Adding route /ndn/multicast nexthop=277 origin=autoreg cost=255</div>
<div class="">nfd.log.4:1503257774.617839 FATAL: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">—</div>
</div>
<div class=""><br class="">
</div>
<div class="">And here is a little more of the log file from that one:</div>
<div class="">
<div class="">1503257410.676895 INFO: [UnicastUdpTransport] [id=0,local=<a href="udp4://128.252.153.194:6363,remote=udp4://141.225.11.173:6363" class="">udp4://128.252.153.194:6363,remote=udp4://141.225.11.173:6363</a>] Creating transport</div>
<div class="">1503257410.676988 INFO: [FaceTable] Added face id=277 remote=<a href="udp4://141.225.11.173:6363" class="">udp4://141.225.11.173:6363</a> local=<a href="udp4://128.252.153.194:6363" class="">udp4://128.252.153.194:6363</a></div>
<div class="">1503257664.566506 INFO: [UnicastUdpTransport] [id=0,local=<a href="udp4://128.252.153.194:6363,remote=udp4://193.147.51.186:6363" class="">udp4://128.252.153.194:6363,remote=udp4://193.147.51.186:6363</a>] Creating transport</div>
<div class="">1503257664.566544 INFO: [FaceTable] Added face id=278 remote=<a href="udp4://193.147.51.186:6363" class="">udp4://193.147.51.186:6363</a> local=<a href="udp4://128.252.153.194:6363" class="">udp4://128.252.153.194:6363</a></div>
<div class="">1503257664.570254 INFO: [UnixStreamTransport] [id=0,local=<a href="unix:///run/nfd.sock,remote=fd://38" class="">unix:///run/nfd.sock,remote=fd://38</a>] Creating transport</div>
<div class="">1503257664.570280 INFO: [FaceTable] Added face id=279 remote=<a href="fd://38" class="">fd://38</a> local=<a href="unix:///run/nfd.sock" class="">unix:///run/nfd.sock</a></div>
<div class="">1503257664.570940 INFO: [RibManager] Adding route /ndn/broadcast nexthop=277 origin=autoreg cost=255</div>
<div class="">1503257664.621400 INFO: [RibManager] Adding route /ndn/multicast nexthop=277 origin=autoreg cost=255</div>
<div class="">1503257774.617839 <b class="">FATAL</b>: [NFD] Non-recoverable error: request timed out code: 10060</div>
<div class="">1503257919.686039 INFO: [CsPolicy] setLimit 10</div>
</div>
<div class=""><br class="">
</div>
<div class="">That last message is the start of a restart of NFD.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">What I see here is that we are trying to add two faces, 278 and 279 and then add a route for /ndn/broadcast and then a route </div>
<div class="">for /ndn/multicast and then 110 seconds later we get a FATAL error.</div>
<div class="">I’ve looked at the NFD code a little bit and it does seem that the error is as others suggested probably related to a timeout in the RibManager.</div>
<div class=""><br class="">
</div>
<div class="">And if I look at this code from fib-updater.cpp: FibUpdater::onUpdateError():</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""> if (code == ndn::nfd::Controller::ERROR_TIMEOUT && nTimeouts < MAX_NUM_TIMEOUTS) {</div>
<div class=""> sendAddNextHopUpdate(update, onSuccess, onFailure, ++nTimeouts);</div>
<div class=""> }</div>
</div>
<div class=""><br class="">
</div>
<div class="">It would make sense that it would keep trying until the 11th time and so if the timeout was 10 seconds</div>
<div class="">this would all make sense. I haven’t found where the timeout period is but I’m guessing it is 10 seconds.</div>
<div class="">And this seems to agree with the FIB Updater section of the NFD dev-guide.</div>
<div class=""><br class="">
</div>
<div class="">So, then the question is, what would cause the RibManager to fail 10 successive times in trying to do a Fib Update?</div>
<div class=""><br class="">
</div>
<div class="">Any thoughts?</div>
<div class=""><br class="">
</div>
<div class="">John</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
</body>
</html>