[Nfd-dev] [Operators] "No buffer space available"

[Dehart, John]

On Sep 7, 2017, at 12:12 PM, Junxiao Shi <shijunxiao at email.arizona.edu<mailto:shijunxiao at email.arizona.edu>> wrote:

Hi John


I am also collecting face information every minute (nfdc face list) and storing
that in a log file so I can see what faces existed around the time of the crash.
My thinking was that when we see the giant interest in the nfd.log I would be able
to tell where it was coming from and track down what was sending it.
You can do so without logging `nfdc face list`. Forwarder DEBUG log tells you the FaceId. FaceTable::add log tells you local and remote FaceUri associated with this FaceId.

Yes, but the problem I was envisioning was that with the log file rotating some of the face adds would be far
enough back in time that the log messages for them would be gone.

John



In this case, during the time that nfd is timing out Fib Updates, I see in my
nfdc log file that nfdc no longer gets any results. I neglected to redirect stderr
to the log file but I suspect that nfdc is failing to connect to get its results.
But just before it starts failing it showed 501 TCP faces.
I have a small number of nodes in the Testbed that have a problem with
fragmentation and for them I run their faces as TCP. That seems to be a problem.

I used to observed similar behavior for unix faces on several nodes' HTTP status page. Occasionally, there were more than 50 unix faces. I believed it's some large monitoring script starting up. I cannot tell how often this occurs.


For this node, SRRU, it has faces to 6 other nodes. But just before it crashed
it had 70-100 faces to each of those 6 nodes.

Are these faces "on-demand" (created because remote node connecting in) or "persistent" (created by a local node)? Do these faces have the same local/remote port number?

Why would the TCP faces not go away when a new one was needed?

If the faces were "persistent" and have same remote port but different local port, this suggests a bug in NLSR. Deployed NLSR creates faces if one does not exist, but it might repeat face creation if face dataset does not reflect new faces.
However, I'm not concerned by this potential bug because this feature is no longer present in NLSR source code. It instead relies on "permanent" faces already created in NFD.

We had another FATAL error overnight.
https://www.arl.wustl.edu/~jdd/nfd.log.SRRU.FATAL.gz
This log does not end with FATAL.
<https://www.arl.wustl.edu/~jdd/nfd.log.SRRU.FATAL.gz>

I’m also very curious about what Eric said in the nfd conference call yesterday.
I believe he said that in integration testing he has seen an instance of a long interest
with empty components. Is that actually something built into the tests or is this something
we should be concerned about?
No, there isn't an integration test scenario that uses a long name.

Yours, Junxiao

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/nfd-dev/attachments/20170907/82826875/attachment.html>