[Nfd-dev] [Operators] "No buffer space available"

[Dehart, John]

On Sep 7, 2017, at 12:12 PM, Junxiao Shi <shijunxiao at email.arizona.edu<mailto:shijunxiao at email.arizona.edu>> wrote:

Hi John


I am also collecting face information every minute (nfdc face list) and storing
that in a log file so I can see what faces existed around the time of the crash.
My thinking was that when we see the giant interest in the nfd.log I would be able
to tell where it was coming from and track down what was sending it.
You can do so without logging `nfdc face list`. Forwarder DEBUG log tells you the FaceId. FaceTable::add log tells you local and remote FaceUri associated with this FaceId.


In this case, during the time that nfd is timing out Fib Updates, I see in my
nfdc log file that nfdc no longer gets any results. I neglected to redirect stderr
to the log file but I suspect that nfdc is failing to connect to get its results.
But just before it starts failing it showed 501 TCP faces.
I have a small number of nodes in the Testbed that have a problem with
fragmentation and for them I run their faces as TCP. That seems to be a problem.

I used to observed similar behavior for unix faces on several nodes' HTTP status page. Occasionally, there were more than 50 unix faces. I believed it's some large monitoring script starting up. I cannot tell how often this occurs.


For this node, SRRU, it has faces to 6 other nodes. But just before it crashed
it had 70-100 faces to each of those 6 nodes.

Are these faces "on-demand" (created because remote node connecting in) or "persistent" (created by a local node)? Do these faces have the same local/remote port number?

There are 6 persistent and the rest are on-demand


Why would the TCP faces not go away when a new one was needed?

If the faces were "persistent" and have same remote port but different local port, this suggests a bug in NLSR. Deployed NLSR creates faces if one does not exist, but it might repeat face creation if face dataset does not reflect new faces.
However, I'm not concerned by this potential bug because this feature is no longer present in NLSR source code. It instead relies on "permanent" faces already created in NFD.

We had another FATAL error overnight.
https://www.arl.wustl.edu/~jdd/nfd.log.SRRU.FATAL.gz
This log does not end with FATAL.

It doesn’t end with the FATAL, but the FATAL is on line 85181.


<https://www.arl.wustl.edu/~jdd/nfd.log.SRRU.FATAL.gz>

I’m also very curious about what Eric said in the nfd conference call yesterday.
I believe he said that in integration testing he has seen an instance of a long interest
with empty components. Is that actually something built into the tests or is this something
we should be concerned about?
No, there isn't an integration test scenario that uses a long name.

OK. I’ll see if I can get some more information from Eric on what he saw. Seems strange
that he saw in his tests what we saw on the Testbed.

Thanks,
John



Yours, Junxiao

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/nfd-dev/attachments/20170907/a35b752e/attachment-0001.html>