[Nfd-dev] Fwd: [Operators] "No buffer space available"

[Dehart, John]

nfd.log file has not been touched in 11 hours:

ndnops at mccoy:/var/log/ndn$ date; ls -l nfd.log
Tue Sep  5 09:22:28 MDT 2017
-rw-r--r-- 1 root root 9852417 Sep  4 20:15 nfd.log
ndnops at mccoy:/var/log/ndn$


And here is a snippet from the end of the nfd.log file:

1504529452.792695 INFO: [FaceTable] Removed face id=3636 remote=fd://41 local=unix:///run/nfd.sock
1504529584.024639 ERROR: [Forwarder] onInterestReject interest=/localhop/ndn/NLSR/LSA/fr/lip6/%C1.Router/ndnhub/name/%00%04K%0F cannot reject forwarded Interest
1504529584.279773 ERROR: [Forwarder] onInterestReject interest=/localhop/ndn/NLSR/LSA/fr/lip6/%C1.Router/ndnhub/name/%00%04K%0F cannot reject forwarded Interest
1504529584.628634 ERROR: [Forwarder] onInterestReject interest=/localhop/ndn/NLSR/LSA/fr/lip6/%C1.Router/ndnhub/name/%00%04K%0F cannot reject forwarded Interest
1504529584.661140 ERROR: [Forwarder] onInterestReject interest=/localhop/ndn/NLSR/LSA/fr/lip6/%C1.Router/ndnhub/coordinate/%00%036%0B cannot reject forwarded Interest
1504529652.749651 WARNING: [TcpTransport] [id=296,local=tcp4://129.82.138.48:48451,remote=tcp4://128.187.81.12:6363] Send or receive operation failed: Connection reset by peer
1504529652.749688 INFO: [Transport] [id=296,local=tcp4://129.82.138.48:48451,remote=tcp4://128.187.81.12:6363] setState UP -> FAILED
1504529652.773699 WARNING: [TcpTransport] [id=297,local=tcp4://129.82.138.48:6363,remote=tcp4://128.187.81.12:50395] Send or receive operation failed: Connection reset by peer
1504529652.773718 INFO: [Transport] [id=297,local=tcp4://129.82.138.48:6363,remote=tcp4://128.187.81.12:50395] setState UP -> FAILED
1504576945.509068 INFO: [Transport] [id=296,local=tcp4://129.82.138.48:48451,remote=tcp4://128.187.81.12:6363] setState FAILED -> CLOSED
1504577758.141391 INFO: [FaceTable] Removed face id=296 remote=tcp4://128.187.81.12:6363 local=tcp4://129.82.138.48:48451



On Sep 5, 2017, at 10:18 AM, Dehart, John <jdd at wustl.edu<mailto:jdd at wustl.edu>> wrote:


One thing I see on CSU is that nfd is huge:
ndnops at mccoy:~/ndn-ops/NOC/bin$ psalxnfd | grep -v grep
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
4     0   705     1  20   0 8435320 2964084 -   Dsl  ?        230:05 /usr/bin/nfd --config /etc/ndn/nfd.conf
ndnops at mccoy:~/ndn-ops/NOC/bin$

ndnops at mccoy:~/ndn-ops/NOC/bin$ nfdc status show
error while connecting to the forwarder (No buffer space available)
ndnops at mccoy:~/ndn-ops/NOC/bin$


On Sep 5, 2017, at 9:59 AM, Dehart, John <jdd at wustl.edu<mailto:jdd at wustl.edu>> wrote:

All:

(Moving this to nfd-dev)

Seems like we are having a lot of FATAL errors and other issues with the
current installation.

I am seeing this error on the CSU node right now.

ndnops at mccoy:~$ nfdc status show
error while connecting to the forwarder (No buffer space available)
ndnops at mccoy:~$


I have also had the following error on UCLA recently:

1504524817.021009 FATAL: [NFD] std::bad_alloc


I’m also investigating other nodes that have restarted in the last couple of days.

I am about to install a scheme for running the nodes in the Testbed in DEBUG and when nfd
restarts, capture the current set of nfd.log files. I have nfd.log being rotated and am only saving
the last 10 copies of 10MB each. So, they can disappear fairly quickly with DEBUG turned on.
But with my scheme I should be able to save copies of anything FATAL that happened recently.
I’ll also be collecting ‘nfdc face list’ info so we can know what faces were pointing where when
something FATAL happened.

I’m open to other suggestions.

John


Begin forwarded message:

From: Junxiao Shi <shijunxiao at email.arizona.edu<mailto:shijunxiao at email.arizona.edu>>
Subject: [Operators] "No buffer space available"
Date: September 5, 2017 at 12:06:46 AM CDT
To: "<operators at lists.named-data.net<mailto:operators at lists.named-data.net>>" <operators at lists.named-data.net<mailto:operators at lists.named-data.net>>

Dear folks

ARIZONA node is not working at the moment. `nfdc status` gives error:
error while connecting to the forwarder (No buffer space available)

The error went away when I restart NFD. However, I wonder what does this error message mean? I've never seen it.
Neither system memory nor disk space was full, and there wasn't high CPU usage.

Yours, Junxiao
_______________________________________________
Operators mailing list
Operators at lists.named-data.net<mailto:Operators at lists.named-data.net>
http://lists.named-data.net/mailman/listinfo/operators

_______________________________________________
Nfd-dev mailing list
Nfd-dev at lists.cs.ucla.edu<mailto:Nfd-dev at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev

_______________________________________________
Nfd-dev mailing list
Nfd-dev at lists.cs.ucla.edu<mailto:Nfd-dev at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/nfd-dev/attachments/20170905/aa7a15e2/attachment.html>