[ndnSIM] ndn::tlv::Error after 600 seconds - too many faces?

[Christian Kreuzberger]

Hi all,

I have a rather large simulation where there are roughly 10000 nodes 
(clients) connected to one intermediate router, which is then connected 
to one server (it is not the most realistic scenario, I know).
Using ndnSIM 2.1 (commit ef77416d5227d6e9123f7c3aaab719793379c2d7), I am 
getting an exception (ndn::tlv::Error) after roughly 600 seconds 
(something between 599.9 and 600.0 seconds, so suspiciously close to 600 
seconds). I've tried with different client start times, it always 
crashes at 600 seconds (my application is set to run much much longer 
than 600 seconds).

 From debugging with gdb I saw that the Interest (or data packet) in 
question has the following name: /localhost/nfd/faces/list


The backtrace (using gdb) looks like this (I've removed non-relevant 
lines like event handling and exception throw handling):


#5  0x00007fffe8e06922 in __cxa_throw () from 
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff23a7814 in 
boost::throw_exception<boost::exception_detail::error_info_injector<ndn::tlv::Error> 
 > (e=...) at /usr/include/boost/throw_exception.hpp:67
#7  0x00007ffff23a6e21 in 
boost::exception_detail::throw_exception_<ndn::tlv::Error> (x=..., 
current_function=0x7ffff26e33f0 <ndn::Block::parse() 
const::__PRETTY_FUNCTION__> "void ndn::Block::parse() const",
     file=0x7ffff26e2990 "../src/ndnSIM/ndn-cxx/src/encoding/block.cpp", 
line=340) at /usr/include/boost/throw_exception.hpp:84
#8  0x00007ffff23a9db1 in ndn::Block::parse (this=0x7fffffffc1e0) at 
../src/ndnSIM/ndn-cxx/src/encoding/block.cpp:340
#9  0x00007ffff23f995c in ndn::nfd::FaceStatus::wireDecode 
(this=0x7fffffffc170, block=...) at 
../src/ndnSIM/ndn-cxx/src/management/nfd-face-status.cpp:121
#10 0x00007ffff23f9738 in ndn::nfd::FaceStatus::FaceStatus 
(this=0x7fffffffc170, block=...) at 
../src/ndnSIM/ndn-cxx/src/management/nfd-face-status.cpp:49
#11 0x00007ffff26059a8 in nfd::rib::RibManager::removeInvalidFaces 
(this=0x5ad20018, buffer=...) at 
../src/ndnSIM/NFD/rib/rib-manager.cpp:704
#12 0x00007ffff260557b in nfd::rib::RibManager::fetchSegments 
(this=0x5ad20018, data=..., buffer=...) at 
../src/ndnSIM/NFD/rib/rib-manager.cpp:679
#13 0x00007ffff2617f41 in boost::_mfi::mf2<void, nfd::rib::RibManager, 
ndn::Data const&, std::shared_ptr<ndn::OBufferStream> >::operator() 
(this=0xa021be20, p=0x5ad20018, a1=..., a2=...)
     at /usr/include/boost/bind/mem_fn_template.hpp:280
#14 0x00007ffff2616040 in 
boost::_bi::list3<boost::_bi::value<nfd::rib::RibManager*>, 
boost::arg<2>, boost::_bi::value<std::shared_ptr<ndn::OBufferStream> > 
 >::operator()<boost::_mfi::mf2<void, nfd::rib::RibManager, ndn::Data const&, std::shared_ptr<ndn::OBufferStream> >, boost::_bi::list2<ndn::Interest const&, ndn::Data&> > (this=0xa021be30, f=..., a=...) at /usr/include/boost/bind/bind.hpp:392
#15 0x00007ffff2612c44 in boost::_bi::bind_t<void, 
boost::_mfi::mf2<void, nfd::rib::RibManager, ndn::Data const&, 
std::shared_ptr<ndn::OBufferStream> >, 
boost::_bi::list3<boost::_bi::value<nfd::rib::RibManager*>, 
boost::arg<2>, boost::_bi::value<std::shared_ptr<ndn::OBufferStream> > > 
 >::operator()<ndn::Interest, ndn::Data> (this=0xa021be20, a1=..., a2=...) at /usr/include/boost/bind/bind_template.hpp:76
#16 0x00007ffff260eeb2 in std::_Function_handler<void (ndn::Interest 
const&, ndn::Data&), boost::_bi::bind_t<void, boost::_mfi::mf2<void, 
nfd::rib::RibManager, ndn::Data const&, 
std::shared_ptr<ndn::OBufferStream> >, 
boost::_bi::list3<boost::_bi::value<nfd::rib::RibManager*>, 
boost::arg<2>, boost::_bi::value<std::shared_ptr<ndn::OBufferStream> > > 
 > >::_M_invoke(std::_Any_data const&, ndn::Interest const&, ndn::Data&) (__functor=..., __args#0=..., __args#1=...)
     at /usr/include/c++/4.8/functional:2071
#17 0x00007ffff23c0d63 in std::function<void (ndn::Interest const&, 
ndn::Data&)>::operator()(ndn::Interest const&, ndn::Data&) const 
(this=0x98f2a648, __args#0=..., __args#1=...) at 
/usr/include/c++/4.8/functional:2471
#18 0x00007ffff23be299 in ndn::PendingInterest::invokeDataCallback 
(this=0x98f2a638, data=...) at 
../src/ndnSIM/ndn-cxx/src/detail/pending-interest.hpp:81
#19 0x00007ffff23bf41d in ndn::Face::Impl::satisfyPendingInterests 
(this=0x5ad1ce20, data=...) at 
../src/ndnSIM/ndn-cxx/src/detail/face-impl.hpp:140
#20 0x00007ffff23be88e in ndn::Face::Impl::NfdFace::sendData(ndn::Data 
const&)::{lambda()#1}::operator()() const (__closure=0x81b20ed0) at 
../src/ndnSIM/ndn-cxx/src/detail/face-impl.hpp:91
#21 0x00007ffff23c3ddc in std::_Function_handler<void (), 
ndn::Face::Impl::NfdFace::sendData(ndn::Data 
const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) 
at /usr/include/c++/4.8/functional:2071
#22 0x00007ffff23c0e4a in std::function<void ()>::operator()() const 
(this=0x9ada39c0) at /usr/include/c++/4.8/functional:2471
#23 0x00007ffff2514135 in ns3::EventImpl* ns3::MakeEvent<void 
(std::function<void ()>::*)() const, std::function<void ()> >(void 
(std::function<void ()>::*)() const, std::function<void 
()>)::EventMemberImpl0::Notify() (this=0x9ada39b0)
     at ./ns3/make-event.h:323



I haven't had much time to debug, but I am suspecting my unusually large 
number of client nodes (10k) is too much for the RIB Manager (?) to 
process. I'm also not sure what this RIB Manager is doing, because I 
haven't explicitly activated it (at least I don't know about it). I did 
find the source that seems to be causing the event after 600 seconds 
though: NFD/rib/remote-registrator.cpp

I will try to provide scenario code to reproduce the problem tomorrow, 
for now I'm just looking for some more pointers for debugging so I can 
produce a better (more helpful) bug report (which I will then, as 
always, submit to redmine).

Thanks
Christian