[ndnSIM] ndn::tlv::Error after 600 seconds - too many faces?

Tue Nov 10 04:35:10 PST 2015

After some heavy debugging (involving coffee, gdb and fprintf), I found the
cause and I have written a bug report:
http://redmine.named-data.net/issues/3326 
In short, it is caused by having many (virtual) faces (so it's not only
related to the number of nodes, but also to the number of applications,
etc...) and having a deactivated or too small content store.

> -----Original Message-----
> From: Christian Kreuzberger [mailto:christian.kreuzberger at itec.aau.at]
> Sent: Tuesday, November 10, 2015 1:35 PM
> To: 'Christian Kreuzberger' <christian.kreuzberger at itec.aau.at>
> Subject: RE: [ndnSIM] ndn::tlv::Error after 600 seconds - too many faces?
> 
> After some heavy debugging (involving coffee, gdb and fprintf), I found
the
> cause and I have written a bug report:
> http://redmine.named-data.net/issues/3326
> In short, it is caused by having many (virtual) faces (so it's not only
related to
> the number of nodes, but also to the number of applications,
> etc...) and having a deactivated or too small content store.
> 
> 
> > -----Original Message-----
> > From: ndnSIM [mailto:ndnsim-bounces at lists.cs.ucla.edu] On Behalf Of
> > Christian Kreuzberger
> > Sent: Monday, November 09, 2015 10:23 PM
> > To: ndnSIM at lists.cs.ucla.edu
> > Subject: [ndnSIM] ndn::tlv::Error after 600 seconds - too many faces?
> >
> > Hi all,
> >
> > I have a rather large simulation where there are roughly 10000 nodes
> > (clients) connected to one intermediate router, which is then connected
to
> > one server (it is not the most realistic scenario, I know).
> > Using ndnSIM 2.1 (commit ef77416d5227d6e9123f7c3aaab719793379c2d7), I
> > am getting an exception (ndn::tlv::Error) after roughly 600 seconds
> > (something between 599.9 and 600.0 seconds, so suspiciously close to 600
> > seconds). I've tried with different client start times, it always
crashes
> at 600
> > seconds (my application is set to run much much longer than 600
seconds).
> >
> >  From debugging with gdb I saw that the Interest (or data packet) in
> question
> > has the following name: /localhost/nfd/faces/list
> >
> >
> > The backtrace (using gdb) looks like this (I've removed non-relevant
> > lines like event handling and exception throw handling):
> >
> >
> > #5  0x00007fffe8e06922 in __cxa_throw () from
> > /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> > #6  0x00007ffff23a7814 in
> >
> boost::throw_exception<boost::exception_detail::error_info_injector<ndn::
> > tlv::Error>
> >  > (e=...) at /usr/include/boost/throw_exception.hpp:67
> > #7  0x00007ffff23a6e21 in
> > boost::exception_detail::throw_exception_<ndn::tlv::Error> (x=...,
> > current_function=0x7ffff26e33f0 <ndn::Block::parse()
> > const::__PRETTY_FUNCTION__> "void ndn::Block::parse() const",
> >      file=0x7ffff26e2990 "../src/ndnSIM/ndn-cxx/src/encoding/block.cpp",
> > line=340) at /usr/include/boost/throw_exception.hpp:84
> > #8  0x00007ffff23a9db1 in ndn::Block::parse (this=0x7fffffffc1e0) at
> > ../src/ndnSIM/ndn-cxx/src/encoding/block.cpp:340
> > #9  0x00007ffff23f995c in ndn::nfd::FaceStatus::wireDecode
> > (this=0x7fffffffc170, block=...) at
> > ../src/ndnSIM/ndn-cxx/src/management/nfd-face-status.cpp:121
> > #10 0x00007ffff23f9738 in ndn::nfd::FaceStatus::FaceStatus
> > (this=0x7fffffffc170, block=...) at
> > ../src/ndnSIM/ndn-cxx/src/management/nfd-face-status.cpp:49
> > #11 0x00007ffff26059a8 in nfd::rib::RibManager::removeInvalidFaces
> > (this=0x5ad20018, buffer=...) at
> > ../src/ndnSIM/NFD/rib/rib-manager.cpp:704
> > #12 0x00007ffff260557b in nfd::rib::RibManager::fetchSegments
> > (this=0x5ad20018, data=..., buffer=...) at
> > ../src/ndnSIM/NFD/rib/rib-manager.cpp:679
> > #13 0x00007ffff2617f41 in boost::_mfi::mf2<void, nfd::rib::RibManager,
> > ndn::Data const&, std::shared_ptr<ndn::OBufferStream> >::operator()
> > (this=0xa021be20, p=0x5ad20018, a1=..., a2=...)
> >      at /usr/include/boost/bind/mem_fn_template.hpp:280
> > #14 0x00007ffff2616040 in
> > boost::_bi::list3<boost::_bi::value<nfd::rib::RibManager*>,
> > boost::arg<2>, boost::_bi::value<std::shared_ptr<ndn::OBufferStream> >
> >  >::operator()<boost::_mfi::mf2<void, nfd::rib::RibManager, ndn::Data
> > const&, std::shared_ptr<ndn::OBufferStream> >,
> > boost::_bi::list2<ndn::Interest const&, ndn::Data&> > (this=0xa021be30,
> > f=..., a=...) at /usr/include/boost/bind/bind.hpp:392
> > #15 0x00007ffff2612c44 in boost::_bi::bind_t<void,
> > boost::_mfi::mf2<void, nfd::rib::RibManager, ndn::Data const&,
> > std::shared_ptr<ndn::OBufferStream> >,
> > boost::_bi::list3<boost::_bi::value<nfd::rib::RibManager*>,
> > boost::arg<2>, boost::_bi::value<std::shared_ptr<ndn::OBufferStream> >
> >
> >  >::operator()<ndn::Interest, ndn::Data> (this=0xa021be20, a1=...,
a2=...)
> at
> > /usr/include/boost/bind/bind_template.hpp:76
> > #16 0x00007ffff260eeb2 in std::_Function_handler<void (ndn::Interest
> > const&, ndn::Data&), boost::_bi::bind_t<void, boost::_mfi::mf2<void,
> > nfd::rib::RibManager, ndn::Data const&,
> > std::shared_ptr<ndn::OBufferStream> >,
> > boost::_bi::list3<boost::_bi::value<nfd::rib::RibManager*>,
> > boost::arg<2>, boost::_bi::value<std::shared_ptr<ndn::OBufferStream> >
> >
> >  > >::_M_invoke(std::_Any_data const&, ndn::Interest const&,
> ndn::Data&)
> > (__functor=..., __args#0=..., __args#1=...)
> >      at /usr/include/c++/4.8/functional:2071
> > #17 0x00007ffff23c0d63 in std::function<void (ndn::Interest const&,
> > ndn::Data&)>::operator()(ndn::Interest const&, ndn::Data&) const
> > (this=0x98f2a648, __args#0=..., __args#1=...) at
> > /usr/include/c++/4.8/functional:2471
> > #18 0x00007ffff23be299 in ndn::PendingInterest::invokeDataCallback
> > (this=0x98f2a638, data=...) at
> > ../src/ndnSIM/ndn-cxx/src/detail/pending-interest.hpp:81
> > #19 0x00007ffff23bf41d in ndn::Face::Impl::satisfyPendingInterests
> > (this=0x5ad1ce20, data=...) at
> > ../src/ndnSIM/ndn-cxx/src/detail/face-impl.hpp:140
> > #20 0x00007ffff23be88e in ndn::Face::Impl::NfdFace::sendData(ndn::Data
> > const&)::{lambda()#1}::operator()() const (__closure=0x81b20ed0) at
> > ../src/ndnSIM/ndn-cxx/src/detail/face-impl.hpp:91
> > #21 0x00007ffff23c3ddc in std::_Function_handler<void (),
> > ndn::Face::Impl::NfdFace::sendData(ndn::Data
> > const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
> (__functor=...)
> > at /usr/include/c++/4.8/functional:2071
> > #22 0x00007ffff23c0e4a in std::function<void ()>::operator()() const
> > (this=0x9ada39c0) at /usr/include/c++/4.8/functional:2471
> > #23 0x00007ffff2514135 in ns3::EventImpl* ns3::MakeEvent<void
> > (std::function<void ()>::*)() const, std::function<void ()> >(void
> > (std::function<void ()>::*)() const, std::function<void
> > ()>)::EventMemberImpl0::Notify() (this=0x9ada39b0)
> >      at ./ns3/make-event.h:323
> >
> >
> >
> > I haven't had much time to debug, but I am suspecting my unusually large
> > number of client nodes (10k) is too much for the RIB Manager (?) to
> > process. I'm also not sure what this RIB Manager is doing, because I
> > haven't explicitly activated it (at least I don't know about it). I did
> > find the source that seems to be causing the event after 600 seconds
> > though: NFD/rib/remote-registrator.cpp
> >
> > I will try to provide scenario code to reproduce the problem tomorrow,
> > for now I'm just looking for some more pointers for debugging so I can
> > produce a better (more helpful) bug report (which I will then, as
> > always, submit to redmine).
> >
> > Thanks
> > Christian
> >
> >
> > _______________________________________________
> > ndnSIM mailing list
> > ndnSIM at lists.cs.ucla.edu
> > http://www.lists.cs.ucla.edu/mailman/listinfo/ndnsim