[Nfd-dev] [EXT] Failure to synchronize using PSync full-sync on Raspberry pi zero.

[Saurab Dulal (sdulal)]

Hello Junxiao,

I compiled (took me about 10hrs) ndn-cxx, nfd, PSync manually on Pi Zero using clang++-3.9, and now everything is working fine. This indeed suggests the packaging might have produced invalid binaries.

Regards,
Saurab Dulal
________________________________
From: Junxiao Shi <shijunxiao at email.arizona.edu>
Sent: Saturday, March 14, 2020 7:48 AM
To: Saurab Dulal (sdulal) <sdulal at memphis.edu>
Cc: <nfd-dev at lists.cs.ucla.edu> <nfd-dev at lists.cs.ucla.edu>
Subject: Re: [EXT][Nfd-dev] Failure to synchronize using PSync full-sync on Raspberry pi zero.

Hi Saurab

While I offered these Raspberry Pi Zero packages, they aren't tested.
The OpenCV3 packages were built on RPi Zero hardware, which took several days<https://yoursunny.com/t/2018/build-OpenCV3>, and many people are using them.
The NDN packages were built on a Xeon server in Docker balenalib/raspberry-pi<https://hub.docker.com/r/balenalib/raspberry-pi> image using this script<https://github.com/yoursunny/docker-nfd/blob/886b036c84ef407fe822ffd29f7e4f11e84a239c/debuild.sh>. If you can confirm the produced invalid binaries, I suggest seeking support from balena<https://www.balena.io/support>.

For RPi Zero hardware, my current recommendation for running C++ software is to use OpenWrt, not Raspbian.
My package feed includes ndn-cxx and NFD: https://github.com/yoursunny/OpenWrt-packages<https://github.com/yoursunny/OpenWrt-packages>
If you need PSync and other software, look at this guide: https://openwrt.org/docs/guide-developer/helloworld/start<https://openwrt.org/docs/guide-developer/helloworld/start>

Yours, Junxiao

On Fri, Mar 13, 2020 at 2:33 PM Saurab Dulal (sdulal) <sdulal at memphis.edu<mailto:sdulal at memphis.edu>> wrote:

External Email

Hello Everyone,

I am trying to synchronize two Raspberry pi zero (ARMv6, Raspbian GNU/Linux 9 (stretch), Linux 4.14.98+) using PSync full-sync.

ndn (0.7.0) is installed using the ndn Debian packages [http://dl.bintray.com/yoursunny/PiZero<http://dl.bintray.com/yoursunny/PiZero>], and the PSync, as well as full-sync application, are compiled using gcc (version 6.3.0 20170516) - have also tried clang. The configuration and setting are same on both the pi's. /sync is used as sync prefix, and the application prefixes are /a and /b respectively for two pi's.

Every time I run the applications on the nodes to synchronize the states, I encounter the following problems.


the full-sync application on one of the node hangs up, and stops publishing, or crash with the following error: (__gnu_cxx::__concurrence_lock_error).


1583881471.476280 DEBUG: [psync.FullProducer] Checking if data will satisfy our own pending interest

1583881471.479401 DEBUG: [psync.FullProducer] Sending Sync Data (stucks here)

1583881471.504475 ERROR: [examples.FullSyncApp] __gnu_cxx::__concurrence_lock_error (or crashes here)

Nothing happens to nfd, it runs perfectly fine even after the app crashes.

(gdb) bt
#0  0xb66b5f10 in __lll_lock_wait (futex=futex at entry=0x56bb4, private=0) at lowlevellock.c:46
#1  0xb66aebd4 in __GI___pthread_mutex_lock (mutex=0x56bb4) at pthread_mutex_lock.c:80
#2  0xb6d097fc in ndn::InMemoryStorageEntry::setData(ndn::Data const&) ()
   from /usr/lib/arm-linux-gnueabihf/libndn-cxx.so.0.7.0
#3  0xb6d126c0 in ndn::InMemoryStorage::insert(ndn::Data const&, boost::chrono::duration<long long, boost::ratio<1ll, 1000ll> > const&) () from /usr/lib/arm-linux-gnueabihf/libndn-cxx.so.0.7.0

#4  0xb66917fc in psync::SegmentPublisher::publish (this=<optimized out>, interestName=..., dataName=...,
    buffer=..., freshness=..., signingInfo=...) at ../PSync/segment-publisher.cpp:91
#5  0xb66857cc in psync::FullProducer::sendSyncData (this=<optimized out>, name=..., block=...)
    at ../PSync/full-producer.cpp:252
#6  0xb6684c24 in psync::FullProducer::satisfyPendingInterests (this=<optimized out>)
    at ../PSync/full-producer.cpp:341
#7  0xb66843d8 in psync::FullProducer::publishName (this=<optimized out>, prefix=..., seq=...)
    at ../PSync/full-producer.cpp:80
#8  0x00018a14 in Producer::doUpdate(ndn::Name const&) ()
#9  0x00018354 in Producer::Producer(ndn::Name const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)::{lambda()#1}::operator()() const ()
#10 0x0001c060 in std::_Function_handler<void (), Producer::Producer(ndn::Name const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
    ()
---Type <return> to continue, or q <return> to quit---
#11 0xb6f0fea8 in ndn::scheduler::Scheduler::executeEvent(boost::system::error_code const&) ()
   from /usr/lib/arm-linux-gnueabihf/libndn-cxx.so.0.7.0
#12 0xb6f10190 in ?? () from /usr/lib/arm-linux-gnueabihf/libndn-cxx.so.0.7.0
#13 0xb6cfb594 in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()
   from /usr/lib/arm-linux-gnueabihf/libndn-cxx.so.0.7.0
#14 0xb6cf75c8 in ndn::Face::doProcessEvents(boost::chrono::duration<long long, boost::ratio<1ll, 1000ll> >, bool)
    () from /usr/lib/arm-linux-gnueabihf/libndn-cxx.so.0.7.0
#15 0x00017bcc in ndn::Face::processEvents(boost::chrono::duration<long long, boost::ratio<1ll, 1000ll> >, bool) ()
#16 0x00018910 in Producer::run() ()
#17 0x000166a4 in main ()

(gdb) i thr
  Id   Target Id         Frame
* 1    Thread 0xb6ff1000 (LWP 1042) "full-sync" 0xb66b5f10 in __lll_lock_wait (futex=futex at entry=0x56bb4, private=0)
    at lowlevellock.c:46
  2    Thread 0xb455fa70 (LWP 1048) "full-sync" syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37

Memory and CPU utilization are in the normal range.

I can't figure out anything from gdb. Any help will be really appreciated.

By the way, everything works perfectly fine on Pi 3 and 4. I have only observed this on pi zero.

Regards,
Saurab Dulal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/nfd-dev/attachments/20200315/7dc797e8/attachment-0001.html>