[Nfd-dev] NDN-RTC: NFD processing logs
shijunxiao at email.arizona.edu
Sun Feb 8 11:16:42 PST 2015
Upgrading to git-HEAD does not prevent you from finding out the root cause
Please use the following procedure to find out the root cause:
1. take a note of commit hash of the current NFD
2. upgrade to git-HEAD
3. test whether the problem exists; if yes, abort these steps, and we
should use git-HEAD for further debugging
4. downgrade one commit
5. test whether the problem re-appear; if yes, abort these steps, and we
have concluded the next commit contains the fix for the problem
6. if current commit is after the commit in note-1, go to step 4
On Sun, Feb 8, 2015 at 11:53 AM, Burke, Jeff <jburke at remap.ucla.edu> wrote:
> Hi Junxiao,
> I don't agree – unless the NFD team is aware of a change in the codebase
> that would potentially fix this problem, and you can selectively apply *only
> those changes* as a patch*, *to see if they fix the problem, simply
> upgrading provides no information on the root cause. Thus, we may
> eliminate the symptom without understanding where it comes from or knowing
> when it might come back.
> Given our goal of having this system work "well enough" to use in real
> meetings, etc. and to do large-scale demos with WUSTL, simply eliminating
> the symptom is insufficient.
> Simplest way to find the root cause is to debug in the current, deployed
> A hybrid approach is to substitute an NFD 0.3 forwarder (on a different,
> identical box) as the hub and see if fixes the problem. This might inform
> which commits to apply/roll back in the testing process. But, somehow, we
> should try methodically to find out *why *this occurs.
> We can provide additional hardware configure per the above, if you'd
> like. But we need the NFD team to engage with debugging the current
> running code on the hub. This will be more productive and more likely to
> get to the root cause than having us make further changes to the setup.
> From: Junxiao Shi <shijunxiao at email.arizona.edu>
> Date: Sun, 8 Feb 2015 11:42:25 -0700
> To: Jeff Burke <jburke at remap.ucla.edu>
> Cc: "Gusev, Peter" <peter at remap.ucla.edu>, "nfd-dev at lists.cs.ucla.edu" <
> nfd-dev at lists.cs.ucla.edu>
> Subject: Re: [Nfd-dev] NDN-RTC: NFD processing logs
> Hi Jeff
> In any case the delay isn't in forwarding pipelines or strategy
> according to logs.
> @Davide is expert on faces and related libraries, and he agreed to
> determine whether the problem is in UdpFace or a lower layer.
> Debugging this problem doesn't depend on new Ubuntu port.
> However, upgrading NFD to latest git-HEAD version is necessary for bug
> reports, because this problem may have been fixed already and it's wasting
> time trying to debug an older version.
> After upgrading NFD and confirming it still exists, please officially file
> the bug on NFD Redmine site.
> Yours, Junxiao
> On Sun, Feb 8, 2015 at 11:33 AM, Burke, Jeff <jburke at remap.ucla.edu>
>> Junxiao & all,
>> I want to strongly advocate for debugging the actual, running system
>> that we have now, rather than waiting for new code and using another setup
>> that may or may not exhibit the same problem.
>> Would it be possible to work with the system *in situ *with remote
>> access rather than waiting on the ubuntu headless build? (Alternatively,
>> I would rather host you at UCLA for a week, or fedex you our Mac test
>> machines, than write new application that just attempts to replicate the
>> What Peter was referring to is an Ubuntu port of ndnrtc that will
>> generate the same type of traffic, not a simulator. We need this for
>> testbed deployment. However, this work is not done yet, and we are trying
>> to move quickly to real world use, and I really don't think we should make
>> this dependent on further app development work.
>> Thank you,
>> Hi Peter
>> On Feb 05 we identified a few possible places that can cause delay:
>> - network
>> - kernel
>> - C++ memory allocator
>> - Boost.Asio
>> - UdpFace or its base class DatagramFace
>> The logs you gathered appear to be complete according to what we
>> @Davide, can you gain some insights from the logs?
>> Regarding the 30-second interval pattern discovered by John: the only
>> 30-second interval I know of in NFD is UDP face idle_timeout, when this
>> option is unset in nfd.conf.
>> Every 30 seconds, UdpChannel will enumerate all its faces, and close
>> all faces that haven't received anything. There's no evidence that any face
>> is created, because FaceIds are consistent throughout the logs. However
>> this scheduled event will take some CPU time, but I don't believe it can
>> cause a delay of more than 5ms.
>> To rule out this possibility, adjust face_system.udp.idle_timeout option
>> in nfd.conf (to "50"), and test again.
>> From an earlier log snippet, I notice that the NFD version you are
>> running is older than September 2014.
>> Please use git-HEAD version for any bug reports.
>> As you told me on Feb 05, you will create a traffic simulator that can
>> have same traffic pattern as NdnCon that can run on headless Ubuntu servers.
>> I'll be able to test on controlled environment when that tool becomes
>> Yours, Junxiao
>> On Fri, Feb 6, 2015 at 4:14 PM, Gusev, Peter <peter at remap.ucla.edu>
>>> Hi guys,
>>> So I’ve run setup today and gathered logs.
>>> I’ve setup NFDs with the following settings
>>> * default_level NONE Forwarder DEBUG UdpFace TRACE …*
>>> And then I run tcpdump on each machine like this:
>>> *$ sudo tcpdump -w ~/dump.pcap -i en4 udp port 6363*
>>> Here is the summary on “problematic” packets:
>>> All the log files and tcpdumps you can find here:
>>> Thanks for all the help!
>>> Peter Gusev
>>> Programmer/Analyst @ REMAP, UCLA
>>> peter at remap.ucla.edu
>>> +1 213 5872748
>>> peetonn_ (skype)
>> _______________________________________________ Nfd-dev mailing list
>> Nfd-dev at lists.cs.ucla.edu
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Nfd-dev