[Nfd-dev] NDN-RTC: NFD processing logs

Davide Pesavento davide.pesavento at lip6.fr
Sun Feb 8 11:15:00 PST 2015


Guys,

I guess I should've written a summary of the live debugging session I
had with Peter after the retreat. Sorry about that.

I'll post our findings in the ticket that Jeff just opened.

Thanks,
Davide

On Sun, Feb 8, 2015 at 11:09 AM, Burke, Jeff <jburke at remap.ucla.edu> wrote:
> I have opened Bug #2492 for this.
> http://redmine.named-data.net/issues/2492
>
> J.
>
> From: Jeff Burke <jburke at remap.ucla.edu>
> Date: Sun, 8 Feb 2015 18:53:14 +0000
> To: Junxiao Shi <shijunxiao at email.arizona.edu>
> Cc: "nfd-dev at lists.cs.ucla.edu" <nfd-dev at lists.cs.ucla.edu>
>
> Subject: Re: [Nfd-dev] NDN-RTC: NFD processing logs
>
>
> Hi Junxiao,
>
> I don't agree – unless the NFD team is aware of a change in the codebase
> that would potentially fix this problem, and you can selectively apply only
> those changes as a patch, to see if they fix the problem, simply upgrading
> provides no information on the root cause.  Thus, we may eliminate the
> symptom without understanding where it comes from or knowing when it might
> come back.
>
> Given our goal of having this system work "well enough" to use in real
> meetings, etc. and to do large-scale demos with WUSTL, simply eliminating
> the symptom is insufficient.
>
> Simplest way to find the root cause is to debug in the current, deployed
> system.
>
> A hybrid approach is to substitute an NFD 0.3 forwarder (on a different,
> identical box) as the hub and see if fixes the problem. This might inform
> which commits to apply/roll back in the testing process.   But, somehow, we
> should try methodically to find out why this occurs.
>
> We can provide additional hardware configure per the above, if you'd like.
> But we need the NFD team to engage with debugging the current running code
> on the hub.  This will be more productive and more likely to get to the root
> cause than having us make further changes to the setup.
>
> Thanks,
> Jeff
>
> From: Junxiao Shi <shijunxiao at email.arizona.edu>
> Date: Sun, 8 Feb 2015 11:42:25 -0700
> To: Jeff Burke <jburke at remap.ucla.edu>
> Cc: "Gusev, Peter" <peter at remap.ucla.edu>, "nfd-dev at lists.cs.ucla.edu"
> <nfd-dev at lists.cs.ucla.edu>
> Subject: Re: [Nfd-dev] NDN-RTC: NFD processing logs
>
> Hi Jeff
>
> In any case the delay isn't in forwarding pipelines or strategy according to
> logs.
> @Davide is expert on faces and related libraries, and he agreed to determine
> whether the problem is in UdpFace or a lower layer.
>
> Debugging this problem doesn't depend on new Ubuntu port.
> However, upgrading NFD to latest git-HEAD version is necessary for bug
> reports, because this problem may have been fixed already and it's wasting
> time trying to debug an older version.
> After upgrading NFD and confirming it still exists, please officially file
> the bug on NFD Redmine site.
>
> Yours, Junxiao
>
> On Sun, Feb 8, 2015 at 11:33 AM, Burke, Jeff <jburke at remap.ucla.edu> wrote:
>>
>> Junxiao & all,
>>
>> I want to strongly advocate for debugging the actual, running system that
>> we have now, rather than waiting for new code and using  another setup that
>> may or may not exhibit the same problem.
>>
>> Would it be possible to work with the system in situ with remote access
>> rather than waiting on the ubuntu headless build? (Alternatively, I would
>> rather host you at UCLA for a week, or fedex you our Mac test machines, than
>> write new application that just attempts to replicate the problem.
>>
>> What Peter was referring to is an Ubuntu port of ndnrtc that will generate
>> the same type of traffic, not a simulator. We need this for testbed
>> deployment.  However, this work is not done yet, and we are trying to move
>> quickly to real world use, and I really don't think we should make this
>> dependent on further app development work.
>>
>> Thank you,
>> Jeff
>>
>> Hi Peter
>>
>> On Feb 05 we identified a few possible places that can cause delay:
>>
>> network
>> kernel
>> C++ memory allocator
>> Boost.Asio
>> UdpFace or its base class DatagramFace
>>
>> The logs you gathered appear to be complete according to what we
>> requested.
>> @Davide, can you gain some insights from the logs?
>>
>>
>> Regarding the 30-second interval pattern discovered by John: the only
>> 30-second interval I know of in NFD is UDP face idle_timeout, when this
>> option is unset in nfd.conf.
>> Every 30 seconds, UdpChannel will enumerate all its faces, and close all
>> faces that haven't received anything. There's no evidence that any face is
>> created, because FaceIds are consistent throughout the logs. However this
>> scheduled event will take some CPU time, but I don't believe it can cause a
>> delay of more than 5ms.
>> To rule out this possibility, adjust face_system.udp.idle_timeout option
>> in nfd.conf (to "50"), and test again.
>>
>>
>> From an earlier log snippet, I notice that the NFD version you are running
>> is older than September 2014.
>> Please use git-HEAD version for any bug reports.
>>
>>
>> As you told me on Feb 05, you will create a traffic simulator that can
>> have same traffic pattern as NdnCon that can run on headless Ubuntu servers.
>> I'll be able to test on controlled environment when that tool becomes
>> available.
>>
>> Yours, Junxiao
>>
>> On Fri, Feb 6, 2015 at 4:14 PM, Gusev, Peter <peter at remap.ucla.edu> wrote:
>>>
>>> Hi guys,
>>>
>>> So I’ve run setup today and gathered logs.
>>> I’ve setup NFDs with the following settings
>>> ...
>>>  default_level NONE
>>>  Forwarder DEBUG
>>>  UdpFace TRACE
>>>>>>
>>> And then I run tcpdump on each machine like this:
>>> $ sudo tcpdump -w ~/dump.pcap -i en4 udp port 6363
>>>
>>> Here is the summary on “problematic” packets:
>>>
>>> https://docs.google.com/document/d/11JAPHI6J4jD6mibMRuWd1EvzP-NY_F2osM2vDF2VaGM/edit?usp=sharing
>>>
>>> All the log files and tcpdumps you can find here:
>>> https://www.dropbox.com/s/1d277xbhqs97o66/logs.zip?dl=0
>>>
>>> Thanks for all the help!
>>>
>>> Thanks,
>>>
>>> --
>>> Peter Gusev
>>> Programmer/Analyst @ REMAP, UCLA
>>>
>>> peter at remap.ucla.edu
>>> +1 213 5872748
>>> peetonn_ (skype)
>>
>>
>> _______________________________________________ Nfd-dev mailing list
>> Nfd-dev at lists.cs.ucla.eduhttp://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev
>
>
> _______________________________________________ Nfd-dev mailing list
> Nfd-dev at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev
>
>
> _______________________________________________
> Nfd-dev mailing list
> Nfd-dev at lists.cs.ucla.edu
> http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev
>




More information about the Nfd-dev mailing list