[Nfd-dev] NDN-RTC: NFD processing logs

Junxiao Shi shijunxiao at email.arizona.edu
Sun Feb 8 11:16:42 PST 2015


Hi Jeff

Upgrading to git-HEAD does not prevent you from finding out the root cause
the problem.

Please use the following procedure to find out the root cause:

   1. take a note of commit hash of the current NFD
   2. upgrade to git-HEAD
   3. test whether the problem exists; if yes, abort these steps, and we
   should use git-HEAD for further debugging
   4. downgrade one commit
   5. test whether the problem re-appear; if yes, abort these steps, and we
   have concluded the next commit contains the fix for the problem
   6. if current commit is after the commit in note-1, go to step 4


Yours, Junxiao

On Sun, Feb 8, 2015 at 11:53 AM, Burke, Jeff <jburke at remap.ucla.edu> wrote:

>
>  Hi Junxiao,
>
>  I don't agree – unless the NFD team is aware of a change in the codebase
> that would potentially fix this problem, and you can selectively apply *only
> those changes* as a patch*, *to see if they fix the problem, simply
> upgrading provides no information on the root cause.  Thus, we may
> eliminate the symptom without understanding where it comes from or knowing
> when it might come back.
>
>  Given our goal of having this system work "well enough" to use in real
> meetings, etc. and to do large-scale demos with WUSTL, simply eliminating
> the symptom is insufficient.
>
>  Simplest way to find the root cause is to debug in the current, deployed
> system.
>
>  A hybrid approach is to substitute an NFD 0.3 forwarder (on a different,
> identical box) as the hub and see if fixes the problem. This might inform
> which commits to apply/roll back in the testing process.   But, somehow, we
> should try methodically to find out *why *this occurs.
>
>  We can provide additional hardware configure per the above, if you'd
> like.  But we need the NFD team to engage with debugging the current
> running code on the hub.  This will be more productive and more likely to
> get to the root cause than having us make further changes to the setup.
>
>  Thanks,
> Jeff
>
>   From: Junxiao Shi <shijunxiao at email.arizona.edu>
> Date: Sun, 8 Feb 2015 11:42:25 -0700
> To: Jeff Burke <jburke at remap.ucla.edu>
> Cc: "Gusev, Peter" <peter at remap.ucla.edu>, "nfd-dev at lists.cs.ucla.edu" <
> nfd-dev at lists.cs.ucla.edu>
> Subject: Re: [Nfd-dev] NDN-RTC: NFD processing logs
>
>  Hi Jeff
>
>  In any case the delay isn't in forwarding pipelines or strategy
> according to logs.
> @Davide is expert on faces and related libraries, and he agreed to
> determine whether the problem is in UdpFace or a lower layer.
>
>  Debugging this problem doesn't depend on new Ubuntu port.
> However, upgrading NFD to latest git-HEAD version is necessary for bug
> reports, because this problem may have been fixed already and it's wasting
> time trying to debug an older version.
> After upgrading NFD and confirming it still exists, please officially file
> the bug on NFD Redmine site.
>
>  Yours, Junxiao
>
> On Sun, Feb 8, 2015 at 11:33 AM, Burke, Jeff <jburke at remap.ucla.edu>
> wrote:
>
>>  Junxiao & all,
>>
>>  I want to strongly advocate for debugging the actual, running system
>> that we have now, rather than waiting for new code and using  another setup
>> that may or may not exhibit the same problem.
>>
>>  Would it be possible to work with the system *in situ *with remote
>> access rather than waiting on the ubuntu headless build? (Alternatively,
>> I would rather host you at UCLA for a week, or fedex you our Mac test
>> machines, than write new application that just attempts to replicate the
>> problem.
>>
>>  What Peter was referring to is an Ubuntu port of ndnrtc that will
>> generate the same type of traffic, not a simulator. We need this for
>> testbed deployment.  However, this work is not done yet, and we are trying
>> to move quickly to real world use, and I really don't think we should make
>> this dependent on further app development work.
>>
>>  Thank you,
>> Jeff
>>
>>    Hi Peter
>>
>>  On Feb 05 we identified a few possible places that can cause delay:
>>
>>    - network
>>    - kernel
>>    - C++ memory allocator
>>    - Boost.Asio
>>    - UdpFace or its base class DatagramFace
>>
>>  The logs you gathered appear to be complete according to what we
>> requested.
>>  @Davide, can you gain some insights from the logs?
>>
>>
>>  Regarding the 30-second interval pattern discovered by John: the only
>> 30-second interval I know of in NFD is UDP face idle_timeout, when this
>> option is unset in nfd.conf.
>>  Every 30 seconds, UdpChannel will enumerate all its faces, and close
>> all faces that haven't received anything. There's no evidence that any face
>> is created, because FaceIds are consistent throughout the logs. However
>> this scheduled event will take some CPU time, but I don't believe it can
>> cause a delay of more than 5ms.
>> To rule out this possibility, adjust face_system.udp.idle_timeout option
>> in nfd.conf (to "50"), and test again.
>>
>>
>>  From an earlier log snippet, I notice that the NFD version you are
>> running is older than September 2014.
>> Please use git-HEAD version for any bug reports.
>>
>>
>>  As you told me on Feb 05, you will create a traffic simulator that can
>> have same traffic pattern as NdnCon that can run on headless Ubuntu servers.
>> I'll be able to test on controlled environment when that tool becomes
>> available.
>>
>>  Yours, Junxiao
>>
>> On Fri, Feb 6, 2015 at 4:14 PM, Gusev, Peter <peter at remap.ucla.edu>
>> wrote:
>>
>>> Hi guys,
>>>
>>>  So I’ve run setup today and gathered logs.
>>> I’ve setup NFDs with the following settings
>>> *...*
>>>
>>>
>>>
>>> * default_level NONE  Forwarder DEBUG  UdpFace TRACE …*
>>>
>>>  And then I run tcpdump on each machine like this:
>>> *$ sudo tcpdump -w ~/dump.pcap -i en4 udp port 6363*
>>>
>>>  Here is the summary on “problematic” packets:
>>>
>>> https://docs.google.com/document/d/11JAPHI6J4jD6mibMRuWd1EvzP-NY_F2osM2vDF2VaGM/edit?usp=sharing
>>>
>>>  All the log files and tcpdumps you can find here:
>>> https://www.dropbox.com/s/1d277xbhqs97o66/logs.zip?dl=0
>>>
>>>  Thanks for all the help!
>>>
>>> Thanks,
>>>
>>> --
>>> Peter Gusev
>>> Programmer/Analyst @ REMAP, UCLA
>>>
>>> peter at remap.ucla.edu
>>> +1 213 5872748
>>> peetonn_ (skype)
>>>
>>
>>    _______________________________________________ Nfd-dev mailing list
>> Nfd-dev at lists.cs.ucla.edu
>> http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/nfd-dev/attachments/20150208/5b40bd36/attachment.html>


More information about the Nfd-dev mailing list