<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Agree with Jeff. It'll be much easier for Junxiao to figure out what's the problem, since Peter already spent a lot of time debugging this and he's not familiar with the internal working of nfd. <br>
<div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<br class="Apple-interchange-newline">
Lan</div>
</span></div>
</div>
</div>
<br>
<div>
<div>On Feb 8, 2015, at 1:26 PM, "Burke, Jeff" <<a href="mailto:jburke@remap.UCLA.EDU">jburke@remap.UCLA.EDU</a>></div>
<div> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; ">
<div><br>
</div>
<div>Hi John,</div>
<div><br>
</div>
<div>Good point. Our hub is not on the same subnet, unfortunately, but we can spin up a 0.3 box if needed. </div>
<div><br>
</div>
<div>But I'd really like to invite the NFD folks to directly debug the installation on the problematic hub.</div>
<div><br>
</div>
<div>Jeff</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223); ">
<span style="font-weight:bold">From: </span>"Dehart, John" <<a href="mailto:jdd@wustl.edu">jdd@wustl.edu</a>><br>
<span style="font-weight:bold">Date: </span>Sun, 8 Feb 2015 19:22:52 +0000<br>
<span style="font-weight:bold">To: </span>Jeff Burke <<a href="mailto:jburke@remap.ucla.edu">jburke@remap.ucla.edu</a>>, "<a href="mailto:peter@remap.ucla.edu">peter@remap.ucla.edu</a>" <<a href="mailto:peter@remap.ucla.edu">peter@remap.ucla.edu</a>><br>
<span style="font-weight:bold">Cc: </span>Junxiao Shi <<a href="mailto:shijunxiao@email.arizona.edu">shijunxiao@email.arizona.edu</a>>, "<a href="mailto:nfd-dev@lists.cs.ucla.edu">nfd-dev@lists.cs.ucla.edu</a>" <<a href="mailto:nfd-dev@lists.cs.ucla.edu">nfd-dev@lists.cs.ucla.edu</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Nfd-dev] NDN-RTC: NFD processing logs<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
Jeff and Peter,
<div><br>
</div>
<div>Another thing you might try is to have your consumer and producer remain on their</div>
<div>current devices with their current NFD installation but have them use REMAP as their hub.</div>
<div>REMAP is still currently running an NFD version of 0.2.0 but will soon be upgraded to </div>
<div>the 0.3.0 release. </div>
<div><br>
</div>
<div>Is it possible to have the consumer and producer laptops on the same subnet as REMAP</div>
<div>to try to come as close as possible to your current configuration?</div>
<div><br>
</div>
<div>If we can repeat the problem with installed Testbed nodes it might make it easier</div>
<div>for others to replicate the problem at their sites.</div>
<div><br>
</div>
<div>I’ll hold off on upgrades of Testbed nodes while we discuss our options.</div>
<div><br>
</div>
<div>John</div>
<div><br>
</div>
<div>
<div>
<div>On Feb 8, 2015, at 12:53 PM, Burke, Jeff <<a href="mailto:jburke@remap.ucla.edu">jburke@remap.ucla.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;">
<div><br>
</div>
<div>Hi Junxiao,</div>
<div><br>
</div>
<div>I don't agree – unless the NFD team is aware of a change in the codebase that would potentially fix this problem, and you can selectively apply
<i>only those changes</i> as a patch<i>, </i>to see if they fix the problem, simply upgrading provides no information on the root cause. Thus, we may eliminate the symptom without understanding where it comes from or knowing when it might come back. </div>
<div><br>
</div>
<div>Given our goal of having this system work "well enough" to use in real meetings, etc. and to do large-scale demos with WUSTL, simply eliminating the symptom is insufficient. </div>
<div><br>
</div>
<div>Simplest way to find the root cause is to debug in the current, deployed system. </div>
<div><br>
</div>
<div>A hybrid approach is to substitute an NFD 0.3 forwarder (on a different, identical box) as the hub and see if fixes the problem. This might inform which commits to apply/roll back in the testing process. But, somehow, we should try methodically to find
out <i>why </i>this occurs. </div>
<div><br>
</div>
<div>We can provide additional hardware configure per the above, if you'd like. But we need the NFD team to engage with debugging the current running code on the hub. This will be more productive and more likely to get to the root cause than having us make
further changes to the setup. </div>
<div><br>
</div>
<div>Thanks,</div>
<div>Jeff</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);">
<span style="font-weight:bold">From: </span>Junxiao Shi <<a href="mailto:shijunxiao@email.arizona.edu">shijunxiao@email.arizona.edu</a>><br>
<span style="font-weight:bold">Date: </span>Sun, 8 Feb 2015 11:42:25 -0700<br>
<span style="font-weight:bold">To: </span>Jeff Burke <<a href="mailto:jburke@remap.ucla.edu">jburke@remap.ucla.edu</a>><br>
<span style="font-weight:bold">Cc: </span>"Gusev, Peter" <<a href="mailto:peter@remap.ucla.edu">peter@remap.ucla.edu</a>>, "<a href="mailto:nfd-dev@lists.cs.ucla.edu">nfd-dev@lists.cs.ucla.edu</a>" <<a href="mailto:nfd-dev@lists.cs.ucla.edu">nfd-dev@lists.cs.ucla.edu</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Nfd-dev] NDN-RTC: NFD processing logs<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div dir="ltr">Hi Jeff
<div><br>
</div>
<div>In any case the delay isn't in forwarding pipelines or strategy according to logs.</div>
<div>@Davide is expert on faces and related libraries, and he agreed to determine whether the problem is in UdpFace or a lower layer.</div>
<div><br>
</div>
<div>Debugging this problem doesn't depend on new Ubuntu port.</div>
<div>However, upgrading NFD to latest git-HEAD version is necessary for bug reports, because this problem may have been fixed already and it's wasting time trying to debug an older version.</div>
<div>After upgrading NFD and confirming it still exists, please officially file the bug on NFD Redmine site.</div>
<div><br>
</div>
<div>Yours, Junxiao<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Feb 8, 2015 at 11:33 AM, Burke, Jeff <span dir="ltr">
<<a href="mailto:jburke@remap.ucla.edu" target="_blank">jburke@remap.ucla.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" type="cite">
<div style="word-wrap: break-word; font-size: 14px; font-family: Calibri, sans-serif;">
<div>Junxiao & all,</div>
<div><br>
</div>
<div>I want to strongly advocate for debugging the actual, running system that we have now, rather than waiting for new code and using another setup that may or may not exhibit the same problem. </div>
<div><br>
</div>
<div>Would it be possible to work with the system <i>in situ </i>with remote access<i> </i>rather than waiting on the ubuntu headless build? (Alternatively, I would rather host you at UCLA for a week, or fedex you our Mac test machines, than write new application
that just attempts to replicate the problem.</div>
<div><br>
</div>
<div>What Peter was referring to is an Ubuntu port of ndnrtc that will generate the same type of traffic, not a simulator. We need this for testbed deployment. However, this work is not done yet, and we are trying to move quickly to real world use, and I really
don't think we should make this dependent on further app development work. </div>
<div><br>
</div>
<div>Thank you,</div>
<div>Jeff</div>
<div><br>
</div>
<span>
<blockquote style="BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5" type="cite">
<div>
<div class="h5">
<div dir="ltr">Hi Peter
<div><br>
</div>
<div>On Feb 05 we identified a few possible places that can cause delay:</div>
<div>
<ul>
<li>network</li><li>kernel</li><li>C++ memory allocator</li><li>Boost.Asio</li><li>UdpFace or its base class DatagramFace</li></ul>
</div>
<div>The logs you gathered appear to be complete according to what we requested.</div>
<div>
<div>@Davide, can you gain some insights from the logs?</div>
<div><br>
</div>
<div><br>
</div>
<div>Regarding the 30-second interval pattern discovered by John: the only 30-second interval I know of in NFD is UDP face idle_timeout, when this option is unset in nfd.conf.<br>
</div>
<div>Every 30 seconds, UdpChannel will enumerate all its faces, and close all faces that haven't received anything. There's no evidence that any face is created, because FaceIds are consistent throughout the logs. However this scheduled event will take some
CPU time, but I don't believe it can cause a delay of more than 5ms.</div>
<div>To rule out this possibility, adjust face_system.udp.idle_timeout option in nfd.conf (to "50"), and test again.</div>
<div><br>
</div>
<div><br>
</div>
<div>From an earlier log snippet, I notice that the NFD version you are running is older than September 2014.</div>
<div>Please use git-HEAD version for any bug reports.</div>
<div><br>
</div>
<div><br>
</div>
<div>As you told me on Feb 05, you will create a traffic simulator that can have same traffic pattern as NdnCon that can run on headless Ubuntu servers.</div>
<div>I'll be able to test on controlled environment when that tool becomes available.</div>
<div><br>
</div>
<div>Yours, Junxiao<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Feb 6, 2015 at 4:14 PM, Gusev, Peter <span dir="ltr">
<<a href="mailto:peter@remap.ucla.edu" target="_blank">peter@remap.ucla.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" type="cite">
<div style="word-wrap:break-word">Hi guys,
<div><br>
</div>
<div>So I’ve run setup today and gathered logs.</div>
<div>I’ve setup NFDs with the following settings</div>
<div><i><span style="white-space:pre-wrap"></span>...</i></div>
<div><i><span style="white-space:pre-wrap"></span> default_level NONE<br>
<span style="white-space:pre-wrap"></span> Forwarder DEBUG<br>
<span style="white-space:pre-wrap"></span> UdpFace TRACE<br>
<span style="white-space:pre-wrap"></span>…</i></div>
<div><br>
</div>
<div>And then I run tcpdump on each machine like this:</div>
<div><i><span style="white-space:pre-wrap"></span>$ sudo tcpdump -w ~/dump.pcap -i en4 udp port 6363</i></div>
<div><br>
</div>
<div>Here is the summary on “problematic” packets:</div>
<div>
<div style="margin:0px"><font face="Menlo"><span style="font-size:11px"><a href="https://docs.google.com/document/d/11JAPHI6J4jD6mibMRuWd1EvzP-NY_F2osM2vDF2VaGM/edit?usp=sharing" target="_blank">https://docs.google.com/document/d/11JAPHI6J4jD6mibMRuWd1EvzP-NY_F2osM2vDF2VaGM/edit?usp=sharing</a></span></font></div>
</div>
<div style="margin:0px"><font face="Menlo"><span style="font-size:11px"><br>
</span></font></div>
<div>All the log files and tcpdumps you can find here:</div>
<div><a href="https://www.dropbox.com/s/1d277xbhqs97o66/logs.zip?dl=0" target="_blank">https://www.dropbox.com/s/1d277xbhqs97o66/logs.zip?dl=0</a></div>
<div><br>
</div>
<div>Thanks for all the help!</div>
<div><span><br>
<div>Thanks, <br>
<br>
-- <br>
Peter Gusev<br>
Programmer/Analyst @ REMAP, UCLA<br>
<br>
<a href="mailto:peter@remap.ucla.edu" target="_blank">peter@remap.ucla.edu</a><br>
<a href="tel:%2B1%20213%205872748" value="+12135872748" target="_blank">+1 213 5872748</a><br>
peetonn_ (skype)<br>
</div>
</span></div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
<span class="">_______________________________________________ Nfd-dev mailing list
<a href="mailto:Nfd-dev@lists.cs.ucla.edu" target="_blank">Nfd-dev@lists.cs.ucla.edu</a><a href="http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev" target="_blank">http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev</a></span></blockquote>
</span></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</span></div>
_______________________________________________<br>
Nfd-dev mailing list<br>
<a href="mailto:Nfd-dev@lists.cs.ucla.edu">Nfd-dev@lists.cs.ucla.edu</a><br>
<a href="http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev">http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</span></div>
_______________________________________________<br>
Nfd-dev mailing list<br>
<a href="mailto:Nfd-dev@lists.cs.ucla.edu">Nfd-dev@lists.cs.ucla.edu</a><br>
http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev<br>
</blockquote>
</div>
<br>
</body>
</html>