[Nfd-dev] NDN-RTC: NFD processing logs

Thu Jan 29 18:56:43 PST 2015

Hi all,

I’m doing tests and explorations on the reasons, why does NdnCon experience rebufferings (i.e. interruptions in video stream which lead to algorithm restart).
I would appreciate any help/suggestions/advices in debugging/analyzing NFD logs and NFD itself.

More specifically, I’d like to hear your comments on the following case.
With the same configuration as before ([Consumer] <—> [NFD1] <==Ethernet_link==> [NFD2] <==Ethernet_link==> [NFD3] <—> [Producer]) consumer experienced interruption after getting frame 640. So I tracked down frame’s 641 interests and data in NFD logs:

NFD1 log: ($ cat consumer/nfd.log | grep "/641/data/%00%00”)

1422581555.423600 DEBUG: [Forwarder] onIncomingInterest face=265 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581555.423943 DEBUG: [Forwarder] onOutgoingInterest face=259 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581555.424205 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=1282205730 from=265 newPitEntry-to=259
^ 294ms (retransmission, confirmed by consumer log)
1422581555.718095 DEBUG: [Forwarder] onIncomingInterest face=265 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581555.718589 DEBUG: [Forwarder] onOutgoingInterest face=259 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581555.718929 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=589980969 from=265 retransmit-retry-to=259
^ 232ms
1422581555.950466 DEBUG: [Forwarder] onIncomingData face=259 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0
1422581555.950763 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581555.950998 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00 inFace=259 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0
1422581555.951438 DEBUG: [Forwarder] onOutgoingData face=265 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0
^ 5ms
1422581555.956230 DEBUG: [Forwarder] onIncomingData face=259 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0
1422581555.956496 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581555.956724 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00 inFace=259 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0

NFD2 log: ($ cat hub/nfd.log | grep "/641/data/%00%00”)

1422581521.773468 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581521.773947 DEBUG: [Forwarder] onOutgoingInterest face=264 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581521.774352 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=1282205730 from=266 newPitEntry-to=264
^ 522ms (??? where is the retransmission interest?)
1422581522.296967 DEBUG: [Forwarder] onIncomingData face=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0
1422581522.297347 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581522.297614 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00 inFace=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0
1422581522.298123 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0
^ 5ms (ah, here it is… how?)
1422581522.303090 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581522.303430 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0

NFD3 log: ($ cat producer/nfd.log | grep "/641/data/%00%00”)

1422581555.255647 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581555.256007 DEBUG: [Forwarder] onOutgoingInterest face=269 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581555.256334 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=1282205730 from=266 newPitEntry-to=269
^ 47ms (!!! interest arrival was not late, data generation was not delayed)
1422581555.303275 DEBUG: [Forwarder] onIncomingData face=269 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0
1422581555.303640 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00
1422581555.303928 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00 inFace=269 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0
1422581555.304458 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0

The questions here are:
1. How did the retransmission interest arrived on NFD2 ~220ms later than was expected and actually after the first interest was already answered?
2. If the data was generated and answered by NFD3 in timely manner, why did data arrived on NFD2 after 522ms but not after ~50ms?

Some peculiar place of NFD2’s log:
$ sed -n 38830,38840p hub/nfd.log

1422581521.916149 DEBUG: [Forwarder] onIncomingData face=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/638/data/%00%0A/14/661/22/0
1422581521.916515 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/638/data/%00%0A
1422581521.916799 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/638/data/%00%0A inFace=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/638/data/%00%0A/14/661/22/0
1422581521.917301 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/638/data/%00%0A/14/661/22/0
^ 360ms
1422581522.277620 DEBUG: [Forwarder] onInterestUnsatisfied interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/499/data/%00%06
1422581522.278080 DEBUG: [Strategy] beforeExpirePendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/499/data/%00%06
1422581522.278524 DEBUG: [Forwarder] onInterestUnsatisfied interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/500/data/%00%06
1422581522.278901 DEBUG: [Strategy] beforeExpirePendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/500/data/%00%06
1422581522.279352 DEBUG: [Forwarder] onInterestUnsatisfied interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/501/data/%00%05
1422581522.279742 DEBUG: [Strategy] beforeExpirePendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/501/data/%00%05
1422581522.280119 DEBUG: [Forwarder] onInterestUnsatisfied interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/501/data/%00%06

I would appreciate any help.
Looking forward for the insights of NFD developers,

Thanks,

PS. full logs are attached

--
Peter Gusev
Programmer/Analyst @ REMAP, UCLA

peter at remap.ucla.edu<mailto:peter at remap.ucla.edu>
+1 213 5872748
peetonn_ (skype)

On Jan 21, 2015, at 12:32 PM, Gusev, Peter <peter at remap.ucla.edu<mailto:peter at remap.ucla.edu>> wrote:

Hi all,

I’ve been testing <https://github.com/remap/ndnrtc/tree/issues/2283-retransmissions> N<https://github.com/remap/ndnrtc/tree/issues/2283-retransmissions>DN-RTC with new consumer algorithm and exploring the problem of rebufferings on consumer side (a state, when consumer is not getting data for some amount of time, flushes everything and starts over again). Consumer rebufferings are perceived as audio/video interruptions for 1-1.5sec by the user and decrease overall user experience.

I'll appreciate your insights on the analysis of the logs I retrieved from NFD.

Setup:

[Consumer] <—> [NFD1] <==WiFi_link==> [NFD2] <==Ethernet_link==> [NFD3] <—> [Producer]

Case:
According to my logs, consumer starved for data after 2656 segment has arrived, so I tracked interests and data for 2657 segment. According to my logs, consumer eventually received this data but too late and rebuffering was already inevitable.

NFD1 log:
initial interest: 1421288806.271931 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.272215 DEBUG: [Forwarder] onOutgoingInterest face=262 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.272469 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=108889115 from=266 newPitEntry-to=262
^ 270ms
retransmission: 1421288806.542289 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.542697 DEBUG: [Forwarder] onOutgoingInterest face=262 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.543032 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=562292066 from=266 retransmit-retry-to=262
^ 286ms
data arrives: 1421288806.829462 DEBUG: [Forwarder] onIncomingData face=262 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0
1421288806.829818 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.830042 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00 inFace=262 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0
1421288806.830449 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0

NFD2 log:
initial interest: 1421288806.015075 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.015845 DEBUG: [Forwarder] onOutgoingInterest face=264 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.016255 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=108889115 from=266 newPitEntry-to=264
^ 545ms
retransmission: 1421288806.561516 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.562233 DEBUG: [Forwarder] onOutgoingInterest face=264 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.562626 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=562292066 from=266 retransmit-retry-to=264
^ 2ms
data arrives: 1421288806.564828 DEBUG: [Forwarder] onIncomingData face=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0
1421288806.565282 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.565594 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00 inFace=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0
1421288806.566213 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0
^ 13ms
data arrives 2:* 1421288806.579547 DEBUG: [Forwarder] onIncomingData face=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0
1421288806.579915 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.580162 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00 inFace=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0

*NOTE: no data was forwarded

NFD3 log:
initial interest: 1421288806.249712 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.250177 DEBUG: [Forwarder] onOutgoingInterest face=269 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.250573 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=108889115 from=266 newPitEntry-to=269
^271ms
data generated: 1421288806.521982 DEBUG: [Forwarder] onIncomingData face=269 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0
1421288806.522531 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.522844 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00 inFace=269 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0
1421288806.523363 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0
^273ms
retransmission: 1421288806.796022 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00
1421288806.796418 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/camera/low/delta/2657/data/%00%00/1/2749/91/0

There are several questions arise:
1. Why do you think the delay b/w initial interest and retransmission for the NFD2 became 545ms (compared to 270ms with NFD1)? Could it be WiFi fault?
2. How come the delay b/w initial interest and data in NFD3 is 271ms and NFD3 forwards this data to NFD2, but NFD2 didn’t get this data and instead got retransmission interest after another 545-271=274ms and later got the same data with 13ms difference? Could it be unreliable logging in NFD?

I attached full NFD logs to this message.

Thank you in advance.
All ideas and thoughts are welcome.

Thanks,

PS. Basically, such things block deployment of NDN-RTC (NdnCon) for the community as it delivers low-quality user experience (i.e. videoconference with interruptions =( ), so I really hope for the help and feedback.

--
Peter Gusev
peter at remap.ucla.edu<mailto:peter at remap.ucla.edu>
+1 213 5872748 (USA)
+7 916 4434826 (Russia)
+37 259 226448 (in case any other number is unavailable)
peetonn_ (skype)

<Archive.zip>_______________________________________________
Nfd-dev mailing list
Nfd-dev at lists.cs.ucla.edu<mailto:Nfd-dev at lists.cs.ucla.edu>
http://www.lists.cs.ucla.edu/mailman/listinfo/nfd-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.cs.ucla.edu/pipermail/nfd-dev/attachments/20150130/d41c392f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Archive.zip
Type: application/zip
Size: 2409896 bytes
Desc: Archive.zip
URL: <http://www.lists.cs.ucla.edu/pipermail/nfd-dev/attachments/20150130/d41c392f/attachment.zip>