<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Junxiao,
<div class=""><br class="">
</div>
<div class="">Thanks for your analysis!</div>
<div class="">You’re right, there’s a clock skew (even though all three machines are NTP-synchronized to the
<a href="http://time1.ucla.edu" class="">time1.ucla.edu</a>). </div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">
<ul class="">
<li class="">HUB log reveals that face 266 (toward CONSUMER) and face 264 (toward PRODUCER) are both UDP and have same local IP address.<br class="">
The delay of Interest 2 from CONSUMER to HUB might be caused by the congestion between HUB and PRODUCER, because part of CONSUMER-HUB logical link shares same IP-layer link.</li></ul>
</div>
</div>
</blockquote>
<div class="">I didn’t quite get this statement - do you mean that consumer and producer run on the same machine? In the scenario I run, there were three machines (consumer 128.97.152.44, hub 131.179.141.51 and producer 131.179.141.33) connected using 1Gbit
switches (LAN). Do you think this could be a congestion problem still?</div>
</div>
<div class="">I’ll run new tests now eliminating one network segment by connecting consumer to the 131.179.141.xx subnet.</div>
<div class=""><br class="">
<div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Thanks, </div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
-- <br class="">
Peter Gusev</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Programmer/Analyst @ REMAP, UCLA</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
<a href="mailto:peter@remap.ucla.edu" class="">peter@remap.ucla.edu</a><br class="">
+1 213 5872748<br class="">
peetonn_ (skype)</div>
</div>
</div>
</div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Feb 1, 2015, at 9:32 PM, Junxiao Shi <<a href="mailto:shijunxiao@email.arizona.edu" class="">shijunxiao@email.arizona.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="">Hi Peter</div>
<div class=""><br class="">
</div>
<div class="">There's a clock skew among three machines.</div>
<div class="">I picked one of the earliest Interests as a sample:</div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px" class="">
<div class=""><font face="monospace, monospace" class="">CONSUMER</font></div>
<div class="">
<div class=""><span style="font-family:monospace,monospace" class="">1422581537.761808 DEBUG: [Forwarder] onOutgoingInterest face=259 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/key/5/data/%00%00</span><br class="">
</div>
</div>
<div class=""><font face="monospace, monospace" class=""><br class="">
</font></div>
<div class=""><font face="monospace, monospace" class="">HUB</font></div>
<div class="">
<div class=""><font face="monospace, monospace" class="">1422581504.112268 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/key/5/data/%00%00</font></div>
</div>
<div class="">
<div class=""><font face="monospace, monospace" class="">1422581504.112966 DEBUG: [Forwarder] onOutgoingInterest face=264 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/key/5/data/%00%00</font></div>
</div>
<div class=""><font face="monospace, monospace" class=""><br class="">
</font></div>
<div class=""><font face="monospace, monospace" class="">PRODUCER</font></div>
<div class="">
<div class=""><font face="monospace, monospace" class="">1422581537.499525 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/key/5/data/%00%00</font></div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Assuming 2ms link delay: clock on HUB is ~33651ms behind of CONSUMER; clock on PRODUCER is ~266ms behind of CONSUMER.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">After compensating for the clock skew, the log snippet below says (in terms of CONSUMER clock, milliseconds):</div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px" class="">
<div class="">CONSUMER</div>
<div class="">55424 send Interest 1</div>
<div class="">55718 send Interest 2</div>
<div class="">55950 receive Data 1</div>
<div class="">55956 receive Data 2</div>
<div class=""><br class="">
</div>
<div class="">HUB</div>
<div class="">55425 receive and send Interest 1</div>
<div class="">55948 receive and send Data 1</div>
<div class="">55954 receive Interest 2, send Data 2 from ContentStore</div>
<div class=""><br class="">
</div>
<div class="">PRODUCER</div>
<div class="">55522 receive Interest 1</div>
<div class="">55570 send Data 1</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Observations:</div>
<div class="">
<ul class="">
<li class="">The link between HUB and PRODUCER is congested.<br class="">
Evidence: Interest 1 is received 97ms after it's sent. Data 1 is received 378ms after it's sent.</li><li class="">Interest 2 from CONSUMER to HUB is delayed.</li><li class="">HUB log reveals that face 266 (toward CONSUMER) and face 264 (toward PRODUCER) are both UDP and have same local IP address.<br class="">
The delay of Interest 2 from CONSUMER to HUB might be caused by the congestion between HUB and PRODUCER, because part of CONSUMER-HUB logical link shares same IP-layer link.</li></ul>
</div>
<div class=""><br class="">
</div>
Yours, Junxiao<br class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Thu, Jan 29, 2015 at 7:56 PM, Gusev, Peter <span dir="ltr" class="">
<<a href="mailto:peter@remap.ucla.edu" target="_blank" class="">peter@remap.ucla.edu</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="">
<div style="word-wrap:break-word" class="">
<div class="">Hi all,</div>
<div class=""><br class="">
</div>
<div class="">I’m doing tests and explorations on the reasons, why does NdnCon experience rebufferings (i.e. interruptions in video stream which lead to algorithm restart).</div>
<div class="">I would appreciate any help/suggestions/advices in debugging/analyzing NFD logs and NFD itself.</div>
<div class=""><br class="">
</div>
<div class="">More specifically, I’d like to hear your comments on the following case.</div>
<div class="">With the same configuration as before ([Consumer] <—> [<b class="">NFD1</b>] <==<i class="">Ethernet</i><i class="">_link</i>==> [<b class="">NFD2</b>] <==<i class="">Ethernet_link</i>==> [<b class="">NFD3</b>] <—> [Producer]) consumer experienced
interruption after getting frame 640. So I tracked down frame’s 641 interests and data in NFD logs:</div>
<div class=""><br class="">
</div>
<div class=""><b class="">NFD1 log: </b><i class="">($ cat consumer/nfd.log | grep "</i><i class="">/641/data/%00%00”)</i></div>
<div class=""><br class="">
</div>
<div class="">1422581555.423600 DEBUG: [Forwarder] onIncomingInterest face=265 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581555.423943 DEBUG: [Forwarder] onOutgoingInterest face=259 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581555.424205 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=<b class="">1282205730</b> from=265 newPitEntry-to=259</div>
<div class=""><b class="">^ 294ms (retransmission, confirmed by consumer log)</b></div>
<div class="">1422581555.718095 DEBUG: [Forwarder] onIncomingInterest face=265 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581555.718589 DEBUG: [Forwarder] onOutgoingInterest face=259 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581555.718929 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=589980969 from=265 retransmit-retry-to=259</div>
<div class=""><b class="">^ 232ms</b></div>
<div class="">1422581555.950466 DEBUG: [Forwarder] onIncomingData face=259 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class="">1422581555.950763 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581555.950998 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00 inFace=259 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class="">1422581555.951438 DEBUG: [Forwarder] onOutgoingData face=265 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class=""><b class="">^ 5ms</b></div>
<div class="">1422581555.956230 DEBUG: [Forwarder] <b class="">onIncomingData</b> face=259 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class="">1422581555.956496 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581555.956724 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00 inFace=259 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class=""><br class="">
</div>
<div class=""><b class="">NFD2 log: </b><i class="">($ cat hub/nfd.log | grep "</i><i class="">/641/data/%00%00”)</i></div>
<div class=""><br class="">
</div>
<div class="">1422581521.773468 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581521.773947 DEBUG: [Forwarder] onOutgoingInterest face=264 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581521.774352 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=<b class="">1282205730</b> from=266 newPitEntry-to=264</div>
<div class=""><b class="">^ 522ms (??? where is the retransmission interest?)</b></div>
<div class="">1422581522.296967 DEBUG: [Forwarder] onIncomingData face=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class="">1422581522.297347 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581522.297614 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00 inFace=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class="">1422581522.298123 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class=""><b class="">^ 5ms (ah, here it is… how?)</b></div>
<div class="">1422581522.303090 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581522.303430 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class=""><br class="">
</div>
<div class=""><b class="">NFD3 log: </b><i class="">($ cat producer/nfd.log | grep "</i><i class="">/641/data/%00%00”)</i></div>
<div class=""><br class="">
</div>
<div class="">1422581555.255647 DEBUG: [Forwarder] onIncomingInterest face=266 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581555.256007 DEBUG: [Forwarder] onOutgoingInterest face=269 interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581555.256334 DEBUG: [BestRouteStrategy2] /ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00?ndn.MustBeFresh=1&ndn.InterestLifetime=5000&ndn.Nonce=<b class="">1282205730</b> from=266 newPitEntry-to=269</div>
<div class=""><b class="">^ 47ms (!!! interest arrival was not late, data generation was not delayed)</b></div>
<div class="">1422581555.303275 DEBUG: [Forwarder] onIncomingData face=269 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class="">1422581555.303640 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00</div>
<div class="">1422581555.303928 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00 inFace=269 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class="">1422581555.304458 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/641/data/%00%00/5/664/22/0</div>
<div class=""> </div>
<div class=""><br class="">
</div>
<div class="">The questions here are:</div>
<div class="">1. How did the retransmission interest arrived on NFD2 ~220ms later than was expected and actually after the first interest was already answered?</div>
<div class="">2. If the data was generated and answered by NFD3 in timely manner, why did data arrived on NFD2 after 522ms but not after ~50ms?</div>
<div class=""><br class="">
</div>
<div class="">Some peculiar place of NFD2’s log:</div>
<div class="">
<div style="margin:0px;font-size:11px;font-family:Menlo" class=""><b class="">$ sed -n 38830,38840p hub/nfd.log </b></div>
</div>
<div style="margin:0px;font-size:11px;font-family:Menlo" class=""><b class=""><br class="">
</b></div>
<div class="">
<div class="">1422581521.916149 DEBUG: [Forwarder] onIncomingData face=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/638/data/%00%0A/14/661/22/0</div>
<div class="">1422581521.916515 DEBUG: [Forwarder] onIncomingData matching=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/638/data/%00%0A</div>
<div class="">1422581521.916799 DEBUG: [Strategy] beforeSatisfyPendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/638/data/%00%0A inFace=264 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/638/data/%00%0A/14/661/22/0</div>
<div class="">1422581521.917301 DEBUG: [Forwarder] onOutgoingData face=266 data=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/638/data/%00%0A/14/661/22/0</div>
<div class="">^ 360ms</div>
<div class="">1422581522.277620 DEBUG: [Forwarder] onInterestUnsatisfied interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/499/data/%00%06</div>
<div class="">1422581522.278080 DEBUG: [Strategy] beforeExpirePendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/499/data/%00%06</div>
<div class="">1422581522.278524 DEBUG: [Forwarder] onInterestUnsatisfied interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/500/data/%00%06</div>
<div class="">1422581522.278901 DEBUG: [Strategy] beforeExpirePendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/500/data/%00%06</div>
<div class="">1422581522.279352 DEBUG: [Forwarder] onInterestUnsatisfied interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/501/data/%00%05</div>
<div class="">1422581522.279742 DEBUG: [Strategy] beforeExpirePendingInterest pitEntry=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/501/data/%00%05</div>
<div class="">1422581522.280119 DEBUG: [Forwarder] onInterestUnsatisfied interest=/ndn/edu/ucla/remap/ndnrtc/user/remap/streams/top_cam/low/delta/501/data/%00%06</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I would appreciate any help.</div>
<div class="">Looking forward for the insights of NFD developers,</div>
<br class="">
<div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
Thanks, </div>
<div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<br class="">
</div>
<div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
PS. full logs are attached</div>
<div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
</div>
</div>
</div>
</div>
</div>
</div>
<div style="word-wrap:break-word" class="">
<div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
</div>
<div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<br class="">
</div>
<div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
-- <br class="">
Peter Gusev</div>
<div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
Programmer/Analyst @ REMAP, UCLA</div>
<div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">
<br class="">
<a href="mailto:peter@remap.ucla.edu" target="_blank" class="">peter@remap.ucla.edu</a><br class="">
<a href="tel:%2B1%20213%205872748" value="+12135872748" target="_blank" class="">+1 213 5872748</a><br class="">
peetonn_ (skype)</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>