Rich's Tongfamily (Richtong.net add-on)

Well now that so many of us are at home, figuring out how to keep your video and other calls running well is more important than ever. Since yesterday, our network speeds have been horrible.

Before the crisis we were getting 190Mbps upstream and 5Mbps downstream. But starting yesterday that had dropped to 30Mbps/1Mbps and holding a Zoom call was pretty hard.

TL;dr

1. Before you start a video conference, test your client. Use Speedtest to see what the client is actually seeing. If you are using Zoom, you need at least 5Mbps downstream and 1Mbps upstream. As an aside, if you are using Google Hangouts then you need 5x more and Facetime more like 10x, which explains why you see more lags and problems on those.
2. Connect to your router’s web administration page to see what is using network while you are running the call because bandwidth fluctuations. This will tell you what the actual outbound traffic is. If you are lucky, you will the ability to do traffic analysis so you can see what each client is doing. As an aside, a very common problem is a rogue client like someone’s iPhone, these can draw 5Mps in the background, so if that’s a problem run it in those phones in Low Data Mode or take it off the network.
3. If the bandwidth is low, it could be the Wifi connection to your device. There are not really great ways to test this, you need to run an benchmark to figure out if your bandwidth has low. Don’t be fooled by the “bars” in the interface, you need to know what the real bandwidth is. The only way I’ve found to do this is to run brew install cask iperf on each machine and then see what the real bandwidth is. Failing that, move as close to the access point as you can and run Speedtest to see if it changes alot.

Problems We Had

We had a problem with speeds down to 30Mbps down and 1Mbps up caused by:

1. iPhones running background insist on uploading photos and things and can use 5Mbps (or 90% of the bandwidth we have here)
2. A rogue task running on a NAS running a backup to the cloud required a full reboot to stop it.
3. A stepped down speed on our main machine so it was limited to 30Mbps on the wifi network
4. A funky access point that needed rebooting.
5. Monitor what is going on at your router, figure out where the router statistics are and monitoring them. With Unifi EdgeRouter Lite using EdgeMAS, go to the second tab called traffic analysis and enable it to see which clients are chewing up bandwidth

Diagnosis Step by Step…

If you have issues like this, here are some diagnostics that you can run and the order

1. Speedtest from a browser just to see what your end devices are seeing. Speedtest has a sharing function and a history function so that you can see how traffic is working over time. For instance, I just ran it and got 27Mbps down and 1.2Mbps yup yesterday.
2. Check your Router Statistics to see what’s flowing, look for rogue traffic. The easiest way to do this is to see the per client traffic. That doesn’t mean too much because you really need to know if it is the network connection that you have particularly if you are using Wifi or it is a problem in the broadband internet. Unfortunately most consumer routers kind of stink at this, but for Unifi EdgeRouter Lite, you get a nice display that shows in their Dashboard how much is being used. It’s a little hard to interpret, but the Internet line is the one that you want. The Local and Local 2 are just the mirrors of the Internet line summed. This gives some great statistics just go to the second tab of EdgeMax called Traffic Analysis and choose to enable as it is off by default. We immediately discovered that 99% of our upload bandwidth was taken by my iPhone. It was uploading a video I had shot. I would never have figured it out otherwise. The easy way to prevent this is to put the phone into Low Power Mode which cuts that off.
3. If there is rogue traffic, figure out what machine it is. Ok so if you have lots of traffic out to the Internet, it’s time to figure out what it is. The simplest thing to do is to start shutting machines down. Or if you have menumeters on your Macs, you can just turn on the network widget and see what is happening on them.
4. Checking on a Synology for a rogue task (found it!). There are quite a few ways for Synology to be doing backups, they include rsync, snapshot replication, hyperbackup, and cloud backup. So the first thing to do is to just look at the widgets on the upper right when you login and turn on performance monitor, you should see immediately the network in and out and also the file log which will tell you what files are written. In our case, there was 1Mbps going upstream so we found the rogue machine! Then you just need to go to each of the applications to figure out what is running. Even then, we were still getting 5Mbps written upstream to something so shutting this off seemed to help. I’m guessing that even though the copy said failed, we were still seeing 1Mbps inbound and 5Mbps to the internet so probably a bug.
5. For fixed smart home devices, if you can’t find it, then try unplugging them. Who knows what they are streaming.
6. At this point in making things work, suddenly the network stopped accepting our usual password. I don’t quite know the issue here, but it looks like the Unifi NanoHD do sometimes need a reboot even though Unifi Controller reports everything as fine.

Ok, if you can confirm that the network is quiescent, then maybe it is something on your client, so time to figure out if it is in your internal network:

1. Wifi Connection speed by looking at the client your testing. The easiest first thing on a Mac is to look at the actual connection speed. Holding the command key when clicking on the Wifi icon tells you the native speed (in our case, with 802.11ac, 700Mbps is not unusual). If you have Unifi, you can then go to the controller side and look at the device and see what the performance is. On Windows, this is a little trickier, but you can use the performance monitor to tell what is going on.
2. Look at the access point data to see if the other problem could be the Wifi is slow or contentious. With Unifi, you can look at interference issues. In our network case, the dashboard showed that things were good.
3. See if your wifi connection has stepped down. Even if the machine reports a high speed, it could be that you are running at a lower one. So the real test is to load iperf onto two machines and see what the speeds really are just to make sure. This is a command line utility that does a true test. You want to test between a wired and a wifi connection to make sure. In our case, it looks like it is really Comcast that is responsible for most of. the slowdown, since sometimes we could get to 200Mbps, but mainly is 30Mbps for download. That is all pretty academic though, what matters for things like zoom is preserving the 5Mbps uplink. Iperf is pretty easy to do, just set up a server with iperf -s and then iperf -c <ip address> to the IP address of the server. In our case, it looks like our main machine’s wifi (even though it says 500Mbps) is actually limited to 30Mbps. That was the fastest we could get with an iperf test.
4. Test with a fast machine. And when we did test with a machine that is one foot from the access point, we ended up getting 176Mbps upstream and 5Mbps downstream.

Net, net, we found a leaking application but also that our download and upload speeds (which are shared on a cable system) to be really pretty variable. At the high we were getting 200Mbps/5Mbps, but averaging 30Mbps/3Mbps which isn’t so great. Probably every streaming like crazy! According to the Speedtest Covid page, there has been a slight average decline in Seattle (I wish I was getting the 160Mbps mean!).