The tinfoil hat wearer inside of me speculates that my internet provider is overloaded and throttling my bandwidth, but immediately prioritizes me when it senses that I'm trying to check if I'm getting what I pay for.
Has anyone else noticed this pattern? Is there a way I can test this more scientifically?
Speedtest kept reporting 100Mb/s range speeds (the tier he was paying for). Local file transfers however were stuck in the 1MB/s range.
After some questions and troubleshooting he discovered that he had used a 10Mb/s hub instead of a 1Gb/s switch that was sitting right next to it. A simple mixup.
His internet connection was also plugged into that same 10Mb/s hub.
There is NO WAY his PC could have reported 100Mb/s to speedtest.
Do not trust them for accurate results.
If you are on a wireless network (either your upstream from a WISP, or internally on your own network), then I'd suspect some wireless air access control syndrome is the cause -- e.g. stations are usually randomly fighting for air, but when you pull down your big speed test file that ends up putting the network in a state where your station gets all the air time. Lo and behold all your traffic has better QoS. Basically your speedtest traffic, because it is a constant stream, shoved everyone else off the air and allowed your other traffic to get through at the same time as a result.
But also worthy of recommendation IHMO is the Cloudflare one (speed.cloudflare.com). Main reason I say that is its one of the few that measures jitter (or at least openly exposes the measurement).
How about the following?:
1. Choose a dozen or so public files on hosts known to have high bandwidth (Google or Microsoft perhaps). Maybe choose a couple of public legal torrents too.
2. Write a script to download all the chosen files (either sequentially or concurrently), and emit timing information.
3. Run that script a couple of times a day at random times. Try to cover weekdays and weekends, days and nights.
4. Repeat 3 but running a speed test first each time. See if there is any statistically significant difference.
All carriers implement various types of throttling or rate limiting, for a lot of reasons (anti-ddos, cost control, etc.)
But; universally, they bypass these rate controls such as Cisco PGW ADC (Application Detection and Control) and LTE eNB ABR for wireless as examples, and many others similar features that limit the rate the user actually gets. And in almost all cases, these systems are designed to allow speed tests to bypass the rate limiting.
Commercial Speed tests are in no way a valid or reliable way of understanding the quality of a network service. I am a little surprised that this is breaking news on HN however. I would think this would be pretty well understood by this community?
BTW. Fast.com does is not always immune to this "Volkswagen diesel" effect. The only way to know for sure is to have a VM or bare metal server someplace and do an iperf to it.
Then added a wire connection to my server and instantly 10x the speed :^]
Dont be like me, have wires set up everywhere where needed.
Eg. http://134.209.196.181:8080 This is an instance of https://github.com/e7d/speedtest running on Digital Ocean in Amsterdam.
Yes, to a certain extent your ISP is watching to see if you run a speedtest and will open up more bandwidth to you in the event you do. However, if they were throttling you, which is not guaranteed, the most likely reason for throttling you is that the network hub for your neighborhood was under an heavy load, and they throttled everyone on your backbone hub to ensure that critical operations don't experience interruptions.
What you might consider critical operations might not agree with what your ISP considers critical of course, but they're a for profit business rather than a utility. So, what you consider a valid reason for throttling you doesn't count for squat. The reason they sped you up is that despite the fact that they carefully crafted their service contract to say they legally can throttle you for any reason or no reason at all, it reduces the number of complaints calls they don't intend to do anything about.
It also gives them plausible deniablity, and weirdly enough a useful metric for identifying hardware problems. They can simply make you think that the problem was the individual web service you were connecting to, or maybe just all of cloud flare. But more than that, they are able to track the number of times in a day that people visited a speedtest site, and whether they were able to get your speed up to the expected level. This functions as an early warning sign that they need to increase the total available bandwidth in an area if those numbers become what your ISP considers unacceptable.
Again, what your ISP considers unacceptable and what you consider unacceptable are not the same, but they're a for profit business rather than a regulated non-profit utility.
You can see what ISP traffic shaping actually does when you have a server on reliable backbone link outside of given ISP network by trying to download or upload large file to it and measuring. Bandwidth changes usually aren't too smooth and often you can see "stairs" on measurement graph as the speed drops the more data you transmit.
This shows how the whole "speed UP TO xyz mbits" marketing trick really works.
Now reload the page and and run it again but as soon as it starts, open another tab and do some browsing.
The second test gives me higher speeds consistently.
Suddenly, my traffic quality was mysteriously better.
No idea what was actually going on, but the whole thing was a bit sus.
I definitely recommend using your own server and profiling tools if you are suspicious of speedtest.
1. If you internet provider has limited bandwidth when it connects to other parts of the internet. I used to have a problem where my ISP has one connection out of its network. That bandwidth was limited and caused real world issues.
2. TCP can and often does involve multiple round trips. As web pages have grown in size it’s had an impact on performance at the network level
Increased bandwidth in the last mile doesn’t solve for these types of issues
1. I used to consult (for two years) in the department of my internet provider that would have had to do the reporting to the people who would have controlled throttling and even worked on the speed test page at times and there was no project to do that.
2. There was evidently no known and talked about throttling project among the hundreds of techies working at the business.
3. This place would not have been able to pull off the coordination between departments required to do this and also keep it a secret, and I'm betting most Internet providers would be even more incapable of doing something so nefarious.
4. The department providing customer support would have had to be in league with whatever department providing throttling, however KPIs for the customer support would of course have been negatively impacted by the many people calling in to report negative bandwidth usage. Thus they would have had to be in league with another division of the company to make sure they underperformed and thus did not get good bonuses etc. That would be an enviable level of dedication I must say.
I gave up with cots routers a while back, and life has been significantly improved. I wrote it up here https://res.rag.pub/2020-11-1-an-home-router.html
So Occam’s Razor makes me doubt it. ISPs do all kinds of tricks to make speed tests work well, co-locating, prioritising that path etc. But I don’t think they’ve complex profiling equipment that slows everything, then removes the slowdown when they see DNS/SNI for speedtest, then apply it again after.
Also been working for ISPs for 20+ years and never heard of it. Only exception may be in cellular with their fancy PCRF boxes, but even there I think it’s much more common to throttle one class, and not another, than to slow everything but switch the throttle on/off if you access a particular site.
If this is true you could easily game it by running speed test all the time. All you gotta do is throttle bandwidth to the speed test servers yourself somehow (so the speed test goes slow and doesn’t eat your available bandwidth.) Then you’ll be permanently in the “they’re doing speedtest give them full speed” category.
Another speedtester you can try: https://www.dslreports.com/speedtest
That day I've found out that my ADSL modem is not very good at making huge amount of DNS queries to outside world, and with a fast DNS resolver, Firefox is actually as fast as Chrome.
So, make sure that you can access your DNS servers with the speed they need to be accessed for a good network experience.
If you want to test whether this is true, make your own speedtest. Download the same Linux ISO from different mirrors, or try different speedtest mirrors to go further than your ISP.
The numbers are often higher than what I pay for, but I think they are honest. When I was having connection issues (dialup-tier download speeds, 20 megabit upload) it was reflected accurately. Since I trust them, I like this arrangement. If I didn't then I wouldn't.
What is true however is that speed test servers are often located right next to you and maybe given priority access. It means that you can get great numbers despite a poor global connection, but it shouldn't make anything else go better.
So someone I know couldnt submit their meter reading at various times throughout the 31st of March so I did it for them simply by doing a tracert to the server handling the meter readings in question for Eon.
In the time it took for the tracert to complete I was able to log in as them and submit their meter reading!
Before the tracert trick, I too was having lots of problems logging in as them trying to submit their meter reading.
I did the tracert because I was going to VPN to physically get closer to their server, possible even within the same data centre and as it happens it was located on a globally recognised brands scalable cloud server so there was no reason really for these server to not handle all these meter readings!
The cynic in me thinks this was an attempt by some energy firms to consolidate their market positions by messing customers around with higher energy bills which reinforces the adage nice guys finish last, or when looking back over several months, the nice companies go bust!
So some IT staff working for Eon will be getting their comeuppance, because they didnt let their servers scale up to meet demand and the so called free british press are extremely quiet on this point, because they dont want to reduce advertising income!
There are so many legal ways to be criminal!
The claim that it affects other tabs...seems wildly unlikely to me though. QoS and shaping is per service and per target.
My ISP does the prioritization whenever you do a DNS lookup for speedtest.net. You don't actually need to run the test.
Hosted version: https://librespeed.org/
Self hosted version: https://github.com/librespeed/speedtest
For those who need to document lower speeds than SLAs from their ISPs, this will help you check for chicanery, shenanigans, and sneaky QoS. Recommend.
So I wrote chargen2p [1] as an extension to the classical chargen protocol.
I use it together with a Prometheus exporter [2] I wrote to periodically check my laptop's connectivity. The actual check runs over Wireguard, since I didn't want to open my chargen2p server to the public. This only checks download speeds, mind you. (The chargen2p library exports upload metrics, but the exporter doesn't use it.)
My graphs tell me the average is ~5 MBps, so 40 Mbps. This is between me (Switzerland) and a Hetzner DC in Germany. speedtest.net just now claimed 160 Mbps.
[1] https://github.com/tommie/chargen2p
[2] https://github.com/tommie/prometheus-connectivity-exporter
you can inspect fast.com and see this in action: It will show up something like "ipv4-c209-sea001-ix.1.oca.nflxvideo.net" Which is their openconnect [1] location in Seattle IX.
In many cases you will probably see a prefix indicating your ISP - Most ISPs with some scale host their own caches or peer with an internet exchange directly.
Incase of speedtest - its the same story most ISPs host servers locally and speedtest's website generally tests from it
Most of my ISP screw ups are involve content which isnt cached and its difficult to convey that when your netflix is able to stream 4K content easily but struggling to download a 100mb file of some AWS region :/
I sometimes use proof.ovh.net and ping.online.net to test using iperf also ec2-reachability.amazonaws.com helps debug if ISP is having issues with connecting to AWS (more open than not thats the case with mine)
--- [1] https://openconnect.netflix.com
Something like this: alias pingspeed="ping -i 4 speedtest.net"
Long story short, most ISPs run or are well peered to data centers nearby, which is where speed tests are normally hosted at. For traffic outside of those hops, peering issues can lead to a multitude of issues... and you don't have to be evil for that to happen. No ISPs control the entire IP path between the client and server (with extremely narrow coincidental exceptions).
Then when your speed test gets popular. ISP's will think they're clever and start prioritizing your site!
That's when you BOOM! switch servers and point the DNS's to the new servers! Flip the servers running your business to the speed test servers and flip the speed test servers to your business!
Be sure to switch it back during off hours. Now you have a SUPER fast business operation and in addition to that a speed test site that is STILL more accurate then other speed test sites.
The default TCP mode is fairly intuitive and usually gives a good idea of what is going on.
For precise independent tests of up and down speeds you need to use UDP mode with increasing bit rates until you start to see significant packet loss in that direction. Repeat in the other direction.
Another consideration is that the endpoint you are testing from (probably some cloud vm or server at work) will likely have different routing to your other endpoint (home) than other sites/services you visit. It is not unusual for eyeball ISP's to sometimes have congested peering with specific networks in specific locations while your work->home path will be clean.
Across the many nodes between you and a destination, all sorts of traffic management systems move data around at different priorities. When something gets overloaded, stalled tabs are a visible system of the breakdown.
By initiating a large, typically sequential, high-speed transfer in both directions, you give each node an amount of time to recover, naturally move up the queue, and identify yourself as needing more resources. Once the speed test is moving along, everything else is 'along for the ride', to an extent.
On a home network, wifi aps are typically the biggest bottleneck, followed by underpowered or overheated routers/gateways. Depending on your isp's tech, there might be a bottleneck at a neighborhood hub, the central servers, their uplink, or more.
Anywhere along the way, this could be due to a mix of technical limits, high traffic, wear, negligence, incompetence, or intentional malice.
So you can assume your isp (or their isp) is throttling you in numerous ways. While some might be to deprioritize specific traffic or for political means, I think more is negligence, incompetence and under investment. When you identify yourself as someone needing more, or experiencing issues, it's in the isp's interest to silently resolve the issue automatically if possible (unless they think they can exploit it for a profit).
This is even more visible with the isp-owned speed tests, as you're naturally communicating to the fastest server available, plus they can shape the traffic and build the test to show you what they want you to see.
iPerf tests are a good tool for testing real world speed. As a basic tool, speedtest.net allows you to select from many test servers, often including your isp, a few local competitiors, and a close by university or tech company server. Individual numbers are less useful than a comparison between numerous tests and destinations.
A VM of smokeping or smokeping-speedtest is useful tool for monitoring these sorts of things as you can go back and look at what drops out when to get a feel conditions across the internet.
As others have said, if your destination has a slow connection, you're pretty screwed. Not to mention lots of slow file transfers = more latency impact.
Transferring 1x1GB file vs 1,000x1MB files is a HUGE difference, the latter will take much longer due to the initialization of the transfer. That's why we used to use FTP to transmit files instead of HTTP way back in the dialup days.
I’ve also seen a lot of Speedtest results for ATT fiber. These allow full gigabit transfers. I do wonder if these results hold up to actual use cases if someone can saturate 1gbit.
[0]https://www.macobserver.com/tips/macos-monterey-includes-net...
But I never assumed malice, even if it is AT&T. I'm not a network engineer but I can see how giving whatever back end networking management system they have a nudge by requesting a fast connection improving things.
Makes me wonder if they have some kind of shady "squeaky wheel" QOS traffic shaping voodoo.
Running certain online games in the background (I think it was final fantasy) fixed problems with the art/drawing/just chatting stream dropping frames.
(Sorry I forget who, I also think it wasn't someone I normally watch so it's unlikely I will remember.)
Instead, I wonder if your router or something else on your network stack is a little wonky. Do speeds increase when you reboot your router or cable modem? When you have slow speeds, does another device on the network also suffer slow speeds?
I never noticed this with any ISP I've used in NL/DE/BE/FI.
Never even questioned whether it was true because it felt so incredibly obvious that the ISP was dethrottling whenever they detected a speed test.
speedtest.net: 19ms ping / 288 Mbps down / 72 Mbps up g speed test : 8ms ping / 620 Mbps down / 28 Mbps up
Xfinity?
All Test Pages mentioned so far:
----------------------------------
https://www.speedtest.net/apps/cli
http://www.dslreports.com/speedtest
https://speed.aussiebroadband.com.au/
Additonal Sites not mentioned yet:
----------------------------------
https://speedtest.xfinity.com/
http://internethealthtest.org/
Tools:
----------------------------------
https://iperf.fr/iperf-servers.php
https://github.com/orz811017/boost_bandwidth_via_speedtest
https://github.com/e7d/speedtest