FYI/A: Bufferbloat 101

[xiaofan]

Finally got an A+ with my work laptop (Dell Precision 3541, Core i7-9850H CPU, 16GB RAM and 512GB SSD), with my Asus RT-AX82U router, using wireless connection.
http://www.dslreports.com/speedtest/66707661

But it is not consistently A+, sometimes it is only A.
http://www.dslreports.com/speedtest/66709067

As the built-in Intel Wireless AC-9560 supports 2*2 AC 160MHz, Ookla speedtest result is very good. With DSL Reports the download speed again is slower but upload is also pretty fast.

 

[xiaofan]

But for my home laptop with Intel AC-7265 wireless card (2*2 AC 80MHz), usually I got the bad result of C grade in Bufferbloat.

https://www.speedtest.net/my-result/d/da048f16-e9c1-48d4-89a2-7b3514c5e60e

Ookla SpeedTest for reference as well.

 

[xiaofan]

Finally got an A+ with my work laptop (Dell Precision 3541, Core i7-9850H CPU, 16GB RAM and 512GB SSD), with my Asus RT-AX82U router, using wireless connection.
http://www.dslreports.com/speedtest/66707661

As the built-in Intel Wireless AC-9560 supports 2*2 AC 160MHz, Ookla speedtest result is very good. With DSL Reports the download speed again is slower but upload is also pretty fast.

With wired, again bufferbloat is bad at Grade C.
http://www.dslreports.com/speedtest/66708944 (link to RT-AX82U)

http://www.dslreports.com/speedtest/66709028 (link to Singtel Mesh router which is the main router)

Ookla speedtest is as expected. Upload is a bit faster than my Acer Windows 10 home Laptop, I think the Ugreen USB3 Gigabit Ethernet used by my Windows 10 limit the upload speed. For the Dell work laptop, it has built-in gigabit ethernet adapter (Intel i219-LM).

 

[bert64]

yes, but the fact remains that the oversaturated buffers are on the ont and not my router, since bufferbloat waa ridiculously bad (thousands of ms) with direct connection to ont (which i have since tested yesterday).

i am happy with the performance of my router as it does not slow down even with torrent and i set the routing rules to fasttrack connections not destined for my vpn

in fact the zte ont is less powerful than the mikrotik. just like huawei onrs, they probably do not run in "true layer 2 bridge" mode but instead run a layer 3 bridge with DHCP relay, zte underpowered processor can easily cause the bottleneck here.

Yeah i've not tried pulling apart an ONT to see what it's doing.. The more unnecessary complexity in the process, the worse performance it's going to get. As far as i know, these devices use a dedicated ASIC for packet forwarding, as the cpus they use would never keep up doing it purely in software.
Usually they don't have large enough buffers that they would ever contain enough data to make a large enough difference on a 1gbps connection, although overloading the hardware in general would achieve a similar latency spike (technically not bufferbloat). I wonder if technical specs are published anywhere for these devices.

If you make an ASIC powered device forward packets in software mode (typically it will do this automatically as a fallback if hardware forwarding is not available due to your config, or fails due to bugs etc) you will see terrible performance, i've spent quite some time debugging this behaviour in various cisco layer 3 switches.

It seems the dslreports cli tool does not attempt to test latency, only throughput, and it defaults to using less connections for upload than download and consequently gets much worse upload stats. If i adjust it to use the same connections for both it gets more symmetrical.

http://www.dslreports.com/speedtest/66712627

Out of interest i ran the dslreports cli version on a linux box with direct 1gbps ethernet connection to a cisco 6500 layer 3 switch with 10gbps uplink:

http://www.dslreports.com/speedtest/66712590

Although the box is based in the uk, and the test servers seem to be in the netherlands and switzerland.

 

[xiaofan]

It seems the dslreports cli tool does not attempt to test latency, only throughput, and it defaults to using less connections for upload than download and consequently gets much worse upload stats. If i adjust it to use the same connections for both it gets more symmetrical.

http://www.dslreports.com/speedtest/66712627

Out of interest i ran the dslreports cli version on a linux box with direct 1gbps ethernet connection to a cisco 6500 layer 3 switch with 10gbps uplink:

http://www.dslreports.com/speedtest/66712590

Although the box is based in the uk, and the test servers seem to be in the netherlands and switzerland.

The CLI tool has been abandoned so no point using the CLI tool as it does not really measure bufferbloat. You will have to use the web version and you can even customize the test.

 

[xiaofan]

Yeah i've not tried pulling apart an ONT to see what it's doing.. The more unnecessary complexity in the process, the worse performance it's going to get. As far as i know, these devices use a dedicated ASIC for packet forwarding, as the cpus they use would never keep up doing it purely in software.
Usually they don't have large enough buffers that they would ever contain enough data to make a large enough difference on a 1gbps connection, although overloading the hardware in general would achieve a similar latency spike (technically not bufferbloat). I wonder if technical specs are published anywhere for these devices.

If you make an ASIC powered device forward packets in software mode (typically it will do this automatically as a fallback if hardware forwarding is not available due to your config, or fails due to bugs etc) you will see terrible performance, i've spent quite some time debugging this behaviour in various cisco layer 3 switches.

The particular ZTE ONT in question is an ONR configured as ONT. Therefore it does need to have a pretty powerful CPU on board to function as a router.

milo is suspecting that SingTel configured the ZTE as an ONT due to performance issues that it may not be able to handle 2Gbps plan. Take note SingTel ONR needs to support 2Gbps plan (aggregated 2Gbps across 4 ports).

As for why ZTE can not even handle 1Gbps well in ONT mode, I am not sure, maybe you are tight that the hardware forwarding is not functioning well above certain speed (somewhere between 800Mbps to 900Mbps).

 

[xiaofan]

By doing quite a bit of speed test, I also find different speedtest server may perform quite differently, even using the same Ookla SpeedTest app on the same computer.

For example, Singtel SpeedTest server is not that consistent but Viewquest SpeedTest server is better.

Same work laptop connected to RT-AX82U wireless.
Viewquest SpeedTest server

Singtel SpeedTest server, upload speed is not consistent and slower than Viewquest significantly

DSLreports, download speed is always slower than upload
http://www.dslreports.com/speedtest/66717565

 

[xiaofan]

today's result:
http://www.dslreports.com/speedtest/66719512

Nice.
Today at least I can still get A with a 900Mbps+ upload speed. This is a pretty beefy Windows 10 Dell Precision 3541 laptop with Core i7-9850H CPU, 16GB RAM and 512GB SSD, Intel AC9560 2*2 AC 160MHz wifi card.

http://www.dslreports.com/speedtest/66720547

Once going to wired, I can only get C.
http://www.dslreports.com/speedtest/66720583

Ookla speedtest results for reference

c:\Utils\ookla-speedtest-1.0.0-win64>speedtest.exe

   Speedtest by Ookla

     Server: Singtel - Singapore (id = 13623)
        ISP: Singtel Fiber
    Latency:     3.11 ms   (0.50 ms jitter)
   Download:   468.63 Mbps (data used: 715.1 MB)
     Upload:   790.73 Mbps (data used: 807.9 MB)
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/6c98d36b-9929-4940-aeda-f730e6822ad1

c:\Utils\ookla-speedtest-1.0.0-win64>speedtest.exe -s 2054

   Speedtest by Ookla

     Server: Viewqwest Pte Ltd - Singapore (id = 2054)
        ISP: Singtel Fiber
    Latency:     4.55 ms   (0.30 ms jitter)
   Download:   905.19 Mbps (data used: 1.0 GB)
     Upload:   856.78 Mbps (data used: 1.2 GB)
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/aab0bf53-7d5d-446f-bc51-9a8f30ad4449

 

[hwzlite]

But for my home laptop with Intel AC-7265 wireless card (2*2 AC 80MHz), usually I got the bad result of C grade in Bufferbloat.

https://www.speedtest.net/my-result/d/da048f16-e9c1-48d4-89a2-7b3514c5e60e

Nice.
Today at least I can still get A with a 900Mbps+ upload speed. This is a pretty beefy Windows 10 Dell Precision 3541 laptop with Core i7-9850H CPU, 16GB RAM and 512GB SSD, Intel AC9560 2*2 AC 160MHz wifi card.

http://www.dslreports.com/speedtest/66720547

[/CODE]

Looks like the Intel AC9560 has better build-in wifi hardware/stack taming bufferbloat.

Actually taming wireless is another beast as described in the ongoing Make Wi-Fi Fast Project , more at https://lwn.net/Articles/705884/

 

[xiaofan]

Looks like the Intel AC9560 has better build-in wifi hardware/stack taming bufferbloat.

Actually taming wireless is another beast as described in the ongoing Make Wi-Fi Fast Project , more at https://lwn.net/Articles/705884/

Interesting article. On the other hand, wifi driver in the Linux world and OpenWRT are still not as good because of vendors like Broadcom does not cooperate well with the open source movements. So many people actually recommend using an x86 based openwrt/pfsense router with a wireless AP.

And yes the computer and the wifi chipset seem to make differences for the bufferbloat test results using wifi.

The Dell Core i7 Gen 9 CPU laptop with Intel AC-9560 usually gets "A" in the testing.

Then for the 3 year old Acer Swift 3 with Intel AC-7265, it is usally "C" and sometimes even "F".

My Mac Mini M1 is pretty powerful. The Wifi 6 adapter does not seem to have very fast speed (400...600Mbps) probably due to the distance to the RT-AX82U router but bufferbloat result is usually "A" grade. I count the 10 times "A", 1 time of "B", "C" and "D" each.

But once I use wired test with either the Dell or the Acer, Singtel ZTE ONT will make the result to be consistently "C". My Mac Mini M1 is in the study room without wired connection so I can not test it with wired connection.

I actually do not use wired for most of my devices as I do not have Ethernet cabling at home. Only the Singtel TV Box and an Xbox One S are connected using wired connection.

 

[hwzlite]

I actually do not use wired for most of my devices as I do not have Ethernet cabling at home. Only the Singtel TV Box and an Xbox One S are connected using wired connection.

For my case I have 2 dedicated wired devices :

- A 8yrs old 24/7 workhorse Intel i5-3570K desktop recently refreshed OS from Win7 to Win10 , acting as HTPC, backup storage, Tor Bridge Relay , RemoteSpark , RDP Wrapper , Torrenting... etc . I used to run vmware virtualized pfsense and Spoho UTM bridged directly to ONT , now replaced by DD-WRT (D-Link DIR-868L).

- RIPE Atlas Probe

 

[xiaofan]

Interesting article. On the other hand, wifi driver in the Linux world and OpenWRT are still not as good because of vendors like Broadcom does not cooperate well with the open source movements. So many people actually recommend using an x86 based openwrt/pfsense router with a wireless AP.

And yes the computer and the wifi chipset seem to make differences for the bufferbloat test results using wifi.

The Dell Core i7 Gen 9 CPU laptop with Intel AC-9560 usually gets "A" in the testing.

Then for the 3 year old Acer Swift 3 with Intel AC-7265, it is usally "C" and sometimes even "F".

I just installed Ubuntu 20.04 to an external USB SSD (HP500 256GB) for the Acer Swift 3 (Core i5-8250U, 8GB RAM, 256GB internal SSD, Intel AC-7265 Wifi card, no Ethernet port but I use a Ugreen USB3 to Gigabit Ethernet).

Interestingly now I got A two times in a row. So Linux wins in this regard.
http://www.dslreports.com/speedtest/66729064

http://www.dslreports.com/speedtest/66729047

Download speed is on par with Singtel speedtest server, but Viewquest server is faster.

mcuee@ubuntu2004:~/build/speedtest$ ./speedtest -s 2054

   Speedtest by Ookla

     Server: Viewqwest Pte Ltd - Singapore (id = 2054)
        ISP: Singtel Fiber
    Latency:     3.91 ms   (0.31 ms jitter)
   Download:   624.44 Mbps (data used: 797.4 MB)                               
     Upload:   446.93 Mbps (data used: 443.2 MB)                               
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/89aa8899-05ce-4af0-8000-faa66754db69

mcuee@ubuntu2004:~/build/speedtest$ ./speedtest -s 13623

   Speedtest by Ookla

     Server: Singtel - Singapore (id = 13623)
        ISP: Singtel Fiber
    Latency:     2.87 ms   (0.26 ms jitter)
   Download:   421.29 Mbps (data used: 468.9 MB)                               
     Upload:   462.10 Mbps (data used: 432.8 MB)                               
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/57f1388d-8366-4b85-a1e6-70a036b17f96

 

[xiaofan]

Under Linux and test with wired using the Ugreen USB 3 gigabit adapter (ASIX AX88179 Gigabit Ethernet chipset), bufferbloat becomes A+ consistently, but there is a big problem for download that it can not sustain the download speed and drop significantly half way. Looks like an ASIX AX88179 Linux driver performance problem. On the other hand, Ookla speedtest is okay.

Ookla test result (BTW, no problem testing under Chrome broswer under Linux).

DSLreport results
http://www.dslreports.com/speedtest/66729150

http://www.dslreports.com/speedtest/66729155

 

[xiaofan]

Finally got a chance to test wired using my Mac Mini M1 by pulling a long Ethernet cable from the RT-AX82U in the living room to the Mac Mini M1 in the study room. I can only test a few minutes because of the trip hazard from the other family members.

Result is not that consisent, 2 times A, one time B and one time F.
Example of "A"
http://www.dslreports.com/speedtest/66742040

Example of "F", really quite bad for the upload.
http://www.dslreports.com/speedtest/66742048

 

[xiaofan]

Just upgrade my Acer Swift 3 Core i5-8250U laptop (bought in Dec 2017) to Intel AX200 based wifi card (AX3000 rated) from stock Intel AC-7265 AC1200.

Bufferbloat testing results seem to be better at Grade A now but apparently the download/upload speed is not that stable, especially upload.
http://www.dslreports.com/speedtest/66767879

http://www.dslreports.com/speedtest/66767891

 

[hwzlite]

LOL as for my weekend homework, after much fiddling....managed to get triple A+ result:

http://www.dslreports.com/speedtest/66767349

This was after CAKE sqm applied on vlan2 rather than the default eth0 (dunno why liddat) interface for WAN.

Can't deny that OpenWrt has great support on documenting CAKE and forum on interpreting its result, too bad my DIR-868L not in their hardware support list...

 

[xiaofan]

Today home internet is slightly quieter and wireless Ookla SpeedTest result is a bit faster than usual with my Acer laptop and Intel AX200 adapter.

DSLReports SpeedTest Bufferbloat is rated A Grade but I can still see drops in the download/upload.
http://www.dslreports.com/speedtest/67013282

 

[xiaofan]

Under Ubutu 20.04 Linux actually the Ookla SpeedTest (using Viewquest SpeedTest server) under Chrome browser is pretty good for the Intel AX200 wireless adapter, with Asus RT-AX82U router

Bufferbloat is actually A+ grade as well. Speed is slower then Ookla though.
http://www.dslreports.com/speedtest/67014206

 

[uncle_josh]

Best Routers For bufferbloat mitigation for 2021

Sent from LG-H930 using GAGT

 

[xiaofan]

Best Routers For bufferbloat mitigation for 2021

Sent from LG-H930 using GAGT

From the article, you need expensive UDM Pro for 500Mbps/1Gbps plan. I guess I will live with the bufferbloat issue which does not really affect the internet access (main issue for SingTel ZTE ONT user is with upload).


****
Our top ‘best router’ recommendation for standard and fibre broadband connections is the affordable Ubiquiti ER-X, which is a bargain at under £60.

For ultrafast broadband connections (with speeds up to 300 Mbps), we recommend the Ubiquiti ER-4.

For the fastest ultrafast broadband connections (up to 910 Mbps), we recommend the outstanding and powerful Ubiquiti UniFi Dream Machine Pro. Rather than purchase a Wi-Fi router, we recommend that you opt for one of these devices and add multiple Wi-Fi Access Points to give the very best coverage and performance.
****