FYI/A: Bufferbloat 101

[xiaofan]

Not the most exciting results.

Also, tested 10x, but only once this Bufferbloat test loaded. Wonder what gives.

Wireless?

 

[firesong]

Wireless?

Yeah. Was too lazy to drag my laptop over to the switch. But speeds are much slower than speedtest's results - guessing it's because it's a connection to a Japanese server.

Shared in the pfSense thread. What's more alarming is that I had a sudden flood of Suricata alerts go off. Looks like a Snort rule that a lot of devices on my LAN suddenly tripped, including a TV, 2 Google Home Minis and one AP. Weird.

FILE-PDF Adobe Acrobat Reader XFA addInstance use after free attempt

 

[xiaofan]

Yeah. Was too lazy to drag my laptop over to the switch. But speeds are much slower than speedtest's results - guessing it's because it's a connection to a Japanese server.

Wireless Download speed from DSL Reports has always been much slower than OOkla SpeedTest, probably due to the test methods. Same for waveform.com but not as bad.

My wireless test results for an Acer laptop with Intel AX200, 3m away from the router
1) RT-AX82U without any QoS settings, both A from DSL Reports (128.6Mbps download, 372.6Mbps upload) and Waveform.com (514.8Mbps download, 369.9Mbps upload). You can see the download speed is very low for DSL Reports.

http://www.dslreports.com/speedtest/70018065https://www.waveform.com/tools/bufferbloat?test-id=839a5be3-dd5c-4c59-bf57-92107842daff
2) Raspberry Pi 400 running OpenWRT with QoS limiting the upload to 500Mbps, using Huawei AX3 Pro as AP.
Score is A for DSL Reports (128.3Mbps download, 338.2Mbps upload) but only C for Waveform.com (398.9Mbps download and 355.1Mbps upload).

http://www.dslreports.com/speedtest/70018360https://www.waveform.com/tools/bufferbloat?test-id=abce7ae2-49ad-483f-8f38-84dc106cf853

 

[Mach3.2]

https://www.waveform.com/tools/bufferbloat?test-id=aa4969b9-cc90-4afc-8f68-ade32f94d5c9
Ran the test a few times, results doesn't seem very consistent. I think might have to further tune the bandwidth number in my codelq limiter.

 

[xiaofan]

https://www.waveform.com/tools/bufferbloat?test-id=94bd2d43-11fa-43dc-be26-90b0dcd7b87f
still on my singtel bb. set rate limit of 700mbps upload (no queueing, packets exceeding will be dropped)
no qos

Are you using wired or wireless? What is the router and device used for the test?

 

[miloaisdino]

Are you using wired or wireless? What is the router and device used for the test?

on wired, my router is mikrotik hapac2 (rate limit was set directly on my tplink tl-sg105e switch)

i might conduct an experiment using xiaomi ax3600 qos sometime soon!

 

[xiaofan]

on wired, my router is mikrotik hapac2 (rate limit was set directly on my tplink tl-sg105e switch)

i might conduct an experiment using xiaomi ax3600 qos sometime soon!

I see. The wired speed (860.4Mbps download and 491.7Mbps upload) is on the low side, I guess that TP-Link TL-SG105E can not properly handle the rate limit.

I remember you can directly set the limit in the Singtel ZTE ONT.

Once you are on M1, maybe you can try the Xiaomi AX3600 QoS to see if it is good or not. It has a very fast CPU.

 

[xiaofan]

https://www.waveform.com/tools/bufferbloat?test-id=aa4969b9-cc90-4afc-8f68-ade32f94d5c9
Ran the test a few times, results doesn't seem very consistent. I think might have to further tune the bandwidth number in my codelq limiter.

You are on a 500Mbps plan and using a powerful pfSense router and good wireless AP, haha, that is not fair.

I assume your test result is with wired. Could you also test wireless?

But I do have a question of all these SQM-QoS method -- how to set the rate limit for upload/download to get good Bufferbloat score for both wired (faster speed, near 1Gbps for 1Gbps plan) and wireless (slower speed, can be 300Mbps/500Mbps/800Mbps/etc), without compromising too much on the download/upload speed.

So I myself am still very much sceptical about the effectiveness of SQM-QoS methods for 1Gbps plan.

Most of the article talking about bufferbloat are for 100Mbps or less plan. Some articles talk about higher speed but I think they only consider wired, for wired, it is not that difficult with a more powerful router with proper SQM-QoS implementation. But I really doubt it will help also on wireless.

Eg: https://www.increasebroadbandspeed.co.uk/bufferbloat-broadband-fix

 

[miloaisdino]

I see. The wired speed (860.4Mbps download and 491.7Mbps upload) is on the low side, I guess that TP-Link TL-SG105E can not properly handle the rate limit.

I remember you can directly set the limit in the Singtel ZTE ONT.

the upload speed will be well below 700mbps as the packets are dropped and not queued, tcp protocol will automatically reduce the transfer rate to prevent packet loss during spikes. for example if i set the limit higher up the actual speed will also increase.

as for the download i think it has to do with the timing of the test (to cloudflare servers) as speedtest to singtel servers still show 900+..

 

[xiaofan]

For wireless Right now I do not know how to get good bufferbloat score with SQM-QoS. I am not so sure if it is due to the Singtel ZTE ONT or not.

But the Asus router side (RT-AX82U) performs better with a score of A without any QoS settings. The results are pretty consistent as well.
1) https://www.waveform.com/tools/bufferbloat?test-id=839a5be3-dd5c-4c59-bf57-92107842daff2) https://www.waveform.com/tools/bufferbloat?test-id=f546936d-f72e-450e-953f-044fd197dccb

Another Router -- Raspberry Pi 400 with Openwrt 21.02, SQM-QoS rate limit set for 500Mbps on the upload, no limit on the download.

I tested the Huawei AX3 Pro AP and the bufferbloat score is bad at C.
https://www.waveform.com/tools/bufferbloat?test-id=abce7ae2-49ad-483f-8f38-84dc106cf853
As I need to redeploy the Huawei AX3 Pro to another place, I changed the AP to Singtel Mesh Router (AC1900), bufferbloat is also not that consistent.
1) Not so good at B.
https://www.waveform.com/tools/bufferbloat?test-id=648a2cbc-37f2-4486-a5b0-8b4b61d7d1b42) Better at A.
https://www.waveform.com/tools/bufferbloat?test-id=c4c948b3-9ff0-4438-a832-9507c1f4f69b
The SQM-QoS inside OpenWRT is apprently very effective for Bufferbloat on wired as I can get A+ (download 679.6Mbps and Upload of 466.1Mbps). So the CPU inside Raspberr Pi 400 is not too bad, it can effectively limit the upload speed to below 500Mbps, even though it also limits the download speed.
https://www.waveform.com/tools/bufferbloat?test-id=e267ceaf-2c71-4755-8c97-4cbcd29f167f
My thinking now is that I need the SQM-QoS on the wireless AP as well. Just setting up on the WAN side does not help in this case. SingTel Mesh Router is a very dumb AP so this goal can not be achieved.

I can not limit on the LAN side because it has both wired and wireless clients through the AP.

I am not so sure if I switch back to pfSense on the J4105 mini PC will help in this case or not. My main clients are wireless. Wired are mostly for testing purposes only since I do not have LAN ports in the rooms.

 

[Mach3.2]

You are on a 500Mbps plan and using a powerful pfSense router and good wireless AP, haha, that is not fair.

I assume your test result is with wired. Could you also test wireless?

But I do have a question of all these SQM-QoS method -- how to set the rate limit for upload/download to get good Bufferbloat score for both wired (faster speed, near 1Gbps for 1Gbps plan) and wireless (slower speed, can be 300Mbps/500Mbps/800Mbps/etc), without compromising too much on the download/upload speed.

So I myself am still very much sceptical about the effectiveness of SQM-QoS methods for 1Gbps plan.

Most of the article talking about bufferbloat are for 100Mbps or less plan. Some articles talk about higher speed but I think they only consider wired, for wired, it is not that difficult with a more powerful router with proper SQM-QoS implementation. But I really doubt it will help also on wireless.

Eg: https://www.increasebroadbandspeed.co.uk/bufferbloat-broadband-fix

Results on wifi is actually kinda wack. But wifi is a best effort service anyway, so long youtube/websites load i guess all is good.

 

[xiaofan]

Results on wifi is actually kinda wack. But wifi is a best effort service anyway, so long youtube/websites load i guess all is good.

Haha, no issues with all things using wireless on myside, including WFH/HBL, Youtube, web browsing, or downloading big files like Linux ISOs.

 

[xiaofan]

Other than DSL Reports, Waveform.com, what other tools can be used for bufferbloat testing?

There are some pointers here. Personally I tend to like command line tools more.
https://www.bufferbloat.net/projects/bloat/wiki/Tests_for_Bufferbloat/

root@OpenWrt:/opt/home/OpenWrtScripts# ./betterspeedtest.sh
2021-11-14 21:50:47 Testing against netperf.bufferbloat.net (ipv4) with 5 simultaneous sessions while pinging gstatic.com (60 seconds in each direction)
...............................................................
 Download: 191.27 Mbps
  Latency: (in msec, 63 pings, 0.00% packet loss)
      Min: 2.721
    10pct: 2.769
   Median: 3.007
      Avg: 3.175
    90pct: 3.230
      Max: 9.484
.............................................................
   Upload: 445.64 Mbps
  Latency: (in msec, 61 pings, 0.00% packet loss)
      Min: 2.939
    10pct: 3.070
   Median: 3.385
      Avg: 5.232
    90pct: 7.206
      Max: 50.404

root@OpenWrt:/opt/home/OpenWrtScripts# ./netperfrunner.sh
2021-11-14 22:01:27 Testing netperf.bufferbloat.net (ipv4) with 4 streams down and up while pinging gstatic.com. Takes about 60 seconds.
 Download:  120.67 Mbps
   Upload:  273.11 Mbps
  Latency: (in msec, 62 pings, 0.00% packet loss)
      Min: 2.714
    10pct: 2.823
   Median: 7.433
      Avg: 16.044
    90pct: 41.023
      Max: 64.027

 

[xiaofan]

More comprehensive test for OpenWRT.
https://forum.openwrt.org/t/speedtest-new-package-to-measure-network-performance/24647
Test results using my Raspberry Pi 400 running OpenWRT 21.02.

root@OpenWrt:/opt/home/OpenWrtScripts# speedtest-netperf.sh
2021-11-14 22:04:36 Starting speedtest for 60 seconds per transfer session.
Measure speed to netperf.bufferbloat.net (IPv4) while pinging gstatic.com.
Download and upload sessions are sequential, each with 5 simultaneous streams.
.............................................................
 Download: 109.04 Mbps
  Latency: [in msec, 61 pings, 0.00% packet loss]
      Min:   2.898
    10pct:   3.023
   Median:   3.215
      Avg:   3.263
    90pct:   3.314
      Max:   7.304
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 58 samples]
     cpu0:   7.3 +/-  3.5  @ 1293 MHz
     cpu1:   5.1 +/-  3.1  @  888 MHz
     cpu2:   3.0 +/-  2.0  @ 1100 MHz
     cpu3:   3.7 +/-  2.3  @  827 MHz
 Overhead: [in % used of total CPU available]
  netperf:   3.4
.............................................................
   Upload: 423.81 Mbps
  Latency: [in msec, 61 pings, 0.00% packet loss]
      Min:   2.431
    10pct:   2.496
   Median:   2.653
      Avg:   8.855
    90pct:  12.528
      Max:  91.897
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 59 samples]
     cpu0:  25.7 +/- 10.6  @ 1518 MHz
     cpu1:   1.7 +/-  1.1  @ 1446 MHz
     cpu2:   3.7 +/-  1.6  @ 1471 MHz
     cpu3:   3.3 +/-  1.3  @ 1413 MHz
 Overhead: [in % used of total CPU available]
  netperf:   2.4

 

[hwzlite]

The SQM-QoS inside OpenWRT is apprently very effective for Bufferbloat on wired as I can get A+ (download 679.6Mbps and Upload of 466.1Mbps). So the CPU inside Raspberr Pi 400 is not too bad, it can effectively limit the upload speed to below 500Mbps, even though it also limits the download speed.
https://www.waveform.com/tools/bufferbloat?test-id=e267ceaf-2c71-4755-8c97-4cbcd29f167f

Interesting.... maybe you can try disable SQM on download (set to "0" setting in GUI?) , in order to free up cpu for SQM upload setting at a high limit.

And then see whether the above setting would max-out cpu by using "TOP" command . ( see FAQ: Measured goodput in speed tests with SQM is considerably lower than without ) during bufferbloat test and hopefully still get a good download and A+ results.

 

[xiaofan]

Interesting.... maybe you can try disable SQM on download (set to "0" setting in GUI?) , in order to free up cpu for SQM upload setting at a high limit.

And then see whether the above setting would max-out cpu by using "TOP" command . ( see FAQ: Measured goodput in speed tests with SQM is considerably lower than without ) during bufferbloat test and hopefully still get a good download and A+ results.

SQM is disabled for download (set to 0 in GUI). Queue disipline is set to fq_codel, script is set to piece_of_cake.qos.

OOkla speedtest download speed is not affected.

Waveform.com Bufferbloat is an A+. Download speed is only 718.8Mbps.
https://www.waveform.com/tools/bufferbloat?test-id=a15edbab-4551-4000-9ddd-0660adf77a19
CPU0 load (using htop) is at most 42% during download and upload. Take node the WAN side is using a Ugreen USB 3.0 to gigabit Ethernet adapter so the CPU load may be on the higher side than those with native gigabit Ethernet WAN port.

 

[xiaofan]

I think it is because waveform.com test server is not able to catch up now. I disabled the QoS SQM rules and the waveform.com bufferbloat test result is still an A+, but I can see that QoS SQM rule is still a bit useful as the results are getting a bit worse even though the result is still an A+.

My test laptop is a new Acer Swift 3 with Ugreen USB 3.0 to 2.5Gbps adapter.

OOkla Speedtest results (CPU0 can go to 100% usage).

Waveform.com Bufferbloat score is still an A+ even though the results seem to be a bit worse. Download is only 693.1 Mbps and upload is only 684.4Mbps. CPU0 load can go to 100% during upload as the speed briefly exceed 1000Mbps.
https://www.waveform.com/tools/bufferbloat?test-id=54afc7fb-fe9c-47c6-9d74-f60bbad15503

 

[xiaofan]

But I can see differences using DSLreports test. It is not a good test for download speed as it is always giving lower speed. But it is a good test for upload as it was able to catch the problem with Singtel ZTE ONT (a bit of issues when upload >800Mbps).

Without SQM QoS -- 490.2/759 Mbps, bufferbloat score is A.
http://www.dslreports.com/speedtest/70039670
WIth SQM QoS -- 398.1/444.7 Mbps bufferbloat score is A+
http://www.dslreports.com/speedtest/70039684

 

[xiaofan]

Just wondering if there is a way to test bufferbloat without using external servers. That will get rid of the impact of the external servers.

Edit to add: it seems to me this is possible with netperf server or flent sever on a Linux PC as the server, and then another PC as the client.

Ref:
https://github.com/richb-hanover/OpenWrtScripts/blob/master/betterspeedtest.shhttps://github.com/richb-hanover/OpenWrtScripts/blob/master/netperfrunner.shhttps://flent.org/intro.html#quick-start

 

[xiaofan]

Just wondering if there is a way to test bufferbloat without using external servers. That will get rid of the impact of the external servers.

Edit to add: it seems to me this is possible with netperf server or flent sever on a Linux PC as the server, and then another PC as the client.

Probably not so meaningful in the end for wired testing. Not so sure if it is useful for wireless testing.
Server -- Intel J4105 mini PC running Ubuntu 21.10.
Client -- Raspbery Pi 400 running OpenWRT 21.02

root@OpenWrt:/opt/home/OpenWrtScripts# ./betterspeedtest.sh -4 -H 192.168.18.169
2021-11-16 19:53:38 Testing against 192.168.18.169 (ipv4) with 5 simultaneous sessions while pinging gstatic.com (60 seconds in each direction)
............................................................
 Download: 941.49 Mbps
  Latency: (in msec, 60 pings, 0.00% packet loss)
      Min: 2.851
    10pct: 2.898
   Median: 3.130
      Avg: 3.151
    90pct: 3.283
      Max: 5.010
............................................................
   Upload: 941.71 Mbps
  Latency: (in msec, 61 pings, 0.00% packet loss)
      Min: 2.777
    10pct: 2.903
   Median: 3.156
      Avg: 3.192
    90pct: 3.505
      Max: 3.772

root@OpenWrt:/opt/home/OpenWrtScripts# ./netperfrunner.sh -4 -H 192.168.18.169
2021-11-16 19:58:39 Testing 192.168.18.169 (ipv4) with 4 streams down and up while pinging gstatic.com. Takes about 60 seconds.
 Download:  934.74 Mbps
   Upload:  901.58 Mbps
  Latency: (in msec, 61 pings, 0.00% packet loss)
      Min: 2.862
    10pct: 2.943
   Median: 3.124
      Avg: 3.156
    90pct: 3.401
      Max: 3.677