FYI/A: Bufferbloat 101

[xiaofan]

As well to reiterate that fq_codel SQM by itself is already enabled internally as the default Queueing mechanism since Linux version is ≥ 4.12 for all of modern Linux-based distros, routers, switches or might even in ONT/R ....etc , where most lay users did not even know of its existence where SQM already working in the background.

fq_codel SQM will have great effect working together with shaper/QOS to limit max bandwidth if it is on the bottleneck/mismatch links, eg. 1Gbps home ethernet link connected to 500Mbps broadband , 2.5Gbps home ethernet link connected to 1Gb broadband.
If both on line rate , eg. 1Gbps home ethernet link connected to 1Gbps broadband , fq_codel SQM will do the job nicely without the need of shaper/QOS .

Indeed this is the default for more recent version of Linux and OpenWRT. Last time I tested it -- only the PVE Linux containers does not use fq_codel.
https://forums.hardwarezone.com.sg/...-gen-alder-lake.6958302/page-2#post-150130537

Just find out Asus also does not use it, probably because the Linux kernel used is old.

xiaofan@RT-AX86U-2D40:/tmp/home/root# tc qdisc show
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth5 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev spu_us_dummy root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev spu_ds_dummy root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth6 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth7 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

xiaofan@RT-AX86U-2D40:/tmp/home/root# uname -a
Linux RT-AX86U-2D40 4.1.52 #2 SMP PREEMPT Mon Feb 26 22:57:01 EST 2024 aarch64 ASUSWRT-Merlin

 

[xiaofan]

I have created an issue in Crusader github.
https://github.com/Zoxc/crusader/issues/17

Issue resolved. Looks like the upload is the issue -- can only accomodate 16 streams.

Still using wireless.

Download test results:

PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test xxxxxxxx.asuscomm.com --download
Connected to server 219.75.xx.xxxx:35481
Latency to server 4.80 ms
Testing download...
Writing data...
Saved raw data as data 2024.03.24 21-16-34.crr
Saved plot as plot 2024.03.24 21-16-34.png

Upload test results: 16 streams only

PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test xxxxxxxx.asuscomm.com --streams 16 --upload
Connected to server 219.75.xx.xxx:35481
Latency to server 3.97 ms
Testing upload...
Writing data...
Saved raw data as data 2024.03.24 21-22-43.crr
Saved plot as plot 2024.03.24 21-22-43.png

 

[xiaofan]

Using OpenWRT stock SQM settings.
WAN/WAN6 Download limit 950Mbps (basically no limit)
WAN/WAN6 Upload limit 500 Mbps (to work around ZTE ONT issue)
Cake -- piece_of_cake_qos

I can not test download + upload at the same time (test failed) as I have only single connection in realit

Crusader Results for download

Crusader Results for upload -- results seem to be improved

SQM rules seems to work.

PS C:\work\speedtest\ookla-speedtest-1.2.0-win64> .\speedtest.exe -s 13623

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     3.93 ms   (jitter: 0.54ms, low: 3.60ms, high: 4.53ms)
    Download:   792.48 Mbps (data used: 1.4 GB)
                 22.54 ms   (jitter: 11.49ms, low: 4.25ms, high: 113.97ms)
      Upload:   446.46 Mbps (data used: 756.5 MB)
                 11.08 ms   (jitter: 7.06ms, low: 3.58ms, high: 88.45ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/37dce68a-bdcf-4256-bcb3-366d1f4ad1aa
PS C:\work\speedtest\ookla-speedtest-1.2.0-win64> .\speedtest.exe -s 13623

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     4.38 ms   (jitter: 0.41ms, low: 4.21ms, high: 5.69ms)
    Download:   800.28 Mbps (data used: 1.4 GB)
                 20.47 ms   (jitter: 9.55ms, low: 4.57ms, high: 66.32ms)
      Upload:   415.86 Mbps (data used: 713.4 MB)
                 14.71 ms   (jitter: 9.04ms, low: 3.27ms, high: 83.66ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/3493521b-6bf0-4618-b7f0-29b7a7c2a367

 

[hwzlite]

@hwzlite
Please help to comment on the above results. Thanks.

Local wireless test: Asus RT-AX86U

Server -- Proxmox PVE 8.0 LxC container (Ubuntu 22.04), connected to RT-AX86U 2.5G LAN port
Client -- Acer Windows 11 machine (2021 model) , Intel AX201 WiFi 6 adapter.

.....

The delta fluctuation at upload test between ~10 to 12 sec mark seems to be high.
Retest to confirm just a one-time “A glitch in the matrix” at the first test?
Can set a longer "load duration" setting (default 5sec) to observe further.

 

[xiaofan]

Looks like upload can be an issue with Local Asus RT-AX86U Wireless Test -- tested 4 times this morning at around 8am to 8:15am.

Local wireless test: Asus RT-AX86U
Server -- Proxmox PVE 8.0 LxC container (Ubuntu 22.04), connected to RT-AX86U 2.5G LAN port
Client -- Acer Windows 11 machine (2021 model) , Intel AX201 WiFi 6 adapter.

Two times with warnings.

PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test 192.168.50.12
Connected to server 192.168.50.12:35481
Latency to server 1.93 ms
Testing download...
Testing upload...
Testing both download and upload...
Warning: Load termination timed out. There may be residual untracked traffic in the background.
Writing data...
Saved raw data as data 2024.03.25 08-01-18.crr
Saved plot as plot 2024.03.25 08-01-18.png
PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test 192.168.50.12
Connected to server 192.168.50.12:35481
Latency to server 1.60 ms
Testing download...
Testing upload...
Testing both download and upload...
Warning: Load termination timed out. There may be residual untracked traffic in the background.
Writing data...
Saved raw data as data 2024.03.25 08-02-09.crr
Saved plot as plot 2024.03.25 08-02-09.png

The other two times got no warnings.

PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test 192.168.50.12
Connected to server 192.168.50.12:35481
Latency to server 2.27 ms
Testing download...
Testing upload...
Testing both download and upload...
Writing data...
Saved raw data as data 2024.03.25 08-10-47.crr
Saved plot as plot 2024.03.25 08-10-47.png
PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test 192.168.50.12
Connected to server 192.168.50.12:35481
Latency to server 1.76 ms
Testing download...
Testing upload...
Testing both download and upload...
Writing data...
Saved raw data as data 2024.03.25 08-11-14.crr
Saved plot as plot 2024.03.25 08-11-15.png

 

[xiaofan]

Local wired test: Asus RT-AX86U
Server -- Proxmox PVE 8.0 LxC container (Ubuntu 22.04), connected to RT-AX86U 2.5G LAN port
Client -- Acer Windows 11 machine (2021 model) , Ugreen USB 3.0 to gigabit Ethernet adapter

Upload is always worse than Download. It is said that the Singtel issued ZTE ZXHN F620 (I got it in Sept 2018) may be the issue.

PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test 192.168.50.12
Connected to server 192.168.50.12:35481
Latency to server 0.95 ms
Testing download...
Testing upload...
Testing both download and upload...
Writing data...
Saved raw data as data 2024.03.25 08-04-07.crr
Saved plot as plot 2024.03.25 08-04-07.png
PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test 192.168.50.12
Connected to server 192.168.50.12:35481
Latency to server 0.89 ms
Testing download...
Testing upload...
Testing both download and upload...
Writing data...
Saved raw data as data 2024.03.25 08-04-34.crr
Saved plot as plot 2024.03.25 08-04-34.png

 

[xiaofan]

Wired test over the internet: two different networks.

Server -- Proxmox PVE 8.0 LxC container (Ubuntu 22.04), connected to RT-AX86U 2.5G LAN port
Client -- Acer Windows 11 machine (2021 model) , Ugreen USB 3.0 to gigabit Ethernet adapter, wired connection to OpenWRT virtual router (another network) through Asus RT-AX82U AP.

Using OpenWRT stock SQM settings.
WAN/WAN6 Download limit 950Mbps (basically no limit)
WAN/WAN6 Upload limit 500 Mbps (to work around ZTE ONT issue)
Cake -- piece_of_cake_qos

I can not test download + upload at the same time (test failed) as I have only single connection in realit

PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test xxxxxxxx.asuscomm.com --download
Connected to server 219.75.xx.xxx:35481
Latency to server 1.89 ms
Testing download...
Writing data...
Saved raw data as data 2024.03.25 08-07-09.crr
Saved plot as plot 2024.03.25 08-07-09.png
PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test xxxxxxxx.asuscomm.com --upload
Connected to server 219.75.xx.xxx:35481
Latency to server 2.42 ms
Testing upload...
Writing data...
Saved raw data as data 2024.03.25 08-07-42.crr
Saved plot as plot 2024.03.25 08-07-42.png

Download

Upload

 

[xiaofan]

I will try a few more tests later this week.

OpenWRT side -- local test and internet test, wired and wireless (using RT-AX82U AP), with and without the SQM rules. For wireless, I will try the strong and weak signal areas.

Asus side -- probably not to do much as I do not trust Asus QoS performance. But I will try also weak signal area tests.

 

[xiaofan]

@hwzlite

You are welcome to comment on the posted results.

For OpenWRT side, you are also welcome to suggest other tests, like other SQM rules, etc.

I actually like Crusader than Waveform.com as I can control the tests and it is not limiting the download/upload speed for internet test.

 

[hwzlite]

I actually like Crusader than Waveform.com as I can control the tests and it is not limiting the download/upload speed for internet test.

Not to mention that both Crusader & Flent RRUL can test bi-directional traffic simulating real-world under worst case conditions, whereas online tests such as speedtest , waveform ..etc usually limit to download & upload only.

Typical non-SQM/rate-limit tuned examples from Crusader bi-directional at end phase test also show the detrimental saturated upload starving the download speed of needed ACK packets.

 

[xiaofan]

Not to mention that both Crusader & Flent RRUL can test bi-directional traffic simulating real-world under worst case conditions, whereas online tests such as speedtest , waveform ..etc usually limit to download & upload only.

Typical non-SQM/rate-limit tuned examples from Crusader bi-directional at end phase test also show the detrimental saturated upload starving the download speed of needed ACK packets.

I will try Flent as well (Flent does not seem to work under Windows but I can use Linux/macOS) using two-machine setup (server and client). The servers hosted on netperf.bufferbloat.net are not good.
https://github.com/tohojo/flent/blob/master/doc/quickstart.rst

Flent Windows issue
https://github.com/tohojo/flent/issues/233

 

[hwzlite]

The servers hosted on netperf.bufferbloat.net are not good.

Hhhmmm..... can test under "singapore.starlink.taht.net" :

"We do not rate limit our servers, however we don’t feel you can trust the results much above 800Mbits in Linode’s cloud. Bare metal in the cloud we have one of, good to 25Gbits, if you want access to that, please let me know, privately. (a huge thank you to Equinix for this donation! )"

Maybe can psycho Mr Dave for more bandwidth if needed

 

[xiaofan]

The good thing about some online tests is that they also start to provide more info about latency, still not as comprehensive as waveform.com though.

Example 1: Cloudflare Speed Test
100kB, 1MB and 10MB download/upload Speed and latency tests
https://speed.cloudflare.com/

Example 2: OOKla SpeedTest CLI version
average latency, jitter, min and max latency during idle/download/upload
https://www.speedtest.net/apps/cli

Reference: SpeedTest tools
https://forums.hardwarezone.com.sg/...e-network-performance.6641164/#post-137961429

 

[xiaofan]

Hhhmmm..... can test under "singapore.starlink.taht.net" :

"We do not rate limit our servers, however we don’t feel you can trust the results much above 800Mbits in Linode’s cloud. Bare metal in the cloud we have one of, good to 25Gbits, if you want access to that, please let me know, privately. (a huge thank you to Equinix for this donation! )"

Maybe can psycho Mr Dave for more bandwidth if needed

Unfortunately it does not seem to work. Tested using a Debian 12 VM under PVE 8.0 on my Intel N100 mini PC. Not sure if has something to do with IPv6.

I have not used flent for quite a few years, need to check again to see if I can resolve the issues or not.

mcuee@debian12vmn100:~/build$ flent rrul -p all_scaled -l 60 -H singapore.starlink.taht.net -t wiredtest_debianvm_asus_ax86u -o wiredtest_debianvm_asus_ax86u.png -L
Starting Flent 2.1.1 using Python 3.11.2.
Starting rrul test. Expected run time: 70 seconds.
WARNING: Command produced no valid data.
Runner class: NetperfDemoRunner
Command: /usr/bin/netperf -P 0 -v 0 -D -0.20 -6 -Y 0x20,0x20 -H singapore.starlink.taht.net -p 12865 -t TCP_STREAM -l 60 -F /dev/urandom -f m   --    -H singapore.starlink.taht.net -k THROUGHPUT,LOCAL_CONG_CONTROL,REMOTE_CONG_CONTROL,TRANSPORT_MSS,LOCAL_TRANSPORT_RETRANS,REMOTE_TRANSPORT_RETRANS,LOCAL_SOCKET_TOS,REMOTE_SOCKET_TOS,DIRECTION,ELAPSED_TIME,PROTOCOL,LOCAL_SEND_SIZE,LOCAL_RECV_SIZE,REMOTE_SEND_SIZE,REMOTE_RECV_SIZE,LOCAL_BYTES_SENT,LOCAL_BYTES_RECVD,REMOTE_BYTES_SENT,REMOTE_BYTES_RECVD
Return code: None
Stdout:
Stderr:
WARNING: Command produced no valid data.
... skipped ...
... skipped ...
... skipped ...
Runner class: NetperfDemoRunner
Command: /usr/bin/netperf -P 0 -v 0 -D -0.20 -6  -H singapore.starlink.taht.net -p 12865 -t UDP_RR -l 70 -F /dev/urandom    -- -e 2   -H singapore.starlink.taht.net -k THROUGHPUT,LOCAL_CONG_CONTROL,REMOTE_CONG_CONTROL,TRANSPORT_MSS,LOCAL_TRANSPORT_RETRANS,REMOTE_TRANSPORT_RETRANS,LOCAL_SOCKET_TOS,REMOTE_SOCKET_TOS,DIRECTION,ELAPSED_TIME,PROTOCOL,LOCAL_SEND_SIZE,LOCAL_RECV_SIZE,REMOTE_SEND_SIZE,REMOTE_RECV_SIZE,LOCAL_BYTES_SENT,LOCAL_BYTES_RECVD,REMOTE_BYTES_SENT,REMOTE_BYTES_RECVD
Return code: None
Stdout:
Stderr:
WARNING: Command produced no valid data.
Runner class: PingRunner
Command: /usr/bin/fping6  -D -p 200 -c 350 -t 140000   singapore.starlink.taht.net
Return code: None
Stdout:
Stderr:
ERROR: No data to aggregate. Run with -L and check log file to investigate.

 

[xiaofan]

Wired test over the internet: two different networks.

Server -- Proxmox PVE 8.0 LxC container (Ubuntu 22.04), connected to RT-AX86U 2.5G LAN port
Client -- Acer Windows 11 machine (2021 model) , Ugreen USB 3.0 to gigabit Ethernet adapter, wired connection to OpenWRT virtual router (another network) through Asus RT-AX82U AP.

Using OpenWRT stock SQM settings.
WAN/WAN6 Download limit 950Mbps (basically no limit)
WAN/WAN6 Upload limit 500 Mbps (to work around ZTE ONT issue)
Cake -- piece_of_cake_qos

I can not test download + upload at the same time (test failed) as I have only single connection in realit

Wireless test over the internet: two different networks.

Server -- Proxmox PVE 8.0 LxC container (Ubuntu 22.04), connected to RT-AX86U 2.5G LAN port
Client -- Acer Windows 11 machine (2021 model) with Intel AX201 WiFi 6 adapter , wireless connection to OpenWRT virtual router (another network) through Asus RT-AX82U AP. Close reange, 4m away from the wireless AP.

Using OpenWRT stock SQM settings.
WAN/WAN6 Download limit 950Mbps (basically no limit)
WAN/WAN6 Upload limit 500 Mbps (to work around ZTE ONT issue)
Cake -- piece_of_cake_qos

I can not test download + upload at the same time (test failed) as I have only single connection in reality.

Results -- there is a spike in the beginning of download/upload, especialy for upload. Overall SQM is quite effective to reduce the bufferbloat for wireless testing (close range).

1) Download with SQM rules ON

2) Upload with SQM rules ON

Once I disable the SQM rules, the results will get much worse, especially for upload.

3) Download with SQM rules OFF

4) Upload with SQM rules OFF

 

[xiaofan]

Wired test over the internet: two different networks.

Server -- Proxmox PVE 8.0 LxC container (Ubuntu 22.04), connected to RT-AX86U 2.5G LAN port
Client -- Acer Windows 11 machine (2021 model) , Ugreen USB 3.0 to gigabit Ethernet adapter, wired connection to OpenWRT virtual router (another network) through Asus RT-AX82U AP.

Using OpenWRT stock SQM settings.
WAN/WAN6 Download limit 950Mbps (basically no limit)
WAN/WAN6 Upload limit 500 Mbps (to work around ZTE ONT issue)
Cake -- piece_of_cake_qos

I can not test download + upload at the same time (test failed) as I have only single connection in reality

From the above test results:

1) Wired Download test with SQM ON

2) Wired Upload test with SQM ON

I just carried out the wired tests with SQM OFF. It seems to me there are no major differences, unlike wireless tests.

3) Wired Download test with SQM OFF

4) Wired Upload test with SQM OFF

 

[hwzlite]

Using OpenWRT stock SQM settings.
WAN/WAN6 Download limit 950Mbps (basically no limit)
WAN/WAN6 Upload limit 500 Mbps (to work around ZTE ONT issue)
Cake -- piece_of_cake_qos
.....
I just carried out the wired tests with SQM OFF. It seems to me there are no major differences, unlike wireless tests.

SQM would not have much effect if physical ethernet link at both ends of OpenWRT & ZTE ONT at the same 1Gbps.

To ensure total "SQM OFF", do make sure all the default fq_codel queue is removed at WAN interface as well. (eg, "tc qdisc del dev eth0 root fq_codel" command )
I guess the results should be similar in latency ranges, but might be more jitter-ish

 

[xiaofan]

One more issues with crusader and IPv6 over the internet.
https://github.com/Zoxc/crusader/issues/9

 

[hwzlite]

@xiaofan
One more issues with crusader and IPv6 over the internet.
https://github.com/Zoxc/crusader/issues/9

Aside TCP & UDP , did u also include "ICMP" in rule on prototol selection?

 

[xiaofan]

Aside TCP & UDP , did u also include "ICMP" in rule on prototol selection?

OpenWRT has the default rules to allow ICMPv6 Input and Forward.

Then I add three more rules.