FYI/A: Bufferbloat 101

xiaofan · Mar 26, 2024

hwzlite said:
To ensure total "SQM OFF", do make sure all the default fq_codel queue is removed at WAN interface as well. (eg, "tc qdisc del dev eth0 root fq_codel" command )
I guess the results should be similar in latency ranges, but might be more jitter-ish

That is a good point. I will probably carry out this test just for info.

Anyway, usually the user will not purposefully disable the default features.

As seen from Asus router, I believe many other consumer wireless routers may still use old Linux kernels and may not have fq_codel queue features.

xiaofan · Mar 26, 2024

@hwzlite

BTW, thanks for the pointer of Crusader. I like it and I may contribute to the testing side of this cross-platform Open Source projects.

I do not know much about programming but I am pretty good at tinkering on different OS (Linux, macOS, Windows, FreeBSD, etc). So I usually contribute on the testing and technical support side.

All my involvements (since 2005) are more focused on USB (I know quite a bit about USB and MCU (Microcontroller, I know a bit of PIC, AVR and Arm Cortex M MCUs) related Open Source project. This may be my first real involvement in networking related project. Probably I can not do too much as I know much less about the fundamentals of networking. But at least I can contribute on the testing side.

xiaofan · Mar 26, 2024

I will also try flent as it is the more established tool.
https://www.bufferbloat.net/projects/bloat/wiki/Tests_for_Bufferbloat/

xiaofan · Mar 26, 2024

For other users who are interested in this topic, maybe the following page is a good start.
https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/

Then you may find that consumer routers may not much to offer on good QoS/SQM and you may go with alternative router/firewall offerings like OpenWRT/pfSense/etc.

xiaofan · Mar 26, 2024

hwzlite said:
SQM would not have much effect if physical ethernet link at both ends of OpenWRT & ZTE ONT at the same 1Gbps.

My PVE 8.0 Intel N100 Mini PC has quad 2.5G NICs. So there will be some mismatches along the chain.

1) Previous test configuration.
Singtel ZTE ONT 1G LAN port -- TP-Link TL-SG105E managed switch (1G LAN ports) -- OpenWRT WAN (2.5G WAN)
OpenWRT (2.5G LAN) -- RT-AX82U (1G WAN)
RT-AX82U (1G LAN) -- Acer Windows 11 laptop (Ugreen gigabit USB Ethernet adapter)

2) I can try another configuration to have more mismatches.

ZTE ONT 1G LAN port -- TP-Link TL-SG105E managed switch (1G LAN ports) -- OpenWRT WAN (2.5G WAN)
OpenWRT (2.5G LAN) -- Acer Windows 11 laptop (Ugreen 2.5G USB Ethernet adapter)

3) FYI on the Asus RT-AX86U side (another home network) -- so there will be mismatches as well in the chain.

Singtel ZTE ONT 1G LAN port -- TP-Link TL-SG105E managed switch (1G LAN ports) -- Asus RT-AX86U (1G WAN)
Asus RT-AX86U (2.5G LAN) -- Acer Windows 11 laptop (Ugreen 1G or 2.5G USB Ethernet adapter) or Intel N100 mini PC with dual 2.5G NICs

4) You can see both home networks share the same upstream (Singtel ONT + TL-SG105E).

hwzlite · Mar 26, 2024

xiaofan said:
WAN/WAN6 Upload limit 500 Mbps (to work around ZTE ONT issue)

xiaofan said:
My PVE 8.0 Intel N100 Mini PC has quad 2.5G NICs. So there will be some mismatches along the chain.

Aha we missed that upload limit (aka shaping) at 500 Mbps , that prolly explained no major differences with SQM ON or OFF , which nicely mentioned at 1:46mins mark of the video:

xiaofan · Mar 27, 2024

hwzlite said:
Aha we missed that upload limit (aka shaping) at 500 Mbps , that prolly explained no major differences with SQM ON or OFF , which nicely mentioned at 1:46mins mark of the video:

When I say SQM OFF, there is no upload limit at all, the SQM instance is not enabled.
When I say SQM ON, then there is upload limit, the SQM instance is enabled.

BTW, QoS is not enabled either.

hwzlite · Mar 27, 2024

xiaofan said:
Singtel ZTE ONT 1G LAN port -- TP-Link TL-SG105E managed switch (1G LAN ports) -- OpenWRT WAN (2.5G WAN)

xiaofan said:
When I say SQM OFF, there is no upload limit at all, the SQM instance is not enabled.
When I say SQM ON, then there is upload limit, the SQM instance is enabled.

Unless something is cushioning the SQM/Bufferbloat effects in TP-Link TL-SG105E which sit between?
By any chance that the switch 802.3x flow control enabled (disabled by default) ?
Or prolly its Packet Buffer Memory of 1.5Mb sized just right after all.

It would be interesting to see the effect on "Singtel ZTE ONT 1G LAN port -- OpenWRT WAN (2.5G WAN)" bypassing TP-Link TL-SG105E .

Ref: interesting read at Sizing Router Buffers - Small is the New Big

xiaofan · Mar 27, 2024

hwzlite said:
Unless something is cushioning the SQM/Bufferbloat effects in TP-Link TL-SG105E which sit between?
By any chance that the switch 802.3x flow control enabled (disabled by default) ?
Or prolly its Packet Buffer Memory of 1.5Mb sized just right after all.

It would be interesting to see the effect on "Singtel ZTE ONT 1G LAN port -- OpenWRT WAN (2.5G WAN)" bypassing TP-Link TL-SG105E .

Ref: interesting read at Sizing Router Buffers - Small is the New Big

Good info. Thanks.

Sorry but I am not going to change my network infrastructure in this case.

I need to check the TL-SG105E settings, I have not touched the settings since March 2021.
https://forums.hardwarezone.com.sg/...h-tplink-sg108e.5746952/page-2#post-132825987

++++++++++++++++++++++++++++++++++
TP-Link TL-SG105E smart switch
Port 1 -- connect to Singtel ZTE ONT (Default VLAN 1) --> no change

Port 2 -- Asus RT-AX82U with public IP 1 (VLAN 10, untagged, PVID=10), or I can leave this unused if I move the RT-AX82U to the pfsense side as an AP. --> not used now

Port 3 -- pfSense with public IP 2 (VLAN 10, untagged, PVID=10) --> OpenWRT/pfSense/OPNsense VM under Proxmox PVE8.0 using Intel N100 mini PC now, mostly using OpenWRT

Port 4 -- meant for Singtel TV box (VLAN 20, untagged or tagged, PVID=20 or 1), not working, unused now. --> no change, still not used

Port 5 -- Singtel Mesh router with public IP 3 -- Singtel TV Box and Xbox One S (default VLAN 1, passthrough) --> Asus RT-AX86U now with Singtel TV Box, using Singtel VLAN profile.
++++++++++++++++++++++++++++++

xiaofan · Mar 27, 2024

@hwzlite

Just wondering ifyou can update the first post so that new users may get better info. Som of the info in first post may be outdated. Thanks.

Good info here for people who are new to bufferbloat.
https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/
https://www.bufferbloat.net/projects/bloat/wiki/Tests_for_Bufferbloat/ [You can add crusader here].

And you can also consolidate some of your links in this thread and other places in the first post.

Example: new bufferbloat video from the first link. It actually mentions that Asus Gaming router lacks SQM support.

xiaofan · Mar 27, 2024

This one seems to be interesting. Need to see how well it works.
https://github.com/lynxthecat/cake-autorate

I do not like the default SQM scripts from OpenWRT because of the following reason.
CAKE problem: fixed bandwith compromise for variable bandwidth connection like wireless connection.

The Problem: CAKE on variable speed connections forces an unpalatable compromise

The Solution: Set CAKE parameters based on load and latency

xiaofan · Mar 28, 2024

xiaofan said:
The Problem: CAKE on variable speed connections forces an unpalatable compromise

Example test resutls: I need to set the limit to quite low for a wireless connection to get A+ score from Waveform.com bufferbloat test.

Singtel ONT -- TL-SG105E smart switch -- OpenWRT virtual router -- Asus RT-AX82U wireless AP

OpenWRT SQM settings --

WAN/WAN6 Download limit: x Mbps
WAN/WAN6 Upload limit: y Mbps
Cake -- piece_of_cake_qos

1) rules disabled, no limit --> D
https://www.waveform.com/tools/bufferbloat?test-id=09f0ae9a-5850-4037-a96c-be1cac99bd0d

2) x = 950, y = 500 --> B
https://www.waveform.com/tools/bufferbloat?test-id=1edc5139-053a-42f0-bb49-063d215e24b5

3) x = 500, y = 300 --> A+
https://www.waveform.com/tools/bufferbloat?test-id=80f4ab48-7691-4e6e-9f8e-1d598b022cdb

4) Still x = 500, y = 300, but with one thick wall in between --> A
https://www.waveform.com/tools/bufferbloat?test-id=0aae8c08-cbcb-4999-9c3e-96ad9fe9f1b6

5) Still x = 500, y = 300, with two walls in between --> F (weak signal, with either 5GHz or 2.4GHz connection)
5GHz connection
https://www.waveform.com/tools/bufferbloat?test-id=bc014104-a6d1-4587-8ae5-fe479130b0c7
2.4GHz connection
https://www.waveform.com/tools/bufferbloat?test-id=52450cf6-1de6-4b29-af6c-e470a4ae4244

Now there is the question whether the user want to accept the limit or not: 500Mbps download and 300Mbps upload.

And when the signal is weak, then the adopted SQM rules can not really help much either.

xiaofan · Mar 29, 2024

@hwzlite

Just wondering if you can reproduce the following crusader issue under your OpenWRT installation. Thanks.
https://github.com/Zoxc/crusader/issues/20#issuecomment-2026589503

hwzlite · Mar 29, 2024

xiaofan said:
@hwzlite

Just wondering ifyou can update the first post so that new users may get better info. Som of the info in first post may be outdated. Thanks.

Good info here for people who are new to bufferbloat.
https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/
https://www.bufferbloat.net/projects/bloat/wiki/Tests_for_Bufferbloat/ [You can add crusader here].

And you can also consolidate some of your links in this thread and other places in the first post.

Example: new bufferbloat video from the first link. It actually mentions that Asus Gaming router lacks SQM support.

Finally updated the first post, thx for the head-out :grin:

xiaofan · Mar 29, 2024

Asus Merlin does support SQM, unfortunately the CPU in my Asus RT-AX86U is not up to task once I enabled Cake SQM.
https://www.snbforums.com/threads/adaptive-cake-qos.75164/
https://github.com/dave14305/FlexQoS

For the Asue Merlin bundled Cake installation, I can only use up to 200Mbps download/upload limit.

Using wired test to make sure the bandwidth limit is working.

Bash:

root@ubuntu2204ct1:~# ./speedtest

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     2.57 ms   (jitter: 0.23ms, low: 2.41ms, high: 2.75ms)
    Download:   197.37 Mbps (data used: 91.6 MB)
                  1.92 ms   (jitter: 0.39ms, low: 1.55ms, high: 6.41ms)
      Upload:   197.40 Mbps (data used: 181.7 MB)
                  1.85 ms   (jitter: 0.55ms, low: 1.28ms, high: 16.03ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/f14a2a19-9d90-444e-bbb9-9ac7aeb98d27

Near range, Acer Windows 11 laptop, wireless connection (3m) -- A+
https://www.waveform.com/tools/bufferbloat?test-id=93c50750-4b0e-4b1b-b8c8-374657b52e88

One thick wall in between, wireless connection -- A
https://www.waveform.com/tools/bufferbloat?test-id=5d1ae214-16a8-4c18-9f4b-cd2b51f8921d

Two walls in between (weak signal, 5GHz band) -- F
https://www.waveform.com/tools/bufferbloat?test-id=2f1f2e16-39b3-4aec-b6e9-d7fde0d1a579

xiaofan · Mar 29, 2024

Asus default Adaptive QoS, Learn-From-Home profile.

Interestingly it does not seem to limit wired bandwidth and wireless bandwidth.

Bash:

root@ubuntu2204ct1:~# ./speedtest -s 13623

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     2.50 ms   (jitter: 0.12ms, low: 2.44ms, high: 2.70ms)
    Download:   935.51 Mbps (data used: 456.1 MB)
                  3.61 ms   (jitter: 0.43ms, low: 2.59ms, high: 6.52ms)
      Upload:   934.85 Mbps (data used: 1.1 GB)
                 32.60 ms   (jitter: 34.71ms, low: 5.01ms, high: 404.93ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/827638d7-600f-4faf-b54e-5d27504b1478

Near range Acer Windows 11 laptop, wireless connection (about 3m away) -- A
https://www.waveform.com/tools/bufferbloat?test-id=d0ea7525-2555-4c0e-ba97-2c489678d084

One thick wall in between -- B
https://www.waveform.com/tools/bufferbloat?test-id=82b911da-1d7a-46de-81db-e0ea4d33d11e

Two walls in between (weak signal, 5GHz band) -- still F
https://www.waveform.com/tools/bufferbloat?test-id=a2558211-e173-482a-97a3-2e5ec23b4e93

xiaofan · Mar 29, 2024

Without any Asus QoS Settings (QoS disabled)

Near range Acer Windows 11 laptop, wireless connection (about 3m away) -- A
https://www.waveform.com/tools/bufferbloat?test-id=3952c126-4c29-414b-9852-a6333ae8ab75

One thick wall in between -- B
https://www.waveform.com/tools/bufferbloat?test-id=39807f3e-3cbe-4c38-8759-5164e484d5a1

Two walls in between (weak signal, 5GHz band) -- still F, but worse than the previous in terms of upload latency
https://www.waveform.com/tools/bufferbloat?test-id=06922732-a5e7-4261-9bb9-5c7b1c5f5db1

xiaofan · Mar 29, 2024

hwzlite said:
Hhhmmm..... can test under "singapore.starlink.taht.net" :

"We do not rate limit our servers, however we don’t feel you can trust the results much above 800Mbits in Linode’s cloud. Bare metal in the cloud we have one of, good to 25Gbits, if you want access to that, please let me know, privately. (a huge thank you to Equinix for this donation! )"

Maybe can psycho Mr Dave for more bandwidth if needed

It was not working from my Linux LXC containers. Today I tried to use macOS (Mac Mini M1 2020) and it does not work either.

Guess the server is down.

But it is interesting to see how Crusader is performing versus flent (local test only).
https://github.com/Zoxc/crusader/issues/14

xiaofan · Mar 29, 2024

hwzlite said:
Hhhmmm..... can test under "singapore.starlink.taht.net" :

"We do not rate limit our servers, however we don’t feel you can trust the results much above 800Mbits in Linode’s cloud. Bare metal in the cloud we have one of, good to 25Gbits, if you want access to that, please let me know, privately. (a huge thank you to Equinix for this donation! )"

Maybe can psycho Mr Dave for more bandwidth if needed

@hwzlite

I guess by Mr Dave you mean Mr Dave Täht, right?

You can see he is testing crusader and recommends users of flent to try out crusader.

Reference:
https://github.com/Zoxc/crusader/issues/14
https://github.com/tohojo/flent/issues/233#issuecomment-1981012766

Probably he has given up on the flent server.

Bash:

mcuee@debian12vmn100new:~/build$ ping -4 -c 4 singapore.starlink.taht.net
PING  (172.105.113.30) 56(84) bytes of data.

---  ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3051ms

mcuee@debian12vmn100new:~/build$ ping -6 -c 4 singapore.starlink.taht.net
PING singapore.starlink.taht.net(2400:8901::f03c:92ff:fef8:9fc0 (2400:8901::f03c:92ff:fef8:9fc0)) 56 data bytes

--- singapore.starlink.taht.net ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3053ms

xiaofan · Mar 29, 2024

BTW, Rich Brown (https://github.com/richb-hanover) is also very active in Crusade testing.
https://github.com/Zoxc/crusader/issues/14

He seems to host the test server netperf.bufferbloat.net which is down as well.

Bash:

mcuee@debian12vmn100new:~/build$ ping netperf.bufferbloat.net -c 4
PING atl.richb-hanover.com (23.226.232.80) 56(84) bytes of data.

--- atl.richb-hanover.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3063ms

FYI/A: Bufferbloat 101

High Supremacy Member

High Supremacy Member

High Supremacy Member

High Supremacy Member

High Supremacy Member

Master Member

High Supremacy Member

Master Member

High Supremacy Member

High Supremacy Member

High Supremacy Member

High Supremacy Member

High Supremacy Member

Master Member

High Supremacy Member

High Supremacy Member

High Supremacy Member

High Supremacy Member

High Supremacy Member

High Supremacy Member