FYI/A: Bufferbloat 101

[xiaofan]

I did adjusted my bandwidth limiter. I tried running without briefly and the latency did spike a little when I saturated the connection.

Thanks for the updates.

pfSense has quite comprehensive traffic shaper functions.
https://docs.netgate.com/pfsense/en/latest/trafficshaper/index.html

When you say bandwidth limiter, do you mean you only use the following and not ALTQ?
https://docs.netgate.com/pfsense/en/latest/trafficshaper/limiters.html

 

[Mach3.2]

When you say bandwidth limiter, do you mean you only use the following and not ALTQ?
https://docs.netgate.com/pfsense/en/latest/trafficshaper/limiters.html

Yes I'm using this.

 

[xiaofan]

For my use case...... the simple answer is no, or maybe I am just not too bothered by it enough to implement some form of QOS/SQM. My personal experiences/use cases as below:​

1. When gaming and BT-ing, my firewall CPU rarely hits 60% load and ping still stays 2-5ms, for local servers, playing games like WWZ and Helldiver II. Even browsing is very responsive since I am also running Unbound on pfsense. I don't play overseas servers since all my (old-timer) gaming kakis are locals. This is on 10G MR.​
2. When off-prem backing my servers/snapshots/data to/from my work-lab, I schedule it late at night around 2 - 6 am, when I am rarely still awake and gaming due to ageing eyes. So there is not much competition for latency and bandwidth during that period.​
3. Since I am on 2x ISPs with all internal traffic segmented to VLANs, where all mobile devices, IOTs, kids' and wifey's systems are all on separate VLANs with all their WAN mapped to an M1 1G fibre. Only the breadwinner (aka me) uses the 10G MR to myself for work and leisure. I will say "rarely" there will be sufficient bandwidth/latency competition to justify for some form of management for a home environment.​

In the end, really up to you. If it works, and you notice the difference (like @Henry Ng browser load a bit faster experience), do continue using it. If you are keen to learn, squeeze every ounce of optimization out of it. Who knows, you learn something you never knew (or I never knew) through self experimenting and I think that mindset is valuable.

For enterprise networks with lots of traffic and bottlenecks, I may consider. But that can be achieved using the L7 (OSI) policy on enterprise network appliances and a well-optimised internal network running multiple L3 switches and routers.​

Thanks a lot for the detailed reply.

I am actually interesting in how your pfSense router CPU can handle the ALTQ or Bandwidth limiter properly or not. For example, if you set the download bandwidth limiter for your desktop PC (LAN client) to 5Gbps and then run OOkla SpeedTest, does it really work at 5Gbps?
https://docs.netgate.com/pfsense/en/latest/trafficshaper/index.html

It is interesting that MediaTek Filogic 880 CPU mentions the following -- HW QoS acceleration. High end Qualcomm CPU for home routers also have NPU. No idea how good the HW QoS performs at 10Gbps (or 8.2Gbps) for consumer 10G capable WiFi 7 routers though.

https://www.mediatek.com/products/broadband-wifi/mediatek-filogic-880
The in-chip, MediaTek-designed Network Processing Unit (NPU) provides hardware QoS acceleration and Tunneling Offload Engine for VLAN / PPTP / L2TP / GRE. It goes beyond others to work with both Wi-Fi and Ethernet connections, ensuring performance at full speeds while also offloading the main application processor.

 

[lobukong]

Status Quo Bias any1 ?

Only for those who have personally experienced the pain bitten by bufferbloat in one form or another...., SHOVEL (SQM) is the ways and as well dig out in whatever "perfect storm" events.

Apparently, a shovel is only considered useful if used by real experts.
If real experts use the shovel, it magically becomes useful, akin to being 开光。
Without 开光, non real experts' opinions are regarded as invalid ("everybody is entitled to his own opinion"). Even analogies are regarded as offensive analogies.
After 开光, useless shovel suddenly became interesting shovels. Even though some real experts haven't evaluated the usefulness of shovels, their feedbacks remains sought after.

So the forum cannot lose real experts just like the West cannot lose Jerusalem.

 

[xiaofan]

Apparently, a shovel is only considered useful if used by real experts.
If real experts use the shovel, it magically becomes useful, akin to being 开光。
Without 开光, non real experts' opinions are regarded as invalid ("everybody is entitled to his own opinion"). Even analogies are regarded as offensive analogies.
After 开光, useless shovel suddenly became interesting shovels. Even though some real experts haven't evaluated the usefulness of shovels, their feedbacks remains sought after.

So the forum cannot lose real experts just like the West cannot lose Jerusalem.

I am not offended by the analogy. I am old enough (through dial-up, ADSL, Cable and Fibre internet, through DOS, Windows 95 to Windows 11, through Slackware/Rednat Linux to Ubuntu) so that I will not be offended easily.

But to be straight -- your comments or analogies like this are without technical merit and like a troll to me already.

Let's focus on the technical side. It is perfectly okay to have different opinons on a technical side. And it is often useful to learn from the other side -- I actually play with different QoS/SQM as well on OpenWRT.

You may want to compare your contribution to this forum with @hwzlite. He has always provided very good technical insight with data to backup, which I will respect even though I may not agree with his point in this paticular issue. In fact I do agree there are some use cases he is actually right.

I am not an expert in home networking area -- that is why I mentioned to quite some pms I do not answer technical questions using pms, but prefer to answer in the forum so that real experts can correct me in case I am wrong.

Prove me wrong using technical side as I see that you are pretty capable in the technical side. But please, do not use useless analogy to try to argue for the sake of argument.

 

[lobukong]

I am not offended by the analogy. I am old enough (through dial-up, ADSL, Cable and Fibre internet, through DOS, Windows 95 to Windows 11, through Slackware/Rednat Linux to Ubuntu) so that I will not be offended easily.

But to be straight -- your comments or analogies like this are without technical merit and like a troll to me already.

Let's focus on the technical side. It is perfectly okay to have different opinons on a technical side. And it is often useful to learn from the other side -- I actually play with different QoS/SQM as well on OpenWRT.

You may want to compare your contribution to this forum with @hwzlite. He has always provided very good technical insight with data to backup, which I will respect even though I may not agree with his point in this paticular issue. In fact I do agree there are some use cases he is actually right.

I am not an expert in home networking area -- that is why I mentioned to quite some pms I do not answer technical questions using pms, but prefer to answer in the forum so that real experts can correct me in case I am wrong.

Prove me wrong using technical side as I see that you are pretty capable in the technical side. But please, do not use useless analogy to try to argue for the sake of argument.

Well, I liked the analogy and I think it make sense. You can elaborate which point in the analogy is trolling? Analogies are usually used to make concepts easier to understand. Accusing using of analogy to be arguing for the sake of argument is beyond my understanding.
But I don't need to reply to you if you don't value my input. So congraz you'll no longer gets trolling messages from me.

 

[xiaofan]

Well, I liked the analogy and I think it make sense. You can elaborate which point in the analogy is trolling? Analogies are usually used to make concepts easier to understand. Accusing using of analogy to be arguing for the sake of argument is beyond my understanding.
But I don't need to reply to you if you don't value my input. So congraz you'll no longer gets trolling messages from me.

Wonderful. The "开光“ word is trolling to me and may be offending to others.

You have very good technical contributions to other thread. Just not this one.

 

[TanKianW]

Status Quo Bias any1 ?

Only for those who have personally experienced the pain bitten by bufferbloat in one form or another...., SHOVEL (SQM) is the ways and as well dig out in whatever "perfect storm" events.

Now still use "shovel" meh? Even the recent sub-sea cabling project I am involved (SG-> Guam ->US) is using excavators and specialised equipment for directional drilling (DD) liao, only initial trial pits still use shovel nia.

Looking at the 10Tbps/fiber-pairs (FPs) coming inland (from beach manhole to landing stations) on rows of fibre network terminals running custom appliance software cures all sense of "bufferbloat". No worries, "bufferbloat antidote" is on the way.

Analogy <-> Reality.​

 

[TanKianW]

Apparently, a shovel is only considered useful if used by real experts.
If real experts use the shovel, it magically becomes useful, akin to being 开光。
Without 开光, non real experts' opinions are regarded as invalid ("everybody is entitled to his own opinion"). Even analogies are regarded as offensive analogies.
After 开光, useless shovel suddenly became interesting shovels. Even though some real experts haven't evaluated the usefulness of shovels, their feedbacks remains sought after.

So the forum cannot lose real experts just like the West cannot lose Jerusalem.

What is your definition of a "real expert"? Gauge by experience, competence, accomplishment or profession? Or just by what some posted on the forum? Or self-proclaimed? I know I will not claim someone as an "expert" just from their forum posts (at most kb warriors). Maybe if real expert, may not even be on this forum liao.

Even if any of us "are" the so-called experts in real life, we are just "learning to be an expert", but not an endpoint or destination, it will be an unending journey. There is always something which "real expert" will not know or learnt, that is the reason we have forum like these that share the knowledge which may also spur discussion in the process (not argument, like what most see it). Actually active and heated discussion is a common thing in the western tech world environment (which I once worked in), just that in Asian culture it seemed/sounded/perceived as more negative.

My own opinion: Based on your analogy, I think "non-real experts" without “开光 shovel” are not regarded as invalid. At least to me. At times I prefer view points from "non-real experts" coz their use case always kept me thinking. Anyone of us who do not use/adopt a certain solution, do not mean it is not valid or not useful, or has the rights to say it is not valid, just "not of use to me". This is a forum where not everyone agrees with everyone, but does not mean everyone is right!? Make sense? Chill bro....I also don't think it is so serious until regarded as offensive analogies. I do prefer analogies at times, but an overdose may make it sound confusing, the right dose is crucial, or maybe I am just stupid?

Hey, there is no need for a "real expert" (I know I'm not) to evaluate/validate the usefulness of the shovel so you can start implementing and using it (long passed that stage). Using whichever suits your use case is more important than following mine. Many times I have already been using viable and better solutions (I think) which most have not even started using for home use, Eg. ZFS/SDS/pfsense, but that does not stop me.

In the end, based on my definition that the forum has no "real experts".........using your lingo...."The West has long lost J*****lem".... ​

 

[xiaofan]

Flent RRUL is the (kind-of-overkill ) way, which i already demo it in my n100 threads.

Another test suite i recently come across is Crusader , lease-overkill which is nicely available in multiplatform :

As for waveform testing, checkout the threads on "[Bloat] We built a new bufferbloat test and keen for feedback" , which got vetting & scrutinized by the gurus there, especially Toke Høiland-Jørgensen who "wrote the book" (literally) and got his PhD from this topic .

As well to reiterate that fq_codel SQM by itself is already enabled internally as the default Queueing mechanism since Linux version is ≥ 4.12 for all of modern Linux-based distros, routers, switches or might even in ONT/R ....etc , where most lay users did not even know of its existence where SQM already working in the background.

fq_codel SQM will have great effect working together with shaper/QOS to limit max bandwidth if it is on the bottleneck/mismatch links, eg. 1Gbps home ethernet link connected to 500Mbps broadband , 2.5Gbps home ethernet link connected to 1Gb broadband.
If both on line rate , eg. 1Gbps home ethernet link connected to 1Gbps broadband , fq_codel SQM will do the job nicely without the need of shaper/QOS .

Very useful info. I will spend more time reading. The waveform.com thread seems to be interesting.

I actually tried to use Flent RRUL last time but the difficulty is to have a valid test server over the internet.

For crusader, how do you set up the server?

My setup was mostly 1Gbps home Ethernet link connected to 1Gbs Fibre Internet. But now I have 2.5G home network connected to 1Gbps Fibre internet. I will carry out some tests to see if tat makes any differences.

 

[hwzlite]

or crusader, how do you set up the server?

Currently I cannot do a wired test cos my n100's SoftRouter is directly connected Internet serving my "c-suites-family" who demand tight SLA

For wireless test i yet to try , should be easy just install crusader(server) in any LAN machine will do.

For the internet (both wired & wireless) , maybe we can psycho those shiny 10Gbps folks (hello bro @TanKianW ? ) to install crusader(server) , lending us bandwidth for testing

 

[TanKianW]

Currently I cannot do a wired test cos my n100's SoftRouter is directly connected Internet serving my "c-suites-family" who demand tight SLA

For wireless test i yet to try , should be easy just install crusader(server) in any LAN machine will do.

For the internet (both wired & wireless) , maybe we can psycho those shiny 10Gbps folks (hello bro @TanKianW ? ) to install crusader(server) , lending us bandwidth for testing

edmw reply: MLMLWML.

I'm in a state of "very hard to psycho". Recently overloaded with my "super extra gey kiang" (NDA) side projects, and focusing on setting up TBs of storage for local privacy focus AI/ML as POCs. Maybe it is just FOMO when this whole AI bubble starts bursting. Still beating my chest out for selling my NVDA shares earlier on..... ​

 

[xiaofan]

Currently I cannot do a wired test cos my n100's SoftRouter is directly connected Internet serving my "c-suites-family" who demand tight SLA

For wireless test i yet to try , should be easy just install crusader(server) in any LAN machine will do.

For the internet (both wired & wireless) , maybe we can psycho those shiny 10Gbps folks (hello bro @TanKianW ? ) to install crusader(server) , lending us bandwidth for testing

Haha, good idea.

I will try internal server first to test wireless.

For WAN side, I will see what I can do.

 

[Mach3.2]

For WAN side, I will see what I can do.

just pull 2 public IPs from singtel and run loh.

But i suspect it will be no better than to run it locally on your own switch anyway.

 

[xiaofan]

just pull 2 public IPs from singtel and run loh.
But i suspect it will be no better than to run it locally on your own switch anyway.

That is the idea. I have two home networks with two public IPv4 addresses from SingTel.

Previously I did some experiments using this way, eg, iperf3/wireguard/librespeed server on one network and client on the other network.

 

[lobukong]

What is your definition of a "real expert"? Gauge by experience, competence, accomplishment or profession? Or just by what some posted on the forum? Or self-proclaimed? I know I will not claim someone as an "expert" just from their forum posts (at most kb warriors). Maybe if real expert, may not even be on this forum liao.

Even if any of us "are" the so-called experts in real life, we are just "learning to be an expert", but not an endpoint or destination, it will be an unending journey. There is always something which "real expert" will not know or learnt, that is the reason we have forum like these that share the knowledge which may also spur discussion in the process (not argument, like what most see it). Actually active and heated discussion is a common thing in the western tech world environment (which I once worked in), just that in Asian culture it seemed/sounded/perceived as more negative.

My own opinion: Based on your analogy, I think "non-real experts" without “开光 shovel” are not regarded as invalid. At least to me. At times I prefer view points from "non-real experts" coz their use case always kept me thinking. Anyone of us who do not use/adopt a certain solution, do not mean it is not valid or not useful, or has the rights to say it is not valid, just "not of use to me". This is a forum where not everyone agrees with everyone, but does not mean everyone is right!? Make sense? Chill bro....I also don't think it is so serious until regarded as offensive analogies. I do prefer analogies at times, but an overdose may make it sound confusing, the right dose is crucial, or maybe I am just stupid?

Hey, there is no need for a "real expert" (I know I'm not) to evaluate/validate the usefulness of the shovel so you can start implementing and using it (long passed that stage). Using whichever suits your use case is more important than following mine. Many times I have already been using viable and better solutions (I think) which most have not even started using for home use, Eg. ZFS/SDS/pfsense, but that does not stop me.

In the end, based on my definition that the forum has no "real experts".........using your lingo...."The West has long lost J*****lem".... ​

thanks, context in below

I have expressed my opinions in quite details whether bufferbloat is a real concern for most home internet users in Singapore or not, as well as whether QoS/SQM measures on typical home routers are useful or not (OpenWRT/pfSense/OPnsense may have good implementations. Netgear DumaOS may be another good one. What lese?). So I am not going to elaborate further in this area.

But I am interested to know more inputs from others. There are for sure different opinions on this topic and I already mentioned I could be wrong. I am especially interested to know the inputs from the real experts in this forum. @Mach3.2 has already shared his opinion and provided a few user cases where bufferbloat can be a real issue. It will be interesting to know from @TanKianW and @bert64 and others.

 

[xiaofan]

Another test suite i recently come across is Crusader , lease-overkill which is nicely available in multiplatform :

@hwzlite
Please help to comment on the above results. Thanks.

Local wireless test: Asus RT-AX86U

Server -- Proxmox PVE 8.0 LxC container (Ubuntu 22.04), connected to RT-AX86U 2.5G LAN port
Client -- Acer Windows 11 machine (2021 model) , Intel AX201 WiFi 6 adapter.

PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test 192.168.50.11
Connected to server 192.168.50.11:35481
Latency to server 1.89 ms
Testing download...
Testing upload...
Testing both download and upload...
Writing data...
Saved raw data as data 2024.03.24 09-10-48.crr
Saved plot as plot 2024.03.24 09-10-48.png

iperf3 results for reference as well.

PS C:\work\speedtest\iperf-3.15-win64> .\iperf3.exe -c 192.168.50.11 -R
Connecting to host 192.168.50.11, port 5201
Reverse mode, remote host 192.168.50.11 is sending
[  5] local 192.168.50.175 port 1674 connected to 192.168.50.11 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   129 MBytes  1.08 Gbits/sec
[  5]   1.00-2.02   sec   148 MBytes  1.22 Gbits/sec
[  5]   2.02-3.00   sec   136 MBytes  1.16 Gbits/sec
[  5]   3.00-4.00   sec   140 MBytes  1.17 Gbits/sec
[  5]   4.00-5.00   sec   130 MBytes  1.08 Gbits/sec
[  5]   5.00-6.00   sec   133 MBytes  1.12 Gbits/sec
[  5]   6.00-7.00   sec   140 MBytes  1.18 Gbits/sec
[  5]   7.00-8.00   sec   128 MBytes  1.07 Gbits/sec
[  5]   8.00-9.00   sec   140 MBytes  1.18 Gbits/sec
[  5]   9.00-10.00  sec   125 MBytes  1.05 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.06  sec  1.32 GBytes  1.13 Gbits/sec   76             sender
[  5]   0.00-10.00  sec  1.32 GBytes  1.13 Gbits/sec                  receiver

iperf Done.
PS C:\work\speedtest\iperf-3.15-win64> .\iperf3.exe -c 192.168.50.11
Connecting to host 192.168.50.11, port 5201
[  5] local 192.168.50.175 port 1678 connected to 192.168.50.11 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   108 MBytes   904 Mbits/sec
[  5]   1.00-2.00   sec   117 MBytes   984 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   943 Mbits/sec
[  5]   3.00-4.00   sec   118 MBytes   985 Mbits/sec
[  5]   4.00-5.00   sec   115 MBytes   969 Mbits/sec
[  5]   5.00-6.00   sec   111 MBytes   925 Mbits/sec
[  5]   6.00-7.00   sec   115 MBytes   966 Mbits/sec
[  5]   7.00-8.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   8.00-9.00   sec   117 MBytes   979 Mbits/sec
[  5]   9.00-10.00  sec   114 MBytes   958 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.12 GBytes   961 Mbits/sec                  sender
[  5]   0.00-10.06  sec  1.12 GBytes   954 Mbits/sec                  receiver

iperf Done.

Another wireless test results (< 1Gbps due the use of gigabit LAN connection), the server is another PVE 8.0 container on another Intel N100 mini PC connected to Asus RT-AX86U gigabit LAN port.

 

[xiaofan]

just pull 2 public IPs from singtel and run loh.

But i suspect it will be no better than to run it locally on your own switch anyway.

That is the idea. I have two home networks with two public IPv4 addresses from SingTel.

Previously I did some experiments using this way, eg, iperf3/wireguard/librespeed server on one network and client on the other network.

@Mach3.2 and @hwzlite
Unfortunately the test failed in this case. Not so sure why.

PS C:\work\speedtest\crusader-x86_64-pc-windows-msvc> .\crusader.exe test xxxxxxxx.asuscomm.com
Connected to server 219.75.xxx.xxx:35481
Latency to server 3.29 ms
thread 'tokio-runtime-worker' panicked at crusader-lib\src\test.rsthread ':tokio-runtime-workerthread '' panicked at tokio-runtime-workercrusader-lib\src\test.rs' panicked at :crusader-lib\src\test.rs947::94722::
22unable to bind TCP socket: Os { code: 10060, kind: TimedOut, message: "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." }:

unable to bind TCP socket: Os { code: 10060, kind: TimedOut, message: "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond." }note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

 

[xiaofan]

@Mach3.2 and @hwzlite
Unfortunately the test failed in this case. Not so sure why.

Same for Linux client, wired connection (another container connected to OpenWRT virtual router).

root@debian12ct2:~# ./crusader test xxxxxxxx.asuscomm.com
Connected to server 219.75.xxx.xxx:35481
Latency to server 1.53 ms
thread 'thread 'tokio-runtime-workertokio-runtime-worker' panicked at ' panicked at crusader-lib/src/test.rscrusader-lib/src/test.rs::947947::2222:
:
unable to bind TCP socket: Os { code: 110, kind: TimedOut, message: "Operation timed out" }unable to bind TCP socket: Os { code: 110, kind: TimedOut, message: "Operation timed out" }

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted

 

[xiaofan]

I have created an issue in Crusader github.
https://github.com/Zoxc/crusader/issues/17