FYI/A: Bufferbloat 101

[xiaofan]

This is an interesting point of view.

You can also say that nice scores of bufferbloat make those who run SQM to have some sort of justifications. For example, in the case of @hwzlite, he would do wonders using the old D-Link router or the Linksys EA8100 v1 router. I think they are very good technical exercises for him but I still doubt the usefulness even in his use cases (he does not seem to use BT a lot). I actually carry out some technical exercises as well to see if I can get good bufferbloat results. But in the end my conclusion is that there are no practical use of such SQM measures for my use cases.

+++++++++++++++
"Bufferbloat" is a significant increase in latency which occurs when either your upload or download bandwidth is saturated. Most people are not going to be operating under these conditions, but it's good for scaring people into thinking their equipment sucks.
++++++++++++++

 

[Mach3.2]

"Bufferbloat" is a significant increase in latency which occurs when either your upload or download bandwidth is saturated. Most people are not going to be operating under these conditions, but it's good for scaring people into thinking their equipment sucks.

it's true you won't run into bufferbloat most of the time, but if your PC is connected using an ethernet cable to your router, you can easily saturate your connection while patching your game on steam/bnet/ea origin, etc since those game launchers usually download files over multiple streams.

But as always, don't fix it if it ain't broke.

If you're experiencing issues, then perhaps investigate whether QoS can mitigate the problem for you boh.

 

[xiaofan]

it's true you won't run into bufferbloat most of the time, but if your PC is connected using an ethernet cable to your router, you can easily saturate your connection while patching your game on steam/bnet/ea origin, etc since those game launchers usually download files over multiple streams.

Sorry I do not play games so I do not know the exact use case here.

Do you still play the games when you patching your games? If not, then there are still no issues. If yes, then good that is one use case, which may reduce my 99% numder to say 95%.

@lobukong likes to use analogy. I am not so sure if the following analogy is correct or not.

Mobile phones generate heat when running games.
Mobile phones generate heat when being charged.
It is better not to play games when charging the mobile phone -- common users.
I want to play games while charging the battery, so I need extra measures -- minority users.

 

[Mach3.2]

Do you still play the games when you patching your games? If not, then there are still no issues. If yes, then good that is one use case, which may reduce my 99% numder to 95%.

Steam doesn't download patches by dafault while you're in game unless you override the default settings. I've heard my friend mention his PS5 will preload patches in the background even when he is gaming.

Another thing is, it's not just about me, but my family members who are using the same connection too. I don't want their connection to become cuis when I saturate the connection with my game patch download.

Another usecase for bandwidth limiter is simply because it's faster to download files if you keep just under the max speed. If you let your upstream ISP do the throttling, they often will just reset your TCP connection or even drop packets to moderate your speed, leading to intermittent pauses while you're downloading files.

 

[lobukong]

Sorry I do not play games so I do not know the exact use case here.

Do you still play the games when you patching your games? If not, then there are still no issues. If yes, then good that is one use case, which may reduce my 99% numder to say 95%.

@lobukong likes to use analogy. I am not so sure if the following analogy is correct or not.

Mobile phones generate heat when running games.
Mobile phones generate heat when being charged.
It is better not to play games when charging the mobile phone -- common users.
I want to play games while charging the battery, so I need extra measures -- minority users.

correct, if I can play games while charging the battery without the phone getting too hot in hand, why not? I will choose the "better" phone even though I may not need to use it that way.

 

[xiaofan]

Steam doesn't download patches by dafault while you're in game unless you override the default settings. I've heard my friend mention his PS5 will preload patches in the background even when he is gaming.

Another thing is, it's not just about me, but my family members who are using the same connection too. I don't want their connection to become cuis when I saturate the connection with my game patch download.

Another usecase for bandwidth limiter is simply because it's faster to download files if you keep just under the max speed. If you let your upstream ISP do the throttling, they often will just reset your TCP connection or even drop packets to moderate your speed, leading to intermittent pauses while you're downloading files.

Another good use case for downloading large files, if your bandwidth limiter is really working fine at 1Gbps or even 800Mbps, for 1Gbps users.

However, you need a good router for the bandwidth limiter to work fine at 800Mbps. As mentioned, Asus routers may drop 1Gbps speed to 300Mbps (those with lower end CPUs) or 600Mbps (those with faster CPUs). In this case, it is hard to say if the bandwidth limiter is helping the download speed or not.

 

[lobukong]

But I thought if you need 800Mbps download then you will benefit from SQM then?

 

[xiaofan]

But I thought if you need 800Mbps download then you will benefit from SQM then?

I am stopping here for this topic. Anyway, everyone is entitled to have his own opinion. Nothing wrong about that.

 

[lobukong]

Well, just wanted to share what I thought was good.

 

[xiaofan]

Well, just wanted to share what I thought was good.

No issues, it will greatly appreciated that you can continue sharing real world experiences where you find SQM really useful, use real data if possible, not analogies.

 

[hwzlite]

Status Quo Bias any1 ?

Only for those who have personally experienced the pain bitten by bufferbloat in one form or another...., SHOVEL (SQM) is the ways and as well dig out in whatever "perfect storm" events.

 

[lobukong]

Well, thank you for believing my test method more than waveform?
I try to download a game on steam, speed is about 150Mbps - 200Mbps, bound by my disk speed. Which is much lower than 1G right. So enabling SQM doesn't affect download speed much. Then I keep pinging 1.1.1.1. Without SQM it even has a packet lost, with SQM the maximum ping is not as bad.

Without SQM:
Ping statistics for 1.1.1.1:
Packets: Sent = 710, Received = 709, Lost = 1 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 33ms, Average = 2ms

With SQM:
Ping statistics for 1.1.1.1:
Packets: Sent = 739, Received = 739, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 19ms, Average = 2ms

 

[xiaofan]

I am stopping here for this topic. Anyway, everyone is entitled to have his own opinion. Nothing wrong about that.

I have expressed my opinions in quite details whether bufferbloat is a real concern for most home internet users in Singapore or not, as well as whether QoS/SQM measures on typical home routers are useful or not (OpenWRT/pfSense/OPnsense may have good implementations. Netgear DumaOS may be another good one. What lese?). So I am not going to elaborate further in this area.

But I am interested to know more inputs from others. There are for sure different opinions on this topic and I already mentioned I could be wrong. I am especially interested to know the inputs from the real experts in this forum. @Mach3.2 has already shared his opinion and provided a few user cases where bufferbloat can be a real issue. It will be interesting to know from @TanKianW and @bert64 and others.

 

[xiaofan]

I am also interested in alternative bufferbloat testing methods from waveform.com. For rexample we already know that it is not good for 10Gbps testing. So what will be the alternative?

My idea is that OOkla SpeedTest CLI version may be a good one for ST/SH/MR 10G plan users (now at about 8.2Gbps with XGS-PON technolog). @TanKianW may be able to help to provide some test results.

Example test results for my Singtel 1Gbps Fibre Internet. We can see that upload latency is not that good compared to download, for both wired and wireless. This was identified as a potential issue with the Singtel provided ZTE ONT used (from @miloaisdino's experiment). On the other hand, waveform.com tends to amplify this effect (high upload latency) which I am not so sure if it is really valid or not, since most of the valid use cases mentioned above focus on download and not on upload (even less use cases for upload at full speed).

1) Wired test result, directly from the OpenWRT virtual router.

root@OpenWrt:~/speedtest# ./speedtest

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     1.47 ms   (jitter: 0.21ms, low: 1.30ms, high: 1.69ms)
    Download:   942.91 Mbps (data used: 422.9 MB)
                  2.80 ms   (jitter: 0.20ms, low: 1.35ms, high: 3.07ms)
      Upload:   946.88 Mbps (data used: 1.1 GB)
                  2.71 ms   (jitter: 0.65ms, low: 1.61ms, high: 16.31ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/7806cc2b-7407-445a-b635-c0eb60757818

2) Wireless test results: ZTE BE7200 Pro+ wireless AP behind OpenWRT virtual router.
Using Acer Swift 3 Windows 11 laptop (2021 model, Core i5-1135G7, 16GB RAM, 512GB SSD).

PS C:\work\speedtest\ookla-speedtest-1.2.0-win64> .\speedtest.exe

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     3.84 ms   (jitter: 0.20ms, low: 3.51ms, high: 3.89ms)
    Download:   903.68 Mbps (data used: 977.9 MB)
                  8.81 ms   (jitter: 3.42ms, low: 3.83ms, high: 30.64ms)
      Upload:   828.15 Mbps (data used: 757.5 MB)
                 16.71 ms   (jitter: 6.16ms, low: 3.25ms, high: 55.30ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/59bd66f2-9650-4fcb-9424-1a40407eb001

 

[xiaofan]

@Mach3.2

Since you upgraded to M1 1Gbps plan from 500Mbps, just wondering if you see any differences about the bufferbloat test results. What are the QoS/SQM measures you are using now? Do you need to adjust when you change from 500Mbps to 1Gbps? Thanks.

Edit: I saw that you were not using any QoS settings last time. No so sure if things changed or not now.

 

[xiaofan]

I have expressed my opinions in quite details whether bufferbloat is a real concern for most home internet users in Singapore or not, as well as whether QoS/SQM measures on typical home routers are useful or not (OpenWRT/pfSense/OPnsense may have good implementations. Netgear DumaOS may be another good one. What lese?). So I am not going to elaborate further in this area.

But I am interested to know more inputs from others. There are for sure different opinions on this topic and I already mentioned I could be wrong. I am especially interested to know the inputs from the real experts in this forum. @Mach3.2 has already shared his opinion and provided a few user cases where bufferbloat can be a real issue. It will be interesting to know from @TanKianW and @bert64 and others.

Old post from @Mach3.2 back in Dec 2020 (Post #28)
https://forums.hardwarezone.com.sg/threads/fyi-a-bufferbloat-101.6427979/page-2#post-131404745

Yup, M1 500Mbps plan.

Didn't apply any QoS settings as I find it unnecessary for fibre plans.

Old post from @bert64 back in Dec 2020 (Post #52)
https://forums.hardwarezone.com.sg/threads/fyi-a-bufferbloat-101.6427979/page-3#post-131456435

Yes, bufferbloat was a thing in the days of dialup.. You'd have a (typically) 64kb transmit buffer for a 33.6kbps (max) modem, so it would take nearly 20 seconds to empty the contents of the buffer. It was designed this way to improve throughput, because operating systems of the time (especially windows) could not respond quickly enough to send more data once the modem was ready to transmit it.

It happened on things like ADSL too, as many providers would give you an extremely weak uplink it may still take a couple of seconds to clear the buffers.

For a 1gbps connection, the buffers are tiny relative to the speed of the line - they should get cleared very quickly. Similarly, your internal devices probably aren't capable of throwing data at the router faster than the upstream line can accept it.

What's more likely a cause of latency, is the router itself not being able to handle the traffic rate. Adding extra complexity like QoS, NAT, firewall rules etc will exacerbate the situation and slow it down further. Remember these devices are built to a price point, with corners cut to make the price more attractive. You'll get better performance with a high end layer 3 switch, but most home users won't pay the price or tolerate the fan noise.

 

[xiaofan]

Another older post from @Qubicfactor in Dec 2020 (Post #40)
https://forums.hardwarezone.com.sg/threads/fyi-a-bufferbloat-101.6427979/page-2#post-131420774

Impact for bufferbloat on uploads are more of you noticing it taking longer to load websites coz the request to the server to send you data is delayed. If doing videoconferencing, other ppl may notice you are lagging since your vide stream to them is not real time but buffered. Bandwidth isnt always everything.

That said, the issue might be something that is tolerable to some and not to others, coz the feeling of lag is still somewhat subjective, although measurable.

I would think a badly implemented QoS/SQM solution is far more harmful than that any supposed harm done on underpowered home routers on gigabit lines, coz the vast majority of ppl dont saturate their connections all the time. Personally I would rather sacrifice some underutilized bandwidth for better latency.

 

[hwzlite]

I am also interested in alternative bufferbloat testing methods from waveform.com. For rexample we already know that it is not good for 10Gbps testing. So what will be the alternative?

​

Flent RRUL is the (kind-of-overkill ) way, which i already demo it in my n100 threads.

Another test suite i recently come across is Crusader , lease-overkill which is nicely available in multiplatform :

As for waveform testing, checkout the threads on "[Bloat] We built a new bufferbloat test and keen for feedback" , which got vetting & scrutinized by the gurus there, especially Toke Høiland-Jørgensen who "wrote the book" (literally) and got his PhD from this topic .

As well to reiterate that fq_codel SQM by itself is already enabled internally as the default Queueing mechanism since Linux version is ≥ 4.12 for all of modern Linux-based distros, routers, switches or might even in ONT/R ....etc , where most lay users did not even know of its existence where SQM already working in the background.

fq_codel SQM will have great effect working together with shaper/QOS to limit max bandwidth if it is on the bottleneck/mismatch links, eg. 1Gbps home ethernet link connected to 500Mbps broadband , 2.5Gbps home ethernet link connected to 1Gb broadband.
If both on line rate , eg. 1Gbps home ethernet link connected to 1Gbps broadband , fq_codel SQM will do the job nicely without the need of shaper/QOS .

 

[TanKianW]

I have expressed my opinions in quite details whether bufferbloat is a real concern for most home internet users in Singapore or not, as well as whether QoS/SQM measures on typical home routers are useful or not (OpenWRT/pfSense/OPnsense may have good implementations. Netgear DumaOS may be another good one. What lese?). So I am not going to elaborate further in this area.

But I am interested to know more inputs from others. There are for sure different opinions on this topic and I already mentioned I could be wrong. I am especially interested to know the inputs from the real experts in this forum. @Mach3.2 has already shared his opinion and provided a few user cases where bufferbloat can be a real issue. It will be interesting to know from @TanKianW and @bert64 and others.

For my use case...... the simple answer is no, or maybe I am just not too bothered by it enough to implement some form of QOS/SQM. My personal experiences/use cases as below:​

1. When gaming and BT-ing, my firewall CPU rarely hits 60% load and ping still stays 2-5ms, for local servers, playing games like WWZ and Helldiver II. Even browsing is very responsive since I am also running Unbound on pfsense. I don't play overseas servers since all my (old-timer) gaming kakis are locals. This is on 10G MR.​
2. When off-prem backing my servers/snapshots/data to/from my work-lab, I schedule it late at night around 2 - 6 am, when I am rarely still awake and gaming due to ageing eyes. So there is not much competition for latency and bandwidth during that period.​
3. Since I am on 2x ISPs with all internal traffic segmented to VLANs, where all mobile devices, IOTs, kids' and wifey's systems are all on separate VLANs with all their WAN mapped to an M1 1G fibre. Only the breadwinner (aka me) uses the 10G MR to myself for work and leisure. I will say "rarely" there will be sufficient bandwidth/latency competition to justify for some form of management for a home environment.​

In the end, really up to you. If it works, and you notice the difference (like @Henry Ng browser load a bit faster experience), do continue using it. If you are keen to learn, squeeze every ounce of optimization out of it. Who knows, you learn something you never knew (or I never knew) through self experimenting and I think that mindset is valuable.

For enterprise networks with lots of traffic and bottlenecks, I may consider. But that can be achieved using the L7 (OSI) policy on enterprise network appliances and a well-optimised internal network running multiple L3 switches and routers.​

 

[Mach3.2]

@Mach3.2

Since you upgraded to M1 1Gbps plan from 500Mbps, just wondering if you see any differences about the bufferbloat test results. What are the QoS/SQM measures you are using now? Do you need to adjust when you change from 500Mbps to 1Gbps? Thanks.

Edit: I saw that you were not using any QoS settings last time. No so sure if things changed or not now.

I did adjusted my bandwidth limiter. I tried running without briefly and the latency did spike a little when I saturated the connection.