Tools for troubleshooting the network performance

[bert64]

You may be right, looks like not everyone is willing to give the free access for example few million users for a friend with few hundred users without any agreement?

Well its not free transit, its free peering - you can only exchange traffic with addresses belonging to the peer and their downstream customers. You can't route traffic upstream via a peering link.

All of the traffic on a peering link is either to or from your customers, it benefits your customers and the remote peer equally.

A lot of large providers do have open peering arrangements, and it doesn't cost them anything if they are already present at a given peering location.

 

[ksw2010]

Well its not free transit, its free peering - you can only exchange traffic with addresses belonging to the peer and their downstream customers. You can't route traffic upstream via a peering link.

All of the traffic on a peering link is either to or from your customers, it benefits your customers and the remote peer equally.

A lot of large providers do have open peering arrangements, and it doesn't cost them anything if they are already present at a given peering location.

Thanks for explanation for the peering connection, but I do not really understand as it does not cost anything for the ISP when they set up free peering, for instance at SGIX? For the connection there should be the cost of connection/device which is connecting to the SGIX POP to peer with other partners there? and also when traffic increase to a certain level upgrading connection may incur the cost?

 

[bert64]

Thanks for explanation for the peering connection, but I do not really understand as it does not cost anything for the ISP when they set up free peering, for instance at SGIX? For the connection there should be the cost of connection/device which is connecting to the SGIX POP to peer with other partners there? and also when traffic increase to a certain level upgrading connection may incur the cost?

The connection to SGIX is a single cost that potentially gives you access to a large number of local peers. For the price of one connection to SGIX you can deliver traffic to most local providers.

The cost of upgrading the connection due to increased traffic is relatively low (SGIX publish pricing on their site), and is much lower than setting up dedicated lines to multiple providers for peering or especially transit.
You would still need transit as well - your transit links would cost more, and be a catchall for anything else that's not covered by the local peering routes. The more traffic you deliver over local peering, the less transit costs you have.

 

[firesong]

Thanks for explanation for the peering connection, but I do not really understand as it does not cost anything for the ISP when they set up free peering, for instance at SGIX? For the connection there should be the cost of connection/device which is connecting to the SGIX POP to peer with other partners there? and also when traffic increase to a certain level upgrading connection may incur the cost?

You're not the only one that does not understand. If we would guess, it's likely because of business reasons, to force customers to peer with them by arbitrarily introducing latency through less efficient peering policies. STIX isn't the cheapest - it's known to pride itself on being "Premium", so we can easily figure out the rest.

So rather than prioritise the good of their customers and Singaporeans in general, they'd rather do silly things.

And many of us who were stuck with Singtel during the lockdown, when the other ISPs were overloaded and collapsed, noted that while there was no outright Singtel outage, there were many occasions where our packets seemed to be thrown haphazardly on the internet and lost, likely dropped. So while we didn't have an actual outage in actuality, what we had was practically the same - except Singtel avoided fines by IMDA because there are no minimum QoS requirements, only uptime requirements. Anything that is not a blatant outage seems to be okay with IMDA.

The irony, even Zoom/Meet/Teams calls during the day would sometimes freeze or drop out because of the instability. It was pretty quickly determined that those of us having problems all were Singtel customers. I would rather have a blatant outage so I can fall back to a contingency, than have a half-assed connection that is neither in nor out. Especially when dealing with clients external to the organisation or to bosses with no patience.

Being stuck with Singtel during the lockdown made me regret porting out of Starhub bitterly. One day of obvious downtime is much more reliable than a working-not-working connection. At least with Starhub we knew we just had to set a public DNS server and 90% of their connectivity problems disappeared.

 

[firesong]

sometimes traffic originating from singtel destined to certain asns are routed via starhub (which peers at sgix), probably if there are too many subscriber complaints.

singtels particpation in sgix could be seen as a "token gesture" to the regulator at the time IDA that spearheaded the project. their port speed is only 1G . it would be disastrous for a telco to oppose any regulator anyway

good read: https://webcache.googleusercontent....blog/sgix-singapore-public-internet-exchange/

noteworthy quotes:
"In particular, there is a concern that mandating multilateral peering between all ISPs would enable smaller ISPs to free-ride off investments in infrastructure and facilities made by larger ISPs, acting as a significant disincentive to such investment.", Singtel
source: https://www.imda.gov.sg/-/media/imda/files/inner/pcdg/consultations/20150213_ippeering/singtel.pdf

That's why Singtel customers who need to access local content on Equinux/SHIX should add their "local" traffic to their frequent flier memberships.

And that Singtel paper is trying to argue their position on a "I has invested more so I deserve more", neglecting to mention that a large part of their earlier investments was due to them being the only IX in Singapore for a period, and that this was publicly funded using taxpayer money back then too - because it benefited all of Singapore to have a good international exchange. The other IXes were eventually opened to provide more cost effective IX to local businesses, further proving the point that STIX is just too expensive for businesses to operate using that backbone. Rather than compete with quality of service and other good strategies, they simply chose to price themselves highly and offer poor local routes. In many other countries, this would be monopolistic business practises that would earn them the ire of their competition watchdogs!

IMDA needs to define not only uptime, but a reasonable QoS at at least 80% of rated speeds to multiple defined local peers behind the different IXes all through the day. If they sell a 1Gbps plan, connection to local peers should be at least 800Mbps with low route pings that are obviously local traffic and not one routed halfway around the world and back. Now with HBL and WFH being more commonplace, having a reliable connection so people can have meetings and attend class without lagging/dropping out should at be the minimum expectation. It's a joke that a lot of our online meetings are local and yet our connections can be so poor, sometimes feeling as if we're on dial-up or satellite in some far flung location instead. That way we customers are not disadvantaged by their own monopolistic business strategies, and the ISPs are forced to up their Quality of Service rather than treat customers as a cash cow.

 

[bert64]

That's why Singtel customers who need to access local content on Equinux/SHIX should add their "local" traffic to their frequent flier memberships.

And that Singtel paper is trying to argue their position on a "I has invested more so I deserve more", neglecting to mention that a large part of their earlier investments was due to them being the only IX in Singapore for a period, and that this was publicly funded using taxpayer money back then too - because it benefited all of Singapore to have a good international exchange. The other IXes were eventually opened to provide more cost effective IX to local businesses, further proving the point that STIX is just too expensive for businesses to operate using that backbone. Rather than compete with quality of service and other good strategies, they simply chose to price themselves highly and offer poor local routes. In many other countries, this would be monopolistic business practises that would earn them the ire of their competition watchdogs!

IMDA needs to define not only uptime, but a reasonable QoS at at least 80% of rated speeds to multiple defined local peers behind the different IXes all through the day. If they sell a 1Gbps plan, connection to local peers should be at least 800Mbps with low route pings that are obviously local traffic and not one routed halfway around the world and back. Now with HBL and WFH being more commonplace, having a reliable connection so people can have meetings and attend class without lagging/dropping out should at be the minimum expectation. It's a joke that a lot of our online meetings are local and yet our connections can be so poor, sometimes feeling as if we're on dial-up or satellite in some far flung location instead. That way we customers are not disadvantaged by their own monopolistic business strategies, and the ISPs are forced to up their Quality of Service rather than treat customers as a cash cow.

Whenever regulations are introduced, they will find ways to bare minimum comply with the letter of the regulation to avoid fines, while doing everything they can do undermine the spirit of the regulation. Take the 2013 IPv6 requirement - wheres other ISPs simply enable it by default, ST make it "available on request" while doing everything they can to discourage users from actually using it.
If you make the regulation specific enough to prevent this bad-faith minimal compliance behaviour, then you are likely to end up causing other problems and having to regularly update the regulation.

One of the biggest problems with online meetings/calls is NAT, which ends up forcing the use of centralised services. There are various standards (SIP, H.323, WebRTC) for fully p2p calls and which used to be commonly used (who remembers microsoft netmeeting?), but NAT breaks them. With a centralised service the provider of that service needs a lot of bandwidth and low latency which makes the service obscenely expensive to provide or scale as well as very slow for users in any country which does not have local servers. With p2p, a large portion of the traffic would remain inside the ISP network itself especially for the larger providers and save them a lot of transit costs. With an expensive centralised service, someone has to pay for it.

Companies get away with poor service because the customers let them. Either they don't understand the technology, or they are unaware that the competition offers a better service, or in some cases there simply is no competition. Where a company offers a quantifiably better service than a competitor, they should be shouting it from the rooftops as a marketing strategy, but for some reason they don't do that. Most users would not understand what lower latency peering, IPv6, lack of CGNAT etc mean unless marketing material points that out. But when you do start marketing something, users will start demanding it - even things that don't even benefit them at all.
Service A comes with a free soft toy, service B does not and costs the same - some users will pick service A just because it comes with "extra" even if its of absolutely no use to them.

The only thing that's going to make providers drop customer-hostile policies, is when the users realise those policies are hostile to them and start moving in significant numbers to other providers.

 

[xiaofan]

The irony, even Zoom/Meet/Teams calls during the day would sometimes freeze or drop out because of the instability. It was pretty quickly determined that those of us having problems all were Singtel customers. I would rather have a blatant outage so I can fall back to a contingency, than have a half-assed connection that is neither in nor out. Especially when dealing with clients external to the organisation or to bosses with no patience.

Being stuck with Singtel during the lockdown made me regret porting out of Starhub bitterly. One day of obvious downtime is much more reliable than a working-not-working connection. At least with Starhub we knew we just had to set a public DNS server and 90% of their connectivity problems disappeared.

Actually I have never encoutered Zoom/Teams/Google Meet calls problem with my Singtel 1Gbps Fibre connection and we are using wireless (using Asus RT-AX82U routers and laptops with good Intel AX200, Intel AX201 or Intel AC9560 wireless adapter, or iPad 2018 and iPad 2019) at home. On the contrary my colleague using Starhub or M1 are having issues with Teams call.

So I guess it all depends on individual family setup. And wireless router may play a bigger part than the ISPs. And it is said that sometimes Zoom itself is the problem.

From what I gather, Singtel Fibre has more issues with gaming due to potentially bad routing, but not general WFH/HBL stuff like Zoom/Teams/Google Meet/etc.

 

[bert64]

Actually I have never encoutered Zoom/Teams/Google Meet calls problem with my Singtel 1Gbps Fibre connection and we are using wireless (using Asus RT-AX82U routers and laptops with good Intel AX200, Intel AX201 or Intel AC9560 wireless adapter, or iPad 2018 and iPad 2019) at home. On the contrary my colleague using Starhub or M1 are having issues with Teams call.

So I guess it all depends on individual family setup. And wireless router may play a bigger part than the ISPs. And it is said that sometimes Zoom itself is the problem.

From what I gather, Singtel Fibre has more issues with gaming due to potentially bad routing, but not general WFH/HBL stuff like Zoom/Teams/Google Meet/etc.

Or just general wireless congestion. Calls are latency sensitive, and wireless congestion can cause collisions and retransmits which will increase latency/jitter. Poor equipment or marginal signal strength can also contribute to latency/jitter.

The other thing to consider is where the call is hosted - while services like zoom/teams/etc do have local servers, the call will be hosted wherever its created - so if someone in the US invites you to a call, your traffic is likely to go to the US based nodes of the given service rather than local nodes. If you are only invited to local calls, the experience is usually going to be better. Even for local users, bad configuration or local congestion could still cause the call to be hosted far away.

The problem is that most users are focused on last mile speedtest apps and don't consider latency, jitter or throughput to far away destinations. Being able to perform a bulk transfer at 1gbps from a local server does you no good for making calls or gaming if the latency varies wildly, a stable 10mbps line with consistent latency would actually be better. This is why business services tend to offer a lower maximum speed, but more stable latency (lower contention etc) and better SLAs.

 

[ksw2010]

Actually I have never encoutered Zoom/Teams/Google Meet calls problem with my Singtel 1Gbps Fibre connection and we are using wireless (using Asus RT-AX82U routers and laptops with good Intel AX200, Intel AX201 or Intel AC9560 wireless adapter, or iPad 2018 and iPad 2019) at home. On the contrary my colleague using Starhub or M1 are having issues with Teams call.

So I guess it all depends on individual family setup. And wireless router may play a bigger part than the ISPs. And it is said that sometimes Zoom itself is the problem.

From what I gather, Singtel Fibre has more issues with gaming due to potentially bad routing, but not general WFH/HBL stuff like Zoom/Teams/Google Meet/etc.

I also have never encountered any issue with general Zoom/Teams/Google Meet/etc as I have been using it everyday for working, studying and supporting Customer as well. It looks like Zoom/Team (MS Azure), Google should already have directly peer connection to ST so it may not have any issue like latency rather than Gamming stuff which ST routing is not good to those gamming sites compared to others local ISPs.

 

[firesong]

Actually I have never encoutered Zoom/Teams/Google Meet calls problem with my Singtel 1Gbps Fibre connection and we are using wireless (using Asus RT-AX82U routers and laptops with good Intel AX200, Intel AX201 or Intel AC9560 wireless adapter, or iPad 2018 and iPad 2019) at home. On the contrary my colleague using Starhub or M1 are having issues with Teams call.

So I guess it all depends on individual family setup. And wireless router may play a bigger part than the ISPs. And it is said that sometimes Zoom itself is the problem.

From what I gather, Singtel Fibre has more issues with gaming due to potentially bad routing, but not general WFH/HBL stuff like Zoom/Teams/Google Meet/etc.

It might not be noticeable for short calls, but a drop out is definitely noticeable when your meetings (cos class) are 4h long. I had 3 memorable moments of call drop outs during 3h class (and this particular lecturer came on 45m early and stayed for 30m after to entertain questions), and a Smokeping log from a friend who happened to be monitoring Singnet during that period suggested dropped packets.

Second and third were via a wired connection direct to the switch in the hall, cos I was tired of seeing laggy lecturers and classmates.

Edit: Went through my screenshots of Smokeping graphs to the same local servers on a Singnet connection. This was on my previous ERL3 so no IDS/IPS running. SP is running on my ARM SBC via an Ethernet connection - dates and times as indicated. Make of it what you will.

My friend's.





My own connection.

 

[firesong]

1. Speedtest related

1) OOkla speedtest apps
https://www.speedtest.net/apps
2) OOkla speedtest cli
https://www.speedtest.net/apps/cli
3) Fast.com app or using website
https://fast.com/
4) DSLreports (also testing bufferbloat)
http://www.dslreports.com/speedtest
5) waveform.com (also testing bufferbloat)
https://www.waveform.com/tools/bufferbloat
6) iperf3
https://iperf.fr/iperf-download.phpNewer version of Windows binary: https://www.neowin.net/forum/topic/1234695-iperf-3101-windows-build/

...

I noticed you left out Smokeping, since it's a graphical monitor over time, a perfect "tracker".

There are various probes that can be configured, so beyond the standard ICMP ping, there's also DNS resolution probes (how long the DNS server takes to resolve addresses). These help give an idea of network performance.

 

[moron+genius]

Would appreciate it if someone could do a step by step iperf tutorial or point me with a link to a good one you know. Thanks

 

[firesong]

Would appreciate it if someone could do a step by step iperf tutorial or point me with a link to a good one you know. Thanks

https://www.dell.com/support/kbdoc/...-test-available-network-bandwidth-using-iperf

 

[bert64]

I noticed you left out Smokeping, since it's a graphical monitor over time, a perfect "tracker".

There are various probes that can be configured, so beyond the standard ICMP ping, there's also DNS resolution probes (how long the DNS server takes to resolve addresses). These help give an idea of network performance.

Yeah smokeping is great, but it requires some level of setup and knowledge, plus a 24/7 running machine to put it on. It's probably beyond the capabilities of a lot of people.

I wonder if there's anything ready made - eg a raspberry pi sdcard image that could be used for this purpose. It might make a useful tool for users if you just need to boot the card image in a pi and enter a list of targets to ping.

 

[ksw2010]

https://www.dell.com/support/kbdoc/...-test-available-network-bandwidth-using-iperf

Just add in additional link which shows steps from installation to testing step: https://www.instructables.com/How-to-measure-network-performance-1/

 

[ksw2010]

It might not be noticeable for short calls, but a drop out is definitely noticeable when your meetings (cos class) are 4h long. I had 3 memorable moments of call drop outs during 3h class (and this particular lecturer came on 45m early and stayed for 30m after to entertain questions), and a Smokeping log from a friend who happened to be monitoring Singnet during that period suggested dropped packets.

Second and third were via a wired connection direct to the switch in the hall, cos I was tired of seeing laggy lecturers and classmates.

Edit: Went through my screenshots of Smokeping graphs to the same local servers on a Singnet connection. This was on my previous ERL3 so no IDS/IPS running. SP is running on my ARM SBC via an Ethernet connection - dates and times as indicated. Make of it what you will.

My friend's.
View attachment 30018

View attachment 30019


My own connection.
View attachment 30020

View attachment 30021

There are some limitation of Smokeping in some situations like IMCP rate-limit at destination hosts or blocking ICMP when Smokeping might not reflect 100% accurate performance when the destination host reaching ICMP traffic threshold, but overall it is still good in visualizing the network monitoring and keep historical graph for reference.

 

[xiaofan]

I noticed you left out Smokeping, since it's a graphical monitor over time, a perfect "tracker".

There are various probes that can be configured, so beyond the standard ICMP ping, there's also DNS resolution probes (how long the DNS server takes to resolve addresses). These help give an idea of network performance.

It is hidden in Post #3 (in the links below).
https://haydenjames.io/linux-networking-commands-scripts/

I have edit the post to expand the list of these 60 useful Linux networking commands, many of them will work under macOS and some of them will work under Windows.

 

[firesong]

There are some limitation of Smokeping in some situations like IMCP rate-limit at destination hosts or blocking ICMP when Smokeping might not reflect 100% accurate performance when the destination host reaching ICMP traffic threshold, but overall it is still good in visualizing the network monitoring and keep historical graph for reference.

This is true. But those servers I shared above are working normally most of the time now. It was really a product of bad routing and internet connectivity during the lockdown. That was the point I was trying to bring across above - that likely contributed to lag spikes and even dropouts during 4h zoom calls.

Those servers that rate limit and give false down reports over time because they drop ICMP packets, I've removed from my monitoring list.

 

[firesong]

It is hidden in Post #3 (in the links below).
https://haydenjames.io/linux-networking-commands-scripts/

I have edit the post to expand the list of these 60 useful Linux networking commands, many of them will work under macOS and some of them will work under Windows.

Ah, I missed it cos it's not a command.

It's a constantly running service that tracks the pings and maps it out.

 

[firesong]

Yeah smokeping is great, but it requires some level of setup and knowledge, plus a 24/7 running machine to put it on. It's probably beyond the capabilities of a lot of people.

It's actually pretty simple on Docker. But yes, it does need a constantly up device. A PiHole or a NAS is fabulous for this. But running it on your PC isn't too bad since it gives an idea of the PC's connectivity as long as it is up.

I'll drop my sanitised docker-compose.yml file for anyone to modify and use - to shorten the setup time. Stipulate the port you want it to run on

---
version: "2.1"
services:
  smokeping:
    image: linuxserver/smokeping:latest
    container_name: smokeping
    hostname: smokeping
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Asia/Singapore
    volumes:
      - ./config:/config
      - ./data:/data
    ports:
      - <openport>:80
#    dns:              # optional if you want to force a DNS server
#      - 1.1.1.1
    restart: unless-stopped

The only other files that need to be edited are the 2 files under ./config, General and Targets

A sanitised Targets file. Feel free to modify and use.

*** Targets ***

probe = FPing

menu = Top
title = Internet Latency Monitoring
remark = Tham Family Home - SmokePing
   
+ Favourites
menu = Favourites
title = Favourites

++ Google
menu = Google
title = Google
host = google.com

++ YouTube
menu = YouTube
title = YouTube
host = youtube.com

++ Spotify
menu = Spotify
title = Spotify
host = spotify.com

++ Facebook
menu = Facebook
title = Facebook
host = facebook.com

++ WhatApp
menu = WhatsApp
title = WhatsApp
host = whatsapp.com

++ Twitter
menu = Twitter
title = Twitter
host = twitter.com

++ Steam
menu = Steam
title = Steam
host = store.steampowered.com

++ GoogleScholar
menu = Google Scholar
title = Google Scholar
host = scholar.google.com

++ PurdueOWL
menu = Purdue OWL
title = Purdue University Online Writing Lab
host = owl.purdue.edu

++ GoogleSearchIpv6
menu = Google IPv6
probe = FPing6
title = ipv6.google.com
host = ipv6.google.com


+ Global
menu = Global
title = Global Connectivity

++ YahooJP
menu = Yahoo JP
title = Yahoo Japan
host = yahoo.co.jp

++ BBC
menu = BBC
title = British Broadcasting Corporation
host = bbc.co.uk

++ CambridgeUni
menu = Cambridge University
title = Cambridge University
host = cam.ac.uk

#++ UEA
#menu = University of East Anglia
#title = University of East Anglia
#host = www.uea.ac.uk

++ CernIXP
menu = CernIXP
title = Cern Internet eXchange Point
host = cixp.web.cern.ch

#++ SBB
#menu = SBB Swiss Federal Railways
#title = SBB Swiss Federal Railways
#host = www.sbb.ch/en

++ MIT
menu = MIT
title = Massachusetts Institute of Technology
host = web.mit.edu

++ IU
menu = Indiana University
title = Indiana University
host = www.indiana.edu

++ UCB
menu = University of California Berkeley
title = University of California Berkeley
host = www.berkeley.edu

++ UCSD
menu = University of California San Diego
title = University of California San Diego
host = ucsd.edu

++ UMN
menu =  University of Minnesota
title = University of Minnesota
host = twin-cities.umn.edu

++ OSUOSL
menu = Oregon State University OSL
title = Oregon State University Open Source Lab
host = osuosl.org


+ DNS
menu = DNS
title = DNS

++ CloudflareDNS1                                
menu = Cloudflare DNS 1                          
title = Cloudflare DNS 1.1.1.1                    
host = 1.1.1.1                                    
                                                 
++ CloudflareDNS2                                
menu = Cloudflare DNS 2                          
title = Cloudflare DNS 1.0.0.1                    
host = 1.0.0.1

++ CloudflareDNS3                                
menu = Cloudflare Family DNS 1                        
title = Cloudflare Family DNS 1.1.1.3                    
host = 1.1.1.3
                                                 
++ CloudflareDNS4                                
menu = Cloudflare Family DNS 2                          
title = Cloudflare Family DNS 1.0.0.3                    
host = 1.0.0.3

++ GoogleDNS1
menu = Google DNS 1
title = Google DNS 8.8.8.8
host = 8.8.8.8

++ GoogleDNS2
menu = Google DNS 2
title = Google DNS 8.8.4.4
host = 8.8.4.4

++ OpenDNS1
menu = OpenDNS 1
title = OpenDNS 1
host = 208.67.222.222

++ OpenDNS2
menu = OpenDNS 2
title = OpenDNS 2
host = 208.67.220.220

++ L3-1
menu = Level3 DNS 1
title = Level3 DNS 4.2.2.1
host = 4.2.2.1

++ L3-2
menu = Level3 DNS 2
title = Level3 DNS 4.2.2.2
host = 4.2.2.2

++ Quad9
menu = Quad9
title = Quad9 DNS 9.9.9.9
host = 9.9.9.9


+ DNSProbes
menu = DNS Probes
title = DNS Probes
probe = DNS

++ CloudflareDNS1                                
menu = Cloudflare DNS 1                          
title = Cloudflare DNS 1.1.1.1                    
host = 1.1.1.1                                    
                                                 
++ CloudflareDNS2                                
menu = Cloudflare DNS 2                          
title = Cloudflare DNS 1.0.0.1                    
host = 1.0.0.1

++ CloudflareDNS3                                
menu = Cloudflare Family DNS 1                          
title = Cloudflare Family DNS 1.1.1.3                    
host = 1.1.1.3
                                                 
++ CloudflareDNS4                                
menu = Cloudflare Family DNS 2                          
title = Cloudflare Family DNS 1.0.0.3                    
host = 1.0.0.3

++ GoogleDNS1
menu = Google DNS 1
title = Google DNS 8.8.8.8
host = 8.8.8.8

++ GoogleDNS2
menu = Google DNS 2
title = Google DNS 8.8.4.4
host = 8.8.4.4

++ OpenDNS1
menu = OpenDNS 1
title = OpenDNS 1
host = 208.67.222.222

++ OpenDNS2
menu = OpenDNS 2
title = OpenDNS 2
host = 208.67.220.220

++ L3-1
menu = Level3 DNS 1
title = Level3 DNS 4.2.2.1
host = 4.2.2.1

++ L3-2
menu = Level3 DNS 2
title = Level3 DNS 4.2.2.2
host = 4.2.2.2

++ Quad9
menu = Quad9
title = Quad9 DNS 9.9.9.9
host = 9.9.9.9