Fastest ISP?

[seowbin]

Which ISP can be lowest latency? Like no such ISP?

Got but expensive.

They market it as ultra low latency network for financial trading. I have 1 link between SG and HK is that category

 

[Henry Ng]

Got but expensive.

They market it as ultra low latency network for financial trading. I have 1 link between SG and HK is that category

You referring to commercial ISP? I think no home user ISP can do that.

 

[seowbin]

You referring to commercial ISP? I think no home user ISP can do that.

Of course la.

Most of them don't even have overseas POP.

 

[froztheart]

MR GAMER

https://www.meter.net/tools/world-ping-test/#0-449391-733f09

 

[seowbin]

 

[probablye]

I wish to consult the network gurus here on something please.

Why does Singtel often perform better (in my experience at least) despite always having the worst results?

Previously I had Viewqwest and Singtel broadband. I now use MyRepublic (Gamer), with the Singtel line expiring soon but I still have access to it.

I don't watch local TV, so I stream video from all around the world, mostly US, UK and Australia, sometimes there's a Switzerland based server also.

Here's my frustration - no matter how much better the latency and routing are on VQ and MR, ST always streams better from these video services in comparison!

There's one particular US service hosted on FDCServers in LA which is particularly problematic. But the thing is, whenever I get buffering on VQ/MR, ST would stream without issue. I've even tried Cloudflare Warp but somehow ST still manages to maintain a decent stream when all else fails.

I put this question to AI and it tells me this:
1) Even though both ST and MR use Tata's uplink to the US, MR's link could be congested / less direct;
2) ST shows some local congestion but their upstream transit has bigger capacity and therefore performs better / does not suffer from congestion overseas;
3) MR has low latency but this is good only for short bursts of low data traffic (like gaming), but for HD video streaming, an ISP with higher bandwidth could work better, even despite lousier routing that hairpins through the US;
4) As a Tier 1 ISP, ST could have a private arrangement for priority traffic even when the routes appear to be the same;
5) Even though MR is now part of Starhub, MR still has its own ASN and may be bottlenecked by the amount of capacity that SH shares with it.
6) Overall, it's still better to use an ISP like ST that has Tier 1 peers and upstream providers, because they own the subsea cables and have excess bandwidth to cater for their large customer base.

I truly hate ST with a vengeance. But if I didn't look at all these traceroutes and ping tests, I have to admit that it tends to work better.

Can anyone explain? Are the points made by AI correct? I would really like to understand this better.

 

[xiaofan]

I wish to consult the network gurus here on something please.

Why does Singtel often perform better (in my experience at least) despite always having the worst results?

Previously I had Viewqwest and Singtel broadband. I now use MyRepublic (Gamer), with the Singtel line expiring soon but I still have access to it.

I don't watch local TV, so I stream video from all around the world, mostly US, UK and Australia, sometimes there's a Switzerland based server also.

Here's my frustration - no matter how much better the latency and routing are on VQ and MR, ST always streams better from these video services in comparison!

There's one particular US service hosted on FDCServers in LA which is particularly problematic. But the thing is, whenever I get buffering on VQ/MR, ST would stream without issue. I've even tried Cloudflare Warp but somehow ST still manages to maintain a decent stream when all else fails.

I put this question to AI and it tells me this:
1) Even though both ST and MR use Tata's uplink to the US, MR's link could be congested / less direct;
2) ST shows some local congestion but their upstream transit has bigger capacity and therefore performs better / does not suffer from congestion overseas;
3) MR has low latency but this is good only for short bursts of low data traffic (like gaming), but for HD video streaming, an ISP with higher bandwidth could work better, even despite lousier routing that hairpins through the US;
4) As a Tier 1 ISP, ST could have a private arrangement for priority traffic even when the routes appear to be the same;
5) Even though MR is now part of Starhub, MR still has its own ASN and may be bottlenecked by the amount of capacity that SH shares with it.
6) Overall, it's still better to use an ISP like ST that has Tier 1 peers and upstream providers, because they own the subsea cables and have excess bandwidth to cater for their large customer base.

I truly hate ST with a vengeance. But if I didn't look at all these traceroutes and ping tests, I have to admit that it tends to work better.

Can anyone explain? Are the points made by AI correct? I would really like to understand this better.

Very good point that benchmarking results may not be the same as real world experiences.

Some possible reasons.

1) IPv6, are you using IPv6 with SingTel? MR does not have IPv6. VQ Static IPv6 is free but you need to request for it.

2) CGNAT, are you paying for Static IPv4 for MR/VQ? Singtel has dynamic IPv4.

3) Have you compared the traceroute or better mtr results?

 

[seowbin]

Singtel have POP in US , maybe the server return via zayo in US via it ? by right viewqwest have POP in US, potentially better quality

The rest of the ISP here no overseas POP. Once handover the packet to upstream provider then quality cannot be assured

That's why my side build our own POP also.

 

[probablye]

so despite all the talk about better latency, it's true then that isn't necessary better in all scenarios?

AFAIK, ST and VQ are the only two ISPs here with their own POPs. can we say then that they generally will perform better?

@seowbin I think everyone has been waiting forever for you to launch your ISP service!

 

[probablye]

1) IPv6, are you using IPv6 with SingTel? MR does not have IPv6. VQ Static IPv6 is free but you need to request for it.

2) CGNAT, are you paying for Static IPv4 for MR/VQ? Singtel has dynamic IPv4.

3) Have you compared the traceroute or better mtr results?

1) I have tried IPv6. Did you realise that in the past few months, ST has removed AS3758's direct IPv6 upstreams? Now they just route everything via the parent AS7473. I have found that to be terrible, because the routing for a lot of services (like FB) go through the US, instead of using the local CDN.

2) Yes I have static IPv4.

3) The frustrating thing is that even when ST's traceroute is far worse, it still streams better.

For example, this route to Zurich. You can see MR's route is optimised with fewer hops but still I couldn't stream anything at all, but ST's connection had no problem even though it goes one big round to the US first. The AI says it could be because MR's route goes via Cogent in Singapore, which is "known to be congested".

Target Name: 142-215-190-186.clients.gthost.com
IP: 186.190.215.142
Hop Sent PL% Min Max Avg Host Name / [IP]
1 11 0 8.87 16.07 11.05 bb151-192-223-199.singnet.com.sg [151.192.223.199]
2 11 0 11.94 30.59 15.99 bb151-192-223-254.singnet.com.sg [151.192.223.254]
3 11 18 11.38 14.18 12.60 165.21.193.234 [165.21.193.234]
4 11 36 11.15 23.56 13.83 165.21.193.233 [165.21.193.233]
5 11 0 11.56 15.72 13.08 165.21.139.169 [165.21.139.169]
6 11 0 10.94 28.86 15.31 165.21.139.134 [165.21.139.134]
7 11 0 10.51 17.65 13.81 203.208.143.129 [203.208.143.129]
8 11 0 11.66 47.39 19.41 203.208.151.38 [203.208.151.38]
9 11 0 201.51 202.73 202.05 203.208.173.106 [203.208.173.106]
10 11 0 183.18 189.33 184.99 203.208.172.234 [203.208.172.234] 11
11 0 200.00 208.94 202.66 ae5.mpr1.pao1.us.zip.zayo.com [64.125.35.189]
12 7 57 201.82 208.07 204.65 ae13.cs4.sjc4.us.zip.zayo.com [64.125.23.208]
13 11 100 0 0 0 [-]
14 11 100 0 0 0 [-]
15 11 0 191.94 197.11 193.47 be3670.ccr22.sfo01.atlas.cogentco.com [154.54.43.13]
16 11 55 218.22 234.52 222.70 be3906.ccr82.slc03.atlas.cogentco.com [154.54.5.153]
17 11 0 221.00 222.88 221.70 be3322.ccr22.den01.atlas.cogentco.com [154.54.163.249]
18 11 0 228.57 233.30 230.37 be3486.ccr82.den01.atlas.cogentco.com [154.54.90.22]
19 11 0 224.87 229.67 226.83 be8568.ccr32.oma02.atlas.cogentco.com [154.54.95.110]
20 11 0 251.59 266.60 254.61 be5068.ccr42.ord01.atlas.cogentco.com [154.54.166.74]
21 11 0 246.80 251.35 248.58 port-channel2718.ccr92.cle04.atlas.cogentco.com [154.54.7.130]
22 11 0 322.10 332.67 325.11 be2994.ccr32.yyz02.atlas.cogentco.com [154.54.31.234]
23 11 0 325.96 334.03 327.30 be3260.ccr22.ymq01.atlas.cogentco.com [154.54.42.90]
24 11 0 334.09 344.28 336.40 be3043.ccr22.lpl01.atlas.cogentco.com [154.54.44.165]
25 11 0 325.92 328.79 327.07 be2183.ccr42.ams03.atlas.cogentco.com [154.54.58.70]
26 11 0 332.18 335.42 333.20 be2950.ccr42.fra05.atlas.cogentco.com [154.54.72.42]
27 11 0 356.60 377.12 363.52 be7944.ccr22.muc03.atlas.cogentco.com [154.54.75.97]
28 11 0 341.16 346.20 342.48 be3073.ccr52.zrh02.atlas.cogentco.com [130.117.0.61]
29 11 0 351.10 354.57 352.80 be2332.rcr61.b021036-0.zrh02.atlas.cogentco.com [154.54.63.74]
30 11 0 353.13 357.96 354.79 gi0-0-0-8.nr11.b047839-0.ams03.atlas.cogentco.com [149.6.176.33]
31 11 0 343.66 358.54 346.14 142-215-190-186.clients.gthost.com [186.190.215.142]

Target Name: 142-215-190-186.clients.gthost.com
IP: 186.190.215.142
Hop Sent PL% Min Max Avg Host Name / [IP]
1 7 0 0.74 1.11 0.94 local [192.168.1.1]
2 8 0 1.89 9.16 3.41 100.81.128.3 [100.81.128.3]
3 8 0 1.67 3.42 2.38 103-6-148-178.myrepublic.com.sg [103.6.148.178]
4 8 0 1.90 2.69 2.38 103-6-148-13.myrepublic.com.sg [103.6.148.13]
5 8 0 2.91 41.15 8.03 203.116.31.145 [203.116.31.145]
6 8 0 2.74 4.56 3.42 203.118.6.125 [203.118.6.125]
7 8 0 2.39 3.31 2.99 203.118.60.110 [203.118.60.110]
8 8 75 3.84 5.16 4.50 hu0-0-0-9.ccr81.sin01.atlas.cogentco.com [154.18.17.57]
9 8 0 138.25 171.62 142.99 be9043.ccr31.mrs02.atlas.cogentco.com [154.54.92.93]
10 8 0 153.57 155.25 154.15 port-channel5890.ccr91.mil02.atlas.cogentco.com [154.54.61.165]
11 8 0 158.46 162.17 159.33 be5894.ccr51.zrh02.atlas.cogentco.com [154.54.72.10]
12 8 0 151.62 152.52 152.04 be2331.rcr61.b021036-0.zrh02.atlas.cogentco.com [154.54.62.126]
13 8 0 157.82 160.00 158.73 gi0-0-0-8.nr11.b047839-0.ams03.atlas.cogentco.com [149.6.176.33]
14 8 25 151.38 153.25 152.33 142-215-190-186.clients.gthost.com [186.190.215.142]

 

[seowbin]

so despite all the talk about better latency, it's true then that isn't necessary better in all scenarios?

AFAIK, ST and VQ are the only two ISPs here with their own POPs. can we say then that they generally will perform better?

@seowbin I think everyone has been waiting forever for you to launch your ISP service!

Not mass market leh. Small company, hard to fight in broadband space. Earn too little

 

[ieatcable]

Singtel have POP in US , maybe the server return via zayo in US via it ? by right viewqwest have POP in US, potentially better quality

The rest of the ISP here no overseas POP. Once handover the packet to upstream provider then quality cannot be assured

That's why my side build our own POP also.

Actually M1 seems to have a US POP connected to Cogent but it is less known to others because it is only used for select destinations only. M1 user can ping `202.65.246.13` which is M1's overseas router in LAX. But this route is quite rare so it's as good as don't have.

                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway                                         0.0%    10    0.6   0.6   0.3   0.9   0.2
  2.|-- 138.75.x.1                                       0.0%    10    3.4   3.4   2.4   6.2   1.1
  3.|-- 182.246.65.202.unknown.m1.com.sg                 0.0%    10    3.4   2.9   2.2   5.2   0.9
  4.|-- 181.246.65.202.unknown.m1.com.sg                 0.0%    10    2.7   3.1   2.1   5.8   1.2
  5.|-- 221.246.65.202.unknown.m1.com.sg                 0.0%    10    3.5   3.1   2.7   3.5   0.3
  6.|-- 13.246.65.202.unknown.m1.com.sg                  0.0%    10  163.5 163.6 162.8 164.4   1.2
  7.|-- te0-10-0-7-0.101.ccr41.lax04.atlas.cogentco.com  0.0%    10  163.9 163.9 163.7 164.1   0.1
  8.|-- gtt.lax04.atlas.cogentco.com                     0.0%    10  164.0 165.0 163.5 172.4   2.7
  9.|-- ae1.cr5-sof1.ip4.gtt.net                        70.0%    10  254.3 254.0 253.5 254.3   0.4

 

[ieatcable]

I wish to consult the network gurus here on something please.

Why does Singtel often perform better (in my experience at least) despite always having the worst results?

Previously I had Viewqwest and Singtel broadband. I now use MyRepublic (Gamer), with the Singtel line expiring soon but I still have access to it.

I don't watch local TV, so I stream video from all around the world, mostly US, UK and Australia, sometimes there's a Switzerland based server also.

Here's my frustration - no matter how much better the latency and routing are on VQ and MR, ST always streams better from these video services in comparison!

There's one particular US service hosted on FDCServers in LA which is particularly problematic. But the thing is, whenever I get buffering on VQ/MR, ST would stream without issue. I've even tried Cloudflare Warp but somehow ST still manages to maintain a decent stream when all else fails.

I put this question to AI and it tells me this:
1) Even though both ST and MR use Tata's uplink to the US, MR's link could be congested / less direct;
2) ST shows some local congestion but their upstream transit has bigger capacity and therefore performs better / does not suffer from congestion overseas;
3) MR has low latency but this is good only for short bursts of low data traffic (like gaming), but for HD video streaming, an ISP with higher bandwidth could work better, even despite lousier routing that hairpins through the US;
4) As a Tier 1 ISP, ST could have a private arrangement for priority traffic even when the routes appear to be the same;
5) Even though MR is now part of Starhub, MR still has its own ASN and may be bottlenecked by the amount of capacity that SH shares with it.
6) Overall, it's still better to use an ISP like ST that has Tier 1 peers and upstream providers, because they own the subsea cables and have excess bandwidth to cater for their large customer base.

I truly hate ST with a vengeance. But if I didn't look at all these traceroutes and ping tests, I have to admit that it tends to work better.

Can anyone explain? Are the points made by AI correct? I would really like to understand this better.

Since you are consuming content, a.k.a downloading, this means the inbound route is a bigger factor here than your outbound, so a traceroute from the server to your ISP could reveal why is that the case. You got any other example IP from other countries with the issues? The IP you provide from Zurich belongs to GTHost which has a looking glass for us to see the routes, but not a download test file

HOST: lg-zrh.gthost.com Loss% Snt Last Avg Best Wrst StDev
  1. 134.195.199.161 0.0% 10 0.3 0.4 0.3 0.6 0.1
  2. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  3. be2331.ccr51.zrh02.atlas.cogentco.com 0.0% 10 1.2 1.3 1.2 1.6 0.1
  4. be3072.ccr21.muc03.atlas.cogentco.com 0.0% 10 6.2 6.2 6.0 6.6 0.2
  5. be5516.ccr41.fra05.atlas.cogentco.com 0.0% 10 11.7 11.6 11.4 11.7 0.1
  6. be3763.agr31.fra05.atlas.cogentco.com 0.0% 10 11.8 11.8 11.7 11.9 0.1
  7. zayo.ae14.mcs1.fra9.de.eth.zayo.com 0.0% 10 25.1 21.5 11.3 50.4 14.8
  8. ae21.cr2.fra9.de.zip.zayo.com 0.0% 10 163.5 163.4 163.2 163.7 0.1
  9. ae20.cr1.cdg11.fr.zip.zayo.com 0.0% 10 162.1 162.1 162.0 162.2 0.1
 10. ae4.cs1.cdg12.fr.zip.zayo.com 0.0% 10 163.3 163.2 163.0 163.5 0.1
 11. ae26.cs3.iad93.us.eth.zayo.com 80.0% 10 163.4 163.4 163.4 163.4 0.0
 12. ae20.cr1.iad21.us.zip.zayo.com 70.0% 10 159.0 158.9 158.9 159.0 0.1
 13. ae14.cr1.dfw1.us.zip.zayo.com 40.0% 10 162.5 162.5 162.4 162.6 0.1
 14. ae0.cr2.dfw7.us.zip.zayo.com 30.0% 10 162.3 162.3 162.1 162.6 0.2
 15. ae2.cr1.lax10.us.zip.zayo.com 30.0% 10 162.6 162.5 162.3 162.6 0.1
 16. ae8.cs1.lax112.us.zip.zayo.com 90.0% 10 162.3 162.3 162.3 162.3 0.0
 17. ae13.mpr1.lax12.us.zip.zayo.com 0.0% 10 168.8 164.7 162.9 168.8 2.0
 18. 64.125.35.198.IPYX-145641-003-ZYO.zip.zayo.com 0.0% 10 157.8 158.0 157.7 158.7 0.4
 19. 203.208.154.69 11.1% 9 332.6 333.8 332.3 342.0 3.4
 20. 203.208.158.10 11.1% 9 346.5 351.6 346.3 383.7 13.0
 21. 203.208.158.9 11.1% 9 326.4 325.6 325.3 326.4 0.3
 22. 203.208.183.90 11.1% 9 342.8 342.6 342.4 342.8 0.1
 23. 203.208.149.26 12.5% 8 332.1 336.6 331.7 364.4 12.2
 24. 203.208.177.214 12.5% 8 327.9 327.4 327.1 327.9 0.2
 25. SN-SINQT1-BO307-ae4.singnet.com.sg 12.5% 8 347.9 358.9 347.7 425.2 29.2
 26. bb151-x-x-1.singnet.com.sg 12.5% 8 332.8 333.0 332.8 333.1 0.1

HOST: lg-zrh.gthost.com Loss% Snt Last Avg Best Wrst StDev
  1. 134.195.199.161 0.0% 10 0.3 0.5 0.3 2.1 0.6
  2. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  3. be2332.ccr52.zrh02.atlas.cogentco.com 0.0% 10 1.7 1.7 1.5 1.9 0.1
  4. port-channel5893.ccr92.mil02.atlas.cogentco.com 0.0% 10 4.6 4.5 4.4 4.6 0.1
  5. be5889.ccr32.mrs02.atlas.cogentco.com 0.0% 10 13.2 13.3 13.2 13.6 0.1
  6. be9042.ccr81.sin01.atlas.cogentco.com 80.0% 10 153.2 153.5 153.2 153.9 0.5
  7. capitalonline.demarc.cogentco.com 0.0% 10 152.0 151.7 151.3 153.7 0.7
  8. 203.118.6.126 0.0% 10 199.1 163.0 158.0 199.1 12.9
  9. 203.116.31.146 30.0% 10 152.9 153.0 152.9 153.1 0.1
 10. 103-6-148-175.myrepublic.com.sg 50.0% 10 151.9 151.9 151.8 152.0 0.1
 11. 3-204-96-66.myrepublic.com.sg 50.0% 10 153.6 153.7 153.6 153.8 0.1
 12. 88-204-96-66.myrepublic.com.sg 10.0% 10 151.7 151.7 151.5 151.8 0.1

@seowbin is right. GTHost route to ST is EU (Cogent>Zayo) > US (ST POP) > SG. For MR and SH is EU (Cogent) > SG (Cogent>SH). For VQ, EU (Hurricane Electric) > SG (Hurricane Electric>VQ). M1 is EU (GTT>PCCW) > SG (PCCW>M1).

But without a download file or sample stream, it is hard to rule out a throughput issue caused by routing. I find it hard to believe that Cogent and HE link from EU to SG is congested, in fact I find that they are the most stable ones so far among Tier 1 (NTT and Arelion are bad lately).

 

[probablye]

Thanks for this info, @ieatcable! Is this routing publicly available information, or you just do a looking glass to check?

The problematic server I encountered was hosted on FDCservers. And you seem to be right! The return path to MR is via NTT (which is causing the issue?)

traceroute to (my MR IP), 30 hops max, 60 byte packets
1 lg-lax (172.31.0.1) 0.084 ms 0.008 ms 0.009 ms
2 * * *
3 * * *
4 ae-20.a02.lsanca20.us.bb.gin.ntt.net (157.238.225.30) 1.080 ms 1.069 ms 1.056 ms
5 ae-6.r26.lsanca07.us.bb.gin.ntt.net (129.250.2.181) 1.191 ms 1.265 ms 1.375 ms
6 * * ae-5.r27.osakjp02.jp.bb.gin.ntt.net (129.250.2.177) 106.553 ms
7 ae-7.r24.sngpsi07.sg.bb.gin.ntt.net (129.250.2.66) 374.634 ms 339.855 ms *
8 ae-15.a03.sngpsi07.sg.bb.gin.ntt.net (129.250.6.65) 437.314 ms 437.298 ms 358.084 ms
9 ce-4-1-3.a03.sngpsi07.sg.ce.gin.ntt.net (128.241.6.249) 349.144 ms 363.902 ms 362.263 ms
10 203.116.3-78.unknown.starhub.net.sg (203.116.3.78) 360.018 ms 337.076 ms 203.118.6.126 (203.118.6.126) 331.272 ms
11 203.116.31.146 (203.116.31.146) 346.443 ms 362.044 ms 354.407 ms
12 103-6-148-14.myrepublic.com.sg (103.6.148.14) 293.939 ms 360.196 ms 332.663 ms

this is Singtel's return path:
traceroute to (my ST IP), 30 hops max, 60 byte packets
1 lg-lax (172.31.0.1) 0.038 ms 0.008 ms 0.008 ms
2 * * *
3 50.7.229.16 (50.7.229.16) 0.239 ms * *
4 ae-20.a02.lsanca20.us.bb.gin.ntt.net (157.238.225.30) 0.925 ms 0.913 ms 0.890 ms
5 ae-6.r27.lsanca07.us.bb.gin.ntt.net (129.250.2.115) 1.185 ms 1.345 ms ae-6.r26.lsanca07.us.bb.gin.ntt.net (129.250.2.181) 1.059 ms
6 ae-1.a03.lsanca20.us.bb.gin.ntt.net (129.250.3.178) 6.482 ms ae-0.a03.lsanca20.us.bb.gin.ntt.net (129.250.3.94) 0.998 ms ae-1.a03.lsanca20.us.bb.gin.ntt.net (129.250.3.178) 1.643 ms
7 * * *
8 * * *
9 * * *
10 64.125.35.202.IPYX-145641-004-ZYO.zip.zayo.com (64.125.35.202) 1.043 ms 1.041 ms 3.643 ms
11 203.208.171.117 (203.208.171.117) 1.236 ms 1.223 ms 1.107 ms
12 203.208.154.69 (203.208.154.69) 168.046 ms 203.208.158.9 (203.208.158.9) 194.153 ms 203.208.154.69 (203.208.154.69) 168.721 ms
13 203.208.158.10 (203.208.158.10) 170.056 ms 170.269 ms 203.208.154.57 (203.208.154.57) 207.828 ms
14 203.208.183.82 (203.208.183.82) 189.696 ms 170.624 ms 203.208.149.26 (203.208.149.26) 170.768 ms
15 * 203.208.158.161 (203.208.158.161) 191.108 ms 190.972 ms
16 203.208.143.130 (203.208.143.130) 209.468 ms 205.136 ms 197.204 ms
17 165.21.139.133 (165.21.139.133) 218.187 ms 203.208.154.57 (203.208.154.57) 211.342 ms 165.21.139.129 (165.21.139.129) 208.759 ms
18 165.21.139.129 (165.21.139.129) 238.715 ms 203.208.143.130 (203.208.143.130) 194.796 ms 165.21.139.170 (165.21.139.170) 211.139 ms
19 165.21.193.234 (165.21.193.234) 196.133 ms 165.21.139.133 (165.21.139.133) 204.390 ms 165.21.193.234 (165.21.193.234) 199.115 ms
20 * * *
21 * 165.21.139.170 (165.21.139.170) 208.323 ms 165.21.193.234 (165.21.193.234) 192.496 ms

 

[lightningroood]

Packet loss rate is the key for smooth streaming. Latency is less important.

 

[ieatcable]

Thanks for this info, @ieatcable! Is this routing publicly available information, or you just do a looking glass to check?

The problematic server I encountered was hosted on FDCservers. And you seem to be right! The return path to MR is via NTT (which is causing the issue?)

traceroute to (my MR IP), 30 hops max, 60 byte packets
1 lg-lax (172.31.0.1) 0.084 ms 0.008 ms 0.009 ms
2 * * *
3 * * *
4 ae-20.a02.lsanca20.us.bb.gin.ntt.net (157.238.225.30) 1.080 ms 1.069 ms 1.056 ms
5 ae-6.r26.lsanca07.us.bb.gin.ntt.net (129.250.2.181) 1.191 ms 1.265 ms 1.375 ms
6 * * ae-5.r27.osakjp02.jp.bb.gin.ntt.net (129.250.2.177) 106.553 ms
7 ae-7.r24.sngpsi07.sg.bb.gin.ntt.net (129.250.2.66) 374.634 ms 339.855 ms *
8 ae-15.a03.sngpsi07.sg.bb.gin.ntt.net (129.250.6.65) 437.314 ms 437.298 ms 358.084 ms
9 ce-4-1-3.a03.sngpsi07.sg.ce.gin.ntt.net (128.241.6.249) 349.144 ms 363.902 ms 362.263 ms
10 203.116.3-78.unknown.starhub.net.sg (203.116.3.78) 360.018 ms 337.076 ms 203.118.6.126 (203.118.6.126) 331.272 ms
11 203.116.31.146 (203.116.31.146) 346.443 ms 362.044 ms 354.407 ms
12 103-6-148-14.myrepublic.com.sg (103.6.148.14) 293.939 ms 360.196 ms 332.663 ms

this is Singtel's return path:
traceroute to (my ST IP), 30 hops max, 60 byte packets
1 lg-lax (172.31.0.1) 0.038 ms 0.008 ms 0.008 ms
2 * * *
3 50.7.229.16 (50.7.229.16) 0.239 ms * *
4 ae-20.a02.lsanca20.us.bb.gin.ntt.net (157.238.225.30) 0.925 ms 0.913 ms 0.890 ms
5 ae-6.r27.lsanca07.us.bb.gin.ntt.net (129.250.2.115) 1.185 ms 1.345 ms ae-6.r26.lsanca07.us.bb.gin.ntt.net (129.250.2.181) 1.059 ms
6 ae-1.a03.lsanca20.us.bb.gin.ntt.net (129.250.3.178) 6.482 ms ae-0.a03.lsanca20.us.bb.gin.ntt.net (129.250.3.94) 0.998 ms ae-1.a03.lsanca20.us.bb.gin.ntt.net (129.250.3.178) 1.643 ms
7 * * *
8 * * *
9 * * *
10 64.125.35.202.IPYX-145641-004-ZYO.zip.zayo.com (64.125.35.202) 1.043 ms 1.041 ms 3.643 ms
11 203.208.171.117 (203.208.171.117) 1.236 ms 1.223 ms 1.107 ms
12 203.208.154.69 (203.208.154.69) 168.046 ms 203.208.158.9 (203.208.158.9) 194.153 ms 203.208.154.69 (203.208.154.69) 168.721 ms
13 203.208.158.10 (203.208.158.10) 170.056 ms 170.269 ms 203.208.154.57 (203.208.154.57) 207.828 ms
14 203.208.183.82 (203.208.183.82) 189.696 ms 170.624 ms 203.208.149.26 (203.208.149.26) 170.768 ms
15 * 203.208.158.161 (203.208.158.161) 191.108 ms 190.972 ms
16 203.208.143.130 (203.208.143.130) 209.468 ms 205.136 ms 197.204 ms
17 165.21.139.133 (165.21.139.133) 218.187 ms 203.208.154.57 (203.208.154.57) 211.342 ms 165.21.139.129 (165.21.139.129) 208.759 ms
18 165.21.139.129 (165.21.139.129) 238.715 ms 203.208.143.130 (203.208.143.130) 194.796 ms 165.21.139.170 (165.21.139.170) 211.139 ms
19 165.21.193.234 (165.21.193.234) 196.133 ms 165.21.139.133 (165.21.139.133) 204.390 ms 165.21.193.234 (165.21.193.234) 199.115 ms
20 * * *
21 * 165.21.139.170 (165.21.139.170) 208.323 ms 165.21.193.234 (165.21.193.234) 192.496 ms

There does seems to be something wrong with connecting to FDC LAX. Some paths are having packet loss and 600ms ping for me (M1).

From M1.

 Host                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 2. 138.75.x.1                                           0.0%    30    3.3   3.8   2.2  13.2   2.7
 3. 182.246.65.202.unknown.m1.com.sg                      0.0%    30    2.6   2.7   2.2   3.6   0.3
 4. 181.246.65.202.unknown.m1.com.sg                      0.0%    30    2.6   5.5   2.2  37.2   7.4
 5. 221.246.65.202.unknown.m1.com.sg                      0.0%    30    3.7   5.1   2.9  26.6   4.4
    134.246.65.202.unknown.m1.com.sg
 6. te0-1-0-8.600.br03.sin02.as3491.net                   0.0%    30    3.8   4.0   3.1   4.8   0.4
 7. Hu-0-0-0-0.br07.sin02.as3491.net                     24.1%    30  1030. 2018.   3.4 7241. 2344.
 8. 63-216-144-10.static.as3491.net                       0.0%    30   16.5   7.9   2.9  40.0   8.2
 9. ae-2.r24.sngpsi07.sg.bb.gin.ntt.net                  13.3%    30  3127. 1442.   3.3 7283. 2078.
10. ae-4.r27.osakjp02.jp.bb.gin.ntt.net                  51.7%    30  7304. 2448.  84.7 7380. 2824.
11. ae-1.r26.lsanca07.us.bb.gin.ntt.net                   0.0%    30  180.5 251.4 175.6 1201. 257.0
12. ae-0.a02.lsanca20.us.bb.gin.ntt.net                   0.0%    30  648.2 691.3 553.5 1708. 197.0
13. ae-0.omnius.lsanca20.us.bb.gin.ntt.net                0.0%    30  684.6 804.6 635.7 3777. 603.0
14. (waiting for reply)
15. 50.7.229.14                                           20.1%    29  664.4 770.9 644.8 1678. 315.4

 Host                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 2. 138.75.x.1                                           0.0%    36    2.4   3.3   2.0  12.6   1.7
 3. 182.246.65.202.unknown.m1.com.sg                      0.0%    36    2.4   3.0   2.1   9.6   1.4
 4. 181.246.65.202.unknown.m1.com.sg                      0.0%    36    2.5   2.8   2.1   7.0   0.8
 5. 134.246.65.202.unknown.m1.com.sg                      0.0%    36    4.5   4.1   2.9   8.0   1.4
    221.246.65.202.unknown.m1.com.sg
 6. te0-1-0-8.600.br03.sin02.as3491.net                   0.0%    36    3.5   4.1   3.5   4.7   0.3
 7. Hu-0-0-0-0.br07.sin02.as3491.net                     34.3%    36    3.5 1306.   3.5 7113. 1758.
 8. 63-216-144-10.static.as3491.net                       0.0%    36    3.4   3.6   2.7   4.4   0.4
 9. ae-2.r24.sngpsi07.sg.bb.gin.ntt.net                   8.6%    35    4.2 356.5   3.0 3175. 679.5
    ae-2.r25.sngpsi07.sg.bb.gin.ntt.net
10. ae-13.r33.tokyjp05.jp.bb.gin.ntt.net                 32.4%    35  1093. 1400.  94.9 7312. 2205.
    ae-3.r23.sydnau05.au.bb.gin.ntt.net
    ae-4.r27.osakjp02.jp.bb.gin.ntt.net
11. ae-1.r26.lsanca07.us.bb.gin.ntt.net                   5.7%    35  199.1 240.4  94.9 1192. 305.5
    ae-4.r27.lsanca07.us.bb.gin.ntt.net
    ae-0.r22.sydnau05.au.bb.gin.ntt.net
12. ae-0.a02.lsanca20.us.bb.gin.ntt.net                   0.0%    35  185.4 311.8 177.8 3241. 539.2
    ae-11.r26.lsanca07.us.bb.gin.ntt.net
    ae-1.a02.lsanca20.us.bb.gin.ntt.net
13. ae-0.omnius.lsanca20.us.bb.gin.ntt.net                0.0%    35  196.8 289.6 176.6 1207. 286.7
    ae-0.a02.lsanca20.us.bb.gin.ntt.net
14. ae-0.omnius.lsanca20.us.bb.gin.ntt.net               50.0%    35  213.8 2412. 213.2 7502. 3055.
15. 50.7.229.14                                          17.1%    35  180.4 301.9 175.4 3409. 597.8

From FDC to M1.

HOST: 1af6c2dd0667                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- lg-lax                     0.0%     5    0.1   0.1   0.0   0.1   0.0
  2.|-- 23.237.111.1               0.0%     5    7.2  17.4   3.1  65.1  26.7
  3.|-- 50.7.229.16               80.0%     5    0.3   0.3   0.3   0.3   0.0
  4.|-- ae-20.a02.lsanca20.us.bb.  0.0%     5    0.9   1.0   0.9   1.4   0.2
  5.|-- ae-6.r26.lsanca07.us.bb.g  0.0%     5    1.3   1.4   1.1   2.0   0.3
  6.|-- ae-4.r22.sydnau05.au.bb.g  0.0%     5  144.7 144.3 144.1 144.7   0.3
  7.|-- ae-0.r23.sydnau05.au.bb.g  0.0%     5  144.8 144.5 144.1 144.8   0.3
  8.|-- ae-10.r24.sngpsi07.sg.bb. 40.0%     5  212.9 213.3 212.9 214.1   0.7
  9.|-- ae-15.a03.sngpsi07.sg.bb.  0.0%     5  213.7 213.6 213.5 213.7   0.1
 10.|-- ntt-as2914-gw1a.sgp.inter  0.0%     5  231.8 223.8 217.2 235.0   8.9
 11.|-- 138.245.65.202.unknown.m1  0.0%     5  209.9 210.1 209.9 211.2   0.6
 12.|-- 202.246.65.202.unknown.m1  0.0%     5  207.7 207.8 207.7 208.0   0.2
 13.|-- 1.32.245.49.unknown.m1.co  0.0%     5  214.2 214.3 214.2 214.8   0.3
 14.|-- 138.75.x.x                 0.0%     5  203.8 204.0 203.8 204.2   0.2

Never seen USA to SG go via AU before. They must be quite desperate to load balance the traffic to SG if they resort to this.

No packet loss issue from my GSL server which bypasses NTT.

HOST:                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                                      100.0    10    0.0   0.0   0.0   0.0   0.0
  2.|-- l880.sg-eqxsg3-cr7.globalsecurelayer.com  0.0%    10    1.0   1.1   1.0   1.2   0.1
  3.|-- ???                                      100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- ???                                      100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- ???                                      100.0    10    0.0   0.0   0.0   0.0   0.0
  6.|-- ???                                      100.0    10    0.0   0.0   0.0   0.0   0.0
  7.|-- 101.203.74.99                             0.0%    10  164.8 266.3 164.7 1178. 320.5
  8.|-- ???                                      100.0    10    0.0   0.0   0.0   0.0   0.0
  9.|-- 50.7.229.14                               0.0%    10  165.3 165.7 165.0 167.5   0.7

 

[seowbin]

@seowbin is right. GTHost route to ST is EU (Cogent>Zayo) > US (ST POP) > SG. For MR and SH is EU (Cogent) > SG (Cogent>SH). For VQ, EU (Hurricane Electric) > SG (Hurricane Electric>VQ). M1 is EU (GTT>PCCW) > SG (PCCW>M1).

But without a download file or sample stream, it is hard to rule out a throughput issue caused by routing. I find it hard to believe that Cogent and HE link from EU to SG is congested, in fact I find that they are the most stable ones so far among Tier 1 (NTT and Arelion are bad lately).

I prepend my as number a lot of times to HE yet this GTHost return traffic via it. Interesting. Either they are in IX or maybe HE is too cheap to resist.

I only want to ask... Can SIC the content ?

 

[Mach3.2]

HE's link to HK and Korea congested is it?

I've been seeing multi week ping spikes to servers in HK and Seoul via HE.

 

[Henry Ng]

HE's link to HK and Korea congested is it?

I've been seeing multi week ping spikes to servers in HK and Seoul via HE.

If the network overseas congested then not our singapore ISP fault. It will need to depend on the overseas service providers.

 

[ieatcable]

I prepend my as number a lot of times to HE yet this GTHost return traffic via it. Interesting. Either they are in IX or maybe HE is too cheap to resist.

I only want to ask... Can SIC the content ?

You having problem with HE or Cogent? Likely is IX connection because from HE looking glass it shows to GTHost prefix is not a customer route. Yes a lot of network prefer IX even with a lot of prepending. Recently Telstra are advertising M1 prefixes in a few overseas IX and messing the inbound routing for us becos their network sucks but other ISP prefer to use this route (even though it must hop thru SH first) because dunnid need to pay expensive transit fees.

HE's link to HK and Korea congested is it?

I've been seeing multi week ping spikes to servers in HK and Seoul via HE.

Interesting. I just try and also having spikes to HK with HE. No issue on GSL. Why everything so congested recently ah? Major attacks going on?

 Host                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. x                                                     0.0%   178   12.2   2.6   0.7  31.6   5.6
 2. e0-50.switch1.sin4.he.net                            84.8%   178   34.2  20.1   1.0  92.7  27.8
 3. port-channel1.core3.sin1.he.net                      91.5%   178    1.6   3.2   0.9  23.8   5.9
 4. port-channel9.core2.hkg1.he.net                      86.4%   178   65.8  80.3  59.9 146.3  22.6
 5. port-channel5.core2.hkg2.he.net                      17.4%   178   63.7  72.2  55.5  89.6   5.7
 6. 9312.hkg.equinix.com                                  0.0%   178   62.9  72.9  32.8  90.2   6.2
 7. 103.193.131.83.static.xtom.com                       41.8%   178   84.3 549.7  36.1 9064. 1654.
 8. hk5.as.node.cdn-perfprod.com                          0.4%   177   74.1  71.0  31.2  78.9   6.0

Traceroute from 45.128.220.79 to x:
1 45.128.220.1 17.129ms 13.769ms 10.418ms
2 45.90.209.34.static.xtom.com (45.90.209.34) 0.695ms 0.67ms 0.671ms
3 103.193.131.76.static.xtom.com (103.193.131.76) 0.249ms 0.37ms 0.31ms
4 103.193.131.87.static.xtom.com (103.193.131.87) 2.346ms 10.29ms 1.505ms
5 9312.hkg.equinix.com (36.255.56.179) 0.362ms 0.418ms 0.388ms
6 * * *
7 * * *
8 * port-channel3.core3.sin1.he.net (184.104.199.182) 31.745ms 32.486ms
9 * * port-channel22.core2.sin1.he.net (184.105.223.133) 31.012ms
10 xx.switch1.sin3.he.net (xx) 68.621ms 76.799ms 75.224ms
11 x (x) 72.045ms 71.58ms 73.459ms

 Host                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 2. l880.sg-eqxsg3-cr7.globalsecurelayer.com              0.0%    50    0.9   1.0   0.9   1.2   0.1
 3. unknown.globalsecurelayer.com                         0.0%    50    1.1   1.1   0.9   1.3   0.1
 4. e50.hk-eqxhk1-bb3.globalsecurelayer.com               0.0%    50   30.7  30.6  30.5  30.9   0.1
 5. po4.hk-eqxhk1-cr2.globalsecurelayer.com               0.0%    50   30.5  30.5  30.4  30.8   0.1
 6. e32.syd.sy5-cr1.gslnetworks.com                       0.0%    50   38.0  38.0  37.7  39.1   0.3
 7. 103.193.131.87.static.xtom.com                        0.0%    50   45.4  42.9  38.0  52.2   3.9
 8. hk5.as.node.cdn-perfprod.com                          0.0%    49   30.7  30.6  30.4  31.2   0.1

If the network overseas congested then not our singapore ISP fault. It will need to depend on the overseas service providers.

False.. ISP has the responsibility to ensure its upstream providers are reliable and have sufficient capacity. If they rely on poor upstream connections, that will reflect degraded service compared to another ISP that optimises their redundancy routes/peering agreements to minimise the impact of overseas congestion. But if you are saying touchwood got some extreme incident like multiple cable cuts or DC burned down which affects all ISPs, then that is a different matter.