Starting pfsense for New Users

[xiaofan]

Just tried the above FQ_CODEL limiters under pfSense CE 2.7.2 (Double NAT behind Asus RT-AX86U), the limiter works but I do not see changes in terms of download/upload latency of wired clients.

Unfortunately Singel has no good connection to Waveform.com bufferbloat test servers, I can not use that one to get meaningful test results.

Therefore I have to use OOkla SpeedTest. I can not see the differences in the following test results using wired client. In fact, it is slightly worse with the bandwidth limiters (900Mbps down, 700Mbps up).

I believe the limters can be useful for some use cases. My results may be related to the fact this is one test installation with Double NAT.

Without traffic shaper:

root@ubuntu2204ct21:~# ./speedtest

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     2.29 ms   (jitter: 0.08ms, low: 2.22ms, high: 2.45ms)
    Download:   916.38 Mbps (data used: 425.0 MB)                                                 
                  3.11 ms   (jitter: 0.68ms, low: 1.25ms, high: 6.49ms)
      Upload:   938.88 Mbps (data used: 1.1 GB)                                                 
                  5.98 ms   (jitter: 0.62ms, low: 4.55ms, high: 15.53ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/c7e5d3eb-2b19-450f-a445-bf8941f2b39d

With traffic shaper:

root@ubuntu2204ct21:~# ./speedtest -s 13623

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     2.37 ms   (jitter: 0.15ms, low: 2.17ms, high: 2.58ms)
    Download:   850.66 Mbps (data used: 385.1 MB)                                                 
                  3.00 ms   (jitter: 6.41ms, low: 1.58ms, high: 208.16ms)
      Upload:   681.82 Mbps (data used: 781.0 MB)                                                 
                  7.66 ms   (jitter: 0.85ms, low: 3.35ms, high: 15.09ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/47ae8267-f1f7-4a3e-8e4f-449c1bd0ce17

 

[xiaofan]

If I use all the defaults with the wizard traffic_shaper_wizard_dedicated.xml and then I can see the CPU (vitual CPU, Proxmox PVE 8.2, Intel N100) is not fast enought to support 900Mbps down and 700Mbps up with PRIQ.
https://docs.netgate.com/pfsense/en/latest/trafficshaper/altq-scheduler-types.html

Virtual CPU:
QEMU Virtual CPU version 2.5+
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: Yes (inactive)
QAT Crypto: No

root@ubuntu2204ct21:~# ./speedtest -s 13623

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     2.50 ms   (jitter: 0.05ms, low: 2.42ms, high: 2.54ms)
    Download:   810.35 Mbps (data used: 385.8 MB)                                             
                  3.82 ms   (jitter: 11.87ms, low: 1.90ms, high: 213.07ms)
      Upload:   343.03 Mbps (data used: 399.3 MB)                                             
                  2.51 ms   (jitter: 11.79ms, low: 1.46ms, high: 415.63ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/364e57b0-2d22-4606-93f9-4b54725253bf

Using CBQ it will be even worse.

root@ubuntu2204ct21:~# ./speedtest -s 13623

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     2.41 ms   (jitter: 0.20ms, low: 2.09ms, high: 2.49ms)
    Download:   222.25 Mbps (data used: 285.8 MB)                                                  
                 12.07 ms   (jitter: 29.13ms, low: 1.87ms, high: 233.29ms)
      Upload:   132.55 Mbps (data used: 132.1 MB)                                                  
                 17.80 ms   (jitter: 35.86ms, low: 1.96ms, high: 429.89ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/ec81fb9d-f830-4e4f-837f-2abdd3e91b7e

I can not get HFSC to work with the wizard.

 

[xiaofan]

@Mach3.2

I know you only use bandwidth limiter. Just wondering if you have tried other traffic shaper using PRIQ/HFSC/CBQ. Thanks. I know your system is more powerful than mine.

 

[xiaofan]

From the IPv6 discussion, I just find out there is another limitation of pfSense (along with OPNsense) that NAT 66 is not supported (even though the underlying FreeBSD OS supports NAT66). NPt is supported but that is not the same as NAT66.

The use case is a bit rare though. On the other hand, I have no problems to get NAT66 to work under OpenWRT.
https://forums.hardwarezone.com.sg/threads/ipv6-discussions.6976522/page-5

All in all, it seems to me you may want to avoid pfSense unless you are using M1. The reports in this forum is that pfSense does not work with native IPv6 with SingTel and Starhub.

I myself tested pfSense with native Singtel IPv6 and it did not work. For Starhub, there were multiple reports.

Edit to update on 31-May-2024: it seems to work with Singtel IPv6 now.
Edit to update on 1-June-2014: it is reported that pfSense works with Starhub IPv6 now.

Not sure why but now my pfSense need to check the tickboxes for the following for IPV6 to work

Send IPv6 prefix hint
Do not wait for a RA

So far I at least can get IPV6 reliably, before that the IPV6 like almost non-existant for quite some time already. Not sure if Starhub did something at the back end recently to require these new settings; last time I just set bogon network off can liao as instructed here and on Zit Seng's blog

I can't test much about speed difference or latency between the two unfortunately but at least it feels snappier? Will need some more time to see how

 

[Mach3.2]

@Mach3.2

I know you only use bandwidth limiter. Just wondering if you have tried other traffic shaper using PRIQ/HFSC/CBQ. Thanks. I know your system is more powerful than mine.

I'll take a look when I'm free, but i remember there's a reason why i went with bandwidth limiter instead

 

[Mach3.2]

@Mach3.2

I know you only use bandwidth limiter. Just wondering if you have tried other traffic shaper using PRIQ/HFSC/CBQ. Thanks. I know your system is more powerful than mine.

So i took a look, and basically i didn't use traffic shaper because it requires ALTQ support on the NIC drivers, which isn't present on both the vmware vmxnet3 and Mellanox mlx5 drivers.


LTS just uploaded a video on traffic limiter too.

 

[xiaofan]

I decided to switch from OpenWRT to pfSense CE 2.7.2 again and I will use it for some time.

Name	pfSensen100new.home.arpa
User	admin@192.168.88.10 (Local Database)
System	KVM Guest Netgate Device ID: 4b8afbd56b01722afa34
Version	2.7.2-RELEASE (amd64) built on Thu Dec 7 4:10:00 +08 2023 FreeBSD 14.0-CURRENT The system is on the latest version. Version information updated at Thu May 30 18:59:04 +08 2024
CPU Type	QEMU Virtual CPU version 2.5+ 2 CPUs: 1 package(s) x 2 core(s) AES-NI CPU Crypto: Yes (active) QAT Crypto: No
Hardware crypto	AES-CBC, AES-CCM, AES-GCM, AES-ICM, AES-XTS
Kernel PTI	Enabled
MDS Mitigation	Inactive

Right now, it is quite plain with only a few packages installed (pfBlockerNG-devel, iperf, wireguard and tailscale). But Singtel native IPv6 does not work as expected -- the WAN interface does not get an IPv6 address from Singtel. I will check again later to see if Singtel IPv6 magically works again.

87% on the AdBlock test for now.
https://d3ward.github.io/toolz/adblock.html

Duck DDNS works fine.
Tailscale is now working.
Basic bandwidth limiter is working.

Somehow Wireguard does not work -- the setup seems to be pretty simple but it does not work...

 

[xiaofan]

Right now, it is quite plain with only a few packages installed (pfBlockerNG-devel, iperf, wireguard and tailscale). But Singtel native IPv6 does not work as expected -- the WAN interface does not get an IPv6 address from Singtel. I will check again later to see if Singtel IPv6 magically works again.

Good, today Singtel IPv6 seems to work well. No issues with IPv6 from the pfSense router or from the wireless clients.

pfSense CE 2.7.2 virtual router running on Proxmox PVE 8.2 Intel N100 mini PC -- ZTE BE7200 PRO+ in AP mode -- wireless clients

From pfSense

[2.7.2-RELEASE][root@pfSensen100new.home.arpa]/root: uname -a
FreeBSD pfSensen100new.home.arpa 14.0-CURRENT FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec  6 20:45:47 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/amd64.amd64/sys/pfSense amd64

[2.7.2-RELEASE][root@pfSensen100new.home.arpa]/root: ping -c 4 ipv6.google.com
PING6(56=40+8+8 bytes) 2400:d802:1c18::3:b7a --> 2404:6800:4003:c1a::71
16 bytes from 2404:6800:4003:c1a::71, icmp_seq=0 hlim=55 time=3.169 ms
16 bytes from 2404:6800:4003:c1a::71, icmp_seq=1 hlim=55 time=3.816 ms
16 bytes from 2404:6800:4003:c1a::71, icmp_seq=2 hlim=55 time=3.041 ms
16 bytes from 2404:6800:4003:c1a::71, icmp_seq=3 hlim=55 time=3.414 ms

--- ipv6.l.google.com ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 3.041/3.360/3.816/0.295 ms

Wireless client

PS C:\work\speedtest> ping ipv6.google.com

Pinging ipv6.l.google.com [2404:6800:4003:c1a::66] with 32 bytes of data:
Reply from 2404:6800:4003:c1a::66: time=5ms
Reply from 2404:6800:4003:c1a::66: time=7ms
Reply from 2404:6800:4003:c1a::66: time=9ms
Reply from 2404:6800:4003:c1a::66: time=5ms

Ping statistics for 2404:6800:4003:c1a::66:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 5ms, Maximum = 9ms, Average = 6ms

 

[Mach3.2]

ipv6 see mood one

 

[xiaofan]

Somehow Wireguard does not work -- the setup seems to be pretty simple but it does not work...

Basically Split tunnel works, my Acer laptop can access the pfSense from another network. However, full tunnel mode does not work. Once I enable full tunnel mode, my Acer laptop would not have access to the Internet.

Luckily Tom Lawrence comes to the rescue again, setting the outbound NAT rule makes that work again,

Wireguard speed is quite decent -- tested using wired connection from anotjer network (Singtel 1Gbps plan).

PS C:\work\speedtest\ookla-speedtest-1.2.0-win64> .\speedtest.exe -s 13623

   Speedtest by Ookla

      Server: Singtel - Singapore (id: 13623)
         ISP: Singtel Fibre
Idle Latency:     3.84 ms   (jitter: 0.33ms, low: 3.50ms, high: 4.26ms)
    Download:   424.59 Mbps (data used: 519.4 MB)
                 11.83 ms   (jitter: 0.77ms, low: 4.18ms, high: 15.97ms)
      Upload:   378.86 Mbps (data used: 456.8 MB)
                 19.05 ms   (jitter: 4.20ms, low: 8.77ms, high: 256.67ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/00472e13-1efc-4be6-99ba-4d6e6dd3dd33

 

[xiaofan]

I wanted to learn to use ntopng and followed the instruction of Tom Lawrence below.

End result -- a complete failure. pfSense gets not so repsonsive, WAN connection got lost packets. Disable ntopng and then reboot the router sorted out the issue. Luckily Singtel IPv6 still works after the reboot.

I guess my virtual pfSense router is just not powerful enough (two virtual core and 6GB RAM, Proxmox PVE 8.2.2, Intel N100).

 

[hwzlite]

I wanted to learn to use ntopng and followed the instruction of Tom Lawrence below.

End result -- a complete failure. pfSense gets not so repsonsive, WAN connection got lost packets. Disable ntopng and then reboot the router sorted out the issue. Luckily Singtel IPv6 still works after the reboot.

I guess my virtual pfSense router is just not powerful enough (two virtual core and 6GB RAM, Proxmox PVE 8.2.2, Intel N100).

Did the usual configured with VirtIO (paravirtualized) ?

Maybe can reduce CPU load by just monitoring LAN interface, for me works pretty well on ntopng.dev channel dockerized under CasaOS.

 

[xiaofan]

Did the usual configured with VirtIO (paravirtualized) ?

Maybe can reduce CPU load by just monitoring LAN interface, for me works pretty well on ntopng.dev channel dockerized under CasaOS.

Good advice.

I was following the video and enable too many things. Now I reduce the monitoring to LAN only and just the default settings, everything is fine now. I will need to learn more about the usage and play with the settings later.

 

[xiaofan]

Good advice.

I was following the video and enable too many things. Now I reduce the monitoring to LAN only and just the default settings, everything is fine now. I will need to learn more about the usage and play with the settings later.

Okay I gave up since it became unresponsive again after two days. Disable ntopng, deleted the data and then removed the package.

 

[TanKianW]

Okay I gave up since it became unresponsive again after two days. Disable ntopng, deleted the data and then removed the package.​

Interested to run ntops can also consider running it on a separate server/container.​

 

[u_ntitled]

Hi guys i need help with choosing the right hardware for pfsense/opnsense. If i run baremetal which variant of cwwk should i get?

 

[xiaofan]

Hi guys i need help with choosing the right hardware for pfsense/opnsense. If i run baremetal which variant of cwwk should i get?

For 1Gbps/2.5Gbps plans -- I think the N100 version with quad 2.5G ports (四网先锋版）will do, 16GB/256GB will be more than enough (yet give you the option to run Linux in the future. I bought 16GB/512GB version and run Proxmox PVE8.2, no issues with running pfSense CE 2.7.2 as a VM for my Singtel 1Gbps version. I think it would work for dual 1Gbps and SIMBA 2.5Gbps plan.

For 10Gbps plan, I think they have two versions of systems with two SFP+ ports. But better check with brother @TanKianW if you want to go with 10Gbps. I have no plan to go with 10Gbs before year 2026.
1) N305 version (N95/N100/N200 will not be good enough),
2) U300E or Core-i5 1240P version (Pentium 8505 may not be good enough)

 

[u_ntitled]

For 1Gbps/2.5Gbps plans -- I think the N100 version with quad 2.5G ports (四网先锋版）will do, 16GB/256GB will be more than enough (yet give you the option to run Linux in the future. I bought 16GB/512GB version and run Proxmox PVE8.2, no issues with running pfSense CE 2.7.2 as a VM for my Singtel 1Gbps version. I think it would work for dual 1Gbps and SIMBA 2.5Gbps plan.

For 10Gbps plan, I think they have two versions of systems with two SFP+ ports. But better check with brother @TanKianW if you want to go with 10Gbps. I have no plan to go with 10Gbs before year 2026.
1) N305 version (N95/N100/N200 will not be good enough),
2) U300E or Core-i5 1240P version (Pentium 8505 may not be good enough)

Thank you, i saw there is many seller on Aliexpress but which one is the official seller and which shape cooler fins should i get?

For all you know bro @TanKianW already on 40Gbps

 

[xiaofan]

Thank you, i saw there is many seller on Aliexpress but which one is the official seller and which shape cooler fins should i get?

For all you know bro @TanKianW already on 40Gbps

I do not use AliExpress, but rather Taobao/Tmall CWWK flagship store (天猫畅网微控旗舰店).

Official website in Chinese
https://www.changwang.com/products.html?typeid=1

The model I bought:
【淘宝】https://m.tb.cn/h.g4uJU4Jml9MYiJ4?tk=OOe9WCsJDJp CZ0015 「畅网微控 N95/N100/N200/I3-N305/N300/无风扇低功耗 微型迷你工控主机软路由 英特尔12代N系列8核新成员」
点击链接直接打开 或者 淘宝搜索直接打开

 

[TanKianW]

Thank you, i saw there is many seller on Aliexpress but which one is the official seller and which shape cooler fins should i get?

For all you know bro @TanKianW already on 40Gbps

I am still on 10G at home btw. Still do not need 40G for my current use case.

Back to topic. These MIC mini PCs are a hit or miss kind of quality. So I can't vouch for any of them. More reputable vendors might be the go to choice instead of the usual no brand OEMs. Or you can go with more reputable system from minisforum. You can check out the reviews of several mini-pcs from "servethehome" website/YouTube. They have pretty much covered quite a few of these MIC mini-pc and their cooling performance. You can consider getting the bare mini-pc box from Taobao/AliExpress, but memory and storage get from local distro. I personally prefer memory from Crucial instead of the "chup ba lang" memory brand that came with the box.

If running bare metal, N100 should suffice for 1/2G. Whether it can go 10G, I have never tried. For reference, I am still using an antique old Xeon E3-1240Lv5 4C/8T CPU which still run excess on a 10G internet with IDS/IPS all on.​