[TanKianW]

With Wifi-6 gradually coming into our homes, availability of Cat6 trunking in modern HDB set up and 10Gbe switches at prices within reach to most. I hope to share my upcoming upgrade to 10Gbe network in a HDB Home setup, also provide a head start to people who are interested in such set up too.

Current network setup:


Existing Network Setup:
1) 1x Netgear GS108PE
2) 2x TPlink SG105E (For living room and kids room)
3) 1x Custom m-itx pc running pfsense 2.4.5
4) 1x Singtel ONR (Bridged at Host side)
5) 1x MR ONT with static IP
6) 2x Ruckus Unleashed R500
7) 1x Ubiquiti EdgeSwitch 10XP (For my home lab)


Project Network Setup:
1) Custom made 5U server rack at 270mm depth
2) 1x Mikrotik CRS312-4C+8XG-RM (On SwOS on WAN side)
3) 1x Mikrotik CRS305-1G-4S+IN (In home Lab)
4) Netgear GS110EMX (In home lab)
5) Transfer custom m-itx build pfsense pc to 1U server chassis
6) 2x Ruckus Unleashed R500
7) 1x Ubiquiti EdgeSwitch 10XP with rack adapter
8) 1x Singtel ONR (Bridged at Host side)
9) 1x MR ONT with static IP
10) 2x TPlink SG105E (For living room and kids room)

All Purchased Gear:

[TanKianW]

Making it a quieter system:
1) Changing the 4x 40mm fans in the Mikrotik switch to Noctua 40x20 PWM fans.
2) Changing the 2x 40mm fans in 1U-1.5U server chassis fans to Noctua 40x20 PWM fans.



Server Arrangement:
Level 5: 1U chassis running pfsense 2.4.5
Level 4: 1U adapter for Ubiquiti EdgeSwitch 10XP
Level 3: 1U Mikrotic CRS312
Level 2: Breath Gap
Level 1: Base tray for Singtel ONR & MR ONT


Mikrotik CRS305 at homelab connected to 10Gbps port on the Netgear GS110EMX via SFP+ to RJ45 modules and direct to PC through DAC.

[TanKianW]

Completed the whole 10G deployment at home including the converting of my pfsense box to a 1U chassis with 2x 40mm noctua fans.





Overall, it is a success, but with some ups and downs. The 10G in my lab and FreeNAS in rooms worked as planned. Quite like the cleaner server rack set up. If I do upgrade to 10G internet, will change the NIC in my pfsense box.

Some unfortunate event unfold during deployment.
1) While changing the chassis, the Kingston SSD boot drive died on me. Changed it to an Intel 240GB SSD.
2) Tested all the Cat6 points around the house and found that 2 out of 6 Ethernet point could not run at 10G. Which may be due to wear and tear of connection point or cable degradation. will get a contractor in to restore these data connection point.

[TanKianW]

Before:


After:




Full working setup:

[zoneguard]

What's the specs of the mITX pfsense router? Can keep up with 2x1Gbps WAN? Why don't you use Mikrotik CCR range of routers?

Remember to show some iPerf figures when the whole setup is completed.

What are you running in your home lab to justify the 10Gbe?

[TanKianW]

zoneguard wrote:

What's the specs of the mITX pfsense router? Can keep up with 2x1Gbps WAN? Why don't you use Mikrotik CCR range of routers?

Remember to show some iPerf figures when the whole setup is completed.

What are you running in your home lab to justify the 10Gbe?

i5-2500K, 8GB RAM, 128GB SSD, GA-HZZN m-itx mobo, 4x 1Gbe Intel NIC. Personal preference geared towards pfsense and more familiar with the GUI.

Both WAN1 and WAN2 are directed to 2 different VLAN (default and VLAN50) within my network. When doing speed test at two VLAN at the same time, I hit close to 1Gbps per WAN. CPU usage will hit 80% when both WAN at peak.

[zoneguard]

I'm running pfSense (virtualized) on i7-4790S with 6 x Intel GBe NICs.

WAN1: M1 1Gbps static IP.
WAN2: Whizcomms 1Gbps.

But severely underutilized the bandwidth so going to decommission 1 WAN soon.

[TanKianW]

zoneguard wrote:

I'm running pfSense (virtualized) on i7-4790S with 6 x Intel GBe NICs.

WAN1: M1 1Gbps static IP.
WAN2: Whizcomms 1Gbps.

But severely underutilized the bandwidth so going to decommission 1 WAN soon.

Your pfsense box.......much much more powerful than my Yr 2011 cpu box! Sure no sweat!

I think yours can still pretty much game very well with a GPU.

[zoneguard]

TanKianW wrote:

Your pfsense box.......much much more powerful than my Yr 2011 cpu box! Sure no sweat!

I think yours can still pretty much game very well with a GPU.

No lah, it's not bare metal like yours. Got other VMs running on it. And it is a mITX motherboard like yours and the sole PCI-e slot is taken up by the 4 x NICs so no GPU.

I don't have server rack also.

Just curious, why you went with so many different vendors instead of say all Mikrotik outfit or all Ubiquiti outfit which is more common in this forum? Like that, you need to learn and be conversant in everything - is Ruckus very different from the rest?

[TanKianW]

zoneguard wrote:

No lah, it's not bare metal like yours. Got other VMs running on it. And it is a mITX motherboard like yours and the sole PCI-e slot is taken up by the 4 x NICs so no GPU.

I don't have server rack also.

Just curious, why you went with so many different vendors instead of say all Mikrotik outfit or all Ubiquiti outfit which is more common in this forum? Like that, you need to learn and be conversant in everything - is Ruckus very different from the rest?

My decision to go server rack: wanted to optimize the available cabinet space. 3yrs ago before moving into the unit, always wanted to go with server rack when setting up my smart home. But along the way, with busy work schedule and new baby coming, resort to OTS devices and no-planning setup! Hope this time round, I can do a revamp!

Hope by sharing how I think about the different brand before I made the decision can provide you with some insight. Brand supporters please dun flame me.

Mikrotik:
Pros: Cheap 10Gbe switches, SwOS is also pretty straight forward. Reliability wise, it just works.
Cons: Don't like their ROS and GUI. Looks ancient. Documentations is not great. If you learning on your own, takes some time to learn the clunky interface and how things work.

Unifi:
Pros: Make good switches other than Netgear. Good APs too, good and simple, nothing more. Good pricing.
Cons: tries to sink you into the whole "Unifi" hole which I am not a fan of. Their UDM and USG is better than most consumer routers out there but compared to firewall/router of the same class, they are still quite far. Lots of on/off setting in their UDM/USG series but you will not know what is going on behind the scene and how they actually work. I will say give it a pass if you want pro features with greater flexibility. There are just much better solution out there.

pfsense:
Pros: Great documentation and community support. Proven reliability and versatility for both consumer and business needs. Stable updates. Fulfill basic needs if you are a novice user. I have been using pfsense since I started learning the basics of firewall, so with that experience, make it hard to switch.
Cons: You probably need to build one yourself. Do take time to understand and learn some of the functions if you are used to consumer routers.

Ruckus:
Pros: reliability, set and forget, proprietary antenna design which performs really well in "noisy" environment, require less APs in a similar environment as compared to other APs (based on experience). Stable firmware updates.
Cons: Expensive unless there is a sale. I managed to use the left-overs from one of my projects, so got it at a discount.

In the end when it comes to decision making I will choose reliability, cost, knowledge and experiences.

[xiaofan]

BTW, just wondering what prompts you to upgrade from 1Gbps to 10 Gbps for the home network. Any real use cases at home? Or is it more for future proof?

Is it possible to save some money if setting up an interim 2.5Gbps home network?

I notice that you are still using 2x1Gbps Internet (SingTel and M1). Is it really required as both are pretty reliable?

[TanKianW]

xiaofan wrote:

BTW, just wondering what prompts you to upgrade from 1Gbps to 10 Gbps for the home network. Any real use cases at home? Or is it more for future proof?

Is it possible to save some money if setting up an interim 2.5Gbps home network?

I notice that you are still using 2x1Gbps Internet (SingTel and M1). Is it really required as both are pretty reliable?

Future proofing, have some fun, transfer large files from PC to 2 NAS library in different rooms (living room and homelab).....more keen towards having some fun to set it up.

Maybe it is only me, I always wanted to start with 10Gbe 3 years ago, knowing the unit's internal cabling is Cat6. Going 1Gbe seems a waste to the existing infrastructure, and I rather future proof it.

For my case, the Mikrotik CRS312 does comes with 8x multi-gig, but my NIC only runs at 1Gbe or 10Gbe, so might as well 10Gbe. Not really sure if 2.5Gbe switches will save some money coz never had any chance to compare. I know QNAP will soon launch an affordable unmanaged 2.5Gbe switch soon. For my case I will prefer managed/smart switches which can be tag with VLANs.

Not using M1. I am running MyRepublic 1Gbps as default ISP for all LAN connected devices, Singtel 1Gbps for wifi, CCTV and IOTs. I do have some critical work linked to my workplace system which I could not have any downtime when working/control remotely from home. Having 2x ISPs are part of the BCP, even my network servers are backed with UPS just in case power trip.

[dodge888]

Very nice. I used to run the following for my home network :

- pfsense 2.4.4 with an old Dell SFF (3rd gen i5) with 4-port Intel gig NIC
- Netgear GS108PE
- Ubiquiti Unifi AC Pro
- VLANs to segment my networks

My gear all 2nd hand except for the Netgear managed switch. pfsense is still my preferred firewall. Could max out my 1gbps Internet using OpenVPN with multi-WAN setup.

[BradenHeat]

Power sia, can feel my wallet screaming at me not to get enticed

Simply epic set up you have man

P.s have you thought of funneling the fans into one way in and one way exhaust?

From the picture it seems it will be all over the place for airflow if you’re trying to keep it cool

Just curious and thinking out loud

Posted from PCWX using iPhone12,5

[TanKianW]

BradenHeat wrote:

Power sia, can feel my wallet screaming at me not to get enticed

Simply epic set up you have man

P.s have you thought of funneling the fans into one way in and one way exhaust?

From the picture it seems it will be all over the place for airflow if you’re trying to keep it cool

Just curious and thinking out loud

Posted from PCWX using iPhone12,5

Yes, will direct all fans to exhaust out of the cabinet in one direction.

The many fans set up you see in the top picture is actually my current set up. Have not upgraded yet.

In this project, all default 40mm fans will all change to Noctua 40mm silent fans. Will be replacing 6 of them, 2 on my 1U pfsense chassis, 4 on Mikrotik CRS312. Will change the direction of the Dynatron 1U CPU blower fan to blow out in same direction too.