Starting and Sharing your Homelab Projects (Include 10G Network Upgrade)

[TanKianW]

With Wifi-6 gradually coming into our homes, availability of Cat6 trunking in modern HDB set up and 10Gbe switches at prices within reach to most. I hope to share my upcoming upgrade to 10Gbe network in a HDB Home setup, also provide a head start to people who are interested in such set up too.

BEFORE:

Existing Network Setup:
1) 1x Netgear GS108PE
2) 2x TPlink SG105E (For living room and kids room)
3) 1x Custom m-itx pc running pfsense 2.4.5
4) 1x Singtel ONR (Bridged at Host side)
5) 1x MR ONT with static IP
6) 2x Ruckus Unleashed R500
7) 1x Ubiquiti EdgeSwitch 10XP (For my home lab)


Project Network Setup:
1) Custom made 5U server rack at 270mm depth
2) 1x Mikrotik CRS312-4C+8XG-RM (On SwOS on WAN side)
3) 1x Mikrotik CRS305-1G-4S+IN (In home Lab)
4) Netgear GS110EMX (In home lab)
5) Transfer custom m-itx build pfsense pc to 1U server chassis
6) 2x Ruckus Unleashed R500
7) 1x Ubiquiti EdgeSwitch 10XP with rack adapter
8) 1x Singtel ONR (Bridged at Host side)
9) 1x MR ONT with static IP
10) 2x TPlink SG105E (For living room and kids room)

All Purchased Gear:

 

[TanKianW]

Making it a quieter system:
1) Changing the 4x 40mm fans in the Mikrotik switch to Noctua 40x20 PWM fans.
2) Changing the 2x 40mm fans in 1U-1.5U server chassis fans to Noctua 40x20 PWM fans.

Server Arrangement:
Level 5: 1U chassis running pfsense 2.4.5
Level 4: 1U adapter for Ubiquiti EdgeSwitch 10XP
Level 3: 1U Mikrotic CRS312
Level 2: Breath Gap
Level 1: Base tray for Singtel ONR & MR ONT

Mikrotik CRS305 at homelab connected to 10Gbps port on the Netgear GS110EMX via SFP+ to RJ45 modules and direct to PC through DAC.

 

[TanKianW]

Sharing a Quick Overview for my Homelab upgrade/overhaul/migration (Part 1):
Sharing a quick overview of my current homelab setup since some are curious to PM me. A few projects are still pending to work out some spare time to set up and migrate. All appliances are backed with APC Back-UPS.

Setup (Updated Oct 2024):
1) Bare-metal pfSense on Xeon E3-1240Lv3 with 16GB ECC RAM, 2x 256 ZFS mirrored boot drive with 2x 1G, 2x 10G (x550) NIC. Internet is running 2x fiber ISPs (M1 1G + MR 10G) and M1 mobile 4G, set up with multi-WAN load-balancing, VLANs mapped to different WANs with cross failover and last resort mobile 4G.

Running mainly as:​

• Network Firewall/Router​
• HAProxy as reverse proxy server with ACME cert (encryption) and Load Balancer​
• VPN server (Tailscale and WireGuard)​
• pfBlocker-NG, for DNS, IP, GeoIP blocking​
• IDS/IPS on Snort​

2) Ruckus R750 x2 APs on Unleashed firmware and software management (powered on 2.5G POE port), 1x Living Room, 1x MBR​

• Optimised for wifi-calling and custom-tuned for optimal coverage and performance with minimal interference between the APs​
• Captive Portal for guests​
• Segregate SSID to different VLANs​

3) MikroTik CRS312 10G core switch (DAC) uplink to pfsense firewall, RJ45 providing 10G to all rooms through the DB patch panel over Cat 6 copper. With all internal 4cm fans changed to Noctua's.

4) Tplink TL-SE2019PB POE switch with 10G (DAC) as uplink and 2.5G POE ports powering the devices​

• Trunk VLANs from MikroTik CRS312 core switch​
• Powering MikroTik 2x HEX-S routers for Kids bedroom and TV-console​

5) PiKVM for remote access to my network appliances:

6) DB area.....might be time to get a proper labelling machine. 10GbE or 2.5GbE+POE provisions to rooms over Cat 6 patch panels.​

7) A pair of MikroTik CRS309 and CRS326 serving the home office for some work/home testing.

Overall setup:

 

[TanKianW]

Sharing a Quick Overview for my Homelab upgrade/overhaul/migration (Part 2):

The plan is to consolidate all the server systems onto one homelab rack. Mock up done.

Top to bottom:
1, Cable Matters Cat 6/6A punchless patch panel. For the lazy homelabers.

2. 2nd MikroTik CRS 312 10G homelab switch connected LACP LAGG (VLAN) Trunk ports to upstream CRS312 core switch

3. (Work in progress) "Host_03" hypervisor backup system (for high availability) Minisforum MS-01 with 2x 10G NIC (X710) as "Host_03" (backup node) with 64GB memory + Tesla P4, installed using 3d-printed bracket. will also be running XCP-ng+XOA

4. (Work in progress) Migrating "Host_01" hypervisor system. Supposed to house my "Host_01" mini-PC with 2x 10G NIC (X550), running AMD Zen 3 5950X (16C/32T), 128GB ECC RAM + GTX1660 Super. Running XCP-ng+XOA. Host_01 current state is still in an m-ITX case.

5. "Host_02" hypervisor system with 2x 10G NIC (X550), running AMD Zen 3 5950X (16C/32T), 128GB ECC RAM + Tesla P4. Running XCP-ng+XOA.

Internals:

6. TrueNAS_03 Scale storage server with AMD Zen 3 5700X (8C/16T), 128GB ECC RAM + Tesla P4, 2x 10G NIC, 12x 16TB Ironwolf Pro to set up 2 vdevs of RAIDZ1, width of 5 drives/vdev, 2 spares, mirrored boot drives, mirrored ZLOGs, setup with SMB, NFS and iSCSI for backing up hypervisor snapshots, periodic ZFS snapshots, deduplication, rsync. Also used for running some container apps with the new "Electric Eel" updates.

Internals:

To better cool the enterprise drives, a 3D-printed 3x 14cm fan custom-designed enclosure was used.

7. (Work in progress) TrueNAS_04 Scale storage server. Supposed to combine (merge) 2x 8-drives NAS on TrueNAS Core into one future 12x 16TB TrueNAS Scale system. Same spec as the above NAS. This will serve as the second backup for the main storage and at the same time run cloud sync task to AWS.

8. 2x 2200VA APC Back-UPS connected to the UPS management server to trigger a proper shutdown to the storage system after power loss for an extended period. 2x TrueNAS storage, 2x hypervisor system will be running 2U 800W redundant switch PSU from FSP or (FSP) OEMs (total of 4 units).

 

[zoneguard]

What's the specs of the mITX pfsense router? Can keep up with 2x1Gbps WAN? Why don't you use Mikrotik CCR range of routers?

Remember to show some iPerf figures when the whole setup is completed.

What are you running in your home lab to justify the 10Gbe?

 

[TanKianW]

What's the specs of the mITX pfsense router? Can keep up with 2x1Gbps WAN? Why don't you use Mikrotik CCR range of routers?

Remember to show some iPerf figures when the whole setup is completed.

What are you running in your home lab to justify the 10Gbe?

i5-2500K, 8GB RAM, 128GB SSD, GA-HZZN m-itx mobo, 4x 1Gbe Intel NIC. Personal preference geared towards pfsense and more familiar with the GUI.

Both WAN1 and WAN2 are directed to 2 different VLAN (default and VLAN50) within my network. When doing speed test at two VLAN at the same time, I hit close to 1Gbps per WAN. CPU usage will hit 80% when both WAN at peak.

 

[zoneguard]

I'm running pfSense (virtualized) on i7-4790S with 6 x Intel GBe NICs.

WAN1: M1 1Gbps static IP.
WAN2: Whizcomms 1Gbps.

But severely underutilized the bandwidth so going to decommission 1 WAN soon.

 

[TanKianW]

I'm running pfSense (virtualized) on i7-4790S with 6 x Intel GBe NICs.

WAN1: M1 1Gbps static IP.
WAN2: Whizcomms 1Gbps.

But severely underutilized the bandwidth so going to decommission 1 WAN soon.

Your pfsense box.......much much more powerful than my Yr 2011 cpu box! Sure no sweat!

I think yours can still pretty much game very well with a GPU.

 

[zoneguard]

Your pfsense box.......much much more powerful than my Yr 2011 cpu box! Sure no sweat!

I think yours can still pretty much game very well with a GPU.

No lah, it's not bare metal like yours. Got other VMs running on it. And it is a mITX motherboard like yours and the sole PCI-e slot is taken up by the 4 x NICs so no GPU.

I don't have server rack also.

Just curious, why you went with so many different vendors instead of say all Mikrotik outfit or all Ubiquiti outfit which is more common in this forum? Like that, you need to learn and be conversant in everything - is Ruckus very different from the rest?

 

[TanKianW]

No lah, it's not bare metal like yours. Got other VMs running on it. And it is a mITX motherboard like yours and the sole PCI-e slot is taken up by the 4 x NICs so no GPU.

I don't have server rack also.

Just curious, why you went with so many different vendors instead of say all Mikrotik outfit or all Ubiquiti outfit which is more common in this forum? Like that, you need to learn and be conversant in everything - is Ruckus very different from the rest?

My decision to go server rack: wanted to optimize the available cabinet space. 3yrs ago before moving into the unit, always wanted to go with server rack when setting up my smart home. But along the way, with busy work schedule and new baby coming, resort to OTS devices and no-planning setup! Hope this time round, I can do a revamp!

Hope by sharing how I think about the different brand before I made the decision can provide you with some insight. Brand supporters please dun flame me.

Mikrotik:
Pros: Cheap 10Gbe switches, SwOS is also pretty straight forward. Reliability wise, it just works.
Cons: Don't like their ROS and GUI. Looks ancient. Documentations is not great. If you learning on your own, takes some time to learn the clunky interface and how things work.

Unifi:
Pros: Make good switches other than Netgear. Good APs too, good and simple, nothing more. Good pricing.
Cons: tries to sink you into the whole "Unifi" hole which I am not a fan of. Their UDM and USG is better than most consumer routers out there but compared to firewall/router of the same class, they are still quite far. Lots of on/off setting in their UDM/USG series but you will not know what is going on behind the scene and how they actually work. I will say give it a pass if you want pro features with greater flexibility. There are just much better solution out there.

pfsense:
Pros: Great documentation and community support. Proven reliability and versatility for both consumer and business needs. Stable updates. Fulfill basic needs if you are a novice user. I have been using pfsense since I started learning the basics of firewall, so with that experience, make it hard to switch.
Cons: You probably need to build one yourself. Do take time to understand and learn some of the functions if you are used to consumer routers.

Ruckus:
Pros: reliability, set and forget, proprietary antenna design which performs really well in "noisy" environment, require less APs in a similar environment as compared to other APs (based on experience). Stable firmware updates.
Cons: Expensive unless there is a sale. I managed to use the left-overs from one of my projects, so got it at a discount.

In the end when it comes to decision making I will choose reliability, cost, knowledge and experiences.

 

[xiaofan]

BTW, just wondering what prompts you to upgrade from 1Gbps to 10 Gbps for the home network. Any real use cases at home? Or is it more for future proof?

Is it possible to save some money if setting up an interim 2.5Gbps home network?

I notice that you are still using 2x1Gbps Internet (SingTel and M1). Is it really required as both are pretty reliable?

 

[TanKianW]

BTW, just wondering what prompts you to upgrade from 1Gbps to 10 Gbps for the home network. Any real use cases at home? Or is it more for future proof?

Is it possible to save some money if setting up an interim 2.5Gbps home network?

I notice that you are still using 2x1Gbps Internet (SingTel and M1). Is it really required as both are pretty reliable?

Future proofing, have some fun, transfer large files from PC to 2 NAS library in different rooms (living room and homelab).....more keen towards having some fun to set it up.

Maybe it is only me, I always wanted to start with 10Gbe 3 years ago, knowing the unit's internal cabling is Cat6. Going 1Gbe seems a waste to the existing infrastructure, and I rather future proof it.

For my case, the Mikrotik CRS312 does comes with 8x multi-gig, but my NIC only runs at 1Gbe or 10Gbe, so might as well 10Gbe. Not really sure if 2.5Gbe switches will save some money coz never had any chance to compare. I know QNAP will soon launch an affordable unmanaged 2.5Gbe switch soon. For my case I will prefer managed/smart switches which can be tag with VLANs.

Not using M1. I am running MyRepublic 1Gbps as default ISP for all LAN connected devices, Singtel 1Gbps for wifi, CCTV and IOTs. I do have some critical work linked to my workplace system which I could not have any downtime when working/control remotely from home. Having 2x ISPs are part of the BCP, even my network servers are backed with UPS just in case power trip.

 

[dodge888]

Very nice. I used to run the following for my home network :

- pfsense 2.4.4 with an old Dell SFF (3rd gen i5) with 4-port Intel gig NIC
- Netgear GS108PE
- Ubiquiti Unifi AC Pro
- VLANs to segment my networks

My gear all 2nd hand except for the Netgear managed switch. pfsense is still my preferred firewall. Could max out my 1gbps Internet using OpenVPN with multi-WAN setup.

 

[BradenHeat]

Power sia, can feel my wallet screaming at me not to get enticed

Simply epic set up you have man

P.s have you thought of funneling the fans into one way in and one way exhaust?

From the picture it seems it will be all over the place for airflow if you’re trying to keep it cool

Just curious and thinking out loud

Posted from PCWX using iPhone12,5

 

[TanKianW]

Power sia, can feel my wallet screaming at me not to get enticed

Simply epic set up you have man

P.s have you thought of funneling the fans into one way in and one way exhaust?

From the picture it seems it will be all over the place for airflow if you’re trying to keep it cool

Just curious and thinking out loud

Posted from PCWX using iPhone12,5

Yes, will direct all fans to exhaust out of the cabinet in one direction.

The many fans set up you see in the top picture is actually my current set up. Have not upgraded yet.

In this project, all default 40mm fans will all change to Noctua 40mm silent fans. Will be replacing 6 of them, 2 on my 1U pfsense chassis, 4 on Mikrotik CRS312. Will change the direction of the Dynatron 1U CPU blower fan to blow out in same direction too.

 

[xiaofan]

https://www.anandtech.com/show/1590...3029d-review-a-fanless-10g-pfsense-powerhouse

This is nice --- super pfsense soft router good enough for 5Gbps internet.

US$1483 triple boot, Windows Server, Ubuntu Server and pfsense, 2x10Gbe and 4x1Gbe.

 

[TanKianW]

https://www.anandtech.com/show/1590...3029d-review-a-fanless-10g-pfsense-powerhouse

This is nice --- super pfsense soft router good enough for 5Gbps internet.

US$1483 triple boot, Windows Server, Ubuntu Server and pfsense, 2x10Gbe and 4x1Gbe.

I will build one on my own when >1Gbps internet becomes more affordable.

 

[Repsol_2]

I'm running pfSense (virtualized) on i7-4790S with 6 x Intel GBe NICs.

WAN1: M1 1Gbps static IP.
WAN2: Whizcomms 1Gbps.

But severely underutilized the bandwidth so going to decommission 1 WAN soon.

should keep both WAN link and do a Load Balance instead. 1 ISP down still got secondary link. Dual ISP full diversity. Just need to make sure "Source and Destination IP Address Binding" so your internet banking or transaction don't get confused.

 

[BradenHeat]

wa Lao eh all the “rich” people now making me gian full setup with NAS and etc

Posted from PCWX using iPhone12,5

 

[leechoonhwee]

I'm also in the midst of upgrading the home network but mine is significantly simple as I only have one NAS and my objective is more towards faster Internet versus faster internal network

So what I got so far is

Pfsense running on a used server from office with 4x sfp+ ports
H3C sfp+ switch
Asus ac5300 ap
10gb Internet from superinternet set as tier 1 gateway
1gb starhub fibre set as tier 2 gateway

Getting an Asus ax89x to connect directly to the pfsense so that hopefully my ax devices can surf past 1gb speed.

Its super overkill setup for home but these hobby setups is really just for prove of concept to bring peace to our own mind and soul