Upgrading your HDB Home Network to 10Gbe Project

lumpyme

Master Member
Joined
Mar 19, 2006
Messages
3,584
Reaction score
638
I like the idea for this, it’s not easy to implement something like this without experience, planning and knowledge. Even if there is no use for it, the lessons are there to be learned and experienced. I really thumbs up ts
 

TanKianW

Supremacy Member
Joined
Apr 21, 2005
Messages
5,907
Reaction score
2,220
I like the idea for this, it’s not easy to implement something like this without experience, planning and knowledge. Even if there is no use for it, the lessons are there to be learned and experienced. I really thumbs up ts

Thanks for the kind words. Hope you learn something from it, that is the objective. :D

"Stay Hungry, Stay Foolish" Steve Jobs.
 

iduncheckmail

Supremacy Member
Joined
Nov 24, 2002
Messages
5,659
Reaction score
2,184
I like the idea for this, it’s not easy to implement something like this without experience, planning and knowledge. Even if ****there is no use for it***, the lessons are there to be learned and experienced. I really thumbs up ts

Well, I’m sure the ones here at least learnt that 1 thing .
 

ShrmnK

Junior Member
Joined
Oct 9, 2011
Messages
13
Reaction score
0
Does the CRS305 run that hot that you need to add a fan? What SFP+ modules are you using?

Context: I recently bought QNAP QSW-M408-2C, kept getting random reboots with
- Ubiquiti's UF-MM-10G uplink to UDM-PRO
- DAC to my NAS (10G)
- and using one of the two 1/2.5/5/10 combo ports to my gaming desktop's Intel i225 (2.5gbe) about 20m of Cat6 cabling away

Random reboots occur regardless of network activity, and even when the gaming desktop is off. Might be getting 10G internet (SFP+ handoff from SuperInternet) in the near future and if it can't even handle minimal traffic, I was worried it can't handle when I start utilising my NAS over WAN (I shuttle between remote offices).

Have a CRS305 on the way to test with above modules, but will be storing it in unventilated open-door cupboard - am worried about heat.
 
Last edited:

TanKianW

Supremacy Member
Joined
Apr 21, 2005
Messages
5,907
Reaction score
2,220
Does the CRS305 run that hot that you need to add a fan? What SFP+ modules are you using?

Context: I recently bought QNAP QSW-M408-2C, kept getting random reboots with
- Ubiquiti's UF-MM-10G uplink to UDM-PRO
- DAC to my NAS (10G)
- and using one of the two 1/2.5/5/10 combo ports to my gaming desktop's Intel i225 (2.5gbe) about 20m of Cat6 cabling away

Random reboots occur regardless of network activity, and even when the gaming desktop is off. Might be getting 10G internet (SFP+ handoff from SuperInternet) in the near future and if it can't even handle minimal traffic, I was worried it can't handle when I start utilising my NAS over WAN (I shuttle between remote offices).

Have a CRS305 on the way to test with above modules, but will be storing it in unventilated open-door cupboard - am worried about heat.

I have not use a QNAP switch before, but you might want to check if it is due to a firmware bug/problem.

Mikrotik does make cheap, good and stable switches. Both my CRS312 and CRS305 runs 24/7, not a single reboot till date.

I am using Mikrotik's DAC and RJ45 modules. You need to check if they are compatible with your 10G switches. But Mikrotik and Ubiquiti ones are quite safe. 10Gtek and FS works for me too.

CRS305 is not that warm on its own. It is the one and only RJ45 modules I am connecting to it that is generating the heat. So I installed a Noctua 12cm fan to cool it down.

I need to use the RJ45 module because the 10G network coming down from my CRS312 is RJ45.
3SVGcYT.jpg


When storing in a cabinet, I will still recommend sufficient airflow using USB powered fans.
ERv04dJ.jpg
 
Last edited:

yosemi

Senior Member
Joined
Nov 20, 2014
Messages
2,063
Reaction score
162
I also trying to overcome the heat generated by CRS305 with 3x SFP+ to ethernet connector running at 10gbps each. really need a fan directly blow at it

the truth of 10gbps speed is only for iperf test, in my real world usage, it is limited by speed of my Nas(raid5), the ssd on my pc, max i get is around 400-500 mb/s transfer rate
 

Mach3.2

Great Supremacy Member
Joined
Apr 8, 2011
Messages
71,499
Reaction score
1,438
Tried the homelab free pricing version of tnsr on a bare metal appliance in the workplace lab. Overall it is much more optimized and better performance as compared to the conventional pfsense.

You can load up the free version to have some fun. It is very CLI based, therefore not for the faint of heart. It geared towards professionals.

Can visit servethehome website to check out more:
https://www.servethehome.com/netgate-tnsr-home-lab-edition-pfsense/

Screenshot from servethehome on the tnsr show interface:
rfFVjZN.jpg
TNSR doesn't have a firewall right? :look:
 

Henry Ng

Arch-Supremacy Member
Joined
Aug 9, 2011
Messages
14,010
Reaction score
220
I also trying to overcome the heat generated by CRS305 with 3x SFP+ to ethernet connector running at 10gbps each. really need a fan directly blow at it

the truth of 10gbps speed is only for iperf test, in my real world usage, it is limited by speed of my Nas(raid5), the ssd on my pc, max i get is around 400-500 mb/s transfer rate

Actually for most users a 1Gbps or 2Gbps already good enough. Even M1 CSO told me not to upgrade to 10Gbps a few years ago.
 

Mach3.2

Great Supremacy Member
Joined
Apr 8, 2011
Messages
71,499
Reaction score
1,438
I also trying to overcome the heat generated by CRS305 with 3x SFP+ to ethernet connector running at 10gbps each. really need a fan directly blow at it

the truth of 10gbps speed is only for iperf test, in my real world usage, it is limited by speed of my Nas(raid5), the ssd on my pc, max i get is around 400-500 mb/s transfer rate
Gigabit is 125MB/s before overhead, you're doing way over gigabit speeds. :s22:

SFP+ to 10GbE is actually out of spec since it draws so much power. If you want it to run cooler consider running a DAC or fibre.
 

TanKianW

Supremacy Member
Joined
Apr 21, 2005
Messages
5,907
Reaction score
2,220
I also trying to overcome the heat generated by CRS305 with 3x SFP+ to ethernet connector running at 10gbps each. really need a fan directly blow at it

the truth of 10gbps speed is only for iperf test, in my real world usage, it is limited by speed of my Nas(raid5), the ssd on my pc, max i get is around 400-500 mb/s transfer rate

Using RJ45 to SFP+ module, sure hot. I using 3x DAC, only one RJ45 to SFP+ module. Heat still manageable. I still use a fan over it to prolonged the lifespan of the module.

DpMs3l8.jpg


Actually Mikrotik don't recommend RJ45 to SFP+ module connected side by side. You are using 3 side by side...I can imagine the heat output.

If you want to connect to 3, good to get a better spaced-out 10G switch (>5 ports) or change your NIC to SFP+.

https://wiki.mikrotik.com/wiki/S%2BRJ10_general_guidance

I hit 600MB/s ->1GB/s on the average. Your pc spec, file size, NAS spec, NIC spec...all contribute to your speed.
f1f7YLd.jpg
 
Last edited:

Mach3.2

Great Supremacy Member
Joined
Apr 8, 2011
Messages
71,499
Reaction score
1,438
Yap. A software router. The documentation says can handle up to 100G.....no chance to test that throughput though.

You want to try?:D
I not so zai, quite CLI adverse, if I can avoid will avoid. :s13:

100G hardware expensive also, mai lah :s13:
 

yosemi

Senior Member
Joined
Nov 20, 2014
Messages
2,063
Reaction score
162
I not so zai, quite CLI adverse, if I can avoid will avoid. :s13:

100G hardware expensive also, mai lah :s13:

CLI may be intimating, but usually all you use is always the few repeated command haha :s13:

unless u configuring something like BGP router
 

bert64

Senior Member
Joined
Jan 20, 2020
Messages
863
Reaction score
352
Yap. A software router. The documentation says can handle up to 100G.....no chance to test that throughput though.

You want to try?:D

TNSR does have firewalling capabilities... But the more complexity you add (firewalling, nat etc) the more work the device has to do and thus you'll need more powerful hardware to achieve the same level of performance.

In terms of 100gbps interfaces, you have to consider the bandwidth of how they connect to the system. For instance a single 100gbps interface will be full duplex so 200gbps plus some overhead, while a pci express 4.0 x16 link can hit 256gbps. You'll need pretty capable hardware to make a 100gbps router with multiple interfaces and capable of driving them all at speed.

When 100mbps nics first came out, the ISA buses in commodity PCs of the day couldn't handle them at full speed.
When 1gbps nics first came out, the 32bit PCI buses in commodity PCs of the day couldn't handle them at full speed.

Highend routers have dedicated ASICs to handle routing, basically layer 3 switches, and such routers are typically not encumbered with any filtering rules etc either.
 

bert64

Senior Member
Joined
Jan 20, 2020
Messages
863
Reaction score
352
I not so zai, quite CLI adverse, if I can avoid will avoid. :s13:

100G hardware expensive also, mai lah :s13:

CLI is often more natural as it's more like having a conversation. Having a conversation with someone who has a limited and strict vocabulary.

It can also be more flexible and efficient in many cases.
 

TanKianW

Supremacy Member
Joined
Apr 21, 2005
Messages
5,907
Reaction score
2,220
TNSR does have firewalling capabilities... But the more complexity you add (firewalling, nat etc) the more work the device has to do and thus you'll need more powerful hardware to achieve the same level of performance.

In terms of 100gbps interfaces, you have to consider the bandwidth of how they connect to the system. For instance a single 100gbps interface will be full duplex so 200gbps plus some overhead, while a pci express 4.0 x16 link can hit 256gbps. You'll need pretty capable hardware to make a 100gbps router with multiple interfaces and capable of driving them all at speed.

When 100mbps nics first came out, the ISA buses in commodity PCs of the day couldn't handle them at full speed.
When 1gbps nics first came out, the 32bit PCI buses in commodity PCs of the day couldn't handle them at full speed.

Highend routers have dedicated ASICs to handle routing, basically layer 3 switches, and such routers are typically not encumbered with any filtering rules etc either.

In a way, yes it does. You explain it clearly. Objective for such routers are geared towards throughput.

It is not designed to replace a UTM or NGFW and deliberately so.
 
Last edited:

Mach3.2

Great Supremacy Member
Joined
Apr 8, 2011
Messages
71,499
Reaction score
1,438
CLI may be intimating, but usually all you use is always the few repeated command haha :s13:

unless u configuring something like BGP router
True lah, but the thought of configuring TNSR/VyOS or any other CLI only routers using CLI only intimidates me :s13:

CLI is often more natural as it's more like having a conversation. Having a conversation with someone who has a limited and strict vocabulary.

It can also be more flexible and efficient in many cases.
Yeah I get that, and in some cases CLI is the only way you can get things done, like to patch your ESXi hosts without a vCenter Server Appliance.

But as an average consumer who doesn't use CLI often, it can be quite intimidating. :o :s13:
 

cscs3

Arch-Supremacy Member
Joined
Jun 4, 2000
Messages
21,676
Reaction score
115
If you are setting up 10G network, is best to use fiber vs cable.
 

Java_Guru

Senior Member
Joined
Aug 4, 2001
Messages
2,229
Reaction score
0
I am also working on my homelab. Can I ask why would we want to load balance or failover two 1GB fibre lines ?

Frankly, as the fibre runs next to each other, if contractor downstairs or in road cut, both lines will be dead.

Has anyone manage to get a 4G modem to bridge to pfsense for failover ? I can't seem to find any locally that does this w/o insisting to use its own router (thereby causing a double NAT).
 
Important Forum Advisory Note
This forum is moderated by volunteer moderators who will react only to members' feedback on posts. Moderators are not employees or representatives of HWZ. Forum members and moderators are responsible for their own posts.

Please refer to our Community Guidelines and Standards, Terms of Service and Member T&Cs for more information.
Top