Data center dream

[Technical Xman]

Hello everyone,

I am planning to set up some sort of data center in my own house. Basically for web hosting and services, so I would be need space for rack and to house my servers.

However, I would like to know does any one already had own it or whom anyone you know had own it or ever thinking of owning it.

Love to know more if you can share, looking forward to get your response!

 

[davidktw]

Nice detail post above. I only have 2 questions which TS need to go think thoroughly about before embarking.

1) Why should your customer choose you over existing commercial webhosting company or even cloud hosting providers like GCP, AWS, Azure?

2) If you are hosting it yourself for your own use only, what advantages are you getting out of it versus an existing commercial offering?

 

[pikelocks]

Hello everyone,

I am planning to set up some sort of data center in my own house. Basically for web hosting and services, so I would be need space for rack and to house my servers.

However, I would like to know does any one already had own it or whom anyone you know had own it or ever thinking of owning it.

Love to know more if you can share, looking forward to get your response!

What is your ultimate goal? if you are just looking to host some few sites, then it will be very expensive to manage a data center. Meanwhile, you can opt for cloud hosting services instead.

 

[pikelocks]

Some general prerequisites:

• BUSINESS Broadband (NOT home broadband!) - it may be more expensive, but the ISP is required to deliver the contracted speed 99.9% of the time. If they fail to perform you may be entitled to damages, but usually it almost never happens. THIS IS VERY IMPORTANT, especially where you need symmetric upload/download speeds, or need LARGER upload than download speeds to serve your pool of clients.
• A good server + array expander (I recommend HP, they have great enterprise support and their products are really good IMO)
• If you are building your own, you can.
  • Lots and lots of enterprise-class disks. Unless you are building a PB array and not in TB, it's better to stick with smaller sized drives because you are looking at a lower AFR. Most enterprise drives have a NRE of 10^-15, avoid consumer drives which have an NRE of 10^-14. If you got the cash, opt for drives with an NRE rate of 10^-16 or better, but these are generally insanely expensive and rare. Better still, if you got deeper pockets, you can spam an SSD array. Realistically, if data size is important to you, getting an enterprise class HDD will suffice. Speeds are not significantly different from 7.2k to 10k in a RAID 5 setting, but 15k makes a difference (I doubt they make them anymore, mainstream enterprise drives are rated at 7.2k RPm which gives you a per drive speed of 160MB/s at best)
  • A good RAID card. I won't recommend LSI if you are working with a large number of disks. Get a HP RAID card. I am a little backward, using the P410i but it still gets the job done.
  • An SAS expander - they are your life savers for when you need to add more disks - the current HP SAS 3 Expander offers up to 48 drives - 12 ports with an SFF-8087 (split into 4 individual data cables)
  • A good casing - A Norco RPC-4220 does the trick
  • For your main server, you would look at server grade motherboards from Supermicro Computer. I have heard cases of people trying with Ryzen Threadripper but I have never done that so don't know.
  • A decent multi-port LAN card - those from intel fare quite well.
  • A managed switch - to perform link aggregation of multiple LAN ports
• Next, you will need to pick a method of data transport: Fibre Channel or iSCSI. The former is insanely expensive and you gotta have additional hardware for that, but you can get insane speeds through Fibre Channel.
• iSCSI is more widely used, is slower but can use existing hardware (the multiport LAN card where you aggregate 4 -> 1) + your managed switch
• And then you gotta pick your OS. Windows Server Standard may lack the features for fibrechannel (IIRC) and high availability - so you may want to consider Windows Server Enterprise in that sense.
• Most sysadmins will go with Linux - e.g. CentOS, SUSE - But you have to configure everything (it's more flexible but more troublesome too)! to set up multipath, high availability.
• I'd recommend you read further on how to configure all these.

Don't forget, your data center needs good ventilation. Get a server cabinet (+ fans) - and you need a dedicated room, air conditioned to keep the servers cool under high load. Some server racks come with liquid cooling to bring the temperatures down further, although you would only see this in massive scale data centers.



The cheapest you can get started would be US$2k - $3k.

Also, on the RAID, would recommend RAID 5 - if you are more with the volume per array, RAID 6 if you can't tolerate any downtime at all.

If you are remain unfazed by the time you get this sentence, then it's time to take up arms and build the first building block of your data center

As I said, if you want to avoid the hassle, go for the cloud. There are plenty of offerings like AWS, etc who have far more resources than you can imagine.


Hope this long post helps.

I welcome all other members to correct me if they wish

This is just awesome. Nothing to add or retract. Thanks for the great insights.

 

[andr3wyong]

Some general prerequisites:

• BUSINESS Broadband (NOT home broadband!) - it may be more expensive, but the ISP is required to deliver the contracted speed 99.9% of the time. If they fail to perform you may be entitled to damages, but usually it almost never happens. THIS IS VERY IMPORTANT, especially where you need symmetric upload/download speeds, or need LARGER upload than download speeds to serve your pool of clients.
• A good server + array expander (I recommend HP, they have great enterprise support and their products are really good IMO)
• If you are building your own, you can.
  • Lots and lots of enterprise-class disks. Unless you are building a PB array and not in TB, it's better to stick with smaller sized drives because you are looking at a lower AFR. Most enterprise drives have a NRE of 10^-15, avoid consumer drives which have an NRE of 10^-14. If you got the cash, opt for drives with an NRE rate of 10^-16 or better, but these are generally insanely expensive and rare. Better still, if you got deeper pockets, you can spam an SSD array. Realistically, if data size is important to you, getting an enterprise class HDD will suffice. Speeds are not significantly different from 7.2k to 10k in a RAID 5 setting, but 15k makes a difference (I doubt they make them anymore, mainstream enterprise drives are rated at 7.2k RPm which gives you a per drive speed of 160MB/s at best)
  • A good RAID card. I won't recommend LSI if you are working with a large number of disks. Get a HP RAID card. I am a little backward, using the P410i but it still gets the job done.
  • An SAS expander - they are your life savers for when you need to add more disks - the current HP SAS 3 Expander offers up to 48 drives - 12 ports with an SFF-8087 (split into 4 individual data cables)
  • A good casing - A Norco RPC-4220 does the trick
  • For your main server, you would look at server grade motherboards from Supermicro Computer. I have heard cases of people trying with Ryzen Threadripper but I have never done that so don't know.
  • A decent multi-port LAN card - those from intel fare quite well.
  • A managed switch - to perform link aggregation of multiple LAN ports
• Next, you will need to pick a method of data transport: Fibre Channel or iSCSI. The former is insanely expensive and you gotta have additional hardware for that, but you can get insane speeds through Fibre Channel. (the fastest is 128Gbps, but are ya willing to fork out $160,000.00 for that?! Only one commercial seller sells it and the contract is highly skewed in their favour. Most FC suppliers will offer 16, 32 and in some cases 64Gbps FC equipment).
• iSCSI is more widely used, is slower but can use existing hardware (the multiport LAN card where you aggregate 4 -> 1) + your managed switch
• And then you gotta pick your OS. Windows Server Standard may lack the features for fibrechannel (IIRC) and high availability - so you may want to consider Windows Server Enterprise in that sense.
• Most sysadmins will go with Linux - e.g. CentOS, SUSE - But you have to configure everything (it's more flexible but more troublesome too)! to set up multipath, high availability.
• I'd recommend you read further on how to configure all these.

Don't forget, your data center needs good ventilation. Get a server cabinet (+ fans) - and you need a dedicated room, air conditioned to keep the servers cool under high load. Some server racks come with liquid cooling to bring the temperatures down further, although you would only see this in massive scale data centers.



The cheapest you can get started would be US$2k - $3k.

Also, on the RAID, would recommend RAID 5 - if you are more with the volume per array, RAID 6 if you can't tolerate any downtime at all.

If you are remain unfazed by the time you get this sentence, then it's time to take up arms and build the first building block of your data center

As I said, if you want to avoid the hassle, go for the cloud. There are plenty of offerings like AWS, etc who have far more resources than you can imagine.


Hope this long post helps.

I welcome all other members to correct me if they wish

Amusingly enough I have a setup that is very contrary to your post (which I would IMHO consider outdated, though I think that's a very subjective/taste thing), but I do have good points to back my preferences up:

• Business broadband - Singapore local loop has gotten so good that this is much less of a problem than before. Outages on local loop increasingly rare but more importantly, affect business and residential last mile alike. The only difference is SLA, but you could probably plan around that. That said, specific features like BGP or public WAN subnet may not be available with residential ISPs (except maybe SuperInternet)
• Servers/RAID expanders - Well, servers have become cheap enough that I prefer to do a scale-out cluster, where you have independent servers doing compute/storage, and scale them accordingly by adding more servers. With Ceph and Gluster (or even a simpler replication setup), this is pretty easy to achieve today, and are self-healing and self-scrubbing (more on that in a bit)
• Disks - You will have to do some math here but the whole point of going to some sort of array is that you don't have to get expensive disks with ultra low BERs/AFR and such; with some exceptions (TLER, head parking), most consumer-grade drives will do just fine
• All that stuff about RAID - Gonna have to stop you right there - if your disks are larger than around 3TB, forget about RAID5/6 and anything that is strongly tied to a fixed array of block devices. RAID5/6 recovery takes DAYS on larger disks (been there, done that; pissed my pants, lost all my data, cursed, swore, etc.). It doesn't offer ANY kind of flexibility with rebalancing your array, and you don't always have the privilege of spare SATA/SAS ports to hot-redesign your array. ZFS improves on RAID in some ways but as you start to fill it up you start to hit many of the same issues with block storage arrays. I’ve moved on to Ceph (though Gluster also solves many of these problems) because of how amazing it is to be able to dynamically configure the level of redundancy I need, while performance and redundancy can grow with additional drives.
• Data transport - I think with the advent of clustered file systems, a lot of this becomes irrelevant. My storage servers are now on 10G and I am converging them to 40G on my compute nodes. It’s incredibly scalable and I can take any node down at any time without causing downtime on my servers. It’s a lot more pleasant to work with than trying to set up any sort of multipathing with FC or iSCSI. Also, 10/40G Ethernet is incredibly cheap if you know where to look.
• OS - I’d stick to some kind of hypervisor OS - my personal preference is Proxmox, but there's quite a number of commercial solutions like the very prevalent ESXi. The isolation that VMs/containers give makes it easy to set up HA or shuffle your environments across different hypervisors, or even P2V/V2P your data.

Again, this is mostly personal preference but also where I feel the industry is heading (or even where it has already headed) as, again, I think your proposed setup tends to be rather monolithic, and the advantages of that are starting to wear rather thin.

I hope it doesn't come across as offensive, because it sure sounds very opinionated, sorry :X

 

[Technical Xman]

Thanks for all your response and high recommendation.

I agree that the word 'data center' is ambiguous. But what about people who doing bitcoin mining, how are their set up like compare?

As I am on a self-funded project, so I wish to set up at home
Looking at around 9/12 U server rack and 1 - 2 servers to run

What are your recommendation?

 

[Technical Xman]

This is just awesome. Nothing to add or retract. Thanks for the great insights.

I have try asking about colo services from the big 3 telco but only 1 of them come back with proper package detail. It seem like they only provide service to registered company which I now understood what some of you tried to said. What a shame.

 

[5star_pundit]

you can start by having a server, couple of workstation and NAS for storage.

as for NAS it usually comes with web server capabilities it is easier to maintain

and has RAID too