Good resume indeed
Thanks!
Suppose I get a "just know how to follow instruction strictly, knows how to walk properly in a data centre, and knows how to open up the rack and replace the harddisk without switching off the power to the server, and knows how to check if the raid has completed" engineer, how much do you think his/her worth is versus your requirement to handle a software raid system, which can range from Linux MD to Solaris ZFS.
Now that's not technology, but that is realism. Since you have >10 years worth of knowledge handling enterprise environments, you should have a good idea what is the pay grade of a good engineer versus one that follows the working schedule strictly. Having downtime can happen any time, any moment, 24 by 7, simple is always the winner.
Yes, I feel your pain when I work with IT engineers who need training wheels, which is common in Singapore.
So the practical solution is to outsource, spend money to solve the problem, like in this case, use hardware RAID instead of software RAID due to incompetent IT engineers.
This will continue to feed the incompetence problem with more reasons to be more incompetent.
I would like to believe otherwise, for Singapore's IT engineers can and should move out of their comfort zone and learn another 'new' technology (software RAID). At least to put a stop to this endless self feeding cycle of incompetency.
Besides, managing a software RAID array on linux is really not that hard. Less then 10 commands to know. And there are plenty of help in the internet for Linux software RAID. Also there is Google search.
how to check if the raid has completed
Code:
cat /proc/mdstat
OR
watch cat /proc/mdstat
I'm not sure how you get to manage such subpar systems, but when we spec for H/W, we look at EOL of the hardware. When I spec for projects, I do question about how old is the H/W since released and we consider how long this project is going to run or expected to run. For maintenance cost as vendor recommendation to customer, if they ask about it, we consider the need to upgrade system when they reach EOL.
Looks good on paper, planned EOL by the hardware manufacturer, but what happens when the distributor stops the product line in Singapore?
What if due to the low sales figures/demand, the hardware RAID card distributor terminates this product/model?
You need to get it from Amazon or ebay, can you wait for the shipment?
My preference for using software RAID is because it uses commodity hardware that is widely available and available all the time in SLS.
With commodity hardware, you don't need planned EOL by a specific hardware manufacturer, because there is always another compatible product from another manufacturer. i.e. No vendor lock-in.
If you ask for me suggestion, I suggest you upgrade them. If you don't have the budget for it, that's another story altogether, isn't it ? I think in the enterprise world, we can't be too shabby at times, especially when company are earning big bucks out of those services, it's there obligation to keep such things in place so that they are continue to earn more.
I work in a SME.
I have seen some clients 'enterprise' solutions and in my opinion, they always overpay for oversized hardware solutions. Because clients will always believe the sales rep for 'enterprise' hardware.
I have not seen any undersized or right-sized 'enterprise' solution that clients use.
Clients like to hear stuff, like "you need to cater for future business growth", "you need to future-proof you hardware" or "what will your boss think if the solution does not perform".
Most of these sales rep will offer these comments without even looking at the numbers, current usage and calculating the disk IOPS requirements.
Fantastic, technologically wise, that's true. I totally agree with it. Today is not like 10years ago. Things are so fast, so is inflation. Now we know CPU as general purpose system are powerful beast, today cores are cheap, but wait a moment, H/W Raid card are cheaper.... So given your resume, I have to ask, how does Oracle, IBM, and all those big companies charge their fantastic applications licenses. If I spend X dollars on the application to license on the CPU, am I silly enough to use even 1 CPU cycle on it for storage management ?
Since we are digressing to 'enterprise' use of hardware RAID (Oracle & IBM applications licenses).
I can only imagine Oracle DB's server uses an external block storage, connected by FC or iSCSI to a SAN storage supplied by EMC?
For the enterprise, EMC SAN storage solution should come with a support contact and SLA.
In this case, I don't care whether if it is hardware or software RAID, cos EMC support contract will take care of it.
However, as we are discussing here, we are responsible the support of the hardware RAID card, so if we cannot get a replacement hardware RAID card for whatever reason, we are screwed.
I myself have design and deploy a whole suite of antivirus + antispam engines running a postfix mail server serving like 20+ domains, each day, we have no less than 100+ mails per minute, it can reach much higher during peak hours like during the start of your working day and towards the end of the day. Somehow people like to rush emails before they knock off. I have seen a software raid 1 system, just raid 1, going super highload with really lousy performance on a pair of 10K SAS Harddisk. Now obviously the question is, how do I conclude that it must be software RAID ? Perhaps it will just suffer just the same under H/W Raid. A higher level question will be, perhaps we have underspec the H/W during sizing.
Great that you have setup and managed mail hosting before.
I have setup mail servers using Exim+ClamAV+SpamAssassin+Dovecot and Exim+ClamAV+SpamAssassin+DBmail.
Yes, I do see disk IO issues on software RAID 1 setup.
But hardware RAID is not the solution to this problem.
If you look at the processes spike when mail volumes are higher, I would bet that it would most likely be SpamAssassin (SA). After SMTP data is received by the MTA (Exim), before it gets routed and saved to the target mailbox, it gets written to a spool file, where SA inspects the file and gives it a spam rating. This is where I believe the bottleneck is.
There are a few (software) MTA tricks you can use to solve this without hardware RAID, if you are interested, PM me or start another thread.
Luckily enough, we have another similar system that is on H/W Raid. For migration reason, we have that ready for production use and we transfer the system over. Immediately we observed a significant decrease in the I/O pressure. I think the scenario is pretty informative. Until you are really running on systems that are eating your I/O like daily meal, one will not observe the value in H/W Raid.
But this brings about another question, does it mean the whole problem is about just that fair bit of performance you can get out of H/W Raid. In the latter part of your post, you have discussed about how better design and more efficient system are better option for the solution. When I discuss infra, I don't come from this background solely, my job scope is in solution architect on overall including the need to size system and develop system from scratch. Therefore having well thought out solution from the highest level in business, to the lowest level in infra is part of my job scope.
Most System Integrators(SI) in Singapore think of systems the way you describe it. Throw high-end and expensive RAM, CPU, Hard disks and hardware RAID card into one expensive box. They always see it as one box. And most SI are unable to correctly identify bottlenecks, and always jump to conclusion and propose more 'enterprise' hardware is required.
If you think out-of-one-box system design, like Google, Backblaze, Apple, etc.
Heard of
GoogleFS - Google's
distributed file system?
You don't need ONE pricey box, you should be using multiple cheap boxes running commodity hardware, not some exotic hardware where you need to call vendor to get a quotation and wait 1-2 weeks for the shipment.
Your first example:
Why did you suggest running storage services and Oracle DB on the same box?
And your second example, your system design:
Did you split the SMTP servers (Exim/Postfix/Sendmail) and IMAP/POP3 services to 2 boxes?
Did you split anti-spam and anti-virus filtering into 2 more boxes?
Are you running SMTP,POP3,IMAP,database on the same box?
Even so, did you run all these different services on different physical hard drives?
For example:
- 6x 1TB HDD (sda, sdb, sdc, sdd, sde, sdf)
- 3x software RAID 1 (mirror) arrays
md0 (sda+sdb)
md1 (sdc+sdd)
md2 (sde+sdf)
- md0 used for SMTP anti-spam and anti-virus spool directory
- md1 used for mail directories (Dovecot mailboxes)
- md2 used for database files
These are just some system optimization that can be done without using hardware RAID, which will give better performance gains then switching to hardware RAID.
Unfortunately while the solution is so simple, it is extremely hard to achieve in real life scenario. For starter, not everyone holds a PhD. Which means, where we can settle the performance in machines, we at times have to use that as a solution to higher level algorithms. I have mention in another thread, today industry has no lack of developers, we have lack of solution architects and a good abundance of superior software engineers. As such, the idea of better design not necessarily is a solution to the problem. The problem of having resources to come up with good design is already a problem itself. While it may not be the best solution, it's a solution that money can buy and its pretty consistent if you ask me. At least it won't ask for a pay rise after 2 years.
I'm certainly not pessimistic, but it's a fact we need to look into and resolve. So how does H/W Raid comes in ? Back to the very early notion of how much do you think you are worth.
Better or faster hardware can never be the solution for terrible slow performance software.
Best example: Windows Vista performance VS Windows XP/7 on the same hardware.
My experience:
Performance gains from better hardware CPU:
Xeon E3-1220 3.1 GHz ($189)
Xeon E3-1290 3.6 GHz ($885)
Speed (GHz) increased by 3.6/3.1=16%
Price increased by 885/189=368%
Performance gains using software methods, add index to database table for slow queries:
Database select query time for 1 record before adding index: 11 seconds
Database select query time for 1 record after adding index: 0.01 seconds
Performance increased by 11/0.01=1100%
As you can see, from my experience, no mater what kind of expensive hardware you can throw at my database server, there is no way you can match my performance gains from a simple software optimization. This is where most SI fail to understand.
My answer to your question is your knowledge to the EOL of the H/W used. We all don't have crystal ball, but we have intuition and current affairs skill sets. Also we have contingency plans if we know some company is going down or having bad reputations. At higher level in the department, I believe one important skill set is to know what is going around in the industry, and be prepared for it. The same you describe about RAID card and it's accessories which are just small components, how about SAN system ? Servers, load-balancers, switches etc. Are these vendors not going to face the risk you mentioned ? Is this therefore a good enough reason to stop using these H/W ? Obviously you can say there is no choice. Not like there are software servers around....... erm is virtual systems also considered virtual... just a side thought. Hence the essence to the problem is not about how long the H/W will last, but how much you know it will last. When procurement is made, is there any black and white involve to ensure support from the vendor. Are these terms and conditions secure during procurement ? If these are not considered, what is there to talk about ?
If I am buying the storage solution with a support contract and SLA, I don't care if it is hardware or software RAID. Just as long as it works.
If I am buying the hardware RAID card, I would reconsider.
And I think this is what the TS is asking.
Well since it's your first time, then count you lucky that I'm making it known to you. Not like it's some school books Q&A. I know the fellow pretty well doing project previously in one of the ISP in SG. It's a couple of years back. The problem is too much iron dusts and ions in the air. Causing electrical shocks to the electrical components in the servers in the data centre. Every now and then, there will be a couple of cases of servers rebooting out of nowhere. The problem is solved only after they perform a through cleanup of the data centre with a good flush of the air in it and eventually stop the outbreak.
I thought data center air-conditioning units have dust filters or some additional HEPA filters. Don't know of much about this to comment further.
The reason for this to be brought up is bit flips while not common are occuring for various reasons. Environmental reasons are also part of it. So electrical inputs are not entirely the equation. UPS protect the system electrical consistency, but they do not protect from kernel panics. Neither do they protect from certain subsystem break down from within. Battery Backed Cache is a performance feature protection mechanism, namely the write back cached from the RAID system down to the HDD. Upon the system is brought back online, the write is replay into the HDD for data consistency. Now this do not means UPS is in any form unnecessary. Like wise the same for BBU. Their usage and what they protect against are different. If you really need a documentation, read up the RedHat recommendation on how write barriers on filesystem are not required if BBU is available. If it's write through cache, we don't even bother about BBU. But even so, one need to understand just because the RAID gives back a success write, doesn't it it's on platter. While we are here dismissing about a technology, do you exactly know which cache are we talking about. We have the kernel file buffer, we have the RAID cache, and we have the disk cache. Which feature protect against which ?
Yes, I do know of disk read/write cache, but how about the performance gains? More then 2x disk IO performance?
Have you looked at 100x performance optimisation before you look at 2x disk optimisation?
Don't just optimize one box, think out-of-one-box, there are better ways as highlighted above.
If your H/W fails before your migration plan, then you should replace your H/W card. Is it a hard decision to make ? If you are testing me on what the H/W card is not available. My question back to you is why is your migration plan not putting it the EOL of the H/W ? Is it because someone screw up somewhere ? What are the options for your migration ? Now let me ask you, what if the server fails before your migration plan and no more servers is available for you to migrate, and oh yes, you are using Solaris and it just stop production yesterday. It's a weird question, but how come you are not alerted that the company is in crisis, or why are you not aware that you have a pretty legacy RAID card that requires migration and this is not brought up to the authority ?
Well, as they say, hindsight is always 20/20, how many here knows that Sun Solaris is going down? How many IT engineers made a bundle when they traded Sun/Palm/Compaq shares if they know this inside news?
Do you know if EMC is crisis? How about 3ware, are they also in crisis?
If you do, you can always bet on it by trading their shares or stock-up on the 3ware hardware RAID cards.
If really your H/W card is not available, can't you DD over ? The world doesn't end right ? Something has to be done if it's not handle properly right from the start. It's easy to say, if I did DD a whole got damn HDD before.
Yes, but there will be unacceptable down time.
Maybe you are right, they are suckers if you ask me. But reality is reality. Your consumer budget just don't fit the price tag of these enterprise game. You want it cheap, then we don't talk business here. I definitely love to have a piece of SAS HDD with the price tag of 100+, but reality says NO. Where do you think the money is ?
Besides being suckers, when we buy technology, we will always be users, our level of technical competency will never improve.
Well like you, I often have to think out of the box, but that doesn't permit me to dismiss about certain technologies. Some goes obsolete, some goes irrelevant with the progress of technologies. But since we are still alive now, we have to look at real situation. What I have described about human resources are real issues we face. When you discuss about enterprise, when you discuss about money, and technology, you don't leave anyone out of the loop. You put them together, you will see the relevance.
If I tell you, any day, I replace you, yes, a couple of H/W card versus your knowledge about Linux MD, or LVM Raid. It sounds ugly, but you are more expensive than a couple of H/W Raid cards. I believe the cost to migrate in half the EOL of each H/W is still cheaper than paying your salary. On the assumption that you are costly and you have higher expectation of your pay.
Conclusion. Software RAID is good, Hardware RAID has its worth. If we are just discussing blindly on one aspect and we dismiss the other, you are the winner. So what does it entail ? You can carry on worship S/W Raid, but you won't want to admit the cost is not just about H/W right ? But if we embrace the whole equation together, we can start to see how an old proven technology comes in useful at times. When systems are young and computation performance are not there, H/W raid comes to the rescue. As processors upgraded, H/W Raid loses its competency in this area, it somehow got hold of itself in another department, mainly consistency, low-tech skillsets and efficiency. We have better options, but is that all ? Better options don't bring new problems ? If they do, what are our options ? Perhaps proven and reliable is what it is, or perhaps breaking out of the box for better options far ahead ? I'm not exactly so optimistic about resources competency if you ask me
It's a hard nut to crack.
I am here to share and learn.
I learn more when my believes or ideas are challenged.
Good to hear your passion for your work and technology here, sometimes hard to find people like this.
This reminds me of another good enterprise topic, "Enterprise Backup: Tape is dead", but that is for another thread.