Yeah, I have a love/hate relationship with Hadoop. I have been working with it for a couple years now, from when we were POCing it 2 years ago, to now having 4 clusters.
Our main production cluster ingests and processes about 600-650GB of data per day. That cluster has a capacity of just under 1PB. Aside from having a spare cluster with the data fully replicated, our cluster is about as redundant as you can get. Replication factor of 3, rack awareness, NN HA, Quorum based storage of NN metadata.
Unfortunately with the amount of data we process, a distributed processing platform such as Hadoop is needed for us to query that data. We tried a massive SQL server with striped FusionIO cards that had a combined throughput of about 250K IOPS, but some of our BI teams queries were taking several hours to run. That's when we started playing with Hadoop.
There was no agreeing to maintenance windows, unfortunately
The business needs for the results of that data processing makes it so that I can only bring the cluster down at certain times, not to mention when I do I have to pause the import process and the upstream SQL servers start running out of space if I have the cluster offline for too long. One of the side effects of big data haha.
A bit of job security though
I am the resident "expert" on Hadoop at my company.
-Brian