on tech

Planning for SaaS Infrastructure Failure – part 1

2010-05-05T09:51:00.000-07:00

There's nothing quite like a good Single Point of Failure (SPOF) during a holiday dinner. Most folks I talk to immediately think of network redundancy when I bring up this topic and tend to look at me strangely when I talk about other parts of their infrastructure. I thought I'd throw a short post together on how to plan for failure in your SaaS infrastructure but, as usual, it's turned into a much longer post than I've intended and I'm going to break this post into three parts. Here we go with part 1.

Start with a meteor strike
Most people laugh when I say this but you should literally start your failure analysis with "what happens if a meteor strike takes out the building or power lines to our data center". In other words, start with a full disaster recovery situation and work your way all down the stack. If you think this is overkill, remember that two highly redundant data centers were both taken out by previously unknown single points of failure.

Before jumping into these examples, I want to point out that Rackspace and 365 Main are two very well respected companies so this isn’t some fly-by-night operation that was taken out. Both providers also did an excellent job of publishing what happened on their websites after the event (see links below).

Rackspace Examples

Rackspace suffered two outages in June and July of 2009 at their DFW facility. In the June outage, a power interruption resulted in a switch to generator power. So far so good as the backup systems kicked in. Unfortunately, the generators failed.

Battery power only lasts for around 10-15 minutes in most of the data centers that I've worked at so you've got 10-15 minutes depending on the power draw (and the state of your UPS batteries) to get the generators running. Not much room for error there.

In the July 2009 outage, Rackspace again switched to generator power due to a power interruption. Unfortunately, a "bus duct" prevented proper operation of a UPS and some customers lost power to their servers for about 20 minutes. They also suffered a loss of network connectivity due to the power disruption.

In 2007, there was also the infamous truck crash where a traffic accident damaged a nearby utility transformer, knocking out power to the Rackspace's Dallas facility. The company switched over to generator power but two chillers failed to start back up again. Servers had to be shutdown to prevent damage from overheating.

365 Main Example

365 Main is a major data center smack in the middle of the San Francisco. This is probably starting to sound familiar -- a power interruption caused a fail-over to generators and three out of ten of the generators failed to start. This outage took down yelp, Craigslist, Technorati, LiveJournal, TypePad, and many others. In this case, the failure was due to a bad setting in the Detroit Diesel Electronic Controller (DDEC) for the generators. The setting "was not allowing the component to correctly reset its memory. Erroneous data left in the DDEC’s memory subsequently caused misfiring or engine start failures on the next diesel engine call to start."

That's right. The backup mechanism for power failure is a diesel generator and this diesel generator has a dependency on an electronic component, presumably with software running on it. I'm fairly ignorant of these diesel generators so let's assume that the DDEC has a redundant controller. Unfortunately, it would appear that the controller has a SPOF failure on a software setting -- or possibly a SPOF on a person that configured the DDEC setting.

I need a backup to my backup, which needs a backup...

There's a pattern here. In all four of these incidents, a primary system failed and there was a problem with the backup system or a system that the backup system relied on. More importantly, the systems that failed were regularly tested. Diesel generators, UPS units, chillers, and CRAC (Computer Room Air Conditioning) units are rigorously maintained and tested at every data center that I've worked in and yet these failures still happened.

Clearly, nodding your head when the colo sales person tells you that you’re getting into a tier-4 data center isn't enough due-diligence on your part. You need to understand a whole lot more about your data center infrastructure, design your physical site properly, and prepare for failure. The facility may be tier-4 but you can very easily wire all your critical systems to the same UPS and create a SPOF. You may even wire-up all of your racks with fully independent paths but then deploy your software across servers that have the same PDU.

Now, a lot of you are probably thinking that you're just out of luck if the building has a power failure and you run into a double or triple failure but I don't believe that's the case unless the building has completely lost power or cooling. In all of the failure cases above, note that only a percentage of the data center was down at any one time. You can avoid being down if you plan for failure across your entire SaaS infrastructure (facility, hardware, and software).

For example, if you assume that a PDU will fail (and include it as part of your failure analysis plan) you will verify that each power drop to your rack is from a separate UPS and that your service will stay up if a UPS battery bank explodes (yes, I’ve had this happen). You also won't assume that the receptacles to each rack are labeled correctly. You'll do a circuit breaker test. You won't even believe that the circuit breaker is rated appropriately and you'll do a power burn-in test before putting the rack into service to make sure you can hit 80% of the circuit without tripping a circuit breaker.

Why do I torture colo vendors with these tests and why do we monitor everything including temperature/humidity on every rack and temperature of every hard disk in the data center?

Because I've answered a phone call when a circuit breaker failed early, a CRAC unit had an airflow problem, rack power was partially lost after flipping the wrong circuit breaker because a circuit was labeled incorrectly, the batteries in a PDU exploded during a building utility breaker replacement, fuses blew when pushed into a not-so-forgiving rack PDU, redundant server power supplies started smoking after failing to failover, and an electrician accidentally turned off both circuit breakers to a rack on the launch of a major product (for the record, these circuits were labeled correctly, it was just a mistake on his part during maintenance).

This doesn't even include the environmental problems like CRAC unit failures or network problems like an ISP black-holing all of your traffic between data centers (happened multiple times now). The truth is, almost every data center experiences these small events a few times a year but most SaaS application owners are either too small to be impacted by these problems or they aren't monitoring well enough to even notice a small event.

Once you start getting into over 20 racks of gear or a few thousand hard disks, you’re going to start seeing regular hardware failures and you need to be prepared.

Now the good news

The good news is that you can protect yourself from many of these problems. In part two of this post, I’ll talk about categories of failures (electrical, environmental, network, hardware, and software), the components of each category, and provide a mitigation step for each component. In part 3 of the post, I’ll see if I can post a checklist for reference.

Tips for Handling Events, Incidents, Outages, and Maintenance

2010-04-30T15:27:00.001-07:00

I get a lot of questions from new service teams about what they should do to prevent downtime but very few people ask for advice on how to handle an incident. This is a bit like asking a boxer for the best way to avoid getting in the ring. It’s not a question of “if” you’re going to be in the ring but “when”. There’s an old saying – the more you bleed in the gym, the less you bleed in the ring and that definitely applies to incident management as well.

Having sat in on more war rooms than I’d like to remember, I thought it might be handy to write down some of the things that my team has found useful over the years. I think every service organization should have a standard approach towards three specific activities:

1.    Tips for Handling Service Incidents (just one service)
2.    Tips for Handling Service Outages (multiple services affected)
3.    Tips for Handling System Maintenance

I hope these posts help you with your handling of incidents, outages, and maintenance. Success here is mostly about being prepared, being calm, good communication, and practice, practice, practice. If you think your service is bullet-proof and you won’t need the practice – you’re wrong :-)

Tips for Handling Service Incidents

2010-04-30T15:26:00.000-07:00

This is the first post related to “Tips for Handling Events, Incidents, Outages, and Maintenance”.

Incidents with a single service are probably the most common type of incident and usually get resolved fairly quickly. Examples here are a blown power supply that results in server failover (or not), bad GBICs that result in dropped packets, daemons that crashed, etc.

The incident usually starts with a email, a text message, a phone call from the NOC, or one of your clients popping their head in your office with their hair on fire. You need to get your head straight and quickly run through a simple process:

Get your head straight

First, stay calm. The worst thing you could do is cause a major outage, destroy some data, or make the existing problem worse in a panic. Simple problems can easily become large complicated problems after a few bad decisions made in haste. Take a breath before continuing. This is especially important with a page at 3AM or if a panicky client is in your office. Tell the client you’ll handle the problem and run through your normal procedure.

Don’t be a hero. Get someone else to run the incident if your judgment is impaired due to lack of sleep, alcohol, or medication.

Remember the prime directive – your job is to restore service as quickly as possible. You are not there to debug interesting problems with your service.

Solve the problem immediately if it’s a simple problem and you can do it in under a minute

It can take up to five minutes to get into a room, get the right people on the line, etc. Just fix it and send out an email after the fact if you can fix it in less than a minute.

For example, let’s say you get an alert from your performance monitoring system that some of your connections are timing out followed by an alert that one of your webservers is running out of memory (but all other webservers are looking fine). Let’s say a large job on the webserver is creating an intermittent high load but it’s not enough to yank the machine out of the load balancer. By all means, yank this webserver out of the load balancer pool so new connections aren’t forwarded to that machine to restore service quickly.

In doing so, you’ve resolved an intermittent problem with the service and you can then send out an advisory alert before debugging what went wrong or coming up with a better solution to the load balancer monitor (or you can go back to sleep).

The danger here is letting this one minute investigation turn into five or ten minutes. If you can’t fix it in a minute, get busy with the normal process.

Do the normal process

Determine the severity, communicate an alert, and get backup.

I’m always stunned when I see someone take a call at 3AM and jump right in to fixing the problem without assessing whether it can wait until morning. Determining the severity is a critical step in ensuring that you don’t turn a problem in the middle of the night into a much larger problem. If it can wait until business hours when you’re awake and more people are around for support, by all means wait until the morning.

You also want backup if it’s an extended event (more than five minutes). The extra person can take over communication duties while you resolve the problem. This two-person approach works well. First, you don’t forget to send regular updates and second you have a co-worker to bounce ideas off of. Just talking through a problem with a co-worker can often help identify the problem. Don’t be a hero. Get help.

Make sure you have a screen sharing room and phone conference reserved for your team. You can’t waste time passing around bad passwords or having multiple people setup phone conference whack-a-mole during an outage. Think of this like a fire drill. Everyone should know how to exit the building and where to assemble.

A normal process means you’ve gone through this before. If you haven’t – practice with simulated events! Also make sure you have standard email templates on multiple servers or your laptop. It’s critical that your email template identifies the incident severity, a short description, and the time of your next update. This will prevent panicky people from bugging you with status requests.

Identifying what is actually wrong is an entire art in itself and something I’ll try to address in a future post

Tips for Handling Service Outages (multiple services affected)

2010-04-30T15:23:00.000-07:00

This is the second post related to “Tips for Handling Events, Incidents, Outages, and Maintenance”.

You’re about to have a interesting day/night. Multiple business-critical services are offline or having intermittent problems. It affects revenue, your company-wide-outage processes have been started, and you’re the lucky person on the on-call roster to lead the outage.

This is a different beast than resolving a problem with an single service. Here, you’re going to have to coordinate across several services and try to get the entire system up and running as soon as possible. The biggest obstacles here are coordination, communication, and discipline. This requires lots of practice before you get good in the role.

I’ve resisted role terminology as much as possible up to this point but I’m going to introduce three roles here: the Incident Manager, the Service Manager, and the Communication Manager. There are lots of names for these roles and you can use any name as long as it’s clear that the Incident Manager is in charge of the entire incident, each Service Manager is responsible for reporting on their service, and the Communication Manager is responsible for handling communication with everyone else and pulling people in as necessary. It’s critical that everyone understand these roles and relationships or a phone call/chat session can turn into a nightmare.

The Incident Manager’s job here is to drive communication, get regular updates from the Service Managers, and look for higher level patterns across the affected services. For example, let’s say three separate services are all having connectivity problems. Each Service Manager is going to be heads down looking at their particular service. The Incident Manager should be looking for a common theme here.

Let’s say that all of the services are having network problems. An Incident Manager might start with the following questions: “Are the services having problems trying to connect to the same place? Are they having problems trying to reach other?” A quick view of a network architecture diagram can tell him what is in common and he can start asking the right questions or he can have the communication manager pull in the right team. In this case, let’s say all the services are connected to each other via a common VPN circuit between data centers and the VPN team isn’t on the call. The incident manager could then pull in the VPN team to verify the systems even if the VPN team’s monitors haven’t gone off. The point is that the Incident Manager has the bandwidth to pull people in and explore ideas while the Service Managers are busy troubleshooting their particular service. The Incident Manager is a critical part of restoring service as quickly as possible and it’s the bandwidth to look at the big picture that makes this possible.

Here are some tips for Incident Managers:

Get your head straight (see previous post on Tips for Handling Service Incidents)
Stay calm and immediately address panic. Your demeanor will affect everyone on the call. Similarly, one panicky person can results in lost time or productivity. There’s no time for panic, stress, or sniping during a call. Address any of these problems immediately.
Control your call. Don’t give out the phone conference number. Non-essential people shouldn’t be joining your meeting. If they feel like they can contribute, they should go through the communication manager.
Don’t assume anything. “Should” is a dirty word during an outage. It’s your job as Incident Manager to validate assumptions while the teams are debugging their specific systems.
Enforce phone discipline. If someone is calling in and has a lot of background noise, tell them to mute, move to a quieter location, or switch to a different phone if possible.
Get comfortable with silence. Most people want to fill the void or speculate during these calls. Let the Service Managers do their job and spend your time as Incident Manager thinking and calling in the right people.
Do a roll-call every five minutes. You should read through the issues that you’re tracking and ask each Service Manager for an update. For example, “Five minute roll-call. We’re tracking two issues. Networking team. You’re a go for update. [update happens]. Thank you. Storage team. You’re a go for update…” These five minute updates help everyone stay focused on the problems and help provide context for anyone joining the call. Framing the list of issues also helps someone correct you if you’ve accidentally forgot about an issue.
Consider an ATC-like-model for communication. This is mostly my preference since I’m a pilot and a nerd but I find that enforcing communication discipline greatly helps during an outage as you don’t lose time to random conversation. Treat the phone as a precious shared resource – something that Service Managers shouldn’t hog with a long question or discussion. They should first request time from the Incident Manager. In the example below, note the control flow and that everyone positively acknowledges their assignment. There’s nothing worse than silence when you ask someone to do something. They should confirm that they understand you and are working on the task. In the following example, John is the Incident Manager and Dave is the Communication Manager:

Storage team: “John, storage team has an update”
Incident Manager: “OK, thanks. Standby. Dave, please call in the VPN team on call engineer”
Communication Manager: “OK. I’m calling in the communication manager”
Incident Manager: “Thanks. Storage team. Go ahead”
Storage team: “We think there may be a network problem outside of our rack”
Incident Manager: “We’re thinking the same. We’re narrowing in on VPN. Please do a traceroute and get back to me with the results”
Storage team: “OK. I’ll callback with the traceroute”
[Incident Manager adds this to his tracking list for the next five minute update]
[Meanwhile, the communication manager is capturing relevant events and sending out updates as required]
[Later, from the storage team…]
Storage team: “Storage team has the traceroute”
Incident Manager: “Go ahead storage”
Storage team: “it looks like we’re getting blackholed with our outbound carrier. We’re working with our colo provider on their BGP routes”
Incident Manager: “Thanks. I’ll track that. Network, please confirm the traceroute blackhole.
[All this can happen in parallel before the VPN team even joins the call]

Carefully track your events. Pen and paper are great but a text editor and a screen-sharing program in a designated room (with Adobe Connect for example) are even better. You can post all information and updates in windows and everyone immediately sees the current status as soon as they join the room.
Make sure you are all referring to a single version of the truth. If you have a monitoring system and an executive dashboard, make sure everyone is looking at the monitoring system and not some fancy dashboard that was thrown together by another group.
Start preparing a shift rotation if it’s a multi-day event. The worst thing you can do is stay up for 48 hours. I’ve done this before and, while it seemed heroic at the time, it was just plain stupid. You want your people at their best if you’re going into extra innings.

Tips and Tricks for System Maintenance

2010-04-30T15:20:00.000-07:00

Tips and Tricks for System Maintenance

This is the third and final post related to “Tips for Handling Events, Incidents, Outages, and Maintenance”.

System Maintenance is a more relaxed version of the previous posts but there’s still a high probability that you might cause an incident during your maintenance. The difference is that you actually get to plan for it. System upgrades and maintenance can be very complicated and involve multiple teams doing many things in parallel. I’ve seen several maintenance periods where there wasn’t a clear plan and people were working on things when they weren’t asked to causing all sorts of problems (imagine your network team killing the firewall before you diverted services to another cluster) and this is never a pretty situation.

As in the previous parts of this post, I highly recommend a second person and an ATC-like communication structure here to save time. I’ve probably taken this to an extreme, but I’ve gone with the flying/pilot/co-pilot metaphor:

Develop a flight-plan

Write down exactly what the goal of the maintenance is, what you’re going to do, and when you’re going to do it (down to the five minutes where each task is going to occur). This helps uncover any gaps in your maintenance (like forgetting to let existing connections finish before restarting a server).

Plan for Bingo

“Bingo” is the point in a flight where you have just enough fuel to get back to the airport. Pick a time that your maintenance has to be done or your rollback. This is mostly for disruptive maintenance but it’s also useful as part of a big upgrade. If you haven’t gotten it right by a certain time, you have to ask whether it’s better to roll-back or whether it’s too dangerous to rollback. The co-pilot role is responsible for enforcing “bingo”

Do a Preflight

Review the maintenance at least one day prior. Everyone should be clear on the order of events and their role

Take-off

Make sure you’re all at the same airport  In this case, make sure you document which data center and which cluster you’re working on. Don’t assume. Have everyone confirm that they are in a shell in the correct data center and cluster.

Start the clock and the announcements so the co-pilot can track progress against the plan and send updates
Read each line of the flight-plan before asking someone to do something. People can jump ahead and accidentally do a step out order. For complicated maintenance, you need to coordinate each activity.

Pass Control

Consider a single control point if your maintenance is complicated with lots of people and coordination. In this model, only one person has the airplane at a time. This prevents multiple people from doing maintenance that interferes with each other. For example:

Pilot: “Storage, please cutover to the new cluster. You have control”.
Storage Team: “Cutting over to the new cluster. I have control”
[The pilot then does the normal five minute updates as with an outage or refers to the expected time on the flight plan and ask the storage team for an update if that time expires]
[Later, when the storage person is done]
Storage Team: “Storage cutover complete. Returning control”.
Pilot: "I have control. Network team, ..."

This is all very overkill with simple upgrades but, with complicated upgrades, I’ve seen this reduce confusion and prevent problems. In the example above, nobody is allowed to do any other type of maintenance while the storage team has control (think of it as a critical section for maintenance).

On the other hand, sometimes you just don’t have this luxury due to time constraints and you need to run things in parallel. There’s nothing wrong with running things in parallel as long as you’ve planned appropriately and understand the risks.

Follow a Landing Checklist

Always follow the appropriate checklist for restarting a service. This can often get skipped after maintenance if you’ve had an interesting upgrade. But, there’s nothing worse than forgetting to turn off a “maintenance” page and getting woken up hours later to find out that you caused multiple hours of downtime by missing this simple check.

Use Autopilot

Finally, if your flight-plan is complicated – automate it! Run scripts instead of doing many manual steps.

HBase Performance Post

2010-04-26T15:02:00.000-07:00

The folks at hstack.org posted a great overview of their latest HBase performance testing. There are some heard-earned performance testing nuggets in the article along with interesting random read/write and map-reduce stats. They also have a small peek at their hardware configuration (lots of spindles!).

SSD Primer and SandForce Controller

2010-04-15T07:40:00.000-07:00

If you're new to SSDs or just want a fantastic in-depth review of the technology along with real-world testing, check out this anandtech.com article. If you're looking for a more recent analysis of SSD products, checkout this March article, also from anandtech.com.

The SandForce controller has been getting a lot of press lately, following on the heels of the Indilinx controller. Here's a review of a new Corsair SSD with the SandForce controller.

The SandForce controller takes a different approach than other SSD controllers and has introduced deduplication and the RAID concept to NAND cells. They call their RAID approach RAISE (Redundant Array of Independent Silicon Elements). Checkout more here.

nosql:eu conference (April 20-22) looks very interesting

2010-04-15T07:15:00.000-07:00

I'm really looking forward to some of the slides and posts from the upcoming nosql conference in London.

The agenda is a drool-fest for any scalable KVP/metadata geek. I like how most of the talks are centered on real-world use of various technologies (Cassandra, CouchDB, Dynamo, HBase, MongoDB, Neo4j, Riak, etc.).

Here's a copy of the agenda:

Tuesday April 20
08.30	- Registration, Coffe and Mingle
09.30	- The Guardian's use of NoSQL - Matthew Wall, The Guardian
10.30	- Coffee break and mingle
10.50	- An overview of NoSQL - Alex Popescu, MyNoSQL
11.50	- Lunch break and mingle
13.00	- Key-value stores and Riak - Bryan Fink, Basho
14.00	- Coffee break and mingle
14.20	- Document-oriented databases and MongoDB - Mathias Stearn, 10gen
15.20	- Coffee break and mingle
15.40	- Column-oriented databases and Cassandra - Jonathan Ellis , Rackspace
16.40	- Coffee break and mingle
17.00	- Graph databases and Neo4j - Emil Eifrem, Neo Technology
18.00	- Evening party with loads of beer and mingle
Wednesday April 21
08.30	- Coffee and mingle
09.30	- On the Birth of Dynamo - Werner Vogels, Amazon
10.30	- Coffee break and mingle
10.50	- Twitter's use of Cassandra, Pig and HBase - Kevin Weil, Twitter
11.50	- Lunch break and mingle
13.00	- CouchDB at the BBC - Enda Farrell, BBC
14.00	- Coffee break and mingle
14.20	- Why Big Enterprises are Interested in NoSQL - Jon Moore, Comcast
15.20	- Coffee break and mingle
15.40	- Memory as the New Disk: Why Redis Rocks - Tim Lossen, Wooga
15.55	- Tokyo Cabinet, Tokyo Tyrant and Kyoto Cabinet - Makoto Inoue
16.10	- Thomas Kuhn Predicted the Fate of the Relational Database - Neil Robbins
16.25	- Notes from the field: NoSQL tools in Production - Matthew Ford
16.40	- Coffee break and mingle
17.00	- Panel debate - Moderated by James Governor, RedMonk
Thursday April 22
08.30	- Registration, Coffee and Mingle
09.00	- Morning workshops - Choose between:
	- MongoDB - Mathias Stearn, 10gen
	- Riak - Bryan Fink, Basho
12:30	- Lunch break and mingle
13:30	- Afternoon workshops - Choose between:
	- Redis - Simon Willison, The Guardian
	- Neo4j - Emil Eifrém, Neo Technology
17.00	- Thank you and see you next year!

NoSQL interview with the hstack.org folks

2010-04-13T11:35:00.000-07:00

Another interesting interview with the hstack.org folks, this time from a NoSQL blog.

Great post on HBase and Adobe

2010-04-09T10:57:00.000-07:00

Cosmin Lehene wrote a great 2-part post about his team's experience with HBase. Here are links to part 1 and part 2. I hear one of his partners-in-crime, Andrei, is working on another interesting post on performance testing related to this work.

FastFlow Parallel Programming Framework

2010-04-09T09:54:00.001-07:00

I’ve been looking into Intel’s thread building blocks during the early morning hours here in Bucharest (jet-lag) and ran across an interesting library that provides non-blocking, lock-free, wait-free, synchronization mechanisms.

Check out this tutorial page with small code snippets and some sample pipelines/farms:

http://calvados.di.unipi.it/dokuwiki/doku.php?id=ffnamespace:usermanual

Here are some background links:

http://en.wikipedia.org/wiki/Fastflow_%28Computer_Science%29
http://calvados.di.unipi.it/dokuwiki/doku.php?id=ffnamespace:about

From the fastflow page:

“FastFlow is a parallel programming framework for multi-core platforms based upon non-blocking lock-free/fence-free synchronization mechanisms. The framework is composed of a stack of layers that progressively abstracts out the programming of shared-memory parallel applications. The goal of the stack is twofold: to ease the development of applications and make them very fast and scalable. FastFlow is particularly targeted to the development of streaming applications.”

From wikipeida:

“Fastflow is implemented as a template library that offers a set of low-level mechanisms to support low-latency and high-bandwidth data flows in a network of threads running on a cache-coherent multi-core.[1] On these architectures, the key performance issues concern memory fences, which are required to keep the various caches coherent. Fastflow provides the programmer with two basic mechanisms: efficient communication channels and a memory allocator. Communication channels, as typical is in streaming applications, are unidirectional and asynchronous. They are implemented via lock-free (and memory fence-free) Multiple-Producer-Multiple-Consumer (MPMC) queues. The memory allocator is built on top of these queues, thus taking advantage of their efficiency.”

AMD 12-core opteron versus 6 core xeon

2010-04-09T09:49:00.000-07:00

I'd like to have seen a larger set of tests thrown at this one, but you have to love all the auto-enthusiast references in this anandtech.com review of the new 12-core Opteron versus the newer 6-core Xeon.

That's two, 6-core Instanbul chips bolted together. Reminds me a bit of the Pentium D with a much larger cache coherency problem (imagine how much of a problem this is going to be as we keep adding cores to chips).

New WD Velociraptor VR200M

2010-04-09T09:38:00.000-07:00

WD has released their next generation VelociRaptor (10K RPM, 2.5" disk). It has a new 6Gbps interface and 600 GB of space. There's an interesting review comparing this disk versus a couple of non-enterprise SSDs here.

SuperMicro 24 core motherboard

2010-04-09T09:31:00.000-07:00

Speaking of 24 core motherboards with loads of RAM, I ran across this new SuperMicro motherboard the other day when doing some research. It's truly terrifying how many cores and RAM you can toss onto one box now.

Assuming one core is dedicated to a Dom0, you could have 23 VMs each with a dedicated core and over 8GB or RAM if you add all 192GB of RAM.

Here are some specs from the link above:

Quad Intel® 64-bit Xeon® MP Support 1066 MHz FSB
Intel® 7300 (Clarksboro) Chipset
Up to 192GB DDR2 ECC FB-DIMM (Fully Buffered DIMM)
Intel® 82575EB Dual-port Gigabit Ethernet Controller
LSI 1068e Dual Channel 8-Port SAS Controller
6x SATA (3 Gbps) Ports via ESB2 Controller
1 (x8) PCI-e (using X16 slot), 1 (x8) PCI-e (using x8 slot) & 1 (x4) PCI-e (using x8 slot) 1x 64-bit 133MHz PCI-X
ATI ES1000 Graphics with 32MB video memory
IPMI 2.0 (SIMSO) Slot

OCZ PCI-e SSD with field-replaceable MLC NAND

2010-04-09T09:22:00.000-07:00

OCZ is ready to mass produce it’s PCI-e SSDs with field replaceable MLC NAND flash modules.

This makes the MLC versus SLC debate a bit moot if you can just replace the NAND when it wears out like a bad disk. Did I mention that it has 8 separate Indlinx controllers, up to 2TBs of space, and has peak transfer rates of 1.4GB/s for reads and writes (that’s gigabytes not gigabits)? I can’t imagine what will happen with a Sandforce controller version of one of these monsters.

This is some seriously interesting temporary storage for a virtualization cluster that needs some fast DAS. With 2 TB, you could carve up 87 gigabytes for 23 VMs on a 24-core virtualization box. That’s mighty interesting.