Buying stuff

We made the decision for our new iSCSI SAN: an EqualLogic PS3700X. It has 16 400GB 10k rpm SAS drives, for a total of 4.8 TB usable space. We were planning on purchasing a PS3600X (300GB drives), and I'm glad we decided to wait as long as possible for newer products. For the extra 1.4 TB we only paid about $1k more. We are also looking at purchasing a PS83E for off-site replication, but that's not a definite yet.

Yesterday there was an announcement that Dell was purchasing EqualLogic. I've heard a few people's opinions on this, and everyone for the most part seems to be apathetic. I don't have the impression that Dell will screw up what a great company EqualLogic is, and I'm also not worried about support. From my perspective, everything should stay the same. Marc Farley from EqualLogic is keeping a good summary of the reactions from the internet over at his Storage @ Work blog. On a side note, our LeftHand salesman (which I had recently informed that they lost to EqualLogic) was emailing me this morning trying to use the situation to their benefit. Nice.

We are also in the process of upgrading our network infrastrucure. We are purchasing a Cisco 4510 with 6 48-port/gigabit/PoE cards and 2 48-port/gigabit cards. We are also purchasing 3 Cisco Catalyst 3560G-48PS switches for our IDF closets, and a 3560G-48TS for our iSCSI SAN.

One of the things we wanted to implement was a way to dynamically assign VLANs based on MAC address. I've already written a script that is currently storing all of our MAC addresses in a SQL database when users log in. There are a couple different technologies that will accomplish this (VMPS, URT, 802.1x), but we haven't picked one yet. It seems that Cisco is trying to phase out VMPS in favor of 802.1x, and I can't find a lot of information on the internet regarding VMPS usage. URT is in the same boat, and one guy on the internet seems to think they were both "Cisco trying to figure out which direction to go in". 802.1x seems great, but it's more than we really need. It dynamically assigns VLANs as an afterthought; it's main purpose is port-security regulated by user authentication. That's great, but what about our 40 or so network printers that don't support 802.1x? What about our thin clients that need to PXE boot? So yeah, I'm still researching other possibilities. When it's done, I'll be sure to post a detailed explanation of how we did it, because I can't really find anything that is straightforward and gives a complete picture (most miss the section about how to get MAC addresses into the actual database).

EqualLogic performance confusion

I spoke with our EqualLogic sales rep yesterday, and they are suggesting we go with a PS300E. It has 7,200 rpm drives. Not 10k. We always avoided 7,200 drives like the plague. But here is EqualLogic, one of the biggest players in the iSCSI industry, telling us we should put their 7,200 rpm drives in. Ok...

We had originally planned on buying a PS3900XV, which has 15k drives. But now it's between the PS3600X (10k drives) and one of the 7,200 rpm models. They are all the same box, but with different drive sizes. Right now it looks like the 7TB raw model (PS300E) is the one we should buy, but we might splurge and get something larger.

But back to the performance...7,200 rpm! I had to ask around, so I threw a post up on to the VMTN forums. And here's what I have so far:

• joergriether tested a PS300E and a PS3900XV, and says they were about the same...
• Yps has 90 VMs running on one PS400E, and has no trouble with it...
• christianZ pointed me over to the thread he had started for SAN comparison, and also suggested the PS3600X over the PS3900XV. Not sure if he meant over the PS300E also...

So right now it looks like the PS300E is going to be what we go with. Sigh...

Configuring LUNs for virtual machines

Two nice folks responded to my post on the VMTN forums regarding how to set up VMFS partitions and LUNS for virtual machines on our SAN.

Before I talk about all of this, Stephan Stelter from LeftHand Networks also included a link about "How to Align Exchange I/O with Storage Track Boundaries". Good to know...

But anyway, this is apparently what most people do:

• Create VMFS partitions for system drives, and place virtual machines that you would back up together on the same LUN.
• For data drives, map to the LUN directly with the OS iSCSI initiator.

I haven't decided if we should place all our system drives on a single LUN or break them up into groups. I think I want to place domain controllers on their own LUN so I can restore our entire AD without the possibility of a USN rollback. And all our other servers don't really fall into groups... if start to think about splitting them up, it gets to the point where I might as well make a LUN for each of them. And from what I've read so far, performance isn't really a problem if you have less than 16 VMs on a LUN.

So with all the system drives (except AD) on a single LUN, restores become a little more complicated. Not too hard.. just harder than using individual LUNs. We would need to bring up a snapshot on a new LUN, mount the LUN on ESX server (or maybe somewhere else), and copy the VMDK of the machine we're restoring over to the production LUN.

If there is a LUN for each system drive, restoring from a snapshot would just require a click on the EqualLogic array's web interface, and everything would be back.

So what are we going to do? I still don't know yet. And there's plenty of time to decide, and neither way is the "wrong way" (nor the "right" way). So we'll see...

EqualLogic wants to sell us stuff

EqualLogic invited themselves in to our conference room yesterday to show us one of their iSCSI SAN arrays. All I can say is...wow. If you're looking at a SAN solution, definitely consider EqualLogic. And they will jump at the chance to give you a 2-hour in-house demo.

But regarding my post the other day...boy was I way off. Apparently I didn't understand storage virtualization. I was debating how to split the drives up, half RAID 5, half RAID 10, or all of it RAID 10...

EqualLogic introduced me to RAID 50. It requires a minimum of 6 drives not including a spare, so I'm not surprised I never came across it before. It basically takes 2 RAID 5 sets and stripes between them (RAID 0).

The EqualLogic PS3900XV has 16 300 GB drives. 2 RAID 5 sets is 8+8. A spare on each leaves us with 7+7. And the parity bit brings us down to 6+6. So out of the 16 drives, we get to use about 12 of them space-wise, at least 3.4 TB. EqualLogic also reserves some space on each drive for housekeeping, but this shouldn't end up being more than 200 gig or so (the tech couldn't remember if it was 20 megs or 20 gigs reserved per drive).

RAID 5 would have given us 4.5 TB but would have been pretty slow on write speed. And RAID 10 would have given us 2.4 TB. So 3.4 is pretty good...

And even better is the fact that its all virtual. We make a LUN for our file server, one for Exchange, one for SQL, etc. But we can grow these LUNs at any time, so instead of saying the file server gets 2 TB and Exchange gets 600 GB, we can just set them to about 150% of what they are right now and then expand them as they need it.

My only question now, and one that EqualLogic couldn't answer for me, was do we put all of our system drive VMDKs on a single LUN or make a seperate LUN for each server? I've posted a question on the VMware forums, and I'm sure a lot of people thought about the same thing while setting up VI3 and a SAN.

LeftHand introduces iSCSI target appliance for ESX server

LeftHand Networks recently announced a software iSCSI target for ESX Server. It basically lets you reclaim all your unused disk space in your ESX box, and stick it on your SAN. I'm excited about this, we maxed out our ESX box to hold us over until we purchase our SAN, and this is a perfect way to use all that space. I'm thinking VM backups in case we lost the SAN for some reason. Or maybe IT storage (ISOs, Ghost images). We'll evaluate our needs after our EqualLogic box gets here and the local drives on the ESX server are emptied.