My old ESXi 4.1 whitebox was nearly begging me to retire him. The maximum of 8gb memory on the Asus P5k mainboard and the 2 sata disks (no raid) spinning at 7200rpm were not keeping up anymore.
After doing the mandatory research on the whitebox HCL sites, I decided to order the following:
- Intel Core i7 2600 3.40GHz 8MB
- Asus P8H67-V Mainboard (which has integrated VGA)
- Kingston 16gb Memory
At first I wanted to go for a sata-600 raid controller and place 2 sata-600 Western Digital VelociRaptor 10000rpm disks in raid 0. Instead I decided to take the plunge and go for 1 sata-600 Intel SSD, plugged in directly on the Asus onboard sata-600 controller.
Just like my old whitebox, I wanted to run ESXi 5.0 from a 2GB Kingston USB stick.
My co-worker Gabrie van Zanten pointed me to an incredibly easy way of installing ESXi to USB: Create a VM in VMware workstation, mount the ESXi 5.0 iso and connect the USB stick. Choose to install ESXi 5.0 to the USB stick and you’re done.
I connected the USB stick to my new system, enabled VT in the BIOS and booted off the USB stick.
As expected the onboard Atheros AR8151 NIC was not recognized, but no “vmkctl.HostCtlException Unable to load module /usr/lib/vmware/vkmod/vmfs3: Failure” errors which my old ESXi 4.1 whitebox threw when it did not recognize the onboard NIC.
Instead ESXi 5.0 reported that my network adapter was not supported, so I used the Intel Pro/1000 GT adapter from my old whitebox and tried again.
After that ESXi 5.0 booted without problems and has been running stable with 8 servers for several days.
Great. I have same situation i have old GIGABYTE GA-P35-DS3R Motherboard with Q6600 Quad Intel Processor with max 8 GB RAM. I really liked your spec above where i am put more memory.
Does your spec above work with vSphere 5.0 HA / FT / DRS Features. thank you in advance… Your config is the best i found googling….
Hi Dewang,
I’m not running vSphere on my whitebox (yet) so I can’t comment on HA, FT and DRS.
Thank you Michel for your reply.
I found some good information for building vmware lab for test and learning which i was looking for.
I am listing the links below in case if you are interested.
http://tinkertry.com/vzilla/
http://tinkertry.com/vmdirectpath/
http://tinkertry.com/files/Z68-Motherboard-VT-d-ESXiNIC-RAID-Test-Matrix.pdf
Hi
Dewang
Where you able to build your Esx lab with Vzilla’s config.
If so , where you able to enable FT and use vt-d vt-x etc..
I going to build 2 servers , but i realy want to have those fonction .
Thanks
Benoit
Hey Michel, how is this Whitebox of yours doing? Curious if it’s been stable for you.
Hi Paul,
I’m still very happy with my whitebox… Haven’t run into any troubles so far.
Running about 10 servers on it without noticing any performance loss.
Hi Michel,
Thanks for sharing your experience with your new whitebox. Was wondering if you could answer a few questions.
Does this Mobo support vt-d? I did a google and it seems it does, can you confirm the option is in the BIOS?
I am considering building 2 boxes for a HA/DRS cluster and will likely use this Mobo and processor combination. Do you have a single box or multiples?
Now you’ve been running this for quite some time. Any other downsides you’ve come across?
Thanks
Steve
Hi Steve,
I checked, but there is no VT-D option in the BIOS.
It’s a single server and I’m not even running vCenter or testing VMware products in my lab, so I can’t comment on that. I’m only using it to host my Window/Linux/freeBSD lab setup.
I haven’t experienced a single thing going wrong and I’m running it 24×7 for several months now.
Compared to my old whitebox, the SSD is what’s really making the difference in this setup.
hmm unless VT-D is just on all the time.
Great to hear it worked out for you. I want to run ESXi on 2 boxes and will use iSCSI for shared storage. I have an Iomega ix2-200 for that but its pretty slow.
SSD’s are amazing, there is no doubt. Now looking to see if I can build another box with a few SSDs to max out 1Gbit iSCSI. RAID0 with 2 or 3 SSD drives would be nice
Will of course backup to a slow 2TB SATA or something because SSDs arent that reliable and RAID0 would hose everything one of those disks died.
SteveJ:
You’ll not even need SSD’s in your iSCSI target to saturate 1GB iSCSI:
Synology iSCSI 1Gb performance
Look specifically for the 4+ disk “+” models.
Conventional disks in RAID5 will already give very nice results. (giving you more storage in return for the same bucks)
I only think in very IOPS demanding cases (lots of VM’s all doing a lot of things at the same time) you’ll really need SSD’s.
But I must admit, my whitebox’s local storage will also have some SDD’s in it, just for the blazinf local fast speed (I hope my SBS 2011 Standard will run like crazy then).
Daniel
Why do people keep asking about i7 motherboards supporting vt-d? the 2600k chip (and all desktop i7′s to my knowledge) don’t support vx-d. They support vt-x but not vt-d.
http://ark.intel.com/products/52214/
Well… the new Ivy Bridge i7 like the i7-3770k with a proper mainboard should be able to support it: http://ark.intel.com/m/products/65719/Intel-Core-i7-3770-Processor-(8M-Cache-up-to-3_90-GHz)
So there’s no real harm in people that keep asking the same question; eventually they might get the answer they’re hoping for.
I have VT-d working on my ASRock mobo and the Intel 2600, seen in screenshots shown here:
http://tinkertry.com/usb3passthru/
Also had an MSI board with VT-d enabled working too (ASUS and Gigabyte didn’t), outlined here
http://tinkertry.com/vmdirectpath
Hi
is the onboard controler reconize by esx .
Can a buy a sata disk and put son vm on it.
Do you think your config work with all the esxi feature like vt-d vt-x ft etc .
thanks
Hi Michael,
I’m trying to build a ESX 5 testlad on a DELL Alienware M17x which have the AR8151 NIC. I have a hard time getting the NIC to function proper.
which driver did you use for the AR8151 NIC ?? and could you please share it
Thanks in advance
Hi Niels,
ESX did not recognize that NIC, so I bought an Intel NIC as you can read in my blogpost.
HI all
With this config, were you able to run everything , vt-d, vt-x ,FT and connect a hardrive
to run VM from it.
Intel Core i7 2600 3.40GHz 8MB
Asus P8H67-V Mainboard (which has integrated VGA)
Kingston 16gb Memory
Thanks.
I am new to this so please excuse the dumb question. Why install ESXi on a USB Stick? Whats the advantage of that? Why not just install on the whitebox?
Thanks
Hi,
Well, if you google around for this you will probably find a lot of opinions.
For me, I just like to seperate esxi from my virtual machines. And since my virtual machines are not running anything important, I can just swap a (defect) disk without loosing my esxi installation.
Another advantage for me is that I can just unplug the usb stick and use my whitebox for other purposes (xenserver, hyper-v) without having to open it up.
hmmm so you saying that i would boot my whitebox via the usb with the esxi install on it? Sounds cool, think i am going to try it. Second question whats a good setup? Is it better to get a box or build one from scratch? Any suggestions? I’d like to keep my budget to $500!?
Thanks
I have the same issue. I am trying to install it at home to learn how to install and configure ESXi. however the workstation I am using is an lenovo with 8 gb of ram and a 1tb hard drive. It has a internal nic card and intel 82579V. However my system only has PCI-express slots.. Does anyone know of a pci-express card that works on ESXi 5.0 or 4.1?
I am trying to not have to buy another system as I was laid off from work a few months ago and that is why I am trying to learn ESXi to add to my resume.
Thank you
Scott Newell
I too have an lab-server, with ESXI 5 on.
The mainboard is a ASUS P5K64-WS P35.
As you all know, the P35 officially supports up to 8gb DDR3, or 16GB DDR2 on some models.
So i decided to buy 16gb DD3 ram, and its worked nice!
The bios and ESXI recognize all 16gb
The next performance killer is the hard drives.
On one sata 7200rpm drive you can run about 3-4vms.
Im planning on buying a Dell Perc 6/i, its supports raid 0,1,5,6 and 10. (you can get a used one for about $100-150 on ebay).
Also buy one with the battery module, so you can get the cache function enabled, increase the preformance a lot!
The Perc 6/ir supports only raid0 and 1.
If going for the internal route a set of 10k/15K (dualport) SAS-drives or SDD’s will be great.
That should give you some IOPS to spare.
SAS drives will also be a better choice than SATA drives in RAID sets, you should see faster initializations and rebuilds than with regular SATA drives.
You’ll have to carefully pick your SSD’s if you’re going to use them in a RAID array; you’ll need SSD’s with a good garbage collector so performance doesn’t degrade (over time) when almost fully loaded with VM’s. TRIM is not yet supported in RAID arrays.
Always try to keep some space free on them to allow for good performance.
I agree with you, SAS / SSD drives would have been the perfect setup.
But they are too expensive, and its only a lab server.
So i think 4x2TB Seagate (not green series) will do the job.
I also have seen screenshots of 4-5 SATA disk in RAID5, and the performance is good enough.
Hi Michel,
Can you please let me know whether the below configuration is supported in ESXi 5.0. or is there any hardware changes or ESXi 5.0 update required…
Intel i7-2600 / 3.4 GHz
ASUS P8Z68-V PRO / GEN3 (With VGA & Intel NIC)
Kingston 16GB Memory
Thanks in Advance….
Hi Andrew,
The CPU and memory are obviously going to work, but I can’t comment on the P8Z68-V PRO.
You’re going to have to crawl various whitebox sites for that. That’s exactly what I did before I bought the hardware from my blogpost.
Good luck!
Hi Michel,
Have you ever had any problem with ESXi booting up with your setup? I was running fine for a short while until one day, I powered the box off, moved it to make way for some diy, then moved it back, powered it on, esxi started booting, but it now won’t get past “initializing scheduler”…it seems to hang then reboot itself. It just cycles like that now. I’ve tried disabling everything I can in the bios but to no avail. I’ve reinstalled esxi 5 onto my usb flash (the installer boots fine), and tried other usb flash drives, but nothing has helped so far. If you have seen this and have any ideas where to go from here, that would be great because I’m starting to get a little stuck.
Thanks
Hi Rob,
I’m sorry I can’t help you with that. Mine has never had the issues you describe.
Hi Michel. Thanks for taking the time to respond, and so quickly. I ended up biting the bullet and spending some money on replacement parts this morning. It turned out to be an issue with the HDD. The old HDD is still spinning up OK, but esxi booted backup without an issue once the HDD was swapped out, so there is obviously some kind of issue there.
I thought I’d let people know incase they have similar issues.
Kind Regards
Hey there,
In regards to SDD, are you running with any SSD drives? I am looking forward to put one on my server but I am not sure it ill be compatible with the ESX. I am neither sure if it is the SATA controller (will be sata 3 6MB/s), the drive or both which have to be compatible in order to work. Any comments on this?
I have the P68Z68 – LX running ESXi5 and it is Great. Intel i5 CPU and Corasir Vengance memory.