jump to navigation

Don’t Leave Virtual CDs in Your Virtual CD Drive (HA Won’t Like it) July 29, 2011

Posted by audiomatron in Virtualization.
trackback

That title is a bit misleading. I’ll explain in due time…

As the sarcastic title of my blog would suggest, I’m not an expert. Although, if I am an expert at anything at all, it is learning things the hard way. Why?

This morning, as I was eating my cereal, and looking at Twitter on my iPad, I received a call from someone at work telling me that a number of our network services were not working – an app running on a physical box, and a couple of apps that run as VMs. I remoted in, and found that all but two of my VMs were running fine. These two VMs were displayed as being powered on, but were grayed out in the vSphere client. When I got to work, I found that some sort of electrical issue had caused one of my UPSes to turn completely off, taking down a couple of regular servers, and my one of my ESXi hosts. An HA event had been triggered, restarting all but these two VMs on different hosts. Luckily, none of these apps are really mission critical. However, it concerned me that these two VMs didn’t restart on other hosts. After all, the other VMs that were residing on that host restarted on other hosts just fine. What was so special about these two VMs?

I began looking at the event logs for these VMs, and noticed that they both had an error stating that there were not enough resources for failover. I was puzzled since upon looking at the utilization of my hosts’ resources, I found there to be plenty of resources available on both of the other hosts – and these are not large VMs by any means! As I searched for an answer, I focused on determining what was different about these two VMs. The difference between these VMs and the other VMs was that I had .iso images on the local datastore attached to their virtual CD/DVD drives (left over from when I installed Windows). I typically remove .isos when I’m finished with them, but apparently that day I got in a hurry and left them there. Was that what caused my problem?

I Googled the issue to no avail. I don’t recall ever reading or hearing anything about this. When I gave it more thought, really, it made perfect sense, which I’ll discuss later. However, how could I test if my theory was correct? To the lab!

The Scenario

I set up two physical hosts, using a VM in VMware workstation on my PC as a vCenter server to manage the hosts. Keeping vCenter separate allowed me to accomplish two things – I could avoid the EVC debacle, as referenced in my very first blog post (I’ve torn the lab down since then, and needed to rebuild it for this experiment), and I could actively monitor vCenter during the experiment. Next, I created a Windows 7 VM, and attached an .iso residing on local storage to it, selecting “Connected” and “Connected at power on” for the CD drive in the VM’s settings dialogue (to simulate the VMs’ configuration in my production environment). I pulled the plug on the host that contained the VM, and waited for the HA event to be triggered. As expected, the VM would not power on on the other host, citing insufficient resources for failover. I powered the host back on. I removed the .iso from the VM, and pulled the plug again. Now, the VM would restart on the other host. I repeated these processes several times with consistent results. I had my answer.

Common Sense?

The whole point to having shared storage in vSphere is so all of the hosts in the cluster can access the files that comprise the VMs running on all of the other hosts – this is how HA is able to restart VMs on other hosts in the event of a host failure. If you have the VM’s .vmdk on local storage, it definitely won’t fail over to another host, especially since local storage is on the host, which in this scenario, is now powered off. What makes an .iso on local storage any different? Nothing, apparently. You can’t even vMotion a VM with any ties to a local datastore. When I look at it in this way, it makes sense that HA wasn’t able to restart these VMs on one of the other hosts.

Solutions

While I was writing this article, I continued to explore this scenario further, and discovered something else interesting. Obviously, the best fix for this problem would be for me to simply remember to disconnect my virtual CD/DVD drives when I’m finished using them, or to store my .iso files on shared storage. However, if you must leave the a .iso on local storage attached to a VM, there is a way to do it so that it will not interfere with HA. in the VM settings dialogue, do not check the box that says “Connected at Power On”. In my lab scenario, HA worked correctly with the CD drive configured this way.

Conclusion

Often, I feel that I write these things at the risk of making myself appear..well..dumb. However, if I can help another vNoob such as myself avoid making the same mistakes, then I’ll gladly look dumb. This was a strange issue that I would never have been able to predict, and as I stated earlier, I couldn’t find any information on it. I know others use local storage to store regularly used .isos, yet I’ve never heard any warnings against leaving the .isos attached (although, it’s likely that I did and just forgot). I was compelled to write an article about it.

So what about you? Do you have any interesting stories/lessons learned the hard way from your virtualization journey? If so, I’d love to hear them!

Advertisements

Comments»

No comments yet — be the first.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: