High Availability (HA)

VMware vSphere 6 – Fault Tolerance hỗ trợ đa vCPU

1. Fault Tolerance đa nhân

Fault Tolerance (FT) đã ra đời từ rất sớm nhưng chỉ hổ trợ 1 vCPU. Tuy nhiên, cách thức kỹ thuật vận hành của FT (Record-Replay) khiến VMware không đa vCPU cho FT.

Sau nhiều năm nghiên cứu Fast Check-Pointing, SMP-FT mới hỗ trợ cho các VM với 4 x vCPU và 64 GB RAM.

Hình 1: FT với VM chỉnh và VM thứ cấp

Với Fast Check-Pointing, VM chính và VM thứ cấp đồng thời thực thi cùng một luồng các dòng lệnh khiến việc đồng bộ trở nên nhanh hơn. Nếu độ trễ của mạng (network latency) cho tính năng FT trở nên quá cao để các VM có thể đồng bộ, VM chính sẽ bị ép chậm lại ở một điểm lệnh mà VM thứ cấp có thể bắt kịp.

Hình 2: Cơ chế Record-Replay giữ VM thứ cấp ở một “virtual lockstep” với VM chính

2. Đặc điểm và giới hạn của SMP-FT

Vẫn có một số giới hạn đối với SMP-FT như

• Một host vật lý bị giới hạn hỗ trợ SMP-FT cho 4 VM hoặc 8 vCPU tổng cộng. Giới hạn này bị áp dụng cho cả VM chính và VM thứ cấp trên host.
• Với VM được cấu hình SMP-FT, vẫn chưa cho phép thêm nóng CPU và RAM.
• Các VM chạy SMP-FT không hỗ trợ trong Storage vMotion, vCloud Director, vSphere Replication, VSAN/vVols và vFlash.
• Card NIC 10Gbps được yêu cầu cho tính năng SMP-FT.

Với cơ chế kỹ thuật mới, SMP-FT sẽ tạo ra 2 VM tách biệt hoàn toàn ở 2 host, bao gồm cả datastore.

• Điều này cho phép bảo vệ VM không chỉ khi xảy sự cố ở host mà còn khi xảy ra sự cố ở datastore với zero downtime.
• Ngoài ra, các yêu cầu cũ về phải sử dụng chung một hệ thống lưu trữ (shared storage) và tạo VM với Eager Zero Thick cũng không còn cần thiết. SMP-FT đã hỗ trợ cho local disk hay bất kỳ dạng storage nào khác.
• VM thứ cấp vận hành cũng sẽ tiêu tốn I/O lên hệ thống lưu trữ gần như VM chính. Cần cân nhắc thông tin này khi thiết kế.
• Ngoài ra, cơ chế mới của SMP-FT cũng sẽ tiêu tốn hơn về năng lực của host, có thể lến đến 10%-30%.

Ngoài ra, các tính năng HA Cluster và DRS cũng được nâng cấp để nhận biết và luôn đảm bảo cặp VM chính và thứ cấp đang chạy FT được chạy trong hệ thống các host.

Nên nhớ, FT không thể thay thế được các kỹ thuật cluster khác hoạt động ở lớp cao hơn như Microsoft Cluster (lớp OS) hay Server Load Balancing (lớp Application).

Follow Up: HA Change (isolation response)

I’ve been asking around why the default isolation response has been changed from “power off” to “leave powered on”. It seems that this is done because a lot of customers had VM’s being powered off unnecessary. This happened because the service console or physical switches weren’t setup redundant and thus caused HA to kick in. In other words, for those having complete redundancy, switches and nics, change the default back to “power off” or use the new option “Shutdown VM”.

Shutdown VM requires VMware Tools to be installed. If HA is unable to shutdown the VM within 5 minutes it will be powered down. I would prefer this option, especially when you virtualized services like Exchange, SQL, Oracle etc.

vSphere HA Isolation response when using IP Storage

I had a question from one of my colleagues last week about the vSphere HA Isolation Response and IP Storage. His customer had an ISCSI storage infrastructure (applies to NFS also) and recently implemented a new vSphere environment. When one of the hosts was isolated virtual machines were restarted and users started reporting strange problems.

What happened was that the vSphere HA Isolation Response was configured to “Leave Powered On” and as both the Management Network and the iSCSI Network were isolated there was no “datastore heartbeating” and no “network heartbeating”. Because the datastores were unavailable the lock on the VMDKs expired (virtual disk files) and HA would be able to restart the VMs.

Now please note that HA/ESXi will power-off (or kill actually) the “ghosted VM” (the host which runs the VMs that has lost network connection) when it detects the locks cannot be re-acquired. It still means that the time between when the restart happens and the time  when the isolation event is resolved potentially the IP Address and the Mac Address of the VM will pop up on the network. Of course this will only happen when your virtual machine network isn’t isolated, and as you can imagine this is not desired.

When you are running IP based storage, it is highly (!!) recommend to configure the isolation response to: power-off! For more details on configuring the isolation response please read this article which lists the best practices / recommendations.

Datastore Heartbeating and preventing Isolation Events?

I was just listening to some of the VMworld sessions and one was about HA. The presenter had a section about Datastore Heartbeats and mentioned that Datastore Heartbeats was added to prevent “Isolation Events”. I’ve heard multiple people make this statement over the last couple of months and I want to make it absolutely clear that this is NOT true. Let me repeat this, Datastore Heartbeats do not prevent an isolation event from occurring.

Lets explain this a bit more in-depth. What happens when a Host is cut off from the network because its NIC which carries the management traffic has just failed?

1. T0 – Isolation of the host (slave)
2. T10s – Slave enters “election state”
3. T25s – Slave elects itself as master
4. T25s – Slave pings “isolation addresses”
5. T30s – Slave declares itself isolated and “triggers” isolation response

Now as you can see the Datastore Heartbeat mechanism plays no role whatsoever in the process for declaring a host isolated, or does it? No from the perspective of the host which is isolated it does not. The Datastore Heartbeat mechanism is used by the master to determine the state of the unresponsive host. The Datastore Heartbeat mechanism allows the the master to determine if the host which stopped sending network heartbeats is isolated or has failed completely. Depending on the determined state the master will take appropriate action.

To summarize, the datastore heartbeat mechanism has been introduced to allow the master to identify the state of hosts and is not use by the “isolated host” to prevent isolation.

vSphere 5 HA – Isolation Response which one to pick?

Last week I did an article about Datastore Heartbeating and the prevention of the Isolation Response being triggered. Apparently this was an eye-opener for some and I received a whole bunch of follow up questions through twitter and email. I figured it might be good to write-up my recommendations around the Isolation Response. Now I would like to stress that these are my recommendations based on my understanding of the product, not based on my understanding of your environment or SLA. When applying these recommendations always validate them against your requirements and constraints. Another thing I want to point out is that most of these details are part of our book, pick it up… the e-book is cheap.

First of all, I want to explain Isolation Response…

Isolation Response is the action HA triggers, per VM, when it is network isolated from the rest of your cluster. Now note the “per VM”, so a host will trigger the configured isolation response per VM, which could be either “power off” or “shutdown”. However before it will trigger the isolation response, and this is new in 5.0, the host will first validate if a master owns the datastore on which the VMs configuration files are stored. If that is not the case then the host will not trigger the isolation response.

Now lets assume for a second that the host has been network isolated but a master doesn’t own the datastore on which the VMs config files are stored, what happens? Nothing happens. Isolation response will not be triggered as the host knows that there is no master which can restart these VMs, in other words there is no point in powering down a VM when it cannot power it on. The host will of course periodically check if the datastore is claimed by a master.

There’s also a scenario where the complete datastore could be unavailable, in the case of a full network isolation and NFS / iSCSI backed storage for instance. In this scenario the host will power off the VM when it has detected another VM has acquired the lock on the VMDK. It will do this to prevent a so-called split brain scenario, as you don’t want to end up with two instances of your VM running in your environment. Keep in mind that in order to detect this lock the “isolation” on the storage layer needs to be resolved. It can only detect this when it has access to the datastore.

I guess there’s at least a couple of you thinking but what about the scenario where a master is network isolated? Well in that case the master will drop responsibility for those VMs and this will allow the newly elected master to claim them and take action if required.

I hope this clarifies things.

Now lets talk configuration settings. As part of the Isolation Response mechanism there are three ways HA could respond to a network isolation:

1. Leave Powered On – no response at all, leave the VMs powered on when there’s a network isolation
2. Shutdown VM – guest initiated shutdown, clean shutdown
3. Power Off VM – hard stop, equivalent to power cord being pulled out

When to use “Leave Powered On”
This is the default option and more than likely the one that fits your organization best as it will work in most scenarios. When you have a Network Isolation event but retain access to your datastores HA will not respond and your virtual machines will keep running. If both your Network and Storage environment are isolated then HA will recognize this and power-off the VMs when it recognizes the lock on the VMDKs of the VMs have been acquired by other VMs to avoid a split brain scenario as explained above. Please note that in order to recognize the lock has been acquired by another host the “isolated” host will need to be able to access the device again. (The power-off won’t happen before the storage has returned!)

When to use “Shutdown VM”
It is recommend to use this option if it is likely that a host will retain access to the VM datastores when it becomes isolated and you wish HA to restart a VM when the isolation occurs. In this scenario, using shutdown allows the guest OS to shutdown in an orderly manner. Further, since datastore connectivity is likely retained during the isolation, it is unlikely that HA will shut down the VM unless there is a master available to restart it. Note that there is a time out period of 5 minutes by default. If the VM has not been gracefully shutdown after 5 minutes a “Power Off” will be initiated.

When to use “Power Off VM”
It is recommend to use this option if it is likely that a host will lose access to the VM datastores when it becomes isolated and you want HA to immediately restart a VM when this condition occurs. This is a hard stop in contrary to “Shutdown VM” which is a guest initiated shutdown and could take up to 5 minutes.

As stated, Leave Powered On is the default and fits most organizations as it prevents unnecessary responses to a Network Isolation but still takes action when the connection to your storage environment is lost at the same time.