High Availability  高可用性

The CRM uses a service state enumeration to record the current service state. This state is displayed on the GUI and can be queried using the ha-manager command-line tool:
CRM 使用服务状态枚举来记录当前的服务状态。该状态会显示在图形界面上，也可以通过 ha-manager 命令行工具查询：

Here is the list of possible states:
以下是可能的状态列表：

The local resource manager (pve-ha-lrm) is started as a daemon on boot and waits until the HA cluster is quorate and thus cluster-wide locks are working.
本地资源管理器（pve-ha-lrm）作为守护进程在启动时启动，并等待直到 HA 集群达到法定人数，从而集群范围的锁定功能生效。

It can be in three states:
它可以处于三种状态：

After the LRM gets in the active state it reads the manager status file in /etc/pve/ha/manager_status and determines the commands it has to execute for the services it owns. For each command a worker gets started, these workers are running in parallel and are limited to at most 4 by default. This default setting may be changed through the datacenter configuration key max_worker. When finished the worker process gets collected and its result saved for the CRM.
当 LRM 进入活动状态后，它会读取位于 /etc/pve/ha/manager_status 的管理器状态文件，并确定它必须为其拥有的服务执行的命令。对于每个命令，都会启动一个工作进程，这些工作进程并行运行，默认最多限制为 4 个。此默认设置可以通过数据中心配置键 max_worker 进行更改。完成后，工作进程会被回收，其结果会保存给 CRM。

Each command requested by the CRM is uniquely identifiable by a UID. When the worker finishes, its result will be processed and written in the LRM status file /etc/pve/nodes/<nodename>/lrm_status. There the CRM may collect it and let its state machine - respective to the commands output - act on it.
CRM 请求的每个命令都由一个唯一的 UID 标识。当工作进程完成后，其结果将被处理并写入 LRM 状态文件 /etc/pve/nodes/<nodename>/lrm_status。CRM 可以在此收集结果，并让其状态机根据命令输出对其进行处理。

The actions on each service between CRM and LRM are normally always synced. This means that the CRM requests a state uniquely marked by a UID, the LRM then executes this action one time and writes back the result, which is also identifiable by the same UID. This is needed so that the LRM does not execute an outdated command. The only exceptions to this behaviour are the stop and error commands; these two do not depend on the result produced and are executed always in the case of the stopped state and once in the case of the error state.
CRM 与 LRM 之间对每个服务的操作通常始终保持同步。这意味着 CRM 请求一个由 UID 唯一标记的状态，LRM 随后执行该操作一次并写回结果，该结果也由相同的 UID 标识。这是为了防止 LRM 执行过时的命令。唯一的例外是停止和错误命令；这两种命令不依赖于产生的结果，在停止状态时总是执行，在错误状态时执行一次。

The cluster resource manager (pve-ha-crm) starts on each node and waits there for the manager lock, which can only be held by one node at a time. The node which successfully acquires the manager lock gets promoted to the CRM master.
集群资源管理器（pve-ha-crm）在每个节点上启动，并在那里等待管理锁，该锁一次只能被一个节点持有。成功获取管理锁的节点将被提升为 CRM 主节点。

It can be in three states:
它可以处于三种状态：

Its main task is to manage the services which are configured to be highly available and try to always enforce the requested state. For example, a service with the requested state started will be started if its not already running. If it crashes it will be automatically started again. Thus the CRM dictates the actions the LRM needs to execute.
它的主要任务是管理被配置为高可用的服务，并尽力始终强制执行请求的状态。例如，请求状态为已启动的服务，如果尚未运行，将会被启动。如果它崩溃了，将会自动重新启动。因此，CRM 决定了 LRM 需要执行的操作。

When a node leaves the cluster quorum, its state changes to unknown. If the current CRM can then secure the failed node’s lock, the services will be stolen and restarted on another node.
当一个节点离开集群法定人数时，其状态会变为未知。如果当前的 CRM 能够获取失败节点的锁，服务将被抢占并在另一个节点上重新启动。

When a cluster member determines that it is no longer in the cluster quorum, the LRM waits for a new quorum to form. Until there is a cluster quorum, the node cannot reset the watchdog. If there are active services on the node, or if the LRM or CRM process is not scheduled or is killed, this will trigger a reboot after the watchdog has timed out (this happens after 60 seconds).
当集群成员确定自己不再处于集群法定人数中时，LRM 会等待新的法定人数形成。在没有集群法定人数之前，节点无法重置看门狗。如果节点上有活动服务，或者 LRM 或 CRM 进程未被调度或被终止，看门狗超时后（60 秒后）将触发重启。

Note that if a node has an active CRM but the LRM is idle, a quorum loss will not trigger a self-fence reset. The reason for this is that all state files and configurations that the CRM accesses are backed up by the clustered configuration file system, which becomes read-only upon quorum loss. This means that the CRM only needs to protect itself against its process being scheduled for too long, in which case another CRM could take over unaware of the situation, causing corruption of the HA state. The open watchdog ensures that this cannot happen.
请注意，如果一个节点有一个活动的 CRM 但 LRM 处于空闲状态，仲裁丢失不会触发自我隔离重置。原因是 CRM 访问的所有状态文件和配置都由集群配置文件系统备份，该文件系统在仲裁丢失时变为只读。这意味着 CRM 只需要防止其进程被调度时间过长，在这种情况下，另一个 CRM 可能会在不知情的情况下接管，导致 HA 状态损坏。开放的看门狗确保这种情况不会发生。

If no service is configured for more than 15 minutes, the CRM automatically returns to the idle state and closes the watchdog completely.
如果超过 15 分钟没有配置任何服务，CRM 会自动返回空闲状态并完全关闭看门狗。

The resource configuration file /etc/pve/ha/resources.cfg stores the list of resources managed by ha-manager. A resource configuration inside that list looks like this:
资源配置文件/etc/pve/ha/resources.cfg 存储由 ha-manager 管理的资源列表。该列表中的资源配置如下所示：

It starts with a resource type followed by a resource specific name, separated with colon. Together this forms the HA resource ID, which is used by all ha-manager commands to uniquely identify a resource (example: vm:100 or ct:101). The next lines contain additional properties:
它以资源类型开头，后跟资源特定名称，两者之间用冒号分隔。组合在一起形成 HA 资源 ID，所有 ha-manager 命令都使用该 ID 来唯一标识资源（例如：vm:100 或 ct:101）。接下来的几行包含附加属性：

Here is a real world example with one VM and one container. As you see, the syntax of those files is really simple, so it is even possible to read or edit those files using your favorite editor:
这里是一个包含一台虚拟机和一个容器的实际示例。如你所见，这些文件的语法非常简单，因此甚至可以使用你喜欢的编辑器来读取或编辑这些文件：

The above config was generated using the ha-manager command-line tool:
上述配置是使用 ha-manager 命令行工具生成的：

The HA group configuration file /etc/pve/ha/groups.cfg is used to define groups of cluster nodes. A resource can be restricted to run only on the members of such group. A group configuration look like this:
HA 组配置文件 /etc/pve/ha/groups.cfg 用于定义集群节点的组。资源可以被限制只在该组的成员上运行。组配置如下所示：

A common requirement is that a resource should run on a specific node. Usually the resource is able to run on other nodes, so you can define an unrestricted group with a single member:
一个常见的需求是资源应运行在特定的节点上。通常资源能够在其他节点上运行，因此你可以定义一个只有单个成员的无限制组：

For bigger clusters, it makes sense to define a more detailed failover behavior. For example, you may want to run a set of services on node1 if possible. If node1 is not available, you want to run them equally split on node2 and node3. If those nodes also fail, the services should run on node4. To achieve this you could set the node list to:
对于较大的集群，定义更详细的故障转移行为是有意义的。例如，你可能希望尽可能在 node1 上运行一组服务。如果 node1 不可用，你希望这些服务在 node2 和 node3 上平均分配运行。如果这些节点也失败，服务应运行在 node4 上。为实现此目的，你可以将节点列表设置为：

Another use case is if a resource uses other resources only available on specific nodes, lets say node1 and node2. We need to make sure that HA manager does not use other nodes, so we need to create a restricted group with said nodes:
另一个用例是，如果一个资源使用的其他资源仅在特定节点上可用，比如 node1 和 node2。我们需要确保 HA 管理器不使用其他节点，因此需要创建一个包含上述节点的受限组：

The above commands created the following group configuration file:
上述命令创建了以下组配置文件：

The nofailback options is mostly useful to avoid unwanted resource movements during administration tasks. For example, if you need to migrate a service to a node which doesn’t have the highest priority in the group, you need to tell the HA manager not to instantly move this service back by setting the nofailback option.
nofailback 选项主要用于避免在管理任务期间资源的非预期移动。例如，如果您需要将服务迁移到组中优先级不是最高的节点，您需要通过设置 nofailback 选项告诉 HA 管理器不要立即将该服务移回。

Another scenario is when a service was fenced and it got recovered to another node. The admin tries to repair the fenced node and brings it up online again to investigate the cause of failure and check if it runs stably again. Setting the nofailback flag prevents the recovered services from moving straight back to the fenced node.
另一种情况是服务被隔离（fenced）后恢复到了另一个节点。管理员尝试修复被隔离的节点并重新上线，以调查故障原因并检查其是否能稳定运行。设置 nofailback 标志可以防止恢复的服务直接移回被隔离的节点。

There are different methods to fence a node, for example, fence devices which cut off the power from the node or disable their communication completely. Those are often quite expensive and bring additional critical components into a system, because if they fail you cannot recover any service.
有多种方法可以对节点进行隔离，例如使用隔离设备切断节点的电源或完全禁用其通信。这些设备通常相当昂贵，并且会为系统带来额外的关键组件，因为如果它们发生故障，您将无法恢复任何服务。

We thus wanted to integrate a simpler fencing method, which does not require additional external hardware. This can be done using watchdog timers.
因此，我们希望集成一种更简单的隔离方法，不需要额外的外部硬件。这可以通过使用看门狗定时器来实现。

Watchdog timers have been widely used in critical and dependable systems since the beginning of microcontrollers. They are often simple, independent integrated circuits which are used to detect and recover from computer malfunctions.
看门狗定时器自微控制器诞生以来就被广泛应用于关键和可靠的系统中。它们通常是简单、独立的集成电路，用于检测和恢复计算机故障。

During normal operation, ha-manager regularly resets the watchdog timer to prevent it from elapsing. If, due to a hardware fault or program error, the computer fails to reset the watchdog, the timer will elapse and trigger a reset of the whole server (reboot).
在正常运行期间，ha-manager 会定期重置看门狗定时器以防止其超时。如果由于硬件故障或程序错误，计算机未能重置看门狗，定时器将超时并触发整个服务器的重置（重启）。

Recent server motherboards often include such hardware watchdogs, but these need to be configured. If no watchdog is available or configured, we fall back to the Linux Kernel softdog. While still reliable, it is not independent of the servers hardware, and thus has a lower reliability than a hardware watchdog.
近年来的服务器主板通常包含此类硬件看门狗，但需要进行配置。如果没有可用或配置的看门狗，我们将回退到 Linux 内核的软狗。虽然软狗仍然可靠，但它不独立于服务器硬件，因此其可靠性低于硬件看门狗。

By default, all hardware watchdog modules are blocked for security reasons. They are like a loaded gun if not correctly initialized. To enable a hardware watchdog, you need to specify the module to load in /etc/default/pve-ha-manager, for example:
默认情况下，出于安全原因，所有硬件看门狗模块都被阻止。它们如果未正确初始化，就像一把上膛的枪。要启用硬件看门狗，需要在 /etc/default/pve-ha-manager 中指定要加载的模块，例如：

This configuration is read by the watchdog-mux service, which loads the specified module at startup.
该配置由 watchdog-mux 服务读取，该服务在启动时加载指定的模块。

After a node failed and its fencing was successful, the CRM tries to move services from the failed node to nodes which are still online.
当节点发生故障且隔离成功后，CRM 会尝试将服务从故障节点迁移到仍在线的节点上。

The selection of nodes, on which those services gets recovered, is influenced by the resource group settings, the list of currently active nodes, and their respective active service count.
这些服务恢复到哪些节点，受资源组设置、当前活跃节点列表及其各自活跃服务数量的影响。

The CRM first builds a set out of the intersection between user selected nodes (from group setting) and available nodes. It then choose the subset of nodes with the highest priority, and finally select the node with the lowest active service count. This minimizes the possibility of an overloaded node.
CRM 首先构建一个集合，该集合是用户选择的节点（来自组设置）与可用节点的交集。然后选择优先级最高的节点子集，最后选择活跃服务数量最少的节点。这样可以最大限度地减少节点过载的可能性。

You can use the manual maintenance mode to mark the node as unavailable for HA operation, prompting all services managed by HA to migrate to other nodes.
您可以使用手动维护模式将节点标记为不可用于 HA 操作，促使所有由 HA 管理的服务迁移到其他节点。

The target nodes for these migrations are selected from the other currently available nodes, and determined by the HA group configuration and the configured cluster resource scheduler (CRS) mode. During each migration, the original node will be recorded in the HA managers' state, so that the service can be moved back again automatically once the maintenance mode is disabled and the node is back online.
这些迁移的目标节点是从其他当前可用的节点中选择的，并由 HA 组配置和配置的集群资源调度器（CRS）模式决定。在每次迁移过程中，原始节点将被记录在 HA 管理器的状态中，以便在维护模式被禁用且节点重新上线后，服务可以自动迁回。

Currently you can enabled or disable the maintenance mode using the ha-manager CLI tool.
目前，您可以使用 ha-manager 命令行工具启用或禁用维护模式。

This will queue a CRM command, when the manager processes this command it will record the request for maintenance-mode in the manager status. This allows you to submit the command on any node, not just on the one you want to place in, or out of the maintenance mode.
这将排队一个 CRM 命令，当管理器处理该命令时，它会在管理器状态中记录维护模式的请求。这允许您在任何节点上提交该命令，而不仅限于您想要进入或退出维护模式的节点。

Once the LRM on the respective node picks the command up it will mark itself as unavailable, but still process all migration commands. This means that the LRM self-fencing watchdog will stay active until all active services got moved, and all running workers finished.
一旦相应节点上的 LRM 接收到命令，它将标记自身为不可用，但仍会处理所有迁移命令。这意味着 LRM 自我隔离看门狗将保持激活状态，直到所有活动服务被迁移，且所有运行中的工作进程完成。

Note that the LRM status will read maintenance mode as soon as the LRM picked the requested state up, not only when all services got moved away, this user experience is planned to be improved in the future. For now, you can check for any active HA service left on the node, or watching out for a log line like: pve-ha-lrm[PID]: watchdog closed (disabled) to know when the node finished its transition into the maintenance mode.
请注意，LRM 状态一旦接收到请求的状态，就会显示为维护模式，而不仅仅是在所有服务迁移完成后才显示，这种用户体验未来计划改进。目前，您可以检查节点上是否还有任何活动的 HA 服务，或者关注类似以下日志行：pve-ha-lrm[PID]: watchdog closed (disabled)，以了解节点何时完成进入维护模式的过渡。

The process of disabling the manual maintenance mode is similar to enabling it. Using the ha-manager CLI command shown above will queue a CRM command that, once processed, marks the respective LRM node as available again.
禁用手动维护模式的过程与启用它类似。使用上面显示的 ha-manager CLI 命令将排队一个 CRM 命令，该命令处理后会将相应的 LRM 节点标记为可用。

If you deactivate the maintenance mode, all services that were on the node when the maintenance mode was activated will be moved back.
如果您停用维护模式，所有在激活维护模式时位于该节点上的服务将被迁移回去。

Below you will find a description of the different HA policies for a node shutdown. Currently Conditional is the default due to backward compatibility. Some users may find that Migrate behaves more as expected.
下面是关于节点关闭的不同 HA 策略的描述。目前由于向后兼容性，默认策略为 Conditional。一些用户可能会发现 Migrate 的行为更符合预期。

The shutdown policy can be configured in the Web UI (Datacenter → Options → HA Settings), or directly in datacenter.cfg:
关机策略可以在网页界面中配置（数据中心 → 选项 → HA 设置），也可以直接在 datacenter.cfg 中配置：

The number of active HA services on each node is used to choose a recovery node. Non-HA-managed services are currently not counted.
每个节点上活动的高可用服务数量用于选择恢复节点。目前不计算非高可用管理的服务。

Static usage information from HA services on each node is used to choose a recovery node. Usage of non-HA-managed services is currently not considered.
使用来自每个节点上高可用（HA）服务的静态使用信息来选择恢复节点。目前不考虑非高可用管理服务的使用情况。

For this selection, each node in turn is considered as if the service was already running on it, using CPU and memory usage from the associated guest configuration. Then for each such alternative, CPU and memory usage of all nodes are considered, with memory being weighted much more, because it’s a truly limited resource. For both, CPU and memory, highest usage among nodes (weighted more, as ideally no node should be overcommitted) and average usage of all nodes (to still be able to distinguish in case there already is a more highly committed node) are considered.
在此选择过程中，依次将每个节点视为服务已在其上运行，使用关联的虚拟机配置中的 CPU 和内存使用情况。然后，对于每个此类备选方案，考虑所有节点的 CPU 和内存使用情况，其中内存权重更大，因为内存是真正有限的资源。对于 CPU 和内存，既考虑节点中最高的使用率（权重更大，因为理想情况下不应有节点过度分配），也考虑所有节点的平均使用率（以便在已有更高负载节点的情况下仍能区分）。

The CRS algorithm is not applied for every service in every round, since this would mean a large number of constant migrations. Depending on the workload, this could put more strain on the cluster than could be avoided by constant balancing. That’s why the Proxmox VE HA manager favors keeping services on their current node.
CRS 算法并不会在每一轮对每个服务都应用，因为这将导致大量的持续迁移。根据工作负载，这可能会给集群带来比持续平衡所能避免的更多的压力。这就是为什么 Proxmox VE HA 管理器更倾向于让服务保持在其当前节点上。

The CRS is currently used at the following scheduling points:
CRS 目前在以下调度点使用：