This document describes an enhancement of Ganeti’s security by restricting the distribution of security-sensitive data to the master and master candidates only.
Note: In this document, we will use the term ‘normal node’ for a node that is neither master nor master-candidate.
Up till 2.10, Ganeti distributes security-relevant keys to all nodes, including nodes that are neither master nor master-candidates. Those keys are the private and public SSH keys for node communication and the SSL certficate and private key for RPC communication. Objective of this design is to limit the set of nodes that can establish ssh and RPC connections to the master and master candidates.
As pointed out in issue 377, this is a security risk. Since all nodes have these keys, compromising any of those nodes would possibly give an attacker access to all other machines in the cluster. Reducing the set of nodes that are able to make ssh and RPC connections to the master and master candidates would significantly reduce the risk simply because fewer machines would be a valuable target for attackers.
Note: For bigger installations of Ganeti, it is advisable to run master candidate nodes as non-vm-capable nodes. This would reduce the attack surface for the hypervisor exploitation.
Currently (as of 2.10), all nodes hold the following information:
Concerning ssh, this setup contains the following security issue. Since all nodes of a cluster can ssh as root into any other cluster node, one compromised node can harm all other nodes of a cluster.
Regarding the SSL encryption of the RPC communication with the node daemon, we currently have the following setup. There is only one certificate which is used as both, client and server certificate. Besides the SSL client verification, we check if the used client certificate is the same as the certificate stored on the server.
This means that any node running a node daemon can also act as an RPC client and use it to issue RPC calls to other cluster nodes. This in turn means that any compromised node could be used to make RPC calls to any node (including itself) to gain full control over VMs. This could be used by an attacker to for example bring down the VMs or exploit bugs in the virtualization stacks to gain access to the host machines as well.
We propose the following design regarding the SSH host key handling. The root keys are untouched by this design.
Each node gets its own ssh private/public key pair, but only the public keys of the master candidates get added to the authorized_keys file of all nodes. This has the advantages, that:
To ensure security while transferring public key information and updating the authorized_keys, there are several other changes necessary:
The following sections describe in detail which Ganeti commands are affected by the proposed changes.
The design goal to limit SSH powers to master candidates conflicts with the current powers a user of the RAPI interface would have. The master_capable flag of nodes can be modified via RAPI. That means, an attacker that has access to the RAPI interface, can make all non-master-capable nodes master-capable, and then increase the master candidate pool size till all machines are master candidates (or at least a particular machine that he is aming for). This means that with RAPI access and a compromised normal node, one can make this node a master candidate and then still have the power to compromise the whole cluster.
To mitigate this issue, we propose the following changes:
With this setup, there are the following definitions of “potential master candidates” depending on the rapi flag:
Note that when the rapi flag is changed, the state of the ganeti_pub_keys file on all nodes has to be updated accordingly. This should be done in the client script gnt_cluster before the RPC call to update the configuration is made, because this way, if someone would try to perform that RPC call on master to trick it into thinking that the flag is enabled, this would not help as the content of the ganeti_pub_keys file is a crucial part in the design of the distribution of the SSH keys.
Note: One could think of always allowing to disable the master-capability via RAPI and just restrict the enabling of it, thus making it possible to RAPI-“freeze” the nodes’ master-capability state once it disabled. However, we think these are rather confusing semantics of the involved flags and thus we go with proposed design.
Note that this change will break RAPI compatibility, at least if the rapi flag is not explicitely set to True. We made this choice to have the more secure option as default, because otherwise it is unlikely to be widely used.
On cluster initialization, the following steps are taken in bootstrap.py:
According to Design for adding a node to a cluster, Ganeti transfers the ssh keys to every node that gets added to the cluster.
Adding a new node will require the following steps.
In LUNodeAdd in cmdlib/node.py:
RPC call execution in noded (on master node):
In case of readding a node that used to be in the cluster before, handling of the SSH keys would basically be the same, in particular also a new SSH key pair is generated for the node, because we cannot be sure that the old key pair has not been compromised while the node was offlined. Note that for reasons of data hygiene, a node’s ganeti_pub_keys file is cleared before the node is readded. Also, Ganeti attempts to remove any Ganeti keys from the authorized_keys file before the node is readded. However, since Ganeti does not keep a list of all keys ever used in the cluster, this applies only to keys which are currently used in the cluster. Note that Ganeti won’t touch any keys that were added to the authorized_keys by other systems than Ganeti.
If the role of a node is changed from ‘normal’ to ‘master_candidate’, the procedure is the same as for adding nodes from the step “In LUNodeAdd ...” on.
If a node gets demoted to ‘normal’, the master daemon makes a similar RPC call to the master node’s node daemon as for adding a node.
In the RPC call, noded will perform the following steps:
This affected the behavior of the following commands:
If the node has been master candidate already before the command to promote it was issued, Ganeti does not do anything.
Note that when you demote a node from master candidate to normal node, another master-capable and normal node will be promoted to master candidate. For this newly promoted node, the same changes apply as if it was explicitely promoted.
The same behavior should be ensured for the corresponding rapi command.
When offlining a node, it immediately loses its role as master or master candidate as well. When it is onlined again, it will become master candidate again if it was so before. The handling of the keys should be done in the same way as when the node is explicitely promoted or demoted to or from master candidate. See the previous section for details.
This affects the command:
For offlining, the removal of the keys is particularly important, as the detection of a compromised node might be the very reason for the offlining. Of course we cannot guarantee that removal of the key is always successful, because the node might not be reachable anymore. Even though it is a best-effort operation, it is still an improvement over the status quo, because currently Ganeti does not even try to remove any keys.
The same behavior should be ensured for the corresponding rapi command.
So far, gnt-cluster verify checks the SSH connectivity of all nodes to all other nodes. We propose to replace this by the following checks:
When upgrading from a version that has the previous SSH setup to the one proposed in this design, the upgrade procedure has to involve the following steps in the post-upgrade hook:
Since this upgrade significantly changes the configuration of the clusters’ nodes, we will add a note to the UPGRADE notes to make the administrator aware of this fact (in case he intends to enable access from normal nodes to master candidates for other reasons than Ganeti uses the machines).
Also, in any operation where Ganeti creates new SSH keys, the old keys will be backed up and not simply overridden.
These downgrading steps will be implemtented from 2.13 to 2.12:
The gnt-cluster renew-crypto command will be extended by a new option --new-ssh-keys, which will renew all SSH keys on all nodes and rebuild the authorized_keys files and the ganeti_pub_keys files according to the previous sections. This operation will be performed considering the already stated security considerations, for example minimizing RPC calls, distribution of keys via SSH only etc.
With this design, Ganeti will do more manipulations of SSH keys and authorized_keys files than before. If this is not feasible in a Ganeti environment, the administrator has the option to prevent Ganeti from performing any manipulations on the SSH setup of the nodes. The options for doing so, are --no-ssh-init for gnt-cluster init, and --no-node-setup for gnt-node add. Note that these options already existed before the implementation of this design, we just confirm that they will be complied to with the new design as well.
Regarding the node daemon certificates, we propose the following changes in the design.
This design has the following advantages:
Drawbacks of this design:
In the following sections, we describe how our design affects various Ganeti operations.
On cluster initialization, so far only the node daemon certificate was created. With our design, two certificates (and corresponding keys) need to be created, a server certificate to be distributed to all nodes and a client certificate only to be used by this particular node. In the following, we use the term node daemon certificate for the server certficate only.
In the cluster configuration, the candidate map is created. It is populated with the respective entry for the master node. It is also written to ssconf.
When a node is added, the server certificate is copied to the node (as before). Additionally, a new client certificate (and the corresponding private key) is created on the new node to be used only by the new node as client certifcate.
If the new node is a master candidate, the candidate map is extended by the new node’s data. As before, the updated configuration is distributed to all nodes (as complete configuration on the master candidates and ssconf on all nodes). Note that distribution of the configuration after adding a node is already implemented, since all nodes hold the list of nodes in the cluster in ssconf anyway.
If the configuration for whatever reason already holds an entry for this node, it will be overriden.
When readding a node, the procedure is the same as for adding a node.
When a normal node gets promoted to be master candidate, an entry to the candidate map has to be added and the updated configuration has to be distributed to all nodes. If there was already an entry for the node, we override it.
On demotion of a master candidate, the node’s entry in the candidate map gets removed and the updated configuration gets redistibuted.
The same procedure applied to onlining and offlining master candidates.
Cluster verify will be extended by the following checks:
Currently, when the cluster’s cryptographic tokens are renewed using the gnt-cluster renew-crypto command the node daemon certificate is renewed (among others). Option --new-cluster-certificate renews the node daemon certificate only.
By adding an option --new-node-certificates we offer to renew the client certificate. Whenever the client certificates are renewed, the candidate map has to be updated and redistributed.
If for whatever reason, the candidate map becomes inconsistent, for example due inconsistent updating after a demotion or offlining), the user can use this option to renew the client certificates and update the candidate certificate map.
Note that renewing the server certificate requires all client certificates being renewed and signed by the new server certificate, because otherwise their signature can not be verified by the server who only has the new server certificate then.
As there was a different design in place in Ganeti 2.12.4 and previous versions, we have to ensure that renew-crypto works on pre 2.12 versions and 2.12.1-4. Users that got hit by Issue 1094 will be encouraged to run renew-crypto at least once after switching to 2.12.5. Those who did not encounter this bug yet, will still get nagged friendly by gnt-cluster verify.
The watcher is a script that is run on all nodes in regular intervals. The changes proposed in this design will not affect the watcher’s implementation, because it behaves differently on the master than on non-master nodes.
Only on the master, it issues query calls which would require a client certificate of a node in the candidate mapping. This is the case for the master node. On non-master node, it’s only external communication is done via the ConfD protocol, which uses the hmac key, which is present on all nodes. Besides that, the watcher does not make any ssh connections, and thus is not affected by the changes in ssh key handling either.
Ganeti handles a couple of other keys/certificates that have not been mentioned in this design so far. Also, other daemons than the ones mentioned so far perform intra-cluster communication. Neither the keys nor the daemons will be affected by this design for several reasons: