Assuming a static cluster, we now have a rack secret that can be computed from K key shares and used to derive child keys. In the following example, K = 2, just so we can keep the diagrams small.

The rack secret protects every key in the system that is used after rack unlock. There are two mechanisms to get the next keys in the hierachy:

Key Derivation we already use to derive any primary child keys from the rack secret. But it can also be used to derive keys from keys. Key wrapping is just taking a key and encrypting it. Each has their benefits and drawbacks.

Key derivation is nice because you never have to store the derived keys on disk. You can just regenerate them. The problem with it is that if the derived key changes any downstream derived keys will now change also change.

Key wrapping is nice because if the wrapper key is rotated, the downstream keys do not have to change. This is particularly useful for encrypting large amounts of data on storage. You don’t want to be forced to decrypt and then re-encrypt just because a parent key was rotated. The downside of key wrapping is that you now have to store the wrapped (encrypted) key on disk somewhere. If the same key is used in multiple places you have to replicate it.

For trust quorum, we have the particular problem that when nodes are added or removed we generate new shares. While it’s possible to maintain the same rack secret and key derivation with new shares, it is less secure over time because any single malicious node who retrieved the rack secret at one time and saved it can now recover any data on any existing drives or any data produced in the future. While this is always an issue, we prefer to allow the rotation of the rack secret in the case of a known compromise. We therefore always generate a new rack secret on every change of the trust quorum (reconfiguration), or other key-share rotation. This way, even if all existing data on the rack is compromised, at least no new new data will be compromised if the compromised sled is removed.

Right now what do we know about our security goals? We want to:

And what are our constraints given these goals? We must:

The first constraint is a given from our use of disk encryption in general. The latter two constraints come from the distributed nature of the reconfiguration problem and are elaborated upon in [RFD 238]. The second constraint makes it impossible to distribute the new key share during the committment, because after a sled commits to a new epoch it should only utilize the new rack secret, and thus it may have to request the new key shares from a sled that has not yet learned that it has committed. There are other security reasons for not distributing key shares during commit that are further fleshed out in [RFD 238]. Therefore, given a 2-phase commit protocol ([RFD 238]), we must distributed the new shares in the prepare message. The third constraint makes it impossible to change the ZFS encryption wrapper keys immediately when learned, since the new rack secret from which those keys are derived may never be committed. Without a committed rack secret, it will be impossible to retrieve the shares necessary to recompute the secret and rederive the ZFS encryption wrapper keys.

Given these goals and constraints, we re-iterate that any sleds that are members of both the old and the new group must have access to the old and new committed rack secrets at the same time. This is necessary to allow them to derive the ZFS encryption wrapper keys for each U.2 device so that they can change the keys. For simplicity of the reconfiguration protocol, and to limit the exposure of the old rack secret we also only want to allow sleds to distribute key shares for the currently committed configuration. The requirement and desires above are in tension, and so we must get creative about how we handle this situation.

The most straightforward way to solve this predicament is via the dealer during a reconfiguration. As described in [RFD 238] we number each configuration with a monotonically increasing epoch. At epoch 1, each sled gets a single share and recomputes the rack secret to derive the original ZFS encryption wrapper keys for the U.2 devices. When a reconfiguration occurs, the dealer retrieves enough shares to recompute the rack secret for the current committed epoch (1 in our example). The dealer generates a new rack secret for epoch 2, and splits it into key shares. The dealer also derives an old-rack-secret encryption key associated with epoch 1 from the epoch 2 rack secret and encrypts the epoch 1 rack secret with the old-rack-secret key. It sends this encrypted secret to sleds that are members of the new group along with the rest of the trust quorum prepare message. Until the new configuration at epoch 2 is committed, the encrypted epoch-1 rack secret cannot be decrypted, because no members will send shares for epoch-2 required to recompute the epoch-2 rack secret and the derived key necessary to decrypt the epoch 1 rack secret. If epoch 2 is never committed this will remain the case.

As soon as a sled sees that a configuration has been committed for a new epoch, it retrieves enough shares to unlock the new epoch rack secret, derives the old-rack-secret encryption key, decrypts the old rack secret protected by this key, derives the old and new U.2 encryption keys from the old and new rack secret respectively and re-configures the ZFS encryption for each U.2 drive. Once the encryption keys have been changed for all the U.2 drives, the encrypted rack secret for the old epoch is securely deleted, along with any other encrypted rack secrets for the old epoch that were prepared but never committed. There is some nuance here around failure modes, but that is not relevant to this RFD and is further fleshed out in [RFD 238].

Our key hierarchy has now been roughly described in prose, and it tolerates failures of sleds during reconfiguration, and allows the changing of encryption wrapper keys used for ZFS encryption on U.2 drives. It also deals with compromise of the rack secret and limits exposure of individual derived key compromise. With this in mind, we can now draw a diagram of the key-hierarchy.