Best practices for key derivation

Marc Ilunga
cryptography

Key derivation is essential in many cryptographic applications, including key exchange, key management, secure communications, and building robust cryptographic primitives. But it’s also easy to get wrong: although standard tools exist for different key derivation needs, our audits often uncover improper uses of these tools that could compromise key security. Flickr’s API signature forgery vulnerability is a famous example of misusing a hash function during key derivation.

These misuses indicate potential misunderstandings about key derivation functions (KDFs). This post covers best practices for using KDFs, including specialized scenarios that require careful treatment of key derivation to achieve the desired security properties. Along the way, we offer advice for answering common questions, like:

  • Do I need to add extra randomness to HKDF?
  • Should I use salt with HKDF?
  • Should I use different salts to derive multiple keys from HKDF?
  • How do I combine multiple sources of keying material?

Before diving into key derivation best practices, we’ll recap some important concepts to help us better understand them.

Sources of keying material

Keyed cryptographic primitives, such as AEADs, require keying material that satisfies certain requirements to guarantee security. In most cases, primitives require that the key is generated uniformly at random or cryptographically close to uniform random. We will distinguish four types of keying material:

  • (Uniform) random, such as 32 bytes generated with the OS CSPRNG
  • Non-uniform but high entropy, such as the output of key exchange
  • Low-entropy, such as passwords and other easily guessable values
  • Sets of several sources, such as pre- and post-quantum shared secrets

Figure 1: A diverse collection of keys (generated with AI)

The last category above is particularly relevant to the current development of quantum-resistant cryptography. Hybrid key exchange protocols combining classical and post-quantum key exchanges are designed to protect against Store Now Decrypt Later attacks.

How key derivation works

Key derivation is the process of generating acceptable keying material for cryptographic usage from some initial keying material (IKM). From a cryptographic perspective, “acceptable” usually means chosen uniformly at random from the set of all possible keys or indistinguishable from a truly random key. There are two main key derivation tasks related to the nature of the initial keying material.

  • Randomness extraction extracts a cryptographic key from an IKM with “enough randomness.” Randomness extraction optionally uses a salt. Naturally, we can apply randomness extraction to a key that is already cryptographically appropriate.
  • Randomness expansion derives subkeys from a cryptographic key. Expansion generally uses a “context” or “info” input unique to each subkey.

This categorization is heavily influenced by the widely used KDF algorithm HKDF; other KDF designs do not necessarily follow the same principles. However, extraction and expansion are well reflected in most KDF applications. Additionally, we will consider an additional KDF task related to complex sources of keying material, such as a set of sources.

Extraction and expansion: a brief look into HKDF

Tip: if you prefer a visual demonstration of HKDF, refer to the animations below.

HKDF was designed to provide both extraction and expansion. HKDF is commonly accessible to applications with an API, such as HKDF(ikm, salt, info, key_len). However, under the hood, the following happens: first, an extraction process generates a pseudo-random key (PRK) from the IKM and salt prk = HKDF.Extract(ikm, salt) = HMAC(salt, ikm). Then, a subkey of length key_len is generated: sub_key = feedback[HMAC](prk, info). Here, feedback[HMAC] is a wrapper around HMAC that generates output as long as desired by repeatedly calling HMAC; in other words, it implements a variable-length pseudorandom function. For a given key, feedback will return a random bit string of the required length for every new info input; a fixed info value will always produce the same output. If info is kept constant but the length is variable, the smaller output will be a prefix of the longer output.

Figure 2: Visualizing the extraction and expansion phases of a KDF

Regarding the extraction salt: the extraction stage of HKDF optionally takes a salt. The extraction salt is a random, non-secret value used to extract sufficient randomness from the keying material. Crucially, the salt cannot be attacker-controlled, since that could lead to catastrophic outcomes for KDFs in general. Hugo Krawczyk provides a theoretical example of attacker-controlled salts breaking the independence between the salt and the IKM, leading to weak extractor construction. However, the consequences can also have practical relevance, as we discuss in the next section. A typical pain point for many applications (except, e.g., authenticated key exchange) is authenticating salts. Therefore, the HKDF standard recommends that most applications use a constant, such as an all-zero-byte string. The price to be paid for not using a salt is making somewhat stronger, albeit still reasonable, assumptions on HMAC.

Addressing KDF misuses

Developers must consider several questions when choosing a KDF, but a misunderstanding of KDFs may lead to choices that introduce security issues. Below, we provide examples of misuse along with best practices to help avoid improper use of KDF.

Should I use different salts to derive multiple subkeys?

With the aforementioned KDF abstraction, subkey generation is better suited to randomness expansion. Given a pseudo-random key (perhaps obtained after an extraction step), subkeys can be obtained with randomness expansion using unique info inputs for each subkey. The salt is used for extraction. Furthermore, as discussed above, attacker-controlled salts can be detrimental to security. Consider a key management application that generates user keys on demand. One implementation might decide to derive a key from a master key using the username as salt. Besides freely choosing their usernames, users may provide a context string (e.g., “file-encryption-key”) that indicates the purpose of the key and ensure that different applications use independent keys. The core functionality is shown in the code snippet below:

# For each subkey
def generate_user_key(username, purpose, key_len):
    ikm = fetch_master_key_from_kms()
sub_key = hkdf(ikm=ikm, salt=username, info=purpose, key_len=key_len)

Figure 3: Key management application using a master key to derive keys on demand

This construction is bad: since the salt is used as an HMAC “key” for extraction, it is first preprocessed by a PAD-or-HASH scheme (key padding, key hashing) to handle variable-length keys. In this implementation, if your username is b”A”*65, and I choose my username to be sha256(b”A”*65), then I will get all your keys!

So what should we do instead? The first thing to avoid is potentially attacker-controlled salts. In the example above, the application could generate a random salt on initialization and retrieve it from a trusted place as needed. Alternatively, the application may also use a constant salt like an all-zero byte string, as RFC 5869 recommends. Notably, for HMAC, if ikm was already a uniform random key, using a constant does not require stronger assumptions. Finally, the issue can also be avoided if the IKM is initially a random key and usernames are restricted to a set of values described in our discussion of dual PRFs.

What should I use as an info value?

The application must ensure that unique info values are used for each new subkey. It is also a good practice to include as much context information as possible in the info value, such as session identifiers or transcript hashes. The encoding of the context into info must be injective, for instance, by paying attention to canonicalization issues.

Do I need extra randomness in the info parameter HKDF?

We often encounter implementations that include extra randomness in the info parameter to generate subkeys. The hope is to make HKDF somewhat more random.

# For each subkey
extra_randomness = random(32)
sub_key = hkdf(ikm=ikm, salt=salt, info=concat(info, extra_randomness), key_len=key_len)

Figure 4: Using additional randomness to derive subkeys

Although this does not hurt, it also does not help much with the initial task of randomness extraction. Note that the extra randomness affects only randomness expansion. Consider the following thought experiment: if the IKM doesn’t have enough entropy or HMAC turns out to be a very bad randomness extractor, the extra randomness will not help create a suitable key to be used during randomness expansion. A far-from-random key for randomness expansion deviates from the security requirements and, therefore, offers no security guarantees. From the above discussion, assuming that HKDF is secure, if ikm has enough randomness, we will extract a random key for it. Then, the expansion will ensure that the sub_key is indistinguishable from a random key of the same length. Furthermore, HKDF does not require the info material to be secret; it only needs to be unique for each subkey.

However, an application may use extra randomness to further guarantee the uniqueness of the info inputs. Unless you do something funny with that extra randomness, you won’t be worse off using it.

Should I use HKDF on a low-entropy input?

No. HKDF consists of only a couple of HMAC calls. Password crackers can fairly efficiently crack massive amounts of passwords for KDFs that aren’t purposefully designed to be slow and memory-intensive. It is best to use slow, memory-hard algorithms like Argon2 for hashing and deriving keys from passwords. Furthermore, it is best to avoid using password hashes as keys to encrypt data. Prefer creating key hierarchies (such as key encryption keys), using the password hash to encrypt a randomly generated key from which further keys can be derived as needed.

Should I use a hash function as a general-purpose KDF?

A hash function should not be used for general purposes in KDF. In scenarios where the information used during key derivation is attacker-controlled, using a hash function as KDF can expose the application to length-extension attacks. These attacks are a major concern for applications that generate randomness from a secret combined with user-provided data (like in Flickr’s API signature forgery vulnerability). Instead, prefer HKDF and other KDFs that were designed specifically for key derivation. Although hashing is acceptable as a KDF in specific cases, we caution against this practice unless the user can reasonably argue formally about the usage of their application. If your application genuinely suffers from one or two extra compression function calls, consult an expert if you do not have a strong justification for using existing KDFs. This advice is also valid for other ad-hoc constructions, such as the YOLO constructions.

Should I use a shared Diffie-Hellman secret key to an AEAD?

The security contract for an AEAD (and most other keyed symmetric algorithms) requires a uniform random bitstring of the appropriate length to provide meaningful security guarantees. DH outputs are high entropy but generally not uniform bitstrings. Therefore, using them as keys deviates from the security contract. Some implementations may allow unsuspecting users to use the wrong key material for a given primitive (e.g., feed a DH output into a Chacha20 cipher). Such usage violates the requirements of the AEAD construction.

Combining keys

A common task in cryptography is combining two instantiations of a primitive so the overall construction is as strong as the strongest. Naturally, this is a highly relevant question for key derivation: can we derive a secret from a set of key materials so the overall secret is secure as long as one of the key materials is secure? Hybrid key exchange protocols are currently relevant use cases of this technique. These protocols combine keys established via both a classical and a post-quantum key exchange primitive to protect against attackers who are harvesting encrypted communications today and hope to decrypt them once a capable quantum computer is available. Such protocols include PQXDH, Apple’s PQ3, and post-quantum Noise. However, key combining is widely used in other contexts unrelated to quantum threats, such as TLS1.3 with preshared keys, the double ratchet algorithm, and MLS.

So, how do we combine secrets? For simplicity, we restrict the following discussion to two secrets, k_1 and k_2. The classical tool for the job is a dual PRF. Like a PRF, a dual PRF takes a key and an input and behaves like a PRF so long as one of the keys or the input contains a uniform secret key. In a dual PRF, you can switch the key and input values without affecting security. In practice, the most common instantiation of a dual PRF is HMAC.

However, using HMAC as a dual PRF requires some caution. The standardized HMAC allows keys of variable lengths, which are processed via a PAD-or-HASH function. PAD-or-HASH is not collision-resistant, and creating HMAC output collisions for unrestricted HMAC keys is trivial. Fortunately, this paper establishes the dual PRF security of HMAC and fully characterizes the set keys for which dual PRF security is expected. In short, a safe dual PRF usage of HMAC requires that the key argument (i.e., what is passed as key to HMAC) is a fixed length bitstring (i.e., all keys must have the same length) or a variable length bitstring as long as all keys have length at least the block length of the underlying hash function.

The dual PRF results apply only when combining two uniform random bitstrings. Although several works argue for using HMAC as a dual PRF with other high-entropy inputs like a Diffie-Hellman shared secret (G^xy), a more conservative usage would apply an initial extraction step to every keying material that requires it. An example of this is prk = HMAC( HKDF.Extract(G^xy, salt), random_kem_secret). Although some analyses do away with the initial extraction step, these uses deviate from the existing security analysis of HMAC and do not directly enjoy the security guarantees.

Another good practice for dual PRF usage is to ensure that the final combined secret depends on as much of the context as possible. The context here can be the Diffie-Hellman shares, the (hash of the) full communication transcript. A good solution is to use an additional expansion step that uses the context as the info input during expansion.

Finally, other combination approaches exist, such as concatenation KDF (CatKDF). CatKDF roughly uses a KDF on the concatenation of secrets. In scenarios where one of the secrets is possibly attacker-controlled, the security of CatKDF falls outside of the existing security analysis. The remarks above do not imply practical attacks but raise awareness around cases where stronger assumptions beyond what is known are sometimes needed. For further discussion on dual PRF usage in practice, see Practical (Post-Quantum) Key Combiners from One-Wayness and Applications to TLS.

Choose the right tool

This blog post examined different KDF tasks, appropriate tools to perform them, and some typical misuses we see in practice. To conclude, we invite you to do the same as you tackle your next KDF task. The invitation is the following: as you face your next KDF tasks, take a step back and consider the higher-level goals and whether a higher-level tool would be better suited.

For example, do you need a KDF because you have established some Diffie-Hellman shared secret and must create a “secure channel”? Consider using an existing battle-tested authenticated key exchange protocol like Noise, TLS 1.3, or EDHOC.

Do you need a KDF to encrypt various chunks of a data stream while expecting some security guarantees for chunks and the overall stream? Consider using a streaming AEAD instead!

Naturally, there comes a time when a novel solution is needed; in that case, ensure that you have a reasonable justification for your proposed solution, then talk to your favorite cryptographer (or come to us)!