Don’t overextend your Oblivious Transfer

By Joop van de Pol

We found a vulnerability in a threshold signature scheme that allows an attacker to recover the signing key of threshold ECDSA implementations that are based on Oblivious Transfer (OT). A malicious participant of the threshold signing protocols could perform selective abort attacks during the OT extension subprotocol, recover the secret values of other parties, and eventually recover the signing key. Using this key, the attacker could assume the identities of users, gain control over critical systems, or pilfer financial assets. While we cannot yet disclose the client software affected by this vulnerability, we believe it is instructive for other developers implementing MPC protocols.

Protecting against this vulnerability is straightforward: since the attack relies on causing selective aborts during several protocol rounds, punishing or excluding participants that cause selective aborts is sufficient. Still, it’s a good example of a common problem that often leads to severe or even critical issues: a disconnect between assumptions made by academics and the implementers trying to build these protocols efficiently in real systems.

Threshold signature schemes

Threshold signature schemes (TSS) are powerful cryptographic objects that allow decentralized control of the signing key for a digital signature scheme. They are a specific application of the more general multi-party computation (MPC), which aims to decentralize arbitrary computation. Each TSS protocol is typically defined for a specific digital signature scheme because different signature schemes require different computations to create signatures.

Current research aims to define efficient TSS protocols for various digital signature schemes. The target for efficiency includes both computation and communication between the different participants. Typically, TSS protocols rely on standard techniques used in MPC, such as secret sharing, zero-knowledge proofs, and multiplicative-to-additive conversion

Threshold ECDSA

The ECDSA signature scheme is widely used. However, threshold schemes for ECDSA are generally more complicated than those for other signature schemes. This is because an ECDSA signature requires the computation of the modular inverse of a secret value.

Various MPC techniques can be used to distribute the computation of this modular inverse. Currently, one line of work in threshold signature schemes for ECDSA uses the homomorphic Paillier encryption scheme for this purpose, as shown in work by Lindell et al., Gennaro et al., and following works. This blog post will focus on schemes that rely on oblivious transfer (OT), such as the work by Doerner et al. and following works, or Cait-Sith.

Before explaining what OT is, it should be noted that the basic variant is relatively inefficient. To mitigate this issue, researchers proposed something called OT extension, where a small number of OTs can efficiently be turned into a larger number of OTs. This feature is eagerly used by the creators of threshold signature schemes, as you can run the setup of the small number of OTs once, and then extend arbitrarily many times.

Oblivious transfer

Oblivious transfer is like the setup of a magician’s card trick. A magician has a bunch of cards and wants you to choose one of them, so they can show off their magic prowess by divining which card you chose. To this end, it is important that the magician does not know which card you chose, but also that you choose exactly one card and cannot claim later that you actually chose another card.

In real life, the magician could let you write something on the card that you chose, forcing you to choose exactly one card. However, this would not be good enough for the cryptographic setting, because the magician could afterwards just look at all the cards (using their impressive sleight of hand to hide this fact) and pick out the card that has text on it. A better solution would be to have the magician write a random word on each card, such that you can choose a card by memorizing this word. Now, in real life the magician might allow you to look at multiple cards before choosing one, whereas in the cryptographic case you have to choose a card blindly such that you only learn the random word written on the card that you chose.

After choosing the card and giving it back to the magician (randomly shuffling the cards to ensure they cannot directly pick out the card that you returned), they can now try to figure out which card you chose. In real life, the magician will use all kinds of tricks to try and pick out your card, whereas in the cryptographic setting, they actually should not be able to.

So, in a nutshell, OT is about a sender (the magician) who wants to give a receiver (you, the mark) a choice between some values (cards). The sender should not learn the receiver’s choice, and the receiver should not learn any other values than the chosen one.

It turns out that OT is a very powerful MPC primitive, as it can be used as a building block to construct protocols for any arbitrary multi-party computation. However, implementing OT without any special assumptions requires asymmetric cryptography, which is relatively expensive. Using expensive building blocks will lead to an inefficient protocol, so something more is needed to be able to use OT in practice.

OT extension

OT requires either asymmetric cryptography or “special assumptions.” This means that OT is possible when the two parties already have access to something called correlated randomness. Even better, this correlated randomness can be created from the output of an OT protocol.

As a result, it is possible to run the expensive OT protocol a number of times, and then to extend these “base” OTs into many more OTs. This extension is possible using only symmetric cryptography (such as hash functions and pseudo-random generators), which makes it more efficient than the expensive asymmetric variants.

For this blog post, we will focus on a particular line of work in OT extension, starting with this paper by Ishai et al. It is a bit too complicated to explain in detail how this scheme works, but the following points are important:

  • It uses only symmetric primitives (pseudo-random generator and hash function).
  • The role of sender and receiver is swapped (sender in base OT becomes receiver in extended OT and vice versa).
  • The protocol includes constructing randomness that is correlated with both the old choices (of the base OTs) and the new choices (of the OT extension).
  • The extended OT sender cannot cheat, but the protocol is not secure against a cheating extended OT receiver.
  • What does this last point mean? The extended OT receiver can cheat and learn the original choice bits belonging to the extended OT sender. Ishai et al. proposed a solution, but it is not very efficient. Therefore, followup works such as by Asharov et al. and by Keller et al. add a kind of consistency check, where the extended OT receiver has to provide some additional information. The extended OT sender can then use this information to verify that the receiver did not cheat.

    These consistency checks restrict how much the receiver can learn about the sender’s secret choices, but they are not perfect. The check that the sender performs to verify the receiver’s information depends on their own secret choices. Therefore, the receiver can still cheat in specific places such that they learn some bits of the sender’s secret choices based on whether or not the sender aborts. This is known as a selective abort attack, as the receiver can selectively try to cause an abort and learns some information from the sender as a result.

    The aforementioned papers acknowledge that this kind of leakage can happen for a cheating receiver. However, the authors choose the parameters of the scheme such that the receiver can never learn enough about the sender’s original choice bits when running the protocol once. Problem solved, right?

    How the vulnerability works

    Recall that in the context of threshold signature schemes based on OT, you want to perform the base OTs once during a set-up phase and reuse this set-up arbitrarily many times to perform OT extension. Since this improves efficiency, implementers will jump on it. What is not mentioned very explicitly, and what caused the vulnerability that we found, is that you can reuse the set-up arbitrarily many times only if the OT extension receiver does not cheat.

    If the receiver cheats, then they can learn a few bits of the secret set-up value of the OT extension sender. This does become a problem if you allow the receiver to do this multiple times over different executions of the protocol. Eventually, the receiver learns all secret sender bits, and the security is completely compromised. Typically, depending on the specific TSS, the receiver can now use the secret sender bits to recover sender shares corresponding to the ECDSA nonce. In a scheme with a threshold of two, this means that the receiver recovers the nonce, and they can recover the ECDSA signing key given a valid signature with this nonce. (In schemes with more parties, the attacker may have to repeat this attack for multiple parties.)

    So what’s the issue here exactly? Selective abort attacks are known and explicitly discussed in OT extension papers, but those papers are not very clear on whether you can reuse the base OTs. Implementers and TSS protocol designers want to reuse the base OTs arbitrarily many times, because that’s efficient. TSS protocol designers know that selective abort attacks are an issue, so they even specify checks and consider the case closed, but they are not very clear on what implementers are supposed to do when checks fail. This kind of vagueness in academic papers invariably leads to attacks on real-world systems.

    In this case, a clear solution would be to throw away the setup for a participant that has attempted to cheat during the OT extension protocol. Looking at some public repositories out there, most OT extension libraries will report something along the lines of “correlation check failed,” which does not tell a user what to do next. In fact, only one library added a warning that a particular check’s failure may represent an attack and that you should not re-run the protocol.

    Bridging the gap between academia and implementations

    Most academic MPC papers provide a general overview of the scheme and corresponding proof of security; however, they don’t have the detail required to constitute a clear specification and aren’t intended to be blueprints for implementations. Making wrong assumptions when interpreting academic papers to create real-world applications can lead to severe issues. We hope that the recent call from NIST for Multi-Party Threshold Cryptography will set a standard for specifications of MPC and TSS and prevent such issues in the future.

    In the meantime, if you’re planning to implement threshold ECDSA, another TSS, or MPC in general, you can contact us to specify, implement, or review those implementations.

    Security flaws in an SSO plugin for Caddy

    By Maciej Domanski, Travis Peters, and David Pokora

    We identified 10 security vulnerabilities within the caddy-security plugin for the Caddy web server that could enable a variety of high-severity attacks in web applications, including client-side code execution, OAuth replay attacks, and unauthorized access to resources.

    During our evaluation, Caddy was deployed as a reverse proxy to provide access to several of our internal services. We explored a plugin configuration that would allow us to handle authentication and authorization with our Google SSO so that we didn’t have to implement this on a per-app basis.

    In this blog post, we will briefly explore the security vulnerabilities we identified in the caddy-security plugin and discuss their potential impact. As with our typical security assessments, for each issue we identified, we present a recommendation for immediately fixing the issue, as well as long-term, more strategic advice for preventing similar issues and improving the overall security posture.

    As security experts, our goal is not only to identify vulnerabilities in specific software but also to contribute to the larger community by sharing our recommendations for fixing these problems. We believe that these recommendations can help developers overcome similar challenges in other SSO systems and improve their security.

    Caddy Background

    Caddy (a.k.a. Caddy Server or Caddy 2) is a modern, open-source web server written in Golang that is designed to be easy to use and highly configurable. Caddy is built to streamline the process of hosting web applications while prioritizing security and performance. It aims to reduce the complexity associated with configuring and deploying web servers.

    The caddy-security plugin is a middleware plugin for the Caddy web server. It provides various security-related functionalities to enhance the overall security posture of web applications. Some of the key features offered by the caddy-security plugin include an authentication plugin for implementing Form-Based, Basic, Local, LDAP, OpenID Connect, OAuth 2.0, SAML Authentication, and an authorization plugin for HTTP request authorization based on JWT/PASETO tokens.

    Findings

    Issue #1: Reflected Cross-Site Scripting (XSS)

    Severity: High

    Reflected XSS occurs when an application includes untrusted data in the HTML response sent to the user’s browser. In this case, the provided /admin%22%3E%3Cscript%3Ealert(document.domain)%3C/script%3E/admin/login or /settings/mfa/delete/<img%20src=x%20onerror=alert(document.domain)> API calls trigger an alert. An attacker can exploit this vulnerability to execute arbitrary JavaScript code within the target user’s browser, potentially leading to further attacks such as session hijacking.

    To immediately address this issue, strategically treat all string values as potentially untrustworthy, regardless of their source, and escape them properly (using the safehtml/template package that generates output-safe HTML).

    In addition to that remediation, we also suggest a few different ways to improve defense in depth:

    • Extend unit tests with potentially malicious XSS payloads. Refer to the Cross-site scripting (XSS) cheat sheet for various attack vectors.
    • Consider using the Active Scanner from Burp Suite Professional in a testing environment for all API calls. Additionally, use the scanning with a live task strategy to have underlying requests scanned automatically when interacting with the web interface.
    • Expand the caddy-security documentation to promote security headers—especially the Content Security Policy (CSP) header that controls which resources can be loaded by the browser, limiting the impact of potential XSS attacks.

    Issue #2: Insecure Randomness

    Severity: High

    The caddy-security plugin uses the math/rand Golang library with a seed based on the Unix timestamp to generate strings for three security-critical contexts in the application, which could possibly be predicted via a brute-force search. Attackers could use the potentially predictable nonce value used for authentication purposes in the OAuth flow to conduct OAuth replay attacks. In addition, insecure randomness is used while generating multifactor authentication (MFA) secrets and creating API keys in the database package.

    To immediately mitigate this vulnerability, use a cryptographically secure random number generator for generating the random strings. Golang’s library crypto/rand is designed for secure random number generation.

    In addition to that fix, we recommend considering the following long-term recommendations:

    • Review the application for other instances where the math/rand package is used for secure context. Create secure wrapping functions and use them throughout the code to serve a cryptographically secure string with the requested length.
    • Avoid duplicating code. Having a single function, such as secureRandomString, rather than multiple duplicate functions makes it easier to audit and verify the system’s security. It also prevents future changes to the codebase from reintroducing issues.
    • Implement Semgrep in the CI/CD. The math-random-used Semgrep rule will catch instances where math/rand is used. Refer to our Testing Handbook on Semgrep for more information.
    • Read textbooks such as Real World Cryptography, as it is a great resource for practical cryptographic considerations.

    Issue #3: IP Spoofing via X-Forwarded-For Header

    Severity: Medium

    By manipulating the X-Forwarded-For header, an attacker can spoof an IP address used in the user identity module (/whoami API endpoint). This could lead to unauthorized access if the system trusts this spoofed IP address.

    To resolve this vulnerability, reimplement the application to not rely on user-provided headers when obtaining a user’s IP address. If user-provided headers are required (e.g., X-Forwarded-For for logging purposes), ensure the header is properly validated (i.e., the value is consistent with IP address format through regular expression) or sanitized (to avoid CRLF log injection attacks, for example).

    In addition to this immediate fix, we recommend considering these long-term recommendations:

    • Implement appropriate checks for potential IP spoofing and X- headers on the unit testing level. Consider other headers that can rewrite IP sources.
    • Cover the IP spoofing scenarios and user-provided header processing in Golang’s native fuzz tests.
    • Use the dynamic testing approach with Burp Suite Professional and the Param Miner extension to identify the processing of hidden headers.
    • Expand the caddy-security documentation to increase user awareness of this type of threat; show an example of misconfiguration, how to resolve, and how to test it.

    Issue #4: Referer-Based Header XSS

    Severity: Medium

    An XSS vulnerability can be triggered by rewriting the Referer header. Although the Referer header is sanitized by escaping some characters that can allow XSS (e.g., [&], [<], [>], ["], [']), it does not account for the attack based on the JavaScript URL scheme (e.g., javascript:alert(document.domain)// payload). Exploiting this vulnerability may not be trivial, but it could lead to the execution of malicious scripts in the context of the target user’s browser, compromising user sessions.

    The mitigation for this issue is identical to issue #1.

    Issue #5: Open Redirection Vulnerability

    Severity: Medium

    When a logged-in user clicks on a specially crafted link with a redirect_url parameter, the user can be redirected to an external website. The user must take an action, such as clicking on a portal button or using the browser’s back button, to trigger the redirection. This could lead to phishing attacks, where an attacker tricks users into visiting a malicious website by crafting a convincing URL.

    To mitigate this vulnerability, perform proper redirect_url parameter validation to ensure that the redirection URLs are allowed only within the same domain or from trusted sources.

    In addition, we also recommend the following long-term fixes:

    • Implement robust unit tests with different bypassing scenarios of redirect_url parameter validation. Refer to the potential URL Format Bypasses. Keep in mind that different components can use different URI parsers, which can lead to parsing confusion.
    • Use Burp Suite Professional with a scanner with both these settings enabled:
      • Audit coverage – maximum: to use the most extensive set of payload variations and insertion point options
      • Audit coverage – thorough: to try more payload variations

    Issue #6: X-Forwarded-Host Header Manipulation

    Severity: Medium

    The caddy-security plugin processes the X-Forwarded-Host header, which could lead to various security vulnerabilities (web cache poisoning, business logic flaws, routing-based server-side request forgery [SSRF], and classic server-side vulnerabilities). Additionally, the caddy-security plugin generates QR codes based on this header, which extends the attack surface.

    To mitigate this issue, do not rely on the Host and X-Forwarded-Host headers in the caddy-security plugin logic. Instead, use the current domain manually specified in the configuration file to generate a QR code.

    In addition, we recommend the following:

    • Use Burp Suite Professional with the Param Miner extension to identify the processing of hidden headers.
    • Extend the caddy-security documentation to increase user awareness of the HTTP Host header attacks.

    Issue #7: X-Forwarded-Proto Header Manipulation

    Severity: Low

    The processing of the X-Forwarded-Proto header results in redirection to the injected protocol. While this scenario may have limited impact, improper handling of such headers could result in unpredictable security risks, such as bypass of security mechanisms or confusion in handling TLS.

    To address this issue, do not rely on the X-Forwarded-Proto header. If it is required, validate the value of the X-Forwarded-Proto header against an allowlist of accepted protocols (e.g., HTTP/HTTPS) and reject unexpected values.

    In addition, consider the long-term recommendations from issue #3.

    Issue #8: 2FA Bypass by Brute-Forcing Verification Codes

    Severity: Low

    The current implementation of the application’s two-factor authentication (2FA) lacks sufficient protection against brute-force attacks. Although the application blocks the user after several failed attempts to provide 2FA codes, attackers can bypass this blocking mechanism by automating the application’s full multistep 2FA process.

    To address this issue effectively, enforce a minimum six-digit code length in the MFA configuration. Additionally, to reduce the risk of automated brute-forcing, implement an account locking mechanism that triggers after a specified number of invalid 2FA code attempts. Finally, enforce reauthentication for critical actions involving sensitive account information or security settings. For actions such as changing passwords or disabling 2FA, users should be required to reauthenticate, either with their password or a 2FA token. An exception can be made for reauthentication if the user has logged in within the last 10 minutes. Check out Getting 2FA Right in 2019 at the Trail of Bits Blog for more information.

    Issue #9: Lack of User Session Invalidation on Logout

    Severity: Low

    The caddy-security plugin lacks proper user session invalidation upon clicking the “Sign Out” button; user sessions remain valid even after requests are sent to /logout and /oauth2/google/logout. Attackers who gain access to an active but supposedly logged-out session can perform unauthorized actions on behalf of the user.

    To address this issue, review the sign-out process to identify the cause of the unexpected behavior. Ensure that the /oauth2/google/logout endpoint correctly terminates the user session and invalidates the associated tokens.

    For more defense in depth, use the OWASP Application Security Verification Standard (V3 Session Management) to check whether the implementation handles sessions securely.

    Issue #10: Multiple Panics when Parsing Caddyfile

    Severity: Low

    Multiple parsing functions do not validate whether their input values are nil before attempting to access elements, which can lead to a panic (index out of range). Panics during the parsing of a Caddyfile may not be perceived as immediate vulnerabilities, but they could indicate improperly enforced security controls (e.g., insufficient data validation), which could lead to issues in other code paths.

    To address these issues, integrate nil checks for input values before element access across all relevant functions.

    To prevent similar issues of this type, add Golang’s native fuzz tests for Caddyfile parsing functions.

    Golang Security for the Community

    We love writing and reviewing Golang codebases at Trail of Bits. Indeed, we are constantly working on Golang-related (Semgrep) resources, rules, and blog posts and look forward to any opportunity to take on pet audits (like this) and client projects where we examine Golang codebases.

    Our aim in publishing our findings is to help protect others who may consider implementing a solution similar to the one we explored and to help them make informed decisions about their security infrastructure.

    If you’re actively implementing a codebase in Golang or have questions, concerns, or other recommendations on open-source software you think we should look at, please contact us.

    Coordinated Disclosure Timeline

    As part of the disclosure process, we reported the vulnerabilities to the caddy-security plugin maintainers first. The timeline of disclosure is provided below:

      • August 7, 2023: We reported our findings to the caddy-security plugin maintainers.
      • August 23, 2023: The caddy-security plugin maintainers confirmed that there were no near-term plans to act on the reported vulnerabilities.
      • September 18, 2023: The disclosure blog post was released and issues were filed with the original project repository.

    Holy Macroni! A recipe for progressive language enhancement

    By Brent Pappas

    Despite its use for refactoring and static analysis tooling, Clang has a massive shortcoming: the Clang AST does not provide provenance information about which CPP macro expansions a given AST node is expanded from; nor does it lower macro expansions down to LLVM Intermediate Representation (IR) code. This makes the construction of macro-aware static analyses and transformations exceedingly difficult and an ongoing area of research.1 Struggle no more, however, because this summer at Trail of Bits, I created Macroni to make it easy to create macro-aware static analyses.

    Macroni allows developers to define the syntax for new C language constructs with macros and provide the semantics for these constructs with MLIR. Macroni uses VAST to lower C code down to MLIR and uses PASTA to obtain macro-to-AST provenance information to lower macros down to MLIR as well. Developers can then define custom MLIR converters to transform Macroni’s output into domain-specific MLIR dialects for more nuanced analyses. In this post, I will present several examples of how to use Macroni to augment C with safer language constructs and build C safety analyses.

    Stronger typedefs

    C typedefs are useful for giving semantically meaningful names to lower-level types; however, C compilers don’t use these names during type checking and perform their type checking only on the lower-level types instead. This can manifest in a simple form of type-confusion bug when the semantic types represent different formats or measures, such as the following:

    typedef double fahrenheit;
    typedef double celsius;
    fahrenheit F;
    celsius C;
    F = C; // No compiler error or warning

    Figure 1: C type checking considers only typedef’s underlying types.

    The above code successfully type checks, but the semantic difference between the types fahrenheit and celsius should not be ignored, as they represent values in different temperature scales. There is no way to enforce this sort of strong typing using C typedefs alone.

    With Macroni, we can use macros to define the syntax for strong typedefs and MLIR to implement custom type checking for them. Here’s an example of using macros to define strong typedefs representing temperatures in degrees Fahrenheit and Celsius:

    #define STRONG_TYPEDEF(name) name
    typedef double STRONG_TYPEDEF(fahrenheit);
    typedef double STRONG_TYPEDEF(celsius);

    Figure 2: Using macros to define syntax for strong C typedefs

    Wrapping a typedef name with the STRONG_TYPEDEF() macro allows Macroni to identify typedefs whose names were expanded from invocations of STRONG_TYPEDEF() and convert them into the types of a custom MLIR dialect (e.g., temp), like so:

    %0 = hl.var "F" : !hl.lvalue<!temp.fahrenheit>
    %1 = hl.var "C" : !hl.lvalue<!temp.celsius>
    %2 = hl.ref %1 : !hl.lvalue<!temp.celsius>
    %3 = hl.ref %0 : !hl.lvalue<!temp.fahrenheit>
    %4 = hl.implicit_cast %3 LValueToRValue : !hl.lvalue<!temp.fahrenheit> -> !temp.fahrenheit
    %5 = hl.assign %4 to %2 : !temp.fahrenheit, !hl.lvalue<!temp.celsius> -> !temp.celsius

    Figure 3: Macroni enables us to lower typedefs to MLIR types and enforce strict typing.

    By integrating these macro-attributed typedefs into the type system, we can now define custom type-checking rules for them. For instance, we could enforce strict type checking for operations between temperature values so that the above program would fail to type check. We could also add custom type-casting logic for temperature values so that casting a temperature value in one scale to a different scale implicitly inserts instructions to convert between them.

    The reason for using macros to add the strong typedef syntax is that macros are both backwards-compatible and portable. While we could identify our custom types with Clang by annotating our typedefs using GNU’s or Clang’s attribute syntax, we cannot guarantee annotate()‘s availability across platforms and compilers, whereas we can make strong assumptions about the presence of a C preprocessor.

    Now, you might be thinking: C already has a form of strong typedef called struct. So we could also enforce stricter type checking by converting our typedef types into structs (e.g., struct fahrenheit { double value; }), but this would alter both the type’s API and ABI, breaking existing client code and backwards-compatibility. If we were to change our typedefs into structs, a compiler may produce completely different assembly code. For example, consider the following function definition:

    fahrenheit convert(celsius temp) { return (temp * 9.0 / 5.0) + 32.0; }

    Figure 4: A definition for a Celsius-to-Fahrenheit conversion function

    If we define our strong typedefs using macro-attributed typedefs, then Clang emits the following LLVM IR for the convert(25) call. The LLVM IR representation of the convert function matches up with its C counterpart, accepting a single double-typed argument and returning a double-typed value.

    tail call double @convert(double noundef 2.500000e+01)

    Figure 5: LLVM IR for convert(25), with macro-attributed typedefs used to define strong typedefs

    Contrast this to the IR that Clang produces when we define our strong typedefs using structs. The function call now accepts four arguments instead of one. That first ptr argument represents the location where convert will store the return value. Imagine what would happen if client code called this new version of convert according to the calling convention of the original.

    call void @convert(ptr nonnull sret(%struct.fahrenheit) align 8 %1,
                       i32 undef, i32 inreg 1077477376, i32 inreg 0)

    Figure 6: LLVM IR for convert(25), with structs used to define strong typedefs

    Weak typedefs that ought to be strong are pervasive in C codebases, including critical infrastructure like libc and the Linux kernel. Preserving API- and ABI-compatibility is essential if you want to add strong type checking to a standard type such as time_t. If you wrapped time_t in a struct (e.g., struct strict_time_t { time_t t; }) to provide strong type checking, then not only would all APIs accessing time_t-typed values need to change, but so would the ABIs of those usage sites. Clients who were already using bare time_t values would need to painstakingly change all the places in their code that use time_t to instead use your struct to activate stronger type checking. On the other hand, if you used a macro-attributed typedef to alias the original time_t (i.e., typedef time_t STRONG_TYPEDEF(time_t)), then time_t‘s API and ABI would remain consistent, and client code using time_t correctly could remain as-is.

    Enhancing Sparse in the Linux Kernel

    In 2003, Linus Torvalds built a custom preprocessor, C parser, and compiler called Sparse. Sparse performs Linux kernel–specific type checking. Sparse relies on macros, such as __user, sprinkled around in kernel code, that do nothing under normal build configurations but expand to uses of __attribute__((address_space(...))) when the __CHECKER__ macro is defined.

    Gatekeeping the macro definitions with __CHECKER__ is necessary because most compilers don’t provide ways to hook into macros or implement custom safety checking … until today. With Macroni, we can hook into macros and perform Sparse-like safety checks and analyses. But where Sparse is limited to C (by virtue of implementing a custom C preprocessor and parser), Macroni applies to any code parseable by Clang (i.e., C, C++, and Objective C).

    The first Sparse macro we’ll hook into is __user. The kernel currently defines __user to an attribute that Sparse recognizes:

    # define __user     __attribute__((noderef, address_space(__user)))

    Figure 7: The Linux kernel’s __user macro

    Sparse hooks into this attribute to find pointers that come from user space, as in the following example. The noderef tells Sparse that these pointers must not be dereferenced (e.g., *uaddr = 1) because their provenance cannot be trusted.

    u32 __user *uaddr;

    Figure 8: Example of using the __user macro to annotate a variable as coming from user space

    Macroni can hook into the macro and expanded attribute to lower the declaration down to MLIR like this:

    %0 = hl.var "uaddr" : !hl.lvalue<!sparse.user<!hl.ptr<!hl.elaborated<!hl.typedef<"u32">>>>>

    Figure 9: Kernel code after being lowered to MLIR by Macroni

    The lowered MLIR code embeds the annotation into the type system by wrapping declarations that come from user space in the type sparse.user. Now we can add custom type-checking logic for user-space variables, similar to how we created strong typedefs previously. We can even hook into the Sparse-specific macro __force to disable strong type checking on an ad hoc basis, as developers do currently:

    raw_copy_to_user(void __user *to, const void *from, unsigned long len)
    {
       return __copy_user((__force void *)to, from, len);
    }
    

    Figure 10: Example use of the __force macro to copy a pointer to user space

    We can also use Macroni to identify RCU read-side critical sections in the kernel and verify that certain RCU operations appear only within these sections. For instance, consider the following call to rcu_dereference():

    rcu_read_lock();
    rcu_dereference(sbi->s_group_desc)[i] = bh;
    rcu_read_unlock();

    Figure 11: A call to rcu_derefernce() in an RCU read-side critical section in the Linux kernel

    The above code calls rcu_derefernce() in a critical section—that is, a region of code beginning with a call to rcu_read_lock() and ending with rcu_read_unlock(). One should call rcu_dereference() only within read-side critical sections; however, there is no way to enforce this constraint.

    With Macroni, we can use rcu_read_lock() and rcu_read_unlock() calls to identify critical sections that form implied lexical code regions and then check that calls to rcu_dereference() appear only within these sections:

    kernel.rcu.critical_section {
     %1 = macroni.parameter "p" : ...
     %2 = kernel.rcu_dereference rcu_dereference(%1) : ...
    }
    

    Figure 12: The result of lowering the RCU-critical section to MLIR, with types omitted for brevity

    The above code turns both the RCU-critical sections and calls to rcu_dereference() into MLIR operations. This makes it easy to check that rcu_dereference() appears only within the right regions.

    Unfortunately, RCU-critical sections don’t always bound neat lexical code regions, and rcu_dereference() is not always called in such regions, as shown in the following example:

    __bpf_kfunc void bpf_rcu_read_lock(void)
    {
           rcu_read_lock();
    }
    

    Figure 13: Kernel code containing a non-lexical RCU-critical section

    static inline struct in_device *__in_dev_get_rcu(const struct net_device *dev)
    {
       return rcu_dereference(dev->ip_ptr);
    }
    

    Figure 14: Kernel code calling rcu_dereference() outside of an RCU-critical section

    We can use the __force macro to permit these sorts of calls to rcu_dereference(), just as we did to escape type checking for user-space pointers.

    Rust-like unsafe regions

    It’s clear that Macroni can help strengthen type checking and even enable application-specific type-checking rules. However, marking types as strong means committing to that level of strength. In a large codebase, such a commitment might require a massive changeset. To make adapting to a stronger type system more manageable, we can design an “unsafety” mechanism for C akin to that of Rust: within the unsafe region, strong type checking does not apply.

    #define unsafe if (0); else
    
    
    fahrenheit convert(celsius C) {
     fahrenheit F;
     unsafe {
             F = (C * 9.0 / 5.0) + 32.0;
     }
     return F;
    }
    

    Figure 15: C code snippet presenting macro-implemented syntax for unsafe regions

    This snippet demonstrates our safety API’s syntax: we call the unsafe macro before potentially unsafe regions of code. All code not listed in an unsafe region will be subject to strong type checking, while we can use the unsafe macro to call out regions of lower-level code that we deliberately want to leave as-is. That’s progressive!

    The unsafe macro provides the syntax only for our safety API, though, and not the logic. To make this leaky abstraction watertight, we would need to transform the macro-marked if statement into an operation in our theoretical safety dialect:

    ...
    "safety.unsafe"() ({
       ...
    }) : () -> ()
    ...
    

    Figure 16: With Macroni, we can lower our safety API to an MLIR dialect and implement safety-checking logic.

    Now we can disable strong type checking on operations nested within the MLIR representation of the unsafe macro.

    Safer signal handling

    By this point, you may have noticed a pattern for creating safer language constructs: we use macros to define syntax for marking certain types, values, or regions of code as obeying some set of invariants, and then we define logic in MLIR to check that these invariants hold.

    We can use Macroni to ensure that signal handlers execute only signal-safe code. For example, consider the following signal handler defined in the Linux kernel:

    static void sig_handler(int signo) {
           do_detach(if_idx, if_name);
           perf_buffer__free(pb);
           exit(0);
    }
    

    Figure 17: A signal handler defined in the Linux kernel

    sig_handler() calls three other functions in its definition, which should all be safe to call in signal-handling contexts. However, nothing in the above code checks that we call signal-safe functions only inside sig_handler()‘s definition—C compilers don’t have a way of expressing semantic checks that apply to lexical regions.

    Using Macroni, we could add macros for marking certain functions as signal handlers and others as signal-safe and then implement logic in MLIR to check that signal handlers call only signal-safe functions, like this:

    #define SIG_HANDLER(name) name
    #define SIG_SAFE(name) name
    
    
    int SIG_SAFE(do_detach)(int, const char*);
    void SIG_SAFE(perf_buffer__free)(struct perf_buffer*);
    void SIG_SAFE(exit)(int);
    
    
    static void SIG_HANDLER(sig_handler)(int signo) { ... }

    Figure 18: Token-based syntax for marking signal handlers and signal-safe functions

    The above code marks sig_handler() as a signal handler and the three functions it calls as signal-safe. Each macro invocation expands to a single token—the name of the function we want to mark. With this approach, Macroni hooks into the expanded function name token to determine if the function is a signal handler or signal-safe.

    An alternative approach would be to define these macros to magic annotations and then hook into these with Macroni:

    #define SIG_HANDLER __attribute__((annotate("macroni.signal_handler")))
    #define SIG_SAFE __attribute__((annotate("macroni.signal_safe")))
    
    
    int SIG_SAFE do_detach(int, const char*);
    void SIG_SAFE perf_buffer__free(struct perf_buffer*);
    void SIG_SAFE exit(int);
    
    
    static void SIG_HANDLER sig_handler(int signo) { ... }

    Figure 19: Alternative attribute syntax for marking signal handlers and signal-safe functions

    With this approach, the macro invocation looks more like a type specifier, which some may find more appealing. The only difference between the token-based syntax and the attribute syntax is that the latter requires compiler support for the annotate() attribute. If this is not an issue, or if __CHECKER__-like gatekeeping is acceptable, then either syntax works fine; the back-end MLIR logic for checking signal safety would be the same regardless of the syntax we choose.

    Conclusion: Why Macroni?

    Macroni lowers C code and macros down to MLIR so that you can avoid basing your analyses on the lackluster Clang AST and instead build them off of a domain-specific IR that has full access to types, control flow, and data flow within VAST’s high-level MLIR dialect. Macroni will lower the domain-relevant macros down to MLIR for you and elide all other macros. This unlocks macro-sensitive static analysis superpowers. You can define custom analyses, transformations, and optimizations, taking macros into account at every step. As this post demonstrates, you can even combine macros and MLIR to define new C syntax and semantics. Macroni is free and open source, so check out its GitHub repo to try it out!

    Acknowledgments

    I thank Trail of Bits for the opportunity to create Macroni this summer. In particular, I would like to thank my manager and mentor Peter Goodman for the initial idea of lowering macros down to MLIR and for suggestions for potential use cases for Macroni. I would also like to thank Lukas Korencik for reviewing Macroni’s code and for providing advice on how to improve it.

    1 See Understanding code containing preprocessor constructs, SugarC: Scalable Desugaring of Real-World Preprocessor Usage into Pure C, An Empirical Analysis of C Preprocessor Use, A Framework for Preprocessor-Aware C Source Code Analyses, Variability-aware parsing in the presence of lexical macros and conditional compilation, Parsing C/C++ Code without Pre-processing, Folding: an approach to enable program understanding of preprocessed languages, and Challenges of refactoring C programs.

    Secure your Apollo GraphQL server with Semgrep

    By Vasco Franco

    tl;dr: Our publicly available Semgrep ruleset has nine new rules to detect misconfigurations of versions 3 and 4 of the Apollo GraphQL server. Try them out with semgrep --config p/trailofbits!

    When auditing several of our clients’ Apollo GraphQL servers, I kept finding the same issues over and over: cross-site request forgery (CSRF) that allowed attackers to perform actions on behalf of users, rate-limiting that allowed attackers to brute-force passwords or MFA tokens, and cross-origin resource sharing (CORS) misconfigurations that allowed attackers to fetch secrets that they shouldn’t have access to. Developers overlook these issues for multiple reasons: bad defaults in version 3 of the Apollo GraphQL server (e.g., the csrfProtection option does not default to true), a lack of understanding or knowledge of certain GraphQL features (e.g., batched queries), and a lack of understanding of certain web concepts (e.g., how the same-origin policy [SOP] and CORS work).

    Finding the same issues repeatedly motivated me to use some internal research and development (IRAD) time to consistently detect some of these issues in our future audits, leaving more time to find deeper, more complex bugs. Semgrep—a static analysis tool used to detect simple patterns that occur in a single file—was the obvious tool for the job because the issues are easy to detect with grep-like constructs and don’t require interprocedural or other types of more complex analysis.

    We open sourced Semgrep rules that find Apollo GraphQL server v3 and v4 misconfigurations. Our rules leverage Semgrep’s taint mode to make them easier to write and to increase their accuracy. Go test your GraphQL servers!

    We previously publicly released Semgrep rules to find Go concurrency bugs and misuses of machine learning libraries.

    Common GraphQL issues

    GraphQL has several design choices that make some vulnerabilities, such as CSRF, more prevalent than in typical REST servers. Of course, GraphQL servers also suffer from all the usual problems: access control issues (e.g., an access control flaw in GitLab that disclosed information about private users, or a bug in HackerOne that allowed attackers to disclose users’ confidential data), SQL injections (e.g., a SQL injection in HackerOne’s GraphQL server), server-side request forgery (SSRF), command injection, and many others.

    This blog post will cover the rules we created to detect CSRF and CORS misconfigurations. We’ll also show how using Semgrep’s taint mode can save you time and increase your rules’ accuracy by reducing the number of patterns you need to define all the ways in which a value can flow into a sink.

    CSRF

    CSRF is an attack that allows malicious actors to trick users into performing unwanted operations (e.g., editing the user’s profile) in websites they’re authenticated to. If you’re unfamiliar with the details, read more about CSRF attacks in PortSwigger’s Web Security Academy CSRF explanation.

    CSRF attacks in the Apollo Server

    CSRF haunted the Apollo GraphQL server until the introduction of the csrfPrevention option. CSRF vulnerabilities are prevalent in the Apollo server because of two factors: developers mislabel mutations as queries, and the Apollo server allows users to issue query operations with GET requests (but not mutation operations). Queries should not change state (like a GET request in a RESTful API), while mutations are expected to change state (like POST, PATCH, PUT, or DELETE). If developers followed this convention, everything would be fine. However, I’ve yet to find a codebase that does not mislabel a mutation as a query, making these mislabeled operations immediately vulnerable to CSRF attacks.

    Thankfully, the Apollo team was very aware of this and, in version 3, added the csrfPrevention option to remove the issue altogether. It prevents CSRF attacks by ensuring that any request must have a Content-Type header different from text/plain, application/x-www-form-urlencoded, or multipart/form-data; a non-empty X-Apollo-Operation-Name header; or a non-empty Apollo-Require-Preflight header. This ensures the request will always be preflighted, which prevents the CSRF attack.

    The csrfPrevention option defaults to false in v3 and to true in v4, so those still using v3 need to consciously add this option in their server initialization, which, in our experience, almost never happens.

    Finding CSRF misconfigurations with Semgrep

    We created two Semgrep rules to find misconfigurations in versions 3 and 4. For v3, we find all ApolloServer initializations where the csrfPrevention option is not set to true.

    patterns:
      - pattern: new ApolloServer({...})
      - pattern-not: |
          new ApolloServer({..., csrfPrevention: true, ...})

    Figure 1.1: Semgrep rule that detects a misconfigured csrfProtection option in version 3 of the Apollo server

    For v4, we find all server initializations with the csrfPrevention option set to false.

    patterns:
      - pattern: |
          new ApolloServer({..., csrfPrevention: false, ...})

    Figure 1.2: Semgrep rule that detects a misconfigured csrfProtection option in version 4 of the Apollo server

    CORS

    CORS allows a server to relax the browser’s SOP. As expected, developers sometimes relax the SOP a bit too far, which can allow attackers to fetch secrets that they should not have access to. If you are unfamiliar with the details, read more about CORS in PortSwigger’s Web Security Academy CORS explanation.

    Setting a CORS policy in an Apollo Server

    In version 3 of the Apollo Server, a developer can set their server’s CORS policy in two ways. First, they can pass the cors argument to their ApolloServer class instance.

    import { ApolloServer } from 'apollo-server';
    
    const apolloServerInstance = new ApolloServer({
        cors: CORS_ORIGIN
    });

    Figure 1.3: Configuring CORS in version 3 of an Apollo GraphQL server

    Alternatively, they can set the CORS policy on the back-end framework they are using. For example, with an Express.js back-end server, the CORS attribute is passed as an argument to the applyMiddleware function.

    import { ApolloServer } from 'apollo-server-express';
    
    const apolloServerInstance = new ApolloServer({});
    
    apolloServerInstance.applyMiddleware({
        app,
        cors: CORS_ORIGIN,
    });

    Figure 1.4: Configuring CORS in version 3 of an Apollo GraphQL server with a back-end Express server

    On version 4 of the Apollo server, the developer must set CORS on the back end itself. Therefore, writing rules for v4 is out of scope for our Apollo-specific Semgrep queries—other Semgrep rules already cover most of those cases.

    Our rules for version 3 cover uses of Express.js and the batteries-included Apollo server back ends, as these were the ones we saw in use the most. If you use a different back-end framework for your Apollo Server, our rules likely won’t work, but we accept PRs at trailofbits/semgrep-rules! It should be effortless to adapt them based on the existing queries. ;)

    Finding missing CORS policies

    The rules for each back end are very similar, so let’s look at one of them—the one that detects CORS misconfigurations in the batteries-included Apollo server. We have two rules in the same file: one to detect cases where a CORS policy is not defined and one to detect a poorly configured CORS policy.

    To detect missing CORS policies, we look for ApolloServer instantiations where the cors argument is undefined. We also need to ensure that the ApolloServer comes from the apollo-server package (the ApolloServer class could also come from the apollo-server-express package, but we don’t want to catch these cases). The query is shown in figure 1.5.

    patterns:
      - pattern-either:
          - pattern-inside: |
              $X = require('apollo-server');
              ...
          - pattern-inside: |
              import 'apollo-server';
              ...
      - pattern: |
          new ApolloServer({...})
      - pattern-not: |
          new ApolloServer({..., cors: ..., ...})

    Figure 1.5: Semgrep rule that detects a missing CORS policy in an Apollo GraphQL server (v3)

    Finding bad CORS policies

    To detect bad CORS policies, it’s not as simple. We have to detect several cases:

    • Cases where the origin is set to true—A true origin tells the server to accept all origins.
    • Cases where the origin is set to null—An attacker can trick a user into making requests from a null origin from, for example, a sandboxed iframe.
    • Cases where the origin is a regex with an unescaped dot character—In regex, a dot matches any character, so if we are using the /api.example.com$/ regex, it will match the apiXexample.com domain, which could potentially be controlled by an attacker.
    • Cases where the origin does not finish with the $ character—In regex, the $ character matches the end of the string, so if we are using the /api.example.com/ regex, it will also match the api.example.com.attacker.com domain, an attacker-controlled domain.

    And these will not cover every possible bad CORS policy (e.g., a bad CORS policy could simply include an attacker domain or a domain that allows an attacker to upload HTML code). We test all the cases described above with the rule in the figure below.

    pattern-either:
      # 'true' mean that every origin is reflected
      - pattern: |
          true
      # the '.' character is not escaped
      - pattern-regex: ^/.*[^\\]\..*/$
      # the regex does not end with '$'
      - pattern-regex: ^/.*[^$]/$
      # An attacker can make requests from ‘null’ origins
      - pattern: |
          'null'

    Figure 1.6: Semgrep pattern that detects bad CORS origins

    These bad origins can be used by themselves or inside an array. To test for both cases, we first check occurrences of the $CORS_SINGLE_ORIGIN metavariable that are isolated or in an array and then use a metavariable-pattern to define what is a bad origin with the pattern we’ve created in figure 1.6.

    pattern-either:
      - patterns:
          # pattern alone or inside an array
          - pattern-either:
              - pattern: |
                  $CORS_SINGLE_ORIGIN
              - pattern: |
                  [..., $CORS_SINGLE_ORIGIN, ...]
          - metavariable-pattern:
              metavariable: $CORS_SINGLE_ORIGIN
              pattern-either:
                 # <The bad origin checks from the previous figure>

    Figure 1.7: Semgrep pattern that detects bad CORS origins in a single entry or in an array

    Finally, we need to find uses of this origin inside an ApolloServer initialization. We do so with the following pattern:

    new ApolloServer({..., cors: $CORS_ORIGIN, ...})

    This $CORS_ORIGIN can be used inline (e.g., cors: true), or it can come from a variable (e.g., cors: corsOriginVariableDefineElsewhere). It is laborious to define all the possible places that the origin could have come from. Thankfully, we don’t need to do so with Semgrep’s taint mode!

    We need to define only the following:

    • pattern-sources: the bad CORS policy—We define it as {origin: $BAD_CORS_ORIGIN} where the $BAD_CORS_ORIGIN metavariable is the pattern we defined above for a bad origin.
    • pattern-sinks: where the bad CORS policy should not flow to—We define it as the $CORS_ORIGIN metavariable in the pattern new ApolloServer({..., cors: $CORS_ORIGIN, ...}).

    With taint mode, we can catch many ways in which the CORS policy can be set: directly (Case 1 in figure 1.8), through a variable that configures the entire CORS policy (Case 2), through a variable that sets only the origin (Case 3), and many other setups that we do not want to define by hand.

    // Case 1: Has a very permissive 'cors' (true)
    const apollo_server_bad_1 = new ApolloServer({
        //ruleid: apollo-graphql-v3-bad-cors
        cors: { origin: true }
    });
    
    // Case 2: Has a very permissive 'cors' from a variable
    const bad_CORS_policy = { origin: true }
    const apollo_server_bad_2 = new ApolloServer({
        //ruleid: apollo-graphql-v3-bad-cors
        cors: bad_CORS_policy
    });
    
    // Case 3: Has a very permissive 'cors' from a variable (just the origin)
    const bad_origin = true;
    const apollo_server_bad_3 = new ApolloServer({
        //ruleid: apollo-graphql-v3-bad-cors
        cors: { origin: bad_origin }</span
    });

    Figure 1.8: Several test cases that Semgrep’s taint mode helps catch for free

    The entire commented rule is shown in figure 1.9.

    mode: taint
    pattern-sources:
      - patterns:
          - pattern-inside: |
              { origin: $BAD_CORS_ORIGIN }
          - metavariable-pattern:
              metavariable: $BAD_CORS_ORIGIN
              pattern-either:
                # 'true' means that every origin is reflected
                - pattern: |
                    true
                - patterns:
                    # pattern alone or inside an array
                    - pattern-either:
                        - pattern: |
                            $CORS_SINGLE_ORIGIN
                        - pattern: |
                            [..., $CORS_SINGLE_ORIGIN, ...]
                    - metavariable-pattern:
                        metavariable: $CORS_SINGLE_ORIGIN
                        pattern-either:
                          # the '.' character is not escaped
                          - pattern-regex: ^/.*[^\\]\..*/$
                          # the regex does not end with '$'
                          - pattern-regex: ^/.*[^$]/$
                          # An attacker can make requests from ‘null’ origins
                          - pattern: |
                              'null'
    pattern-sinks:
      - patterns:
          # The ApolloServer comes from the 'apollo-server' package
          - pattern-either:
              - pattern-inside: |
                  $X = require('apollo-server');
                  ...
              - pattern-inside: |
                  import 'apollo-server';
                  ...
          # The sink is the ApolloServer's cors argument
          - pattern: |
              new ApolloServer({..., cors: $CORS_ORIGIN, ...})
          # This tells Semgrep that the sink is only the $CORS_ORIGIN variable
          - focus-metavariable: $CORS_ORIGIN

    Figure 1.9: Semgrep rule that detects a bad CORS policy in an Apollo GraphQL server (v3)

    We have also created a Semgrep rule for auditors and security engineers that want to review their Apollo server’s CORS policy in detail, even when the policy might be safe. This rule reports any CORS policy that is not false or an empty array—obviously good CORS policies. It is helpful when you want to check all the hard-coded origins by hand, but it is not something that you want to integrate in your CI pipeline since it will report false positives (an audit rule). You can find the rule at trailofbits.javascript.apollo-graphql.v3-cors-audit.v3-potentially-bad-cors.

    Finishing thoughts

    Semgrep excels in finding simple patterns that happen in a single file like the ones we’ve described in this post. For more complex analysis, you may want to use a tool such as CodeQL, which has its disadvantages as well: it involves a more difficult learning curve, it uses different APIs for different languages, it requires compiling the code, and it does not support some languages that Semgrep does (e.g., Rust).

    One of Semgrep’s biggest limitations is that it lacks interfile and interprocedural analysis. For example, the rules above won’t catch cases where the CORS policy is set in one file and the Apollo Server initialization occurs in another file. This may now be possible with Semgrep Pro Engine (previously called DeepSemgrep), which enhances the Semgrep engine with interfile analysis capabilities. However, this feature is currently limited to paid customers and to a limited number of languages.

    At Trail of Bits, we extensively use static analysis tools and usually end up writing custom rules and queries specific to our clients’ codebases. These can provide great value because they can find patterns specific to your codebase and even enforce your organization’s engineering best practices. When the rules we write are useful to the community, we like to open source them. Check them out at https://github.com/trailofbits/semgrep-rules.

    Use our new Apollo GraphQL rules with semgrep --config p/trailofbits, and try writing your own custom rules!

    iVerify is now an independent company!

    We’re proud to announce that iVerify is now an independent company following its four-year incubation at Trail of Bits. Originally developed in-house to ensure that our personal phones, which store data essential to our work and private lives, were secured to the standards of security professionals, iVerify quickly showed that it could be valuable to the public:

    “The mobile security market has a problem. Simply put, current solutions fail to meet the sophistication of modern threats or the growing privacy desires of mobile device users… Forensics tools are limited, and researchers are not necessarily equipped to rapidly discover and alert targets that their device has been compromised in a timely manner.

    [iVerify’s] vision is to arm organizations and individuals with the protection they need in this changing world. We are building the first mobile threat hunting company focused on countering the emerging mobile spyware threat.” – iVerify

    Trail of Bits launched the iVerify security toolkit for iPhones in 2019, an enterprise product in 2020, then an Android app in 2021. Now, with $4 million in seed funding, iVerify plans to expand its capabilities to reach more enterprise customers.

    At the helm are four Trail of Bits alumni: Matthias Frielingsdorf, Jelmer de Hen, and Vlad Orlov, who join CEO Danny Rogers. Their contributions to making iVerify an essential consumer and enterprise product motivated investors, including Mischief Ventures, Mantis Venture Capital, Altman Capital, and others.

    “It’s rare for a seed startup to have the impressive portfolio of enterprise customers that iVerify already has. It’s also rare to find a founding team with the technical prowess and the business mindset to build something that is both technically sound and commercially viable.” – Dustin Moring, General Partner at Mischief Ventures (lead investor).

    We couldn’t agree more. Thank you to the team and everyone who has contributed to iVerify’s success. We’re excited to watch from the sidelines as iVerify leads the cause in safeguarding individual and organizational device security.

    ###

    Read the entire press release to learn more about iVerify’s product offerings and plans for growth, and check out additional coverage from around the web:

    • Why The Chainsmokers Invest in—and Party With—Niche Cybersecurity Companies
    • iVerify Raises $4M to Take On the Growing Threat of Mercenary Spyware
    • Introducing iVerify, the first mobile threat hunting company
    • Decipher Podcast: iVerify CEO Danny Rogers and COO Rocky Cole join Dennis Fisher to discuss the spinout of the iVerify mobile security tool as a standalone company, the scourge of mercenary spyware, and how enterprises can protect their users.
    • Danny Rogers: “With the commercialization of advanced spyware, the narrative around mobile security has changed. That change demands new approaches, and iVerify is setting out to be the world’s first true mobile threat hunting company focused on combating mercenary spyware.”
    • Matthias Frielingsdorf: “I’m very proud to announce the launch of the first mobile threat hunting company, iVerify, which we’re building to harmonize security and privacy in the face of a new class of mobile security threats!”
    • Rocky Cole: “… the security industry still looks a lot like it did when I first joined the hacking community, straight out of college… It’s time for something new and I’m beyond thrilled to be working with … our growing and exceptionally talented team at iVerify to bring the world the mobile security product it deserves.”
    • Alex Pall (Chainsmokers): “Since we’re all about propelling ingenious solutions that crack real-world problems, iVerify – which was incubated by security research firm Trail of Bits – was an obvious bet”
    • Gabriel Jacoby-Cooper: “It’s exceedingly difficult to fight back against the mercenary spyware, but the iVerify team is simply the best to do so.”
    • William Knowles: “iVerify is the one smartphone app everyone concerned with their security should install and make well advised updates to their devices.”

    If you’re as enthusiastic about the mission and potential of iVerify as we are, consider becoming a part of their groundbreaking journey. They’re on the lookout for talented individuals who can help turn their vision for a safer mobile ecosystem into reality. Here are the current opportunities they have open:

    If you’re an interested potential user of iVerify, follow along with them on Twitter, LinkedIn, or Mastodon for product updates.

    The Engineer’s Guide to Blockchain Finality

    By Benjamin Samuels

    Many security-critical off-chain applications use a simple block delay to determine finality: the point at which a transaction becomes immutable in a blockchain’s ledger (and is impossible to “undo” without extreme economic cost). But this is inadequate for most networks, and can become a single point of failure for the centralized exchanges, multi-chain bridges, and L2 scaling solutions that rely on transaction finality. Without proper consideration of a blockchain’s finality criteria, transactions that appear final can be expunged from the blockchain by a malicious actor in an event called a re-org, leading to double-spend attacks and value stolen from the application.

    We researched several off-chain applications and L2 networks and discovered two L2 clients, Juno and Pathfinder, that either were not checking for finality or incorrectly used block delays to detect whether Ethereum blocks were finalized. We disclosed our findings to each product team, and fixes were published shortly after disclosure in version v0.4.0 for Juno and v0.6.2 for Pathfinder. This blog post gathers the knowledge and insights we gained from this research. It explains the dangers of reorgs, the differences between distinct finality mechanisms, and how to prevent double-spend attacks when writing applications that consume data from different kinds of blockchains.

    Understanding re-orgs

    When a user submits a transaction to a blockchain, it follows a lifecycle that is nearly identical across all blockchains. First, their transaction is gossiped across the blockchain’s peer-to-peer network to a block proposer. Once a block proposer receives the transaction and includes it in a block, the block is broadcast across the network.

    Here is where the problems begin: some blockchains don’t explicitly define who the next block proposer should be, and the ones that do need a way to recover if that proposer is offline. These conditions lead to situations where there are two or more valid ways for the blockchain to proceed (a fork), and the network has to figure out which fork should be canonical.

    Figure 1: Two miners on a PoW network propose a valid block for slot 4 at the same time.

    Blockchains are designed with these issues in mind and define a fork choice rule to determine which fork should be considered canonical. Forks can sometimes last for multiple blocks, where different portions of the network consider a different chain to be canonical.

    Figure 2: A PoW network where block candidates for slots 4, 5, and 6 were mined in quick succession and built on different parents, leading to a fork.

    Assuming there is no bug in the network’s software, the fork will eventually be reconciled, leading to a single fork becoming canonical. The other forks, their blocks, and their transactions are expunged from the blockchain’s history, called a re-org.

    When a transaction is expunged from the chain via a re-org, that transaction may either be re-queued for inclusion in a new block, or otherwise have its ordering or block number changed to whatever it is in the canonical chain. Attackers can leverage these changes to modify a transaction’s behavior or cancel the transaction entirely based on which fork it is included on.

    Figure 3: A network after a three-block re-org. Transactions in blocks 4a, 5a, and 6a are no longer part of the canonical chain.

    Re-orgs are a normal part of a blockchain’s lifecycle, and can happen regularly due to factors like block production speed, network latency, and network health. However, attackers can take advantage of (and even orchestrate!) re-orgs to perform double-spend attacks, a category of attack where an attacker submits a deposit transaction, waits for it to be included in a block, then orchestrates a re-org to expunge their transaction from the canonical chain while still receiving credit for their deposit on the off-chain application.

    It is for this reason that finality considerations are important. If a centralized exchange or bridge indexes a deposit transaction that is not final, it is vulnerable to double-spend attacks by way of an attacker causing a re-org of the blockchain.

    Blockchains use a variety of different consensus algorithms and thus have a variety of different finality conditions that should be considered for each chain.

    Probabilistic finality

    Examples: Bitcoin, Binance Smart Chain (pre-BEP-126), Polygon PoS, Avalanche – or generally any PoW-based blockchain

    Chains using probabilistic finality are unique in that their blocks are never actually finalized—instead, they become probabilistically final, or more “final” as time goes on. Given enough time, the probability that a previous block will be re-orged off the chain approaches zero, and thus the block becomes final.

    In most probabilistically final chains, the fork choice rule that determines the canonical chain is based on whichever fork has the most blocks built on top of it, called Nakamoto consensus. Under Nakamoto consensus, the chain may re-org if a longer chain is broadcast to the network, even if the longer chain excludes blocks/transactions that were already included in the shorter chain.

    Double-spend attacks on probabilistic proof-of-work networks

    The classic attack against proof-of-work networks is a 51% re-org attack. This attack requires an off-chain exchange, bridge, or other application that indexes deposit transactions very quickly, ideally indexing blocks as soon as they are produced or with an exceedingly short delay.

    The attacker must accumulate, purchase, or rent enough computing resources so the attacker controls the majority of the hash power on the network. This means the attacker has enough resources to privately mine a chain that’s longer than the honest canonical chain. Note that this is a probabilistic attack; control over 51% of the network’s hash power makes the attack an eventual certainty. An attacker could theoretically perform double-spend attacks with much less than 51% of the network’s hash power, but it may require many attempts before the attack succeeds.

    Once the mining resources are ready, the attacker submits a transaction on the public, canonical chain to deposit funds from their wallet to the exchange/bridge.

    Immediately afterward, the attacker must create a second, conflicting transaction that transfers funds from their wallet to another attacker-controlled address. The attacker configures their mining resources to mine a new fork that includes the transfer transaction instead of the deposit transaction.

    Figure 4: The attacker creates a private fork that includes their transfer transaction instead of the deposit transaction.

    Given that the attacker controls the majority of the hash power on the network, eventually their private fork will have more blocks than the canonical fork. Once they have received credit for the deposit transaction and their private fork has more blocks than the canonical chain, the attacker instructs their network to publish the private chain’s blocks to the honest nodes following the canonical chain.

    The honest nodes apply the “longest chain” fork choice rule, triggering a re-org around the attacker’s longer chain and excluding the blocks that contained the attacker’s deposit transaction.

    Figure 5: Once the attacker publishes their private fork, the network reorgs and expunges the fork that includes their deposit transaction.

    In effect, this allows the attacker to “double-spend” their coins: the exchange or bridge credits the attacker for the coins, while the coins are still present in an attacker-controlled wallet.

    Measuring probabilistic finality

    Since probabilistically final chains don’t define finality conditions, finality must be measured probabilistically based on the number of blocks that have elapsed since the target transaction/ancestor block. The more blocks that have been built on top of a given ancestor, the higher the cost of a re-org is for an attacker.

    The correct block delay should be based on both historical factors and economic factors, selecting the greater of the two for the application’s indexing delay.

    Among historical factors, off-chain application developers should consider how often the chain has reorgs and how large the reorgs typically are. If block production is probabilistic (as in Proof-of-work networks), then a larger delay should be factored in to compensate for the chance that many blocks are created in an unusually short period of time.

    Among economic factors, one should consider the cost to execute a re-org attack for a transaction of a given economic value. Given the probabilistic nature of 51% attacks, engineers should build in a safety margin and consider the cost of a 25% attack instead of 51%. For example, if it costs a minimum of $500k USD to execute a 25% attack, six-block re-org attack against Bitcoin, then an off-chain application that receives a $500k deposit should wait for at least six blocks before considering the transaction final and indexing it.

    Using Crypto51, we can calculate the block delays required for deposits and withdrawals of $75k with a 25% attack threshold (as of June 2023).

    • Bitcoin: Two blocks for finality (~20 minutes)
    • Litecoin/Dogecoin: 48 blocks for finality (~Two hours)
    • Bitcoin Cash: 103 blocks for finality (~17 hours)
    • Ethereum Classic: 3,031 blocks for finality (~11 hours)
    • Ethereum PoW: 23,600 blocks for finality (~89 hours)
    • Zcash: 1,881 blocks for finality (~40 hours)

    These delays represent a single data point in time for a specific deposit amount. As the hash power on each network increases or decreases, the time-to-finality will change as well and must be updated accordingly. Mining hash power on each network correlates highly with the chain token price, so the amount of hash power can drop very quickly, requiring monitoring and fast response by integrating applications. Failure to adjust finality delays in a timely manner will lead to double-spending attacks.

    Finality delays for proof-of-work chains may be reduced for certain exchange-like applications using on-chain monitoring, automated system halts, trading limits, and withdrawal delays. However, these mechanisms may overcomplicate the application’s logic and make it vulnerable to other forms of attack.

    It should be noted that chains with extraordinarily low hash rates may easily be attacked with much more than 51% of the network’s hash rate. Existing services offer an easy-to-rent hashing capacity that may exceed a chain’s hash rate many times over. In cases like this, it is recommended to either avoid integration with the chain, or base finality calculations on the available-for-rent hashing capacity.

    Probabilistic chains using proof-of-stake/proof-of-authority require slightly different considerations, since block proposers cannot freely enter the proposer set and may have different fork choice rules than proof-of-work networks.

    • Binance Smart Chain: Blocks are considered final once the number of blocks that have passed is equal to two-thirds the number of validators in their validator set. Twenty blocks are required for finality (~60 seconds).
    • Polygon PoS: Integrators should use L1 state root checkpoints as a measure of finality. When a state root checkpoint containing a given transaction is finalized on the L1, the Polygon transaction may be considered final. State root checkpoints occur roughly every 30 minutes.

    Provable finality

    Delayed finality examples: Ethereum PoS (Casper FFG), Polkadot (GRANDPA), Solana
    Instant Finality Examples: Cosmos, Celestia, Algorand, Aptos

    Systems using provable finality make special considerations for finality to ensure it happens more quickly and with better economic assurances than most probabilistically final chain constructions.

    There are two types of provable finality: chains with instantly provable finality, and chains with delayed provable finality.

    Chains with instant finality don’t need special finality considerations by off-chain applications. All blocks published by the network are immediately provably final by definition.

    Chains with delayed finality have separate consensus mechanisms for newly produced vs. finalized blocks. These chains usually have superior liveness properties compared to instant finality chains, but at the cost of added complexity, vulnerability to re-orgs, and more complex integration considerations for off-chain applications.

    Figure 6: A delayed-finality chain. Blocks to the right of the finality boundary may be re-orged and should not be indexed by exchanges or bridges.

    Double-spend attacks on delayed finality chains

    Historically, most blockchains haven’t had provable finality, so bridges, exchanges, and other off-chain applications would use a block delay for measuring the finality of any new chains they integrate with.

    However, for chains with provable delayed finality, there are situations where the finality mechanism may stall or fail, as occurred in the May 2023 incident where Ethereum’s finality gadget, Casper FFG, stalled. When finality mechanisms fail, the chain may continue to produce blocks, creating long strings of unfinalized blocks that may be reorged by a bug or an attacker.

    During the Ethereum incident, the chain’s finality mechanism was stalled by nine epochs—the equivalent of 139 blocks’ worth of confirmations (after controlling for missed slot proposals). At this time, most bridges/centralized exchanges used a block-delay rule to determine the finality of a transaction on Ethereum, with delays ranging from 14 blocks to 100 blocks.

    Had the Ethereum finality incident been orchestrated by an attacker, the attacker may have been able to perform double-spend attacks against these bridges/exchanges by orchestrating exceedingly long re-orgs.

    Checking for finality

    For delayed-finality chains, as illustrated in the previous example, block delays are not an adequate way to “wait” for blocks to become final. Instead, applications must query the chain’s RPC for the exact finality condition to ensure the block being indexed is actually final.

    Ethereum proof-of-stake

    The Ethereum JSON RPC defines a “default block” parameter for various endpoints that should be set to “finalized” to query the most recent finalized block. To obtain the most recent finalized block, use eth_getBlockByNumber(“finalized”, ...). This parameter may be used for other endpoints, such as eth_call, eth_getBalance, and eth_getLogs.

    Polkadot

    Call chain.getFinalizedHead() to get the block hash of the latest finalized block, then use chain.getBlock() to get the block associated with the hash.

    Solana

    Use the getBlocks() RPC method with the commitment level set as finalized.

    When provable finality lies

    One major caveat of provable finality/proof-of-stake systems is they have no way to provide strong subjectivity guarantees. A blockchain’s subjectivity refers to whether a node syncing from genesis will always arrive at the same chain head and whether an attacker can manipulate the end state of the syncing node.

    In proof-of-work blockchains, the cost of creating an alternate chain for partially synced nodes to follow is equal to all of the work performed by miners from genesis to the canonical chain head, making any subjectivity attack impractical against proof-of-work networks.

    However, in proof-of-stake networks, the cost of creating an alternate chain has only one requirement with an unknown, and possibly zero cost: the private keys of the chain’s historical validators. The keys for historical validators may be acquired by a number of means; private keys may be leaked or brute-forced, or validators who no longer use their keys may offer them up for sale.

    This re-use of old validator keys creates the possibility for long-range sync attacks, in which a newly synced node may behave as though a specific transaction is submitted and finalized when in reality, it was never submitted to the canonical chain in the first place.

    To protect against long-range sync attacks, operations teams should always begin node sync from weak subjectivity checkpoints. These checkpoints are essentially used as genesis blocks, providing a trusted starting point for nodes to sync from. Weak subjectivity checkpoints may be acquired from already-synced nodes or via social processes.

    The special case of L2s

    Examples: Arbitrum, Optimism, StarkNet, Scroll, ZKSync, Polygon zkEVM

    L2 networks are unique in that they don’t have consensus mechanisms in the way a normal blockchain does. In a normal blockchain, the validator set must come to a consensus on the output of a state transition function. In an L2 network, it is the underlying L1 network that is responsible for verifying the state transition function. Ultimately, this means the finality condition for an L2 network is dependent on the finality condition of the underlying L1.

    When an L2 sequencer or prover receives a transaction, it sequences/generates a proof for the transaction, then returns an L2 transaction receipt. Once the sequencer/prover has received enough transactions, it assembles the transactions into a batch that is submitted to the L1 network.

    Figure 7: The flow of a user’s transaction on an L2 network. Notably, sequencers/provers provide users with transaction receipts far in advance of the transaction’s inclusion or finality on the L1.

    For ZK-Rollups, the batch contains a proof representing the execution of every transaction in the batch. The L1 contract verifies the proof, and once the batch transaction is final, all of the L2 transactions included in the proof are final as well.

    For Optimistic Rollups, the batch contains the calldata for every transaction in the batch. The L1 contract does not run any state transition function or verification that the calldata is valid. Instead, Optimistic Rollups use a challenge mechanism to allow L2 nodes to contest an L1 batch. This means a transaction submitted to an Optimistic Rollup may be considered final only once it’s been included in a batch on the L1, the batch and its parents are valid, and the L1 transaction is final.

    Checking for finality

    To determine the finality of an L2 transaction, one must verify that the commitment/proof transaction has both been included on and finalized by the L1. L2 providers often offer convenient RPC methods that off-chain integrators can use to determine the finality of a given L2 transaction.

    Arbitrum Nitro/Optimism

    Both Arbitrum and Optimism nodes implement the Ethereum JSON RPC, including the “finalized” block parameter. As a result, eth_getBlockByNumber(“finalized”, ...) can be used to determine finality.

    StarkNet

    StarkNet’s sequencer provider has a getTransactionStatus() function that reports the transaction’s status in the StarkNet transaction lifecycle. Transactions whose tx_status is ACCEPTED_ON_L1 may be considered final.

    ZkSync Classic

    ZkSync’s v0.2 API has several endpoints that accept finalization parameters.

    • /accounts/{accountIdOrAddress}/{stateType} may have the stateType set to finalized.
    • /blocks/{blockNumber} accepts lastFinalized as the blockNumber parameter.
    • /blocks/{blockNumber}/transactions{?from,limit,direction} accepts lastFinalized as the blockNumber parameter.

    Practicing safe finality

    Like other recent innovations in the blockchain space, provable finality has drastically changed the kinds of security assurances a blockchain can provide. However, developers of off-chain or multi-chain applications must be cognizant of the specific finality requirements of different architectures and, where necessary, use the correct techniques to determine whether transactions are final.

    Older techniques of determining finality, such as block delays, are not adequate for newer architectures, and using incorrect finality criteria may put applications at risk of double-spend attacks.

    If you’re designing a new blockchain or off-chain application and have concerns about finality, please contact us.

    Can you pass the Rekt test?

    One of the biggest challenges for blockchain developers is objectively assessing their security posture and measuring how it progresses. To address this issue, a working group of Web3 security experts, led by Trail of Bits CEO Dan Guido, met earlier this year to create a simple test for profiling the security of blockchain teams. We call it the Rekt Test.

    The Rekt Test is modeled after The Joel Test. Developed 25 years ago by software developer Joel Spolsky, The Joel Test replaced a Byzantine process for determining the maturity and quality of a software team with 12 simple yes-or-no questions. The blockchain industry needs something similar because today’s complex guidance does more to frustrate than to inform.

    The Rekt Test focuses on the simplest, most universally applicable security controls to help teams assess security posture and measure progress. The more an organization can answer “yes” to these questions, the more they can trust the quality of their operations. This is not a definitive checklist for blockchain security teams, but it’s a way to start an informed discussion about important security controls.

    At the Gathering of Minds conference earlier this year, a group of industry leaders were challenged to address the lack of cybersecurity standards in the blockchain ecosystem. One of these discussions was led by Dan Guido, CEO of Trail of Bits. Other participants included Nathan McCauley (Anchorage Digital), Lee Mount (Euler Labs), Shahar Madar (Fireblocks), Mitchell Amador (Immunefi), Nick Shalek (Ribbit Capital), and others. Through their discussions, the Rekt Test was created:

    The Rekt Test

    1. Do you have all actors, roles, and privileges documented?
    2. Do you keep documentation of all the external services, contracts, and oracles you rely on?
    3. Do you have a written and tested incident response plan?
    4. Do you document the best ways to attack your system?
    5. Do you perform identity verification and background checks on all employees?
    6. Do you have a team member with security defined in their role?
    7. Do you require hardware security keys for production systems?
    8. Does your key management system require multiple humans and physical steps?
    9. Do you define key invariants for your system and test them on every commit?
    10. Do you use the best automated tools to discover security issues in your code?
    11. Do you undergo external audits and maintain a vulnerability disclosure or bug bounty program?
    12. Have you considered and mitigated avenues for abusing users of your system?

    The landscape of blockchain technology is diverse, extending beyond blockchains to include decentralized protocols, wallets, custody systems, and more, each with unique security nuances. The subsequent explanations of the Rekt Test questions reflect the consensus of best practices agreed to by this group, and are by no means exhaustive or absolute. The intent of the Rekt Test is not to establish rigid benchmarks but to stimulate meaningful conversations about security in the blockchain community. Thus, consider this interpretation as a stepping stone in this critical dialogue.

    1. Do you have all actors, roles, and privileges documented?

    Comprehensive documentation of all actors, roles, and privileges affecting the blockchain product is crucial, as this clarifies who can access system resources and what actions they are authorized to perform. Actors refer to entities interacting with the system; roles are predefined sets of permissions assigned to actors or groups; and privileges define specific rights and permissions.

    Thorough documentation of these entities facilitates comprehensive testing, allowing developers (and external auditors) to identify security gaps, improper access controls, the degree of decentralization, and potential exposure in specific compromise scenarios. Addressing these issues enhances the overall security and integrity of the system. The documentation also serves as a reference point for auditors to compare the actual access privileges with the documented ones, identify any discrepancies, and investigate potential security risks.

    2. Do you keep documentation of all the external services, contracts, and oracles you rely on?

    Interactions with external smart contracts, oracles, and bridges are fundamental to many key functionalities expected from blockchain applications. A new blockchain application or service may also rely on the assumed security posture of a financial token developed outside of your organization, which increases its complexity and attack surface. As a result, even organizations that integrate the best security procedures into their software development process can fall victim to a destructive security incident.

    It is crucial to document all external services (like cloud hosting services and wallet providers), contracts (like DeFi protocols), and oracles (like pricing information) used by a blockchain system in order to identify risk exposure and mitigate incident impact. Doing so will help you answer the following essential questions:

    • How will we know when an external dependency suffers a security incident?
    • What are the specific conditions under which we declare a security incident?
    • What steps will we take when we detect one?

    Answering these questions will help you be prepared when, inevitably, a security incident affects a dependency outside of your control. You should be able to notice any change, innocuous or not, in a dependency’s output, interface, or assumed program state; assess it for security impact; and take the necessary next steps. This will limit the security impact on your system and help ensure its uninterrupted operation.

    3. Do you have a written and tested incident response plan?

    While security in the blockchain space differs from traditional product security (where more centralized or closed systems may be easier to control), both require an effective incident response plan to help remain resilient in the face of a security incident. The plan should include steps to identify, contain, and remediate the incident through automated and manual procedures. An organization should provide training to ensure that all team members are familiar with the plan, and it should include steps for communicating incidents over internal and out-of-band channels. This plan should be regularly tested to ensure it is up-to-date and effective, especially given how quickly the blockchain security world can change. You should create your own incident response (IR) plan, and can use this Trail of Bits guide as a resource.

    For blockchain systems, it is especially important that IR plans mitigate key person risk by ensuring the organization is not overly reliant on any single individual. The plan should anticipate scenarios where key personnel may be unavailable or coerced, and outline steps to ensure continuity of operations. Developers should consider decentralizing access controls, implementing quorum-based approvals, and documenting procedures so that multiple team members are prepared to respond.

    For blockchain systems, it is especially important that incident response be proactive, not only reactive. The contracts should be designed alongside the creation of the IR plan using strategies like guarded launches to incrementally deploy new code. The developers should consider if they want—or not—pausable features in their contracts, and what part of the protocol should—or should not—be upgradeable or decentralized, as this will influence the team’s capabilities during an incident.

    4. Do you document the best ways to attack your system?

    By constructing a threat model that documents all potential avenues to attack the system, you can understand whether your existing security controls are sufficient to mitigate attacks. The threat model should visually lay out a product’s entire ecosystem, including information from beyond software development, such as applications, systems, networks, distributed systems, hardware, and business processes. It should identify all of the system’s weak points and clearly explain how an attacker can exploit them, incorporating information from relevant real-world attacks to help you avoid making the same mistakes.

    This information will help you understand whether you are concentrating your efforts in the right spots—i.e., whether your current efforts to mitigate attacks are aligned with how and where they are most likely to occur. It will help you understand what a successful attack on your system will look like, and whether you are sufficiently prepared to detect, respond to, and contain it. A good threat model should eliminate surprise and enable your team to deliberately plan mitigations.

    5. Do you perform identity verification and background checks on all employees?

    Pseudonymous development is commonplace in the blockchain industry, but it impedes accountability, contractual enforcement, and engendering trust in and among stakeholders in a blockchain product. Malicious actors can exploit a lack of identity verification and background checks to interfere with a product’s development, steal funds, or cause other serious harm, and institutions will have no or limited means to punish them. In recent years, North Korean hackers have applied to real positions using fake Linkedin accounts and impersonated companies to offer fraudulent positions. These practices have directly led to severe hacks, including Axie Infinity’s $540 million loss.

    As a result, companies must know the identities of and perform background checks on all of their employees, including those who use public pseudonyms. Companies must also reach additional maturity in their access controls and monitoring; for example, they should make prudent decisions surrounding operational security based on an employee’s role, background, and the territory they reside in (i.e., considering local laws and jurisdiction).

    6. Do you have a team member with security defined in their role?

    There needs to be a person on the team who is accountable for ensuring the safety and security of the blockchain system. Threats against blockchain technology evolve rapidly, and even a single security incident can be devastating. Only a dedicated security engineer has the time, knowledge, and skill set necessary to identify threats, triage incidents, and remediate vulnerabilities, which helps instill trust in your product as it develops.

    Ideally, this person will create and oversee a dedicated team with security at the forefront of their job responsibilities, ultimately owning initiatives to get an organization to answer “yes” to other questions on this list. They will oversee cross-departmental efforts, working with developers, administrators, project leads, executives, and others to ensure security practices are included in all aspects of the organization.

    7. Do you require hardware security keys for production systems?

    Credential stuffing, SIM swap attacks, and spear phishing have nearly neutralized the protective capability of passwords and SMS/push two-factor authentication. For high-risk organizations with value at stake, phishing-resistant hardware keys are the only reasonable option. Special hardware keys should be used to access the company’s resources, including email, chat, servers, and software development platforms. Special care should be taken to protect any operation in production that is very difficult or impossible to reverse.

    Using these keys inside your organization is a leading indicator of competent off-chain infrastructure management. Do not be deterred if this seems like a high-volume lift for your IT team. In 2016, Google released a study that showed that implementing these keys was simple, well-received among its 50,000 employees, and strong against malicious attacks. U2F hardware tokens, such as YubiKey and Google Titan, are good choices for hardware keys.

    8. Does your key management system require multiple humans and physical steps?

    If a single individual maintains the keys that control your system, they can unilaterally make changes that have an outsized impact, without a consensus of the relevant stakeholders. And if an attacker compromises their credentials, they can gain full control of core assets.

    Instead, key management should be set up to require a consensus or quorum of multiple people and physical access for important decisions. Multi-person integrity is an effective security policy used in high-risk industries like defense and traditional finance; they protect against compromise via attackers, insider threats (e.g., rogue employees), and coercion, all in one fell swoop. When selecting the trusted set of individuals for a quorum-based setup, it’s crucial to choose those who are both trustworthy and properly incentivized, as including ill-suited or misaligned individuals can undermine the system’s resistance to coercion. By additionally requiring physical key management (e.g., using a physical safe or air-gapped device to store keys), you will significantly reduce the risk of fraud, theft, misuse, or errors by any individual or if any individual’s key or key fragment is compromised.

    Blockchain organizations should employ the use of multi-signature or multi-party computation (MPC) controls and cold storage solutions for, at a minimum, the central wallets that hold most of their assets, or opt to use a qualified custodian, depending on specific regulations and needs. The keys to unlock a multi-signature wallet should be stored on trusted hardware, such as a hardware security module (HSM), secure enclave, or a tamper-resistant hardware wallet.

    It’s imperative that the deployment and configuration of the trusted hardware is done carefully to limit its attack surface. For example, secrets should never be extractable and network connections should be avoided. Organizations should also establish a strict procedure for moving funds based on parameters like thresholds, affected wallets, destination, and key person(s) initiating the transaction.

    9. Do you define key invariants for your system and test them on every commit?

    An invariant is a condition that must remain true throughout the program’s execution. Invariants can target the system as a whole (e.g., no user should have more tokens than the total supply) or target a specific function (e.g., a compute_price function cannot lead to free assets). Understanding and defining invariants helps developers be explicit about the system’s expected behaviors and helps security engineers evaluate whether those behaviors measure up to expectations. This provides a roadmap for security testing and reduces the likelihood of unexpected outcomes and failures.

    Defining invariants starts with documenting the assumptions made about the system in plain English. These invariants should cover a breadth of functional and cryptographic properties and their valid states, state transitions, and high-level behaviors. Well-specified systems may have hundreds of properties: you should focus on the most important ones first and continuously work to improve their depth of coverage. To ensure that the code follows the invariants, they must be tested with an automated tool (such as a fuzzer or a tool based on formal methods) throughout the development process. 

    10. Do you use the best automated tools for discovering security issues in your code?

    Automated security tools are a baseline requirement in a successful security strategy. Fully automated tools, such as static analyzers, automatically find common mistakes and require low maintenance, while semi-automated tools, like fuzzers, allow developers to go one step further and check for logical issues. Many such tools are available, but we recommend using those that are actively used by top-tier security engineers, for which a proven track record of discovered bugs is available.

    Trail of Bits’ smart contract security tools use state-of-the-art technology and can be integrated into your CI systems and the edit/test/debug cycle. They include Echidna, a smart contract fuzzer for Solidity smart contracts, and Slither, a static analyzer for Solidity smart contracts. Automating the use of these tools during development and testing helps developers catch critical security bugs before deployment.

    11. Do you undergo external audits and maintain a vulnerability disclosure or bug bounty program?

    To identify vulnerabilities in blockchain code, it isn’t enough to rely on internal security teams. Instead, organizations must work with external auditors who possess in-depth knowledge of modern blockchain technology, spanning low-level implementations, financial products and their underlying assumptions, and the libraries, services, bridges, and other infrastructure that power modern applications. (Websites that track blockchain security incidents are filled with companies that did not seek external guidance for sometimes complex changes.)

    Security auditors help to identify vulnerabilities and provide advice for restructuring your development and testing workflow so these vulnerabilities do not come back. When looking for an audit, it’s important to clarify which components are under review, which are excluded, and the level of effort that should be applied, including through the use of tooling. By understanding the benefits and limitations of an audit, an organization can focus on additional areas needed for improvements once the audit has concluded.

    Additionally, a vulnerability disclosure or bug bounty program can enhance your security posture by providing a publicly accessible option for users or researchers to contact you if they uncover a bug. By establishing these programs, organizations show a willingness to engage with independent bug hunters—and without them, they may instead publicly disclose the bugs on social media or even exploit them for their own gain. While these programs offer many benefits, it is important to consider their limitations and pitfalls. For example, bug bounty hunters will not provide recommendations for improving the security maturity of the system nor for reducing the likelihood of bugs in the long term. In addition, your team will still be responsible for triaging bug submissions, which can require constant dedicated resources.

    12. Have you considered and mitigated avenues for abusing users of your system?

    Many attacks against blockchain, such as phishing, Twitter/Discord scams, and “pig butchering,” attempt to fool users into taking irreparable actions while using your products. Even if an organization has the most expertly designed security system to protect itself, its own users may still be vulnerable. For example, blockchain applications often rely on cryptographic signatures that increase the likelihood of phishing attempts. Developers should consider making the signatures easily identifiable (for example, with EIP-712) and should create and promote guidance for their users to minimize the risk of abuse.

    To avoid such attacks, an organization’s security strategy should include abusability testing, where your team considers how attackers can inflict social, psychological, and physical harm. Understanding the risks of significant financial or societal harms will help your team to evaluate necessary processes and mitigations. For example, if your protocol’s users include high-impact stakeholders, such as retirement funds, creating an assurance fund based on the protocol’s fees may help to make the users whole in case of compromise.

    Don’t get rekt

    These 12 controls are not the only actions that can determine your security posture, but we’re confident that they will enhance every developer’s software and operational security, even as blockchain technology rapidly innovates. This test should not serve as a one-time exercise; these questions have lasting value and should give organizations a roadmap as they continue to grow and develop new products. Answering “yes” to these questions doesn’t mean you will completely avoid a security incident, but it can empower you and your team to steer clear of the worst label in the industry: getting rekt.

    Use our suite of eBPF libraries

    By Artem Dinaburg

    Trail of Bits has developed a suite of open-source libraries designed to streamline the creation and deployment of eBPF applications. These libraries facilitate efficient process and network event monitoring, function tracing, kernel debug symbol parsing, and eBPF code generation.

    Previously, deploying portable, dependency-free eBPF applications posed significant challenges due to Linux kernel version disparities and the need for external tools for C-to-eBPF bytecode translation. We’ve addressed these issues with our innovative libraries, which use the latest eBPF and Linux kernel features to reduce external dependencies. These tools, ideal for creating on-machine agents and enabling cloud-native monitoring, are actively maintained and compatible with a variety of Linux distributions and kernel releases. Some are even integral to the functionality of osquery, the renowned endpoint visibility framework.

    Our eBPF libraries

    The libraries in this suite are linuxevents, ebpfpub, btfparse, and ebpf-common. Together they can be used to develop streamlined event monitoring with a high degree of accuracy and efficiency. Their applications range from network event monitoring, function tracing, and kernel debug symbol parsing to assisting in generating and using eBPF code.

    linuxevents: A container-aware library for process monitoring with no runtime dependencies

    The linuxevents library showcases how eBPF can monitor events without requiring accurate kernel headers or other external dependencies. No more shipping kernel headers, multiple copies of precompiled eBPF bytecode, or dependencies on BCC! The linuxevents library supports runtime code generation to create custom probes at runtime, not just during build. It is also much faster than traditional system-call-based hooking, an essential feature when monitoring events from multiple containers on a single machine. How does linuxevents do this?

    First, linuxevents uses the Linux kernel’s BTF debugging data (via our btfparse library) to accurately identify function prototypes and kernel data structures. This allows linuxevents to automatically adjust to variances in data structure layout and to hook arbitrary non-public symbols in a way that greatly simplifies tracing.

    This approach is faster than traditional system call based hooking not only because it has to hook fewer things (sched_process_exec vs execve, execveat, etc.) but also because it can avoid expensive correlations. For example, to trace which program on disk is executed via execve, one would normally have to correlate a file descriptor passed to execve with an open call and multiple chdir calls to get the full path of a program. The correlation is computationally expensive, especially on a machine with multiple active containers. The linuxevents library uses an accurate kernel data structure representation to hook just one function and simply extract the path from the kernel’s vfs layer.

    A recording of the linuxevents library being used as a part of the execsnoop example that comes with the library

    The linuxevents library is still a proof of concept; it is in use by osquery as a toggleable experiment. The library also has a canonical example of tracing executed processes with cross-container visibility.

    ebpfpub: A function-tracing library for Linux

    The ebpfpub library allows for monitoring system calls across multiple Linux kernel versions while relying on minimal external runtime dependencies. In ebpfpub, eBPF probes are autogenerated from function prototypes defined via a simple custom language, which can be created from tracepoint descriptors. This approach required proper headers for the running kernel and it came with performance penalties, such as the need to match file descriptors with system calls.

    Depending on the desired target event, ebpfpub can use either kernel tracepoints, kprobes, or uprobes as the underlying tracing mechanism. The library includes the following examples:

    • execsnoop: Shows how to use Linux kernel tracepoints to detect program execution via execve()
    • kprobe_execsnoop: Like execsnoop, but uses a different hooking mechanism (kprobes instead of tracepoints)
    • readline_trace: Uses uprobes to hook the user-mode readline library, which can enable use cases such as monitoring whenever a live shell is used on a machine
    • sockevents: An example of how to trace sockets through a series of connect/accept/bind calls that establish connectivity to a remote machine
    • systemd_resolved: Shows how to use uprobes to hook into systemd’s DNS service (systemd-resolved), which will show in real time the domains being looked up by your local machine

    The ebpfpub library is currently used by osquery to capture process and socket events by tracing executed system calls. While ebpfpub is still maintained and useful in specific circumstances (like the need to support older kernels and use runtime code generation), new projects should use the linuxevents approach instead.

    btfparse: A C++ library that parses kernel debug symbols in BTF format

    BTF, or the Binary Type Format, is a compact binary format for representing type information in the Linux kernel. BTF stores data such as structures, unions, enumerations, and typedefs. Debuggers and other tools can use BTF data to enable richer debugging features by understanding complex C types and expressions. BTF was introduced in Linux 4.20 and is generated from source code and traditional debugging information like DWARF. BTF is more compact than DWARF, and it improves the debugging experience by conveying more semantic type information than was previously available. The standardized BTF format also allows new debugging tools to leverage type data across compilers, enabling more consistent quality of introspection across languages.

    The btfparse library allows you to read BTF data in your C++ projects and generate header files directly in memory without any external application. The library also comes with a tool, called btf-dump, that serves both as an example of using btfparse and as a standalone tool that can dump BTF data present in a Linux kernel image.

    ebpf-common: A C++ library to help write new eBPF-based tools

    The ebpf-common library is a set of utilities that assist with generating, loading, and using eBPF code. It is the common substrate that underpins all of our eBPF-related tooling. Use epbf-common to create your own runtime, eBPF-based tools!

    The ebpf-common library’s main job is to compile C code to eBPF bytecode and to provide helpful abstractions that make writing eBPF hooks easier. Here are some of the features ebpf-common provides:

    • It uses LLVM and clang as libraries that write to in-memory scratch buffers.
    • It includes abstractions to make accessing eBPF data structures (hash maps, arrays, ring buffers, etc.) simple. These data structures are used to exchange data between eBPF and your application.
    • It includes abstractions to create and read perf outputs, another way eBPF can communicate with the tracing application.
    • It allows for the management of events (like kprobes, uprobes, and tracepoints) that trigger the execution of eBPF programs.
    • Finally, it includes functions that implement eBPF helpers and related functionality via LLVM.

    The ebpf-common library is used as the core of all of our other eBPF tools, which serve as library clients and examples of use cases for ebpf-common for your applications. Refer to our blog post All your tracing are belong to BPF for additional guidance and examples for how to use ebpf-common.

    Our eBPF tools

    ebpfault: A Linux system call fault injector built on top of eBPF

    ebpfault is a system-wide fault injector that does not require risky kernel drivers that could crash the system. It can start a specific program, target running processes, or target all processes except those on a specific list. A simple configuration file in JSON format lets you configure faults by using the syscall name, the probability of injecting a fault, and the error code that should be returned.

    A recording of ebpfault running against the htop process and causing faults via a specific configuration

    BPF deep-dives and talks

    Most of the material available online is geared toward using the command-line sample tools that demonstrate eBPF, which work mostly as standalone demonstrations and not as reusable libraries. We wanted to fill in the gaps for developers and provide a step-by-step guide on how to actually integrate eBPF from scratch from the point of view of a developer writing a tracing tool. The documentation focuses on runtime code generation using the LLVM libraries.

    The All your tracing belong to BPF blog post and our companion code guide show how to use epbf-common to create a tool that uses eBPF to count system calls, with each example increasing in complexity, starting from simple counting, to using maps to store data, to finally using perf events for outputs.

    Monitoring Linux events is a talk by Alessandro Gario about using eBPF for event monitoring. Alessandro describes how to dynamically decide on what to monitor and to generate your own eBPF bytecode directly from C++. He touches on some of the intricacies of eBPF maps, perf events, and practical considerations for using our eBPF tools and libraries.

    eBPF public contributions

    The world of eBPF continues to expand and find applications in various domains. We have explored eBPF as it relates to interesting tasks beyond tracing and performance monitoring, such as improving CI/CD for eBPF bytecode, writing an eBPF-to-ARM64 JIT compiler for the Solana platform, and improving the experience of building eBPF projects on Windows.

    ebpf-verifier: Sometimes it is necessary to bundle prebuilt eBPF programs. The Linux kernel “verifies” eBPF programs at load time and rejects any that it deems unsafe. Bundling eBPF bytecode is a CI/CD nightmare because every kernel’s verification is ever so slightly different. ebpf-verifier aims to eliminate that nightmare by executing the eBPF verifier outside of the running kernel and opens the door to the possibility of testing eBPF programs across different kernel versions.

    Solana eBPF-to-ARM64 JIT compiler: eBPF makes an appearance in many surprising places! The Solana blockchain uses an eBPF virtual machine to run its smart contracts, and it uses a JIT compiler to compile the eBPF bytecode to native architectures. Trail of Bits ported the Solana eBPF JIT compiler to ARM64 to allow Solana applications to natively run on the now very popular ARM64 platforms like Apple Silicon.

    Adding CMake support to eBPF for Windows: eBPF also works on Windows! To make Windows development easier, we ported the prior Visual Studio–based build system to CMake. The improvements include better handling of transitive dependencies and properties, better packaging, and enhanced build settings for a more efficient development experience.

    Conclusion

    We’ve used eBPF to provide rapid, high-quality monitoring data for system instrumentation agents like osquery. Our intention is that the frameworks and tools we’ve created will assist developers in integrating eBPF into their applications more seamlessly. eBPF is a useful technology with a bright future in a variety of fields, including increasingly in cloud-native monitoring and observation.

    We plan to share more of our lessons learned from eBPF tool development in the near future, and we hope to apply some of these lessons to the problems of cloud-native monitoring and observability.

    A mistake in the bulletproofs paper could have led to the theft of millions of dollars

    By Jim Miller

    We discovered a critical vulnerability in Incognito Chain that would allow an attacker to mint arbitrary tokens and drain user funds. Incognito offers confidential transactions through zero-knowledge proofs, so an attacker could have stolen millions of dollars of shielded funds without ever being detected or identified. The vulnerability stemmed from an insecure implementation of the Fiat-Shamir transformation of Incognito’s bulletproofs. 

    This is not the first instance of this vulnerability; last year we disclosed a series of these same types of issues, which we dubbed Frozen Heart vulnerabilities. These vulnerabilities, as we detailed in our earlier blog series, resulted from a mistake in the original bulletproofs paper. The mistake was in the paper for over four years before it was corrected last year in response to our disclosure.

    Since posting that blog series, Trail of Bits has continued research with Paul Grubbs (University of Michigan) and Quang Dao (Carnegie Mellon University) to review more codebases and proof systems for this issue, resulting in a paper recently accepted for publication at IEEE S&P 2023. The vulnerability in Incognito Chain was identified during this research.

    After discovering the issue, we informed multiple members of the Incognito Chain team, who patched the vulnerability and released a new version of their privacy protocol. This new version has since been adopted by their validators, effectively patching the vulnerability and, to our knowledge, securing all funds.

    Understanding bulletproofs, Fiat-Shamir, Frozen Heart, and confidential transactions

    Bulletproofs are a special kind of zero-knowledge proof, also known as a range proof. They permit a prover to validate that an encrypted value lies within a specific range. These proofs serve as a crucial foundation for confidential transactions.

    Fiat-Shamir is a transformation we detailed in a series of previous blog posts. These posts delve into how it can be incorrectly implemented and exploited across several zero-knowledge proof systems.

    Frozen Heart vulnerabilities break the security of zero-knowledge proofs. When exploited, an attacker can forge zero-knowledge proofs, tricking the verifier into accepting incorrect proofs. This compromise can have severe implications for the protocol’s security.

    Confidential transactions are a significant feature of Incognito Chain. Similar to Monero, these transactions hide the amount being transferred and the identities of both the sender and receiver.

    For a more detailed examination of the Incognito Chain protocol and the cryptographic primitives it contains, you can check out their forum. A brief explanation is that coins are the encryption of their underlying value (technically, a commitment scheme instead of an encryption scheme). The identities of the sender and receiver are concealed by using a ring signature, which proves the transaction came from a group (or ring) of keys, one of which is controlled by the sender. To hide the receiver’s identity, the protocol relies on stealth or one-time addresses; the sender in the transaction generates one-time public keys that only the receiver can access.

    A transaction includes a set of input coins, output coins, and a transaction fee. For a transaction to be valid, the sum of the inputs should equal the sum of the outputs plus the transaction fee. However, the values of the coins are encrypted. This challenge is overcome through the protocol’s reliance on homomorphic encryption, which allows the values of inputs and outputs to be added and subtracted while remaining encrypted. This balance check can be performed homomorphically, allowing the verified transaction to validate that the sums are equal without knowing their actual values.

    There is a caveat to this balance check. Homomorphic encryption schemes ensure sums are equal modulo some value, typically a 256-bit or larger prime value (the group order). An adept attacker could manipulate this fact to execute an attack that functions essentially as an integer overflow, where the overflow occurs modulo the large prime. Consequently, instead of the transaction not generating additional funds (as a secure protocol requires), an attacker could mint extra funds—a substantial amount equivalent to a 256-bit prime number.

    The protocol designers anticipated this issue, employing bulletproofs as a safeguard. Bulletproofs, as range proofs, confirm that all input and output values are less than a specific maximum. Thus, if the group order is a 256-bit prime, we can avert this overflow attack by using bulletproofs that validate that the values of each coin are at most 264. An overflow attack would then be unattainable when using a reasonable number of inputs and outputs.

    Vulnerability details

    Bulletproofs are an essential part of confidential transactions. They limit the underlying value of the privacy coins, thus safeguarding the system from attackers minting money illicitly. Like most zero-knowledge proofs, bulletproofs use the Fiat-Shamir transformation to be noninteractive. Our prior blog post highlights how an error in the original bulletproofs paper led to insecure implementations of the Fiat-Shamir protocol. If this mistake is implemented as instructed, an attacker can forge bulletproofs for values outside the range.

    The original bulletproofs forgery results in a coin that has a uniformly random value. Although this could complicate exploitation since you can’t control the value exactly, it’s not entirely impossible. As we elaborate in the Attacking Mimblewimble section of our recently published paper, Wagner’s k-sum algorithm could be employed to engineer such an exploit.

    However, Incognito Chain uses a variant of bulletproofs known as aggregate bulletproofs. As the name suggests, this variant aggregates multiple bulletproofs into a single proof, allowing more efficient verification. When this variant of bulletproofs is vulnerable to Frozen Heart, the severity escalates notably because it grants an attacker the liberty to select arbitrary values for the coins instead of being confined to random values. With this level of control over these values, an attacker can effortlessly solve the balance equation, thereby generating free money. What amplifies the concern surrounding this vulnerability is its target: confidential transactions, which inherently hide most information from external observers, make practical detection of exploitation a formidable challenge.

    Coordinated disclosure

    We notified multiple members of the Incognito Chain team of this vulnerability on April 25, 2023. They responded swiftly, confirmed the issue, and started working on a fix. On April 26, 2023, Incognito Chain submitted this patch and other commits that fixed the bulletproofs implementation. 

    The Incognito Chain team released this patch as part of a new version (v3) of their privacy protocol to prevent future exploits. However, this initial patch inadvertently introduced a bug that caused a temporary network outage. The team detailed this issue on their forum, the problem has been resolved, and the network is operational once again.

    We appreciate the quick and efficient response of the Incognito Chain team in addressing this issue.

    How AI will affect cybersecurity: What we told the CFTC

    Dan Guido, CEO

    The second meeting of the Commodity Futures Trading Commission’s Technology Advisory Committee (TAC) on July 18 focused on the effects of AI on the financial sector. During the meeting, I explained that AI has the potential to fundamentally change the balance between cyber offense and defense, and that we need security-focused benchmarks and taxonomies to properly assess AI capabilities and risks.

    • The widespread availability of capable AI models presents new offensive opportunities that defenders must now account for. AI will make certain attacks dramatically easier, upsetting the equilibrium of offense and defense. We must reevaluate our defenses given this new reality.
    • Many think AI is either magical or useless, but the truth lies between these extremes. AI augments human capabilities; it does not wholly replace human judgment and expertise. One key question is: can a mid-level practitioner operate at an expert level with the help of AI? Our experience suggests yes.
      • AI models can do many helpful things: decompile code into high-level languages, identify and trigger bugs, and write scripts to launch exploits. But to leverage it effectively, we must ask the right questions (e.g., with knowledge of the subject matter and prompt engineering techniques) and evaluate progress correctly (is AI better than state-of-the-art techniques)?
      • It’s also necessary to choose the right problems. AI is better for problems that require breadth of knowledge and where mistakes are acceptable (e.g., document this function, write a phishing email). It’s not great at problems that require mastery and correctness (e.g., find and exploit this iOS 0-day).
    • Bug bounties, phishing defenses, antivirus, IDS, and attribution will be among the first fields impacted as AI confers a greater advantage to attackers in the near term. For example, AI can mass produce tailored phishing messages, for every target, in their native language, and without errors. We can’t just regulate these problems away; alignment and attempts to restrict model availability won’t work, since impressively capable open-source models are already here.
    • What’s needed now is a systematic measurement of these models’ capabilities that focuses on cybersecurity, not programming. We need benchmarks that let us compare AI versus existing state-of-the-art tools and human experts, and taxonomies that map advancements to opportunities and risks.

    The full video is available here:

    Finally, I am honored to have been named the co-chair of the Subcommittee on Cybersecurity. I look forward to continuing our work with the committee. We will continue studying the risks and opportunities of AI, supply chain security, and authentication technology in the finance industry.

    Read our prior coverage of the CFTC TAC’s first meeting, which focused on blockchain risks. For our work on AI-enabled cybersecurity, see the links below: