Security flaws in an SSO plugin for Caddy

By Maciej Domanski, Travis Peters, and David Pokora

We identified 10 security vulnerabilities within the caddy-security plugin for the Caddy web server that could enable a variety of high-severity attacks in web applications, including client-side code execution, OAuth replay attacks, and unauthorized access to resources.

During our evaluation, Caddy was deployed as a reverse proxy to provide access to several of our internal services. We explored a plugin configuration that would allow us to handle authentication and authorization with our Google SSO so that we didn’t have to implement this on a per-app basis.

In this blog post, we will briefly explore the security vulnerabilities we identified in the caddy-security plugin and discuss their potential impact. As with our typical security assessments, for each issue we identified, we present a recommendation for immediately fixing the issue, as well as long-term, more strategic advice for preventing similar issues and improving the overall security posture.

As security experts, our goal is not only to identify vulnerabilities in specific software but also to contribute to the larger community by sharing our recommendations for fixing these problems. We believe that these recommendations can help developers overcome similar challenges in other SSO systems and improve their security.

Caddy Background

Caddy (a.k.a. Caddy Server or Caddy 2) is a modern, open-source web server written in Golang that is designed to be easy to use and highly configurable. Caddy is built to streamline the process of hosting web applications while prioritizing security and performance. It aims to reduce the complexity associated with configuring and deploying web servers.

The caddy-security plugin is a middleware plugin for the Caddy web server. It provides various security-related functionalities to enhance the overall security posture of web applications. Some of the key features offered by the caddy-security plugin include an authentication plugin for implementing Form-Based, Basic, Local, LDAP, OpenID Connect, OAuth 2.0, SAML Authentication, and an authorization plugin for HTTP request authorization based on JWT/PASETO tokens.


Issue #1: Reflected Cross-Site Scripting (XSS)

Severity: High

Reflected XSS occurs when an application includes untrusted data in the HTML response sent to the user’s browser. In this case, the provided /admin%22%3E%3Cscript%3Ealert(document.domain)%3C/script%3E/admin/login or /settings/mfa/delete/<img%20src=x%20onerror=alert(document.domain)> API calls trigger an alert. An attacker can exploit this vulnerability to execute arbitrary JavaScript code within the target user’s browser, potentially leading to further attacks such as session hijacking.

To immediately address this issue, strategically treat all string values as potentially untrustworthy, regardless of their source, and escape them properly (using the safehtml/template package that generates output-safe HTML).

In addition to that remediation, we also suggest a few different ways to improve defense in depth:

  • Extend unit tests with potentially malicious XSS payloads. Refer to the Cross-site scripting (XSS) cheat sheet for various attack vectors.
  • Consider using the Active Scanner from Burp Suite Professional in a testing environment for all API calls. Additionally, use the scanning with a live task strategy to have underlying requests scanned automatically when interacting with the web interface.
  • Expand the caddy-security documentation to promote security headers—especially the Content Security Policy (CSP) header that controls which resources can be loaded by the browser, limiting the impact of potential XSS attacks.

Issue #2: Insecure Randomness

Severity: High

The caddy-security plugin uses the math/rand Golang library with a seed based on the Unix timestamp to generate strings for three security-critical contexts in the application, which could possibly be predicted via a brute-force search. Attackers could use the potentially predictable nonce value used for authentication purposes in the OAuth flow to conduct OAuth replay attacks. In addition, insecure randomness is used while generating multifactor authentication (MFA) secrets and creating API keys in the database package.

To immediately mitigate this vulnerability, use a cryptographically secure random number generator for generating the random strings. Golang’s library crypto/rand is designed for secure random number generation.

In addition to that fix, we recommend considering the following long-term recommendations:

  • Review the application for other instances where the math/rand package is used for secure context. Create secure wrapping functions and use them throughout the code to serve a cryptographically secure string with the requested length.
  • Avoid duplicating code. Having a single function, such as secureRandomString, rather than multiple duplicate functions makes it easier to audit and verify the system’s security. It also prevents future changes to the codebase from reintroducing issues.
  • Implement Semgrep in the CI/CD. The math-random-used Semgrep rule will catch instances where math/rand is used. Refer to our Testing Handbook on Semgrep for more information.
  • Read textbooks such as Real World Cryptography, as it is a great resource for practical cryptographic considerations.

Issue #3: IP Spoofing via X-Forwarded-For Header

Severity: Medium

By manipulating the X-Forwarded-For header, an attacker can spoof an IP address used in the user identity module (/whoami API endpoint). This could lead to unauthorized access if the system trusts this spoofed IP address.

To resolve this vulnerability, reimplement the application to not rely on user-provided headers when obtaining a user’s IP address. If user-provided headers are required (e.g., X-Forwarded-For for logging purposes), ensure the header is properly validated (i.e., the value is consistent with IP address format through regular expression) or sanitized (to avoid CRLF log injection attacks, for example).

In addition to this immediate fix, we recommend considering these long-term recommendations:

  • Implement appropriate checks for potential IP spoofing and X- headers on the unit testing level. Consider other headers that can rewrite IP sources.
  • Cover the IP spoofing scenarios and user-provided header processing in Golang’s native fuzz tests.
  • Use the dynamic testing approach with Burp Suite Professional and the Param Miner extension to identify the processing of hidden headers.
  • Expand the caddy-security documentation to increase user awareness of this type of threat; show an example of misconfiguration, how to resolve, and how to test it.

Issue #4: Referer-Based Header XSS

Severity: Medium

An XSS vulnerability can be triggered by rewriting the Referer header. Although the Referer header is sanitized by escaping some characters that can allow XSS (e.g., [&], [<], [>], ["], [']), it does not account for the attack based on the JavaScript URL scheme (e.g., javascript:alert(document.domain)// payload). Exploiting this vulnerability may not be trivial, but it could lead to the execution of malicious scripts in the context of the target user’s browser, compromising user sessions.

The mitigation for this issue is identical to issue #1.

Issue #5: Open Redirection Vulnerability

Severity: Medium

When a logged-in user clicks on a specially crafted link with a redirect_url parameter, the user can be redirected to an external website. The user must take an action, such as clicking on a portal button or using the browser’s back button, to trigger the redirection. This could lead to phishing attacks, where an attacker tricks users into visiting a malicious website by crafting a convincing URL.

To mitigate this vulnerability, perform proper redirect_url parameter validation to ensure that the redirection URLs are allowed only within the same domain or from trusted sources.

In addition, we also recommend the following long-term fixes:

  • Implement robust unit tests with different bypassing scenarios of redirect_url parameter validation. Refer to the potential URL Format Bypasses. Keep in mind that different components can use different URI parsers, which can lead to parsing confusion.
  • Use Burp Suite Professional with a scanner with both these settings enabled:
    • Audit coverage – maximum: to use the most extensive set of payload variations and insertion point options
    • Audit coverage – thorough: to try more payload variations

Issue #6: X-Forwarded-Host Header Manipulation

Severity: Medium

The caddy-security plugin processes the X-Forwarded-Host header, which could lead to various security vulnerabilities (web cache poisoning, business logic flaws, routing-based server-side request forgery [SSRF], and classic server-side vulnerabilities). Additionally, the caddy-security plugin generates QR codes based on this header, which extends the attack surface.

To mitigate this issue, do not rely on the Host and X-Forwarded-Host headers in the caddy-security plugin logic. Instead, use the current domain manually specified in the configuration file to generate a QR code.

In addition, we recommend the following:

  • Use Burp Suite Professional with the Param Miner extension to identify the processing of hidden headers.
  • Extend the caddy-security documentation to increase user awareness of the HTTP Host header attacks.

Issue #7: X-Forwarded-Proto Header Manipulation

Severity: Low

The processing of the X-Forwarded-Proto header results in redirection to the injected protocol. While this scenario may have limited impact, improper handling of such headers could result in unpredictable security risks, such as bypass of security mechanisms or confusion in handling TLS.

To address this issue, do not rely on the X-Forwarded-Proto header. If it is required, validate the value of the X-Forwarded-Proto header against an allowlist of accepted protocols (e.g., HTTP/HTTPS) and reject unexpected values.

In addition, consider the long-term recommendations from issue #3.

Issue #8: 2FA Bypass by Brute-Forcing Verification Codes

Severity: Low

The current implementation of the application’s two-factor authentication (2FA) lacks sufficient protection against brute-force attacks. Although the application blocks the user after several failed attempts to provide 2FA codes, attackers can bypass this blocking mechanism by automating the application’s full multistep 2FA process.

To address this issue effectively, enforce a minimum six-digit code length in the MFA configuration. Additionally, to reduce the risk of automated brute-forcing, implement an account locking mechanism that triggers after a specified number of invalid 2FA code attempts. Finally, enforce reauthentication for critical actions involving sensitive account information or security settings. For actions such as changing passwords or disabling 2FA, users should be required to reauthenticate, either with their password or a 2FA token. An exception can be made for reauthentication if the user has logged in within the last 10 minutes. Check out Getting 2FA Right in 2019 at the Trail of Bits Blog for more information.

Issue #9: Lack of User Session Invalidation on Logout

Severity: Low

The caddy-security plugin lacks proper user session invalidation upon clicking the “Sign Out” button; user sessions remain valid even after requests are sent to /logout and /oauth2/google/logout. Attackers who gain access to an active but supposedly logged-out session can perform unauthorized actions on behalf of the user.

To address this issue, review the sign-out process to identify the cause of the unexpected behavior. Ensure that the /oauth2/google/logout endpoint correctly terminates the user session and invalidates the associated tokens.

For more defense in depth, use the OWASP Application Security Verification Standard (V3 Session Management) to check whether the implementation handles sessions securely.

Issue #10: Multiple Panics when Parsing Caddyfile

Severity: Low

Multiple parsing functions do not validate whether their input values are nil before attempting to access elements, which can lead to a panic (index out of range). Panics during the parsing of a Caddyfile may not be perceived as immediate vulnerabilities, but they could indicate improperly enforced security controls (e.g., insufficient data validation), which could lead to issues in other code paths.

To address these issues, integrate nil checks for input values before element access across all relevant functions.

To prevent similar issues of this type, add Golang’s native fuzz tests for Caddyfile parsing functions.

Golang Security for the Community

We love writing and reviewing Golang codebases at Trail of Bits. Indeed, we are constantly working on Golang-related (Semgrep) resources, rules, and blog posts and look forward to any opportunity to take on pet audits (like this) and client projects where we examine Golang codebases.

Our aim in publishing our findings is to help protect others who may consider implementing a solution similar to the one we explored and to help them make informed decisions about their security infrastructure.

If you’re actively implementing a codebase in Golang or have questions, concerns, or other recommendations on open-source software you think we should look at, please contact us.

Coordinated Disclosure Timeline

As part of the disclosure process, we reported the vulnerabilities to the caddy-security plugin maintainers first. The timeline of disclosure is provided below:

    • August 7, 2023: We reported our findings to the caddy-security plugin maintainers.
    • August 23, 2023: The caddy-security plugin maintainers confirmed that there were no near-term plans to act on the reported vulnerabilities.
    • September 18, 2023: The disclosure blog post was released and issues were filed with the original project repository.

Holy Macroni! A recipe for progressive language enhancement

By Brent Pappas

Despite its use for refactoring and static analysis tooling, Clang has a massive shortcoming: the Clang AST does not provide provenance information about which CPP macro expansions a given AST node is expanded from; nor does it lower macro expansions down to LLVM Intermediate Representation (IR) code. This makes the construction of macro-aware static analyses and transformations exceedingly difficult and an ongoing area of research.1 Struggle no more, however, because this summer at Trail of Bits, I created Macroni to make it easy to create macro-aware static analyses.

Macroni allows developers to define the syntax for new C language constructs with macros and provide the semantics for these constructs with MLIR. Macroni uses VAST to lower C code down to MLIR and uses PASTA to obtain macro-to-AST provenance information to lower macros down to MLIR as well. Developers can then define custom MLIR converters to transform Macroni’s output into domain-specific MLIR dialects for more nuanced analyses. In this post, I will present several examples of how to use Macroni to augment C with safer language constructs and build C safety analyses.

Stronger typedefs

C typedefs are useful for giving semantically meaningful names to lower-level types; however, C compilers don’t use these names during type checking and perform their type checking only on the lower-level types instead. This can manifest in a simple form of type-confusion bug when the semantic types represent different formats or measures, such as the following:

typedef double fahrenheit;
typedef double celsius;
fahrenheit F;
celsius C;
F = C; // No compiler error or warning

Figure 1: C type checking considers only typedef’s underlying types.

The above code successfully type checks, but the semantic difference between the types fahrenheit and celsius should not be ignored, as they represent values in different temperature scales. There is no way to enforce this sort of strong typing using C typedefs alone.

With Macroni, we can use macros to define the syntax for strong typedefs and MLIR to implement custom type checking for them. Here’s an example of using macros to define strong typedefs representing temperatures in degrees Fahrenheit and Celsius:

#define STRONG_TYPEDEF(name) name
typedef double STRONG_TYPEDEF(fahrenheit);
typedef double STRONG_TYPEDEF(celsius);

Figure 2: Using macros to define syntax for strong C typedefs

Wrapping a typedef name with the STRONG_TYPEDEF() macro allows Macroni to identify typedefs whose names were expanded from invocations of STRONG_TYPEDEF() and convert them into the types of a custom MLIR dialect (e.g., temp), like so:

%0 = hl.var "F" : !hl.lvalue<!temp.fahrenheit>
%1 = hl.var "C" : !hl.lvalue<!temp.celsius>
%2 = hl.ref %1 : !hl.lvalue<!temp.celsius>
%3 = hl.ref %0 : !hl.lvalue<!temp.fahrenheit>
%4 = hl.implicit_cast %3 LValueToRValue : !hl.lvalue<!temp.fahrenheit> -> !temp.fahrenheit
%5 = hl.assign %4 to %2 : !temp.fahrenheit, !hl.lvalue<!temp.celsius> -> !temp.celsius

Figure 3: Macroni enables us to lower typedefs to MLIR types and enforce strict typing.

By integrating these macro-attributed typedefs into the type system, we can now define custom type-checking rules for them. For instance, we could enforce strict type checking for operations between temperature values so that the above program would fail to type check. We could also add custom type-casting logic for temperature values so that casting a temperature value in one scale to a different scale implicitly inserts instructions to convert between them.

The reason for using macros to add the strong typedef syntax is that macros are both backwards-compatible and portable. While we could identify our custom types with Clang by annotating our typedefs using GNU’s or Clang’s attribute syntax, we cannot guarantee annotate()‘s availability across platforms and compilers, whereas we can make strong assumptions about the presence of a C preprocessor.

Now, you might be thinking: C already has a form of strong typedef called struct. So we could also enforce stricter type checking by converting our typedef types into structs (e.g., struct fahrenheit { double value; }), but this would alter both the type’s API and ABI, breaking existing client code and backwards-compatibility. If we were to change our typedefs into structs, a compiler may produce completely different assembly code. For example, consider the following function definition:

fahrenheit convert(celsius temp) { return (temp * 9.0 / 5.0) + 32.0; }

Figure 4: A definition for a Celsius-to-Fahrenheit conversion function

If we define our strong typedefs using macro-attributed typedefs, then Clang emits the following LLVM IR for the convert(25) call. The LLVM IR representation of the convert function matches up with its C counterpart, accepting a single double-typed argument and returning a double-typed value.

tail call double @convert(double noundef 2.500000e+01)

Figure 5: LLVM IR for convert(25), with macro-attributed typedefs used to define strong typedefs

Contrast this to the IR that Clang produces when we define our strong typedefs using structs. The function call now accepts four arguments instead of one. That first ptr argument represents the location where convert will store the return value. Imagine what would happen if client code called this new version of convert according to the calling convention of the original.

call void @convert(ptr nonnull sret(%struct.fahrenheit) align 8 %1,
                   i32 undef, i32 inreg 1077477376, i32 inreg 0)

Figure 6: LLVM IR for convert(25), with structs used to define strong typedefs

Weak typedefs that ought to be strong are pervasive in C codebases, including critical infrastructure like libc and the Linux kernel. Preserving API- and ABI-compatibility is essential if you want to add strong type checking to a standard type such as time_t. If you wrapped time_t in a struct (e.g., struct strict_time_t { time_t t; }) to provide strong type checking, then not only would all APIs accessing time_t-typed values need to change, but so would the ABIs of those usage sites. Clients who were already using bare time_t values would need to painstakingly change all the places in their code that use time_t to instead use your struct to activate stronger type checking. On the other hand, if you used a macro-attributed typedef to alias the original time_t (i.e., typedef time_t STRONG_TYPEDEF(time_t)), then time_t‘s API and ABI would remain consistent, and client code using time_t correctly could remain as-is.

Enhancing Sparse in the Linux Kernel

In 2003, Linus Torvalds built a custom preprocessor, C parser, and compiler called Sparse. Sparse performs Linux kernel–specific type checking. Sparse relies on macros, such as __user, sprinkled around in kernel code, that do nothing under normal build configurations but expand to uses of __attribute__((address_space(...))) when the __CHECKER__ macro is defined.

Gatekeeping the macro definitions with __CHECKER__ is necessary because most compilers don’t provide ways to hook into macros or implement custom safety checking … until today. With Macroni, we can hook into macros and perform Sparse-like safety checks and analyses. But where Sparse is limited to C (by virtue of implementing a custom C preprocessor and parser), Macroni applies to any code parseable by Clang (i.e., C, C++, and Objective C).

The first Sparse macro we’ll hook into is __user. The kernel currently defines __user to an attribute that Sparse recognizes:

# define __user     __attribute__((noderef, address_space(__user)))

Figure 7: The Linux kernel’s __user macro

Sparse hooks into this attribute to find pointers that come from user space, as in the following example. The noderef tells Sparse that these pointers must not be dereferenced (e.g., *uaddr = 1) because their provenance cannot be trusted.

u32 __user *uaddr;

Figure 8: Example of using the __user macro to annotate a variable as coming from user space

Macroni can hook into the macro and expanded attribute to lower the declaration down to MLIR like this:

%0 = hl.var "uaddr" : !hl.lvalue<!sparse.user<!hl.ptr<!hl.elaborated<!hl.typedef<"u32">>>>>

Figure 9: Kernel code after being lowered to MLIR by Macroni

The lowered MLIR code embeds the annotation into the type system by wrapping declarations that come from user space in the type sparse.user. Now we can add custom type-checking logic for user-space variables, similar to how we created strong typedefs previously. We can even hook into the Sparse-specific macro __force to disable strong type checking on an ad hoc basis, as developers do currently:

raw_copy_to_user(void __user *to, const void *from, unsigned long len)
   return __copy_user((__force void *)to, from, len);

Figure 10: Example use of the __force macro to copy a pointer to user space

We can also use Macroni to identify RCU read-side critical sections in the kernel and verify that certain RCU operations appear only within these sections. For instance, consider the following call to rcu_dereference():

rcu_dereference(sbi->s_group_desc)[i] = bh;

Figure 11: A call to rcu_derefernce() in an RCU read-side critical section in the Linux kernel

The above code calls rcu_derefernce() in a critical section—that is, a region of code beginning with a call to rcu_read_lock() and ending with rcu_read_unlock(). One should call rcu_dereference() only within read-side critical sections; however, there is no way to enforce this constraint.

With Macroni, we can use rcu_read_lock() and rcu_read_unlock() calls to identify critical sections that form implied lexical code regions and then check that calls to rcu_dereference() appear only within these sections:

kernel.rcu.critical_section {
 %1 = macroni.parameter "p" : ...
 %2 = kernel.rcu_dereference rcu_dereference(%1) : ...

Figure 12: The result of lowering the RCU-critical section to MLIR, with types omitted for brevity

The above code turns both the RCU-critical sections and calls to rcu_dereference() into MLIR operations. This makes it easy to check that rcu_dereference() appears only within the right regions.

Unfortunately, RCU-critical sections don’t always bound neat lexical code regions, and rcu_dereference() is not always called in such regions, as shown in the following example:

__bpf_kfunc void bpf_rcu_read_lock(void)

Figure 13: Kernel code containing a non-lexical RCU-critical section

static inline struct in_device *__in_dev_get_rcu(const struct net_device *dev)
   return rcu_dereference(dev->ip_ptr);

Figure 14: Kernel code calling rcu_dereference() outside of an RCU-critical section

We can use the __force macro to permit these sorts of calls to rcu_dereference(), just as we did to escape type checking for user-space pointers.

Rust-like unsafe regions

It’s clear that Macroni can help strengthen type checking and even enable application-specific type-checking rules. However, marking types as strong means committing to that level of strength. In a large codebase, such a commitment might require a massive changeset. To make adapting to a stronger type system more manageable, we can design an “unsafety” mechanism for C akin to that of Rust: within the unsafe region, strong type checking does not apply.

#define unsafe if (0); else

fahrenheit convert(celsius C) {
 fahrenheit F;
 unsafe {
         F = (C * 9.0 / 5.0) + 32.0;
 return F;

Figure 15: C code snippet presenting macro-implemented syntax for unsafe regions

This snippet demonstrates our safety API’s syntax: we call the unsafe macro before potentially unsafe regions of code. All code not listed in an unsafe region will be subject to strong type checking, while we can use the unsafe macro to call out regions of lower-level code that we deliberately want to leave as-is. That’s progressive!

The unsafe macro provides the syntax only for our safety API, though, and not the logic. To make this leaky abstraction watertight, we would need to transform the macro-marked if statement into an operation in our theoretical safety dialect:

"safety.unsafe"() ({
}) : () -> ()

Figure 16: With Macroni, we can lower our safety API to an MLIR dialect and implement safety-checking logic.

Now we can disable strong type checking on operations nested within the MLIR representation of the unsafe macro.

Safer signal handling

By this point, you may have noticed a pattern for creating safer language constructs: we use macros to define syntax for marking certain types, values, or regions of code as obeying some set of invariants, and then we define logic in MLIR to check that these invariants hold.

We can use Macroni to ensure that signal handlers execute only signal-safe code. For example, consider the following signal handler defined in the Linux kernel:

static void sig_handler(int signo) {
       do_detach(if_idx, if_name);

Figure 17: A signal handler defined in the Linux kernel

sig_handler() calls three other functions in its definition, which should all be safe to call in signal-handling contexts. However, nothing in the above code checks that we call signal-safe functions only inside sig_handler()‘s definition—C compilers don’t have a way of expressing semantic checks that apply to lexical regions.

Using Macroni, we could add macros for marking certain functions as signal handlers and others as signal-safe and then implement logic in MLIR to check that signal handlers call only signal-safe functions, like this:

#define SIG_HANDLER(name) name
#define SIG_SAFE(name) name

int SIG_SAFE(do_detach)(int, const char*);
void SIG_SAFE(perf_buffer__free)(struct perf_buffer*);
void SIG_SAFE(exit)(int);

static void SIG_HANDLER(sig_handler)(int signo) { ... }

Figure 18: Token-based syntax for marking signal handlers and signal-safe functions

The above code marks sig_handler() as a signal handler and the three functions it calls as signal-safe. Each macro invocation expands to a single token—the name of the function we want to mark. With this approach, Macroni hooks into the expanded function name token to determine if the function is a signal handler or signal-safe.

An alternative approach would be to define these macros to magic annotations and then hook into these with Macroni:

#define SIG_HANDLER __attribute__((annotate("macroni.signal_handler")))
#define SIG_SAFE __attribute__((annotate("macroni.signal_safe")))

int SIG_SAFE do_detach(int, const char*);
void SIG_SAFE perf_buffer__free(struct perf_buffer*);
void SIG_SAFE exit(int);

static void SIG_HANDLER sig_handler(int signo) { ... }

Figure 19: Alternative attribute syntax for marking signal handlers and signal-safe functions

With this approach, the macro invocation looks more like a type specifier, which some may find more appealing. The only difference between the token-based syntax and the attribute syntax is that the latter requires compiler support for the annotate() attribute. If this is not an issue, or if __CHECKER__-like gatekeeping is acceptable, then either syntax works fine; the back-end MLIR logic for checking signal safety would be the same regardless of the syntax we choose.

Conclusion: Why Macroni?

Macroni lowers C code and macros down to MLIR so that you can avoid basing your analyses on the lackluster Clang AST and instead build them off of a domain-specific IR that has full access to types, control flow, and data flow within VAST’s high-level MLIR dialect. Macroni will lower the domain-relevant macros down to MLIR for you and elide all other macros. This unlocks macro-sensitive static analysis superpowers. You can define custom analyses, transformations, and optimizations, taking macros into account at every step. As this post demonstrates, you can even combine macros and MLIR to define new C syntax and semantics. Macroni is free and open source, so check out its GitHub repo to try it out!


I thank Trail of Bits for the opportunity to create Macroni this summer. In particular, I would like to thank my manager and mentor Peter Goodman for the initial idea of lowering macros down to MLIR and for suggestions for potential use cases for Macroni. I would also like to thank Lukas Korencik for reviewing Macroni’s code and for providing advice on how to improve it.

1 See Understanding code containing preprocessor constructs, SugarC: Scalable Desugaring of Real-World Preprocessor Usage into Pure C, An Empirical Analysis of C Preprocessor Use, A Framework for Preprocessor-Aware C Source Code Analyses, Variability-aware parsing in the presence of lexical macros and conditional compilation, Parsing C/C++ Code without Pre-processing, Folding: an approach to enable program understanding of preprocessed languages, and Challenges of refactoring C programs.

Secure your Apollo GraphQL server with Semgrep

By Vasco Franco

tl;dr: Our publicly available Semgrep ruleset has nine new rules to detect misconfigurations of versions 3 and 4 of the Apollo GraphQL server. Try them out with semgrep --config p/trailofbits!

When auditing several of our clients’ Apollo GraphQL servers, I kept finding the same issues over and over: cross-site request forgery (CSRF) that allowed attackers to perform actions on behalf of users, rate-limiting that allowed attackers to brute-force passwords or MFA tokens, and cross-origin resource sharing (CORS) misconfigurations that allowed attackers to fetch secrets that they shouldn’t have access to. Developers overlook these issues for multiple reasons: bad defaults in version 3 of the Apollo GraphQL server (e.g., the csrfProtection option does not default to true), a lack of understanding or knowledge of certain GraphQL features (e.g., batched queries), and a lack of understanding of certain web concepts (e.g., how the same-origin policy [SOP] and CORS work).

Finding the same issues repeatedly motivated me to use some internal research and development (IRAD) time to consistently detect some of these issues in our future audits, leaving more time to find deeper, more complex bugs. Semgrep—a static analysis tool used to detect simple patterns that occur in a single file—was the obvious tool for the job because the issues are easy to detect with grep-like constructs and don’t require interprocedural or other types of more complex analysis.

We open sourced Semgrep rules that find Apollo GraphQL server v3 and v4 misconfigurations. Our rules leverage Semgrep’s taint mode to make them easier to write and to increase their accuracy. Go test your GraphQL servers!

We previously publicly released Semgrep rules to find Go concurrency bugs and misuses of machine learning libraries.

Common GraphQL issues

GraphQL has several design choices that make some vulnerabilities, such as CSRF, more prevalent than in typical REST servers. Of course, GraphQL servers also suffer from all the usual problems: access control issues (e.g., an access control flaw in GitLab that disclosed information about private users, or a bug in HackerOne that allowed attackers to disclose users’ confidential data), SQL injections (e.g., a SQL injection in HackerOne’s GraphQL server), server-side request forgery (SSRF), command injection, and many others.

This blog post will cover the rules we created to detect CSRF and CORS misconfigurations. We’ll also show how using Semgrep’s taint mode can save you time and increase your rules’ accuracy by reducing the number of patterns you need to define all the ways in which a value can flow into a sink.


CSRF is an attack that allows malicious actors to trick users into performing unwanted operations (e.g., editing the user’s profile) in websites they’re authenticated to. If you’re unfamiliar with the details, read more about CSRF attacks in PortSwigger’s Web Security Academy CSRF explanation.

CSRF attacks in the Apollo Server

CSRF haunted the Apollo GraphQL server until the introduction of the csrfPrevention option. CSRF vulnerabilities are prevalent in the Apollo server because of two factors: developers mislabel mutations as queries, and the Apollo server allows users to issue query operations with GET requests (but not mutation operations). Queries should not change state (like a GET request in a RESTful API), while mutations are expected to change state (like POST, PATCH, PUT, or DELETE). If developers followed this convention, everything would be fine. However, I’ve yet to find a codebase that does not mislabel a mutation as a query, making these mislabeled operations immediately vulnerable to CSRF attacks.

Thankfully, the Apollo team was very aware of this and, in version 3, added the csrfPrevention option to remove the issue altogether. It prevents CSRF attacks by ensuring that any request must have a Content-Type header different from text/plain, application/x-www-form-urlencoded, or multipart/form-data; a non-empty X-Apollo-Operation-Name header; or a non-empty Apollo-Require-Preflight header. This ensures the request will always be preflighted, which prevents the CSRF attack.

The csrfPrevention option defaults to false in v3 and to true in v4, so those still using v3 need to consciously add this option in their server initialization, which, in our experience, almost never happens.

Finding CSRF misconfigurations with Semgrep

We created two Semgrep rules to find misconfigurations in versions 3 and 4. For v3, we find all ApolloServer initializations where the csrfPrevention option is not set to true.

  - pattern: new ApolloServer({...})
  - pattern-not: |
      new ApolloServer({..., csrfPrevention: true, ...})

Figure 1.1: Semgrep rule that detects a misconfigured csrfProtection option in version 3 of the Apollo server

For v4, we find all server initializations with the csrfPrevention option set to false.

  - pattern: |
      new ApolloServer({..., csrfPrevention: false, ...})

Figure 1.2: Semgrep rule that detects a misconfigured csrfProtection option in version 4 of the Apollo server


CORS allows a server to relax the browser’s SOP. As expected, developers sometimes relax the SOP a bit too far, which can allow attackers to fetch secrets that they should not have access to. If you are unfamiliar with the details, read more about CORS in PortSwigger’s Web Security Academy CORS explanation.

Setting a CORS policy in an Apollo Server

In version 3 of the Apollo Server, a developer can set their server’s CORS policy in two ways. First, they can pass the cors argument to their ApolloServer class instance.

import { ApolloServer } from 'apollo-server';

const apolloServerInstance = new ApolloServer({
    cors: CORS_ORIGIN

Figure 1.3: Configuring CORS in version 3 of an Apollo GraphQL server

Alternatively, they can set the CORS policy on the back-end framework they are using. For example, with an Express.js back-end server, the CORS attribute is passed as an argument to the applyMiddleware function.

import { ApolloServer } from 'apollo-server-express';

const apolloServerInstance = new ApolloServer({});

    cors: CORS_ORIGIN,

Figure 1.4: Configuring CORS in version 3 of an Apollo GraphQL server with a back-end Express server

On version 4 of the Apollo server, the developer must set CORS on the back end itself. Therefore, writing rules for v4 is out of scope for our Apollo-specific Semgrep queries—other Semgrep rules already cover most of those cases.

Our rules for version 3 cover uses of Express.js and the batteries-included Apollo server back ends, as these were the ones we saw in use the most. If you use a different back-end framework for your Apollo Server, our rules likely won’t work, but we accept PRs at trailofbits/semgrep-rules! It should be effortless to adapt them based on the existing queries. ;)

Finding missing CORS policies

The rules for each back end are very similar, so let’s look at one of them—the one that detects CORS misconfigurations in the batteries-included Apollo server. We have two rules in the same file: one to detect cases where a CORS policy is not defined and one to detect a poorly configured CORS policy.

To detect missing CORS policies, we look for ApolloServer instantiations where the cors argument is undefined. We also need to ensure that the ApolloServer comes from the apollo-server package (the ApolloServer class could also come from the apollo-server-express package, but we don’t want to catch these cases). The query is shown in figure 1.5.

  - pattern-either:
      - pattern-inside: |
          $X = require('apollo-server');
      - pattern-inside: |
          import 'apollo-server';
  - pattern: |
      new ApolloServer({...})
  - pattern-not: |
      new ApolloServer({..., cors: ..., ...})

Figure 1.5: Semgrep rule that detects a missing CORS policy in an Apollo GraphQL server (v3)

Finding bad CORS policies

To detect bad CORS policies, it’s not as simple. We have to detect several cases:

  • Cases where the origin is set to true—A true origin tells the server to accept all origins.
  • Cases where the origin is set to null—An attacker can trick a user into making requests from a null origin from, for example, a sandboxed iframe.
  • Cases where the origin is a regex with an unescaped dot character—In regex, a dot matches any character, so if we are using the /$/ regex, it will match the domain, which could potentially be controlled by an attacker.
  • Cases where the origin does not finish with the $ character—In regex, the $ character matches the end of the string, so if we are using the / regex, it will also match the domain, an attacker-controlled domain.

And these will not cover every possible bad CORS policy (e.g., a bad CORS policy could simply include an attacker domain or a domain that allows an attacker to upload HTML code). We test all the cases described above with the rule in the figure below.

  # 'true' mean that every origin is reflected
  - pattern: |
  # the '.' character is not escaped
  - pattern-regex: ^/.*[^\\]\..*/$
  # the regex does not end with '$'
  - pattern-regex: ^/.*[^$]/$
  # An attacker can make requests from ‘null’ origins
  - pattern: |

Figure 1.6: Semgrep pattern that detects bad CORS origins

These bad origins can be used by themselves or inside an array. To test for both cases, we first check occurrences of the $CORS_SINGLE_ORIGIN metavariable that are isolated or in an array and then use a metavariable-pattern to define what is a bad origin with the pattern we’ve created in figure 1.6.

  - patterns:
      # pattern alone or inside an array
      - pattern-either:
          - pattern: |
          - pattern: |
              [..., $CORS_SINGLE_ORIGIN, ...]
      - metavariable-pattern:
          metavariable: $CORS_SINGLE_ORIGIN
             # <The bad origin checks from the previous figure>

Figure 1.7: Semgrep pattern that detects bad CORS origins in a single entry or in an array

Finally, we need to find uses of this origin inside an ApolloServer initialization. We do so with the following pattern:

new ApolloServer({..., cors: $CORS_ORIGIN, ...})

This $CORS_ORIGIN can be used inline (e.g., cors: true), or it can come from a variable (e.g., cors: corsOriginVariableDefineElsewhere). It is laborious to define all the possible places that the origin could have come from. Thankfully, we don’t need to do so with Semgrep’s taint mode!

We need to define only the following:

  • pattern-sources: the bad CORS policy—We define it as {origin: $BAD_CORS_ORIGIN} where the $BAD_CORS_ORIGIN metavariable is the pattern we defined above for a bad origin.
  • pattern-sinks: where the bad CORS policy should not flow to—We define it as the $CORS_ORIGIN metavariable in the pattern new ApolloServer({..., cors: $CORS_ORIGIN, ...}).

With taint mode, we can catch many ways in which the CORS policy can be set: directly (Case 1 in figure 1.8), through a variable that configures the entire CORS policy (Case 2), through a variable that sets only the origin (Case 3), and many other setups that we do not want to define by hand.

// Case 1: Has a very permissive 'cors' (true)
const apollo_server_bad_1 = new ApolloServer({
    //ruleid: apollo-graphql-v3-bad-cors
    cors: { origin: true }

// Case 2: Has a very permissive 'cors' from a variable
const bad_CORS_policy = { origin: true }
const apollo_server_bad_2 = new ApolloServer({
    //ruleid: apollo-graphql-v3-bad-cors
    cors: bad_CORS_policy

// Case 3: Has a very permissive 'cors' from a variable (just the origin)
const bad_origin = true;
const apollo_server_bad_3 = new ApolloServer({
    //ruleid: apollo-graphql-v3-bad-cors
    cors: { origin: bad_origin }</span

Figure 1.8: Several test cases that Semgrep’s taint mode helps catch for free

The entire commented rule is shown in figure 1.9.

mode: taint
  - patterns:
      - pattern-inside: |
          { origin: $BAD_CORS_ORIGIN }
      - metavariable-pattern:
          metavariable: $BAD_CORS_ORIGIN
            # 'true' means that every origin is reflected
            - pattern: |
            - patterns:
                # pattern alone or inside an array
                - pattern-either:
                    - pattern: |
                    - pattern: |
                        [..., $CORS_SINGLE_ORIGIN, ...]
                - metavariable-pattern:
                    metavariable: $CORS_SINGLE_ORIGIN
                      # the '.' character is not escaped
                      - pattern-regex: ^/.*[^\\]\..*/$
                      # the regex does not end with '$'
                      - pattern-regex: ^/.*[^$]/$
                      # An attacker can make requests from ‘null’ origins
                      - pattern: |
  - patterns:
      # The ApolloServer comes from the 'apollo-server' package
      - pattern-either:
          - pattern-inside: |
              $X = require('apollo-server');
          - pattern-inside: |
              import 'apollo-server';
      # The sink is the ApolloServer's cors argument
      - pattern: |
          new ApolloServer({..., cors: $CORS_ORIGIN, ...})
      # This tells Semgrep that the sink is only the $CORS_ORIGIN variable
      - focus-metavariable: $CORS_ORIGIN

Figure 1.9: Semgrep rule that detects a bad CORS policy in an Apollo GraphQL server (v3)

We have also created a Semgrep rule for auditors and security engineers that want to review their Apollo server’s CORS policy in detail, even when the policy might be safe. This rule reports any CORS policy that is not false or an empty array—obviously good CORS policies. It is helpful when you want to check all the hard-coded origins by hand, but it is not something that you want to integrate in your CI pipeline since it will report false positives (an audit rule). You can find the rule at trailofbits.javascript.apollo-graphql.v3-cors-audit.v3-potentially-bad-cors.

Finishing thoughts

Semgrep excels in finding simple patterns that happen in a single file like the ones we’ve described in this post. For more complex analysis, you may want to use a tool such as CodeQL, which has its disadvantages as well: it involves a more difficult learning curve, it uses different APIs for different languages, it requires compiling the code, and it does not support some languages that Semgrep does (e.g., Rust).

One of Semgrep’s biggest limitations is that it lacks interfile and interprocedural analysis. For example, the rules above won’t catch cases where the CORS policy is set in one file and the Apollo Server initialization occurs in another file. This may now be possible with Semgrep Pro Engine (previously called DeepSemgrep), which enhances the Semgrep engine with interfile analysis capabilities. However, this feature is currently limited to paid customers and to a limited number of languages.

At Trail of Bits, we extensively use static analysis tools and usually end up writing custom rules and queries specific to our clients’ codebases. These can provide great value because they can find patterns specific to your codebase and even enforce your organization’s engineering best practices. When the rules we write are useful to the community, we like to open source them. Check them out at

Use our new Apollo GraphQL rules with semgrep --config p/trailofbits, and try writing your own custom rules!

iVerify is now an independent company!

We’re proud to announce that iVerify is now an independent company following its four-year incubation at Trail of Bits. Originally developed in-house to ensure that our personal phones, which store data essential to our work and private lives, were secured to the standards of security professionals, iVerify quickly showed that it could be valuable to the public:

“The mobile security market has a problem. Simply put, current solutions fail to meet the sophistication of modern threats or the growing privacy desires of mobile device users… Forensics tools are limited, and researchers are not necessarily equipped to rapidly discover and alert targets that their device has been compromised in a timely manner.

[iVerify’s] vision is to arm organizations and individuals with the protection they need in this changing world. We are building the first mobile threat hunting company focused on countering the emerging mobile spyware threat.” – iVerify

Trail of Bits launched the iVerify security toolkit for iPhones in 2019, an enterprise product in 2020, then an Android app in 2021. Now, with $4 million in seed funding, iVerify plans to expand its capabilities to reach more enterprise customers.

At the helm are four Trail of Bits alumni: Matthias Frielingsdorf, Jelmer de Hen, and Vlad Orlov, who join CEO Danny Rogers. Their contributions to making iVerify an essential consumer and enterprise product motivated investors, including Mischief Ventures, Mantis Venture Capital, Altman Capital, and others.

“It’s rare for a seed startup to have the impressive portfolio of enterprise customers that iVerify already has. It’s also rare to find a founding team with the technical prowess and the business mindset to build something that is both technically sound and commercially viable.” – Dustin Moring, General Partner at Mischief Ventures (lead investor).

We couldn’t agree more. Thank you to the team and everyone who has contributed to iVerify’s success. We’re excited to watch from the sidelines as iVerify leads the cause in safeguarding individual and organizational device security.


Read the entire press release to learn more about iVerify’s product offerings and plans for growth, and check out additional coverage from around the web:

  • Why The Chainsmokers Invest in—and Party With—Niche Cybersecurity Companies
  • iVerify Raises $4M to Take On the Growing Threat of Mercenary Spyware
  • Introducing iVerify, the first mobile threat hunting company
  • Decipher Podcast: iVerify CEO Danny Rogers and COO Rocky Cole join Dennis Fisher to discuss the spinout of the iVerify mobile security tool as a standalone company, the scourge of mercenary spyware, and how enterprises can protect their users.
  • Danny Rogers: “With the commercialization of advanced spyware, the narrative around mobile security has changed. That change demands new approaches, and iVerify is setting out to be the world’s first true mobile threat hunting company focused on combating mercenary spyware.”
  • Matthias Frielingsdorf: “I’m very proud to announce the launch of the first mobile threat hunting company, iVerify, which we’re building to harmonize security and privacy in the face of a new class of mobile security threats!”
  • Rocky Cole: “… the security industry still looks a lot like it did when I first joined the hacking community, straight out of college… It’s time for something new and I’m beyond thrilled to be working with … our growing and exceptionally talented team at iVerify to bring the world the mobile security product it deserves.”
  • Alex Pall (Chainsmokers): “Since we’re all about propelling ingenious solutions that crack real-world problems, iVerify – which was incubated by security research firm Trail of Bits – was an obvious bet”
  • Gabriel Jacoby-Cooper: “It’s exceedingly difficult to fight back against the mercenary spyware, but the iVerify team is simply the best to do so.”
  • William Knowles: “iVerify is the one smartphone app everyone concerned with their security should install and make well advised updates to their devices.”

If you’re as enthusiastic about the mission and potential of iVerify as we are, consider becoming a part of their groundbreaking journey. They’re on the lookout for talented individuals who can help turn their vision for a safer mobile ecosystem into reality. Here are the current opportunities they have open:

If you’re an interested potential user of iVerify, follow along with them on Twitter, LinkedIn, or Mastodon for product updates.

The Engineer’s Guide to Blockchain Finality

By Benjamin Samuels

Many security-critical off-chain applications use a simple block delay to determine finality: the point at which a transaction becomes immutable in a blockchain’s ledger (and is impossible to “undo” without extreme economic cost). But this is inadequate for most networks, and can become a single point of failure for the centralized exchanges, multi-chain bridges, and L2 scaling solutions that rely on transaction finality. Without proper consideration of a blockchain’s finality criteria, transactions that appear final can be expunged from the blockchain by a malicious actor in an event called a re-org, leading to double-spend attacks and value stolen from the application.

We researched several off-chain applications and L2 networks and discovered two L2 clients, Juno and Pathfinder, that either were not checking for finality or incorrectly used block delays to detect whether Ethereum blocks were finalized. We disclosed our findings to each product team, and fixes were published shortly after disclosure in version v0.4.0 for Juno and v0.6.2 for Pathfinder. This blog post gathers the knowledge and insights we gained from this research. It explains the dangers of reorgs, the differences between distinct finality mechanisms, and how to prevent double-spend attacks when writing applications that consume data from different kinds of blockchains.

Understanding re-orgs

When a user submits a transaction to a blockchain, it follows a lifecycle that is nearly identical across all blockchains. First, their transaction is gossiped across the blockchain’s peer-to-peer network to a block proposer. Once a block proposer receives the transaction and includes it in a block, the block is broadcast across the network.

Here is where the problems begin: some blockchains don’t explicitly define who the next block proposer should be, and the ones that do need a way to recover if that proposer is offline. These conditions lead to situations where there are two or more valid ways for the blockchain to proceed (a fork), and the network has to figure out which fork should be canonical.

Figure 1: Two miners on a PoW network propose a valid block for slot 4 at the same time.

Blockchains are designed with these issues in mind and define a fork choice rule to determine which fork should be considered canonical. Forks can sometimes last for multiple blocks, where different portions of the network consider a different chain to be canonical.

Figure 2: A PoW network where block candidates for slots 4, 5, and 6 were mined in quick succession and built on different parents, leading to a fork.

Assuming there is no bug in the network’s software, the fork will eventually be reconciled, leading to a single fork becoming canonical. The other forks, their blocks, and their transactions are expunged from the blockchain’s history, called a re-org.

When a transaction is expunged from the chain via a re-org, that transaction may either be re-queued for inclusion in a new block, or otherwise have its ordering or block number changed to whatever it is in the canonical chain. Attackers can leverage these changes to modify a transaction’s behavior or cancel the transaction entirely based on which fork it is included on.

Figure 3: A network after a three-block re-org. Transactions in blocks 4a, 5a, and 6a are no longer part of the canonical chain.

Re-orgs are a normal part of a blockchain’s lifecycle, and can happen regularly due to factors like block production speed, network latency, and network health. However, attackers can take advantage of (and even orchestrate!) re-orgs to perform double-spend attacks, a category of attack where an attacker submits a deposit transaction, waits for it to be included in a block, then orchestrates a re-org to expunge their transaction from the canonical chain while still receiving credit for their deposit on the off-chain application.

It is for this reason that finality considerations are important. If a centralized exchange or bridge indexes a deposit transaction that is not final, it is vulnerable to double-spend attacks by way of an attacker causing a re-org of the blockchain.

Blockchains use a variety of different consensus algorithms and thus have a variety of different finality conditions that should be considered for each chain.

Probabilistic finality

Examples: Bitcoin, Binance Smart Chain (pre-BEP-126), Polygon PoS, Avalanche – or generally any PoW-based blockchain

Chains using probabilistic finality are unique in that their blocks are never actually finalized—instead, they become probabilistically final, or more “final” as time goes on. Given enough time, the probability that a previous block will be re-orged off the chain approaches zero, and thus the block becomes final.

In most probabilistically final chains, the fork choice rule that determines the canonical chain is based on whichever fork has the most blocks built on top of it, called Nakamoto consensus. Under Nakamoto consensus, the chain may re-org if a longer chain is broadcast to the network, even if the longer chain excludes blocks/transactions that were already included in the shorter chain.

Double-spend attacks on probabilistic proof-of-work networks

The classic attack against proof-of-work networks is a 51% re-org attack. This attack requires an off-chain exchange, bridge, or other application that indexes deposit transactions very quickly, ideally indexing blocks as soon as they are produced or with an exceedingly short delay.

The attacker must accumulate, purchase, or rent enough computing resources so the attacker controls the majority of the hash power on the network. This means the attacker has enough resources to privately mine a chain that’s longer than the honest canonical chain. Note that this is a probabilistic attack; control over 51% of the network’s hash power makes the attack an eventual certainty. An attacker could theoretically perform double-spend attacks with much less than 51% of the network’s hash power, but it may require many attempts before the attack succeeds.

Once the mining resources are ready, the attacker submits a transaction on the public, canonical chain to deposit funds from their wallet to the exchange/bridge.

Immediately afterward, the attacker must create a second, conflicting transaction that transfers funds from their wallet to another attacker-controlled address. The attacker configures their mining resources to mine a new fork that includes the transfer transaction instead of the deposit transaction.

Figure 4: The attacker creates a private fork that includes their transfer transaction instead of the deposit transaction.

Given that the attacker controls the majority of the hash power on the network, eventually their private fork will have more blocks than the canonical fork. Once they have received credit for the deposit transaction and their private fork has more blocks than the canonical chain, the attacker instructs their network to publish the private chain’s blocks to the honest nodes following the canonical chain.

The honest nodes apply the “longest chain” fork choice rule, triggering a re-org around the attacker’s longer chain and excluding the blocks that contained the attacker’s deposit transaction.

Figure 5: Once the attacker publishes their private fork, the network reorgs and expunges the fork that includes their deposit transaction.

In effect, this allows the attacker to “double-spend” their coins: the exchange or bridge credits the attacker for the coins, while the coins are still present in an attacker-controlled wallet.

Measuring probabilistic finality

Since probabilistically final chains don’t define finality conditions, finality must be measured probabilistically based on the number of blocks that have elapsed since the target transaction/ancestor block. The more blocks that have been built on top of a given ancestor, the higher the cost of a re-org is for an attacker.

The correct block delay should be based on both historical factors and economic factors, selecting the greater of the two for the application’s indexing delay.

Among historical factors, off-chain application developers should consider how often the chain has reorgs and how large the reorgs typically are. If block production is probabilistic (as in Proof-of-work networks), then a larger delay should be factored in to compensate for the chance that many blocks are created in an unusually short period of time.

Among economic factors, one should consider the cost to execute a re-org attack for a transaction of a given economic value. Given the probabilistic nature of 51% attacks, engineers should build in a safety margin and consider the cost of a 25% attack instead of 51%. For example, if it costs a minimum of $500k USD to execute a 25% attack, six-block re-org attack against Bitcoin, then an off-chain application that receives a $500k deposit should wait for at least six blocks before considering the transaction final and indexing it.

Using Crypto51, we can calculate the block delays required for deposits and withdrawals of $75k with a 25% attack threshold (as of June 2023).

  • Bitcoin: Two blocks for finality (~20 minutes)
  • Litecoin/Dogecoin: 48 blocks for finality (~Two hours)
  • Bitcoin Cash: 103 blocks for finality (~17 hours)
  • Ethereum Classic: 3,031 blocks for finality (~11 hours)
  • Ethereum PoW: 23,600 blocks for finality (~89 hours)
  • Zcash: 1,881 blocks for finality (~40 hours)

These delays represent a single data point in time for a specific deposit amount. As the hash power on each network increases or decreases, the time-to-finality will change as well and must be updated accordingly. Mining hash power on each network correlates highly with the chain token price, so the amount of hash power can drop very quickly, requiring monitoring and fast response by integrating applications. Failure to adjust finality delays in a timely manner will lead to double-spending attacks.

Finality delays for proof-of-work chains may be reduced for certain exchange-like applications using on-chain monitoring, automated system halts, trading limits, and withdrawal delays. However, these mechanisms may overcomplicate the application’s logic and make it vulnerable to other forms of attack.

It should be noted that chains with extraordinarily low hash rates may easily be attacked with much more than 51% of the network’s hash rate. Existing services offer an easy-to-rent hashing capacity that may exceed a chain’s hash rate many times over. In cases like this, it is recommended to either avoid integration with the chain, or base finality calculations on the available-for-rent hashing capacity.

Probabilistic chains using proof-of-stake/proof-of-authority require slightly different considerations, since block proposers cannot freely enter the proposer set and may have different fork choice rules than proof-of-work networks.

  • Binance Smart Chain: Blocks are considered final once the number of blocks that have passed is equal to two-thirds the number of validators in their validator set. Twenty blocks are required for finality (~60 seconds).
  • Polygon PoS: Integrators should use L1 state root checkpoints as a measure of finality. When a state root checkpoint containing a given transaction is finalized on the L1, the Polygon transaction may be considered final. State root checkpoints occur roughly every 30 minutes.

Provable finality

Delayed finality examples: Ethereum PoS (Casper FFG), Polkadot (GRANDPA), Solana
Instant Finality Examples: Cosmos, Celestia, Algorand, Aptos

Systems using provable finality make special considerations for finality to ensure it happens more quickly and with better economic assurances than most probabilistically final chain constructions.

There are two types of provable finality: chains with instantly provable finality, and chains with delayed provable finality.

Chains with instant finality don’t need special finality considerations by off-chain applications. All blocks published by the network are immediately provably final by definition.

Chains with delayed finality have separate consensus mechanisms for newly produced vs. finalized blocks. These chains usually have superior liveness properties compared to instant finality chains, but at the cost of added complexity, vulnerability to re-orgs, and more complex integration considerations for off-chain applications.

Figure 6: A delayed-finality chain. Blocks to the right of the finality boundary may be re-orged and should not be indexed by exchanges or bridges.

Double-spend attacks on delayed finality chains

Historically, most blockchains haven’t had provable finality, so bridges, exchanges, and other off-chain applications would use a block delay for measuring the finality of any new chains they integrate with.

However, for chains with provable delayed finality, there are situations where the finality mechanism may stall or fail, as occurred in the May 2023 incident where Ethereum’s finality gadget, Casper FFG, stalled. When finality mechanisms fail, the chain may continue to produce blocks, creating long strings of unfinalized blocks that may be reorged by a bug or an attacker.

During the Ethereum incident, the chain’s finality mechanism was stalled by nine epochs—the equivalent of 139 blocks’ worth of confirmations (after controlling for missed slot proposals). At this time, most bridges/centralized exchanges used a block-delay rule to determine the finality of a transaction on Ethereum, with delays ranging from 14 blocks to 100 blocks.

Had the Ethereum finality incident been orchestrated by an attacker, the attacker may have been able to perform double-spend attacks against these bridges/exchanges by orchestrating exceedingly long re-orgs.

Checking for finality

For delayed-finality chains, as illustrated in the previous example, block delays are not an adequate way to “wait” for blocks to become final. Instead, applications must query the chain’s RPC for the exact finality condition to ensure the block being indexed is actually final.

Ethereum proof-of-stake

The Ethereum JSON RPC defines a “default block” parameter for various endpoints that should be set to “finalized” to query the most recent finalized block. To obtain the most recent finalized block, use eth_getBlockByNumber(“finalized”, ...). This parameter may be used for other endpoints, such as eth_call, eth_getBalance, and eth_getLogs.


Call chain.getFinalizedHead() to get the block hash of the latest finalized block, then use chain.getBlock() to get the block associated with the hash.


Use the getBlocks() RPC method with the commitment level set as finalized.

When provable finality lies

One major caveat of provable finality/proof-of-stake systems is they have no way to provide strong subjectivity guarantees. A blockchain’s subjectivity refers to whether a node syncing from genesis will always arrive at the same chain head and whether an attacker can manipulate the end state of the syncing node.

In proof-of-work blockchains, the cost of creating an alternate chain for partially synced nodes to follow is equal to all of the work performed by miners from genesis to the canonical chain head, making any subjectivity attack impractical against proof-of-work networks.

However, in proof-of-stake networks, the cost of creating an alternate chain has only one requirement with an unknown, and possibly zero cost: the private keys of the chain’s historical validators. The keys for historical validators may be acquired by a number of means; private keys may be leaked or brute-forced, or validators who no longer use their keys may offer them up for sale.

This re-use of old validator keys creates the possibility for long-range sync attacks, in which a newly synced node may behave as though a specific transaction is submitted and finalized when in reality, it was never submitted to the canonical chain in the first place.

To protect against long-range sync attacks, operations teams should always begin node sync from weak subjectivity checkpoints. These checkpoints are essentially used as genesis blocks, providing a trusted starting point for nodes to sync from. Weak subjectivity checkpoints may be acquired from already-synced nodes or via social processes.

The special case of L2s

Examples: Arbitrum, Optimism, StarkNet, Scroll, ZKSync, Polygon zkEVM

L2 networks are unique in that they don’t have consensus mechanisms in the way a normal blockchain does. In a normal blockchain, the validator set must come to a consensus on the output of a state transition function. In an L2 network, it is the underlying L1 network that is responsible for verifying the state transition function. Ultimately, this means the finality condition for an L2 network is dependent on the finality condition of the underlying L1.

When an L2 sequencer or prover receives a transaction, it sequences/generates a proof for the transaction, then returns an L2 transaction receipt. Once the sequencer/prover has received enough transactions, it assembles the transactions into a batch that is submitted to the L1 network.

Figure 7: The flow of a user’s transaction on an L2 network. Notably, sequencers/provers provide users with transaction receipts far in advance of the transaction’s inclusion or finality on the L1.

For ZK-Rollups, the batch contains a proof representing the execution of every transaction in the batch. The L1 contract verifies the proof, and once the batch transaction is final, all of the L2 transactions included in the proof are final as well.

For Optimistic Rollups, the batch contains the calldata for every transaction in the batch. The L1 contract does not run any state transition function or verification that the calldata is valid. Instead, Optimistic Rollups use a challenge mechanism to allow L2 nodes to contest an L1 batch. This means a transaction submitted to an Optimistic Rollup may be considered final only once it’s been included in a batch on the L1, the batch and its parents are valid, and the L1 transaction is final.

Checking for finality

To determine the finality of an L2 transaction, one must verify that the commitment/proof transaction has both been included on and finalized by the L1. L2 providers often offer convenient RPC methods that off-chain integrators can use to determine the finality of a given L2 transaction.

Arbitrum Nitro/Optimism

Both Arbitrum and Optimism nodes implement the Ethereum JSON RPC, including the “finalized” block parameter. As a result, eth_getBlockByNumber(“finalized”, ...) can be used to determine finality.


StarkNet’s sequencer provider has a getTransactionStatus() function that reports the transaction’s status in the StarkNet transaction lifecycle. Transactions whose tx_status is ACCEPTED_ON_L1 may be considered final.

ZkSync Classic

ZkSync’s v0.2 API has several endpoints that accept finalization parameters.

  • /accounts/{accountIdOrAddress}/{stateType} may have the stateType set to finalized.
  • /blocks/{blockNumber} accepts lastFinalized as the blockNumber parameter.
  • /blocks/{blockNumber}/transactions{?from,limit,direction} accepts lastFinalized as the blockNumber parameter.

Practicing safe finality

Like other recent innovations in the blockchain space, provable finality has drastically changed the kinds of security assurances a blockchain can provide. However, developers of off-chain or multi-chain applications must be cognizant of the specific finality requirements of different architectures and, where necessary, use the correct techniques to determine whether transactions are final.

Older techniques of determining finality, such as block delays, are not adequate for newer architectures, and using incorrect finality criteria may put applications at risk of double-spend attacks.

If you’re designing a new blockchain or off-chain application and have concerns about finality, please contact us.

Can you pass the Rekt test?

One of the biggest challenges for blockchain developers is objectively assessing their security posture and measuring how it progresses. To address this issue, a working group of Web3 security experts, led by Trail of Bits CEO Dan Guido, met earlier this year to create a simple test for profiling the security of blockchain teams. We call it the Rekt Test.

The Rekt Test is modeled after The Joel Test. Developed 25 years ago by software developer Joel Spolsky, The Joel Test replaced a Byzantine process for determining the maturity and quality of a software team with 12 simple yes-or-no questions. The blockchain industry needs something similar because today’s complex guidance does more to frustrate than to inform.

The Rekt Test focuses on the simplest, most universally applicable security controls to help teams assess security posture and measure progress. The more an organization can answer “yes” to these questions, the more they can trust the quality of their operations. This is not a definitive checklist for blockchain security teams, but it’s a way to start an informed discussion about important security controls.

At the Gathering of Minds conference earlier this year, a group of industry leaders were challenged to address the lack of cybersecurity standards in the blockchain ecosystem. One of these discussions was led by Dan Guido, CEO of Trail of Bits. Other participants included Nathan McCauley (Anchorage Digital), Lee Mount (Euler Labs), Shahar Madar (Fireblocks), Mitchell Amador (Immunefi), Nick Shalek (Ribbit Capital), and others. Through their discussions, the Rekt Test was created:

The Rekt Test

  1. Do you have all actors, roles, and privileges documented?
  2. Do you keep documentation of all the external services, contracts, and oracles you rely on?
  3. Do you have a written and tested incident response plan?
  4. Do you document the best ways to attack your system?
  5. Do you perform identity verification and background checks on all employees?
  6. Do you have a team member with security defined in their role?
  7. Do you require hardware security keys for production systems?
  8. Does your key management system require multiple humans and physical steps?
  9. Do you define key invariants for your system and test them on every commit?
  10. Do you use the best automated tools to discover security issues in your code?
  11. Do you undergo external audits and maintain a vulnerability disclosure or bug bounty program?
  12. Have you considered and mitigated avenues for abusing users of your system?

The landscape of blockchain technology is diverse, extending beyond blockchains to include decentralized protocols, wallets, custody systems, and more, each with unique security nuances. The subsequent explanations of the Rekt Test questions reflect the consensus of best practices agreed to by this group, and are by no means exhaustive or absolute. The intent of the Rekt Test is not to establish rigid benchmarks but to stimulate meaningful conversations about security in the blockchain community. Thus, consider this interpretation as a stepping stone in this critical dialogue.

1. Do you have all actors, roles, and privileges documented?

Comprehensive documentation of all actors, roles, and privileges affecting the blockchain product is crucial, as this clarifies who can access system resources and what actions they are authorized to perform. Actors refer to entities interacting with the system; roles are predefined sets of permissions assigned to actors or groups; and privileges define specific rights and permissions.

Thorough documentation of these entities facilitates comprehensive testing, allowing developers (and external auditors) to identify security gaps, improper access controls, the degree of decentralization, and potential exposure in specific compromise scenarios. Addressing these issues enhances the overall security and integrity of the system. The documentation also serves as a reference point for auditors to compare the actual access privileges with the documented ones, identify any discrepancies, and investigate potential security risks.

2. Do you keep documentation of all the external services, contracts, and oracles you rely on?

Interactions with external smart contracts, oracles, and bridges are fundamental to many key functionalities expected from blockchain applications. A new blockchain application or service may also rely on the assumed security posture of a financial token developed outside of your organization, which increases its complexity and attack surface. As a result, even organizations that integrate the best security procedures into their software development process can fall victim to a destructive security incident.

It is crucial to document all external services (like cloud hosting services and wallet providers), contracts (like DeFi protocols), and oracles (like pricing information) used by a blockchain system in order to identify risk exposure and mitigate incident impact. Doing so will help you answer the following essential questions:

  • How will we know when an external dependency suffers a security incident?
  • What are the specific conditions under which we declare a security incident?
  • What steps will we take when we detect one?

Answering these questions will help you be prepared when, inevitably, a security incident affects a dependency outside of your control. You should be able to notice any change, innocuous or not, in a dependency’s output, interface, or assumed program state; assess it for security impact; and take the necessary next steps. This will limit the security impact on your system and help ensure its uninterrupted operation.

3. Do you have a written and tested incident response plan?

While security in the blockchain space differs from traditional product security (where more centralized or closed systems may be easier to control), both require an effective incident response plan to help remain resilient in the face of a security incident. The plan should include steps to identify, contain, and remediate the incident through automated and manual procedures. An organization should provide training to ensure that all team members are familiar with the plan, and it should include steps for communicating incidents over internal and out-of-band channels. This plan should be regularly tested to ensure it is up-to-date and effective, especially given how quickly the blockchain security world can change. You should create your own incident response (IR) plan, and can use this Trail of Bits guide as a resource.

For blockchain systems, it is especially important that IR plans mitigate key person risk by ensuring the organization is not overly reliant on any single individual. The plan should anticipate scenarios where key personnel may be unavailable or coerced, and outline steps to ensure continuity of operations. Developers should consider decentralizing access controls, implementing quorum-based approvals, and documenting procedures so that multiple team members are prepared to respond.

For blockchain systems, it is especially important that incident response be proactive, not only reactive. The contracts should be designed alongside the creation of the IR plan using strategies like guarded launches to incrementally deploy new code. The developers should consider if they want—or not—pausable features in their contracts, and what part of the protocol should—or should not—be upgradeable or decentralized, as this will influence the team’s capabilities during an incident.

4. Do you document the best ways to attack your system?

By constructing a threat model that documents all potential avenues to attack the system, you can understand whether your existing security controls are sufficient to mitigate attacks. The threat model should visually lay out a product’s entire ecosystem, including information from beyond software development, such as applications, systems, networks, distributed systems, hardware, and business processes. It should identify all of the system’s weak points and clearly explain how an attacker can exploit them, incorporating information from relevant real-world attacks to help you avoid making the same mistakes.

This information will help you understand whether you are concentrating your efforts in the right spots—i.e., whether your current efforts to mitigate attacks are aligned with how and where they are most likely to occur. It will help you understand what a successful attack on your system will look like, and whether you are sufficiently prepared to detect, respond to, and contain it. A good threat model should eliminate surprise and enable your team to deliberately plan mitigations.

5. Do you perform identity verification and background checks on all employees?

Pseudonymous development is commonplace in the blockchain industry, but it impedes accountability, contractual enforcement, and engendering trust in and among stakeholders in a blockchain product. Malicious actors can exploit a lack of identity verification and background checks to interfere with a product’s development, steal funds, or cause other serious harm, and institutions will have no or limited means to punish them. In recent years, North Korean hackers have applied to real positions using fake Linkedin accounts and impersonated companies to offer fraudulent positions. These practices have directly led to severe hacks, including Axie Infinity’s $540 million loss.

As a result, companies must know the identities of and perform background checks on all of their employees, including those who use public pseudonyms. Companies must also reach additional maturity in their access controls and monitoring; for example, they should make prudent decisions surrounding operational security based on an employee’s role, background, and the territory they reside in (i.e., considering local laws and jurisdiction).

6. Do you have a team member with security defined in their role?

There needs to be a person on the team who is accountable for ensuring the safety and security of the blockchain system. Threats against blockchain technology evolve rapidly, and even a single security incident can be devastating. Only a dedicated security engineer has the time, knowledge, and skill set necessary to identify threats, triage incidents, and remediate vulnerabilities, which helps instill trust in your product as it develops.

Ideally, this person will create and oversee a dedicated team with security at the forefront of their job responsibilities, ultimately owning initiatives to get an organization to answer “yes” to other questions on this list. They will oversee cross-departmental efforts, working with developers, administrators, project leads, executives, and others to ensure security practices are included in all aspects of the organization.

7. Do you require hardware security keys for production systems?

Credential stuffing, SIM swap attacks, and spear phishing have nearly neutralized the protective capability of passwords and SMS/push two-factor authentication. For high-risk organizations with value at stake, phishing-resistant hardware keys are the only reasonable option. Special hardware keys should be used to access the company’s resources, including email, chat, servers, and software development platforms. Special care should be taken to protect any operation in production that is very difficult or impossible to reverse.

Using these keys inside your organization is a leading indicator of competent off-chain infrastructure management. Do not be deterred if this seems like a high-volume lift for your IT team. In 2016, Google released a study that showed that implementing these keys was simple, well-received among its 50,000 employees, and strong against malicious attacks. U2F hardware tokens, such as YubiKey and Google Titan, are good choices for hardware keys.

8. Does your key management system require multiple humans and physical steps?

If a single individual maintains the keys that control your system, they can unilaterally make changes that have an outsized impact, without a consensus of the relevant stakeholders. And if an attacker compromises their credentials, they can gain full control of core assets.

Instead, key management should be set up to require a consensus or quorum of multiple people and physical access for important decisions. Multi-person integrity is an effective security policy used in high-risk industries like defense and traditional finance; they protect against compromise via attackers, insider threats (e.g., rogue employees), and coercion, all in one fell swoop. When selecting the trusted set of individuals for a quorum-based setup, it’s crucial to choose those who are both trustworthy and properly incentivized, as including ill-suited or misaligned individuals can undermine the system’s resistance to coercion. By additionally requiring physical key management (e.g., using a physical safe or air-gapped device to store keys), you will significantly reduce the risk of fraud, theft, misuse, or errors by any individual or if any individual’s key or key fragment is compromised.

Blockchain organizations should employ the use of multi-signature or multi-party computation (MPC) controls and cold storage solutions for, at a minimum, the central wallets that hold most of their assets, or opt to use a qualified custodian, depending on specific regulations and needs. The keys to unlock a multi-signature wallet should be stored on trusted hardware, such as a hardware security module (HSM), secure enclave, or a tamper-resistant hardware wallet.

It’s imperative that the deployment and configuration of the trusted hardware is done carefully to limit its attack surface. For example, secrets should never be extractable and network connections should be avoided. Organizations should also establish a strict procedure for moving funds based on parameters like thresholds, affected wallets, destination, and key person(s) initiating the transaction.

9. Do you define key invariants for your system and test them on every commit?

An invariant is a condition that must remain true throughout the program’s execution. Invariants can target the system as a whole (e.g., no user should have more tokens than the total supply) or target a specific function (e.g., a compute_price function cannot lead to free assets). Understanding and defining invariants helps developers be explicit about the system’s expected behaviors and helps security engineers evaluate whether those behaviors measure up to expectations. This provides a roadmap for security testing and reduces the likelihood of unexpected outcomes and failures.

Defining invariants starts with documenting the assumptions made about the system in plain English. These invariants should cover a breadth of functional and cryptographic properties and their valid states, state transitions, and high-level behaviors. Well-specified systems may have hundreds of properties: you should focus on the most important ones first and continuously work to improve their depth of coverage. To ensure that the code follows the invariants, they must be tested with an automated tool (such as a fuzzer or a tool based on formal methods) throughout the development process. 

10. Do you use the best automated tools for discovering security issues in your code?

Automated security tools are a baseline requirement in a successful security strategy. Fully automated tools, such as static analyzers, automatically find common mistakes and require low maintenance, while semi-automated tools, like fuzzers, allow developers to go one step further and check for logical issues. Many such tools are available, but we recommend using those that are actively used by top-tier security engineers, for which a proven track record of discovered bugs is available.

Trail of Bits’ smart contract security tools use state-of-the-art technology and can be integrated into your CI systems and the edit/test/debug cycle. They include Echidna, a smart contract fuzzer for Solidity smart contracts, and Slither, a static analyzer for Solidity smart contracts. Automating the use of these tools during development and testing helps developers catch critical security bugs before deployment.

11. Do you undergo external audits and maintain a vulnerability disclosure or bug bounty program?

To identify vulnerabilities in blockchain code, it isn’t enough to rely on internal security teams. Instead, organizations must work with external auditors who possess in-depth knowledge of modern blockchain technology, spanning low-level implementations, financial products and their underlying assumptions, and the libraries, services, bridges, and other infrastructure that power modern applications. (Websites that track blockchain security incidents are filled with companies that did not seek external guidance for sometimes complex changes.)

Security auditors help to identify vulnerabilities and provide advice for restructuring your development and testing workflow so these vulnerabilities do not come back. When looking for an audit, it’s important to clarify which components are under review, which are excluded, and the level of effort that should be applied, including through the use of tooling. By understanding the benefits and limitations of an audit, an organization can focus on additional areas needed for improvements once the audit has concluded.

Additionally, a vulnerability disclosure or bug bounty program can enhance your security posture by providing a publicly accessible option for users or researchers to contact you if they uncover a bug. By establishing these programs, organizations show a willingness to engage with independent bug hunters—and without them, they may instead publicly disclose the bugs on social media or even exploit them for their own gain. While these programs offer many benefits, it is important to consider their limitations and pitfalls. For example, bug bounty hunters will not provide recommendations for improving the security maturity of the system nor for reducing the likelihood of bugs in the long term. In addition, your team will still be responsible for triaging bug submissions, which can require constant dedicated resources.

12. Have you considered and mitigated avenues for abusing users of your system?

Many attacks against blockchain, such as phishing, Twitter/Discord scams, and “pig butchering,” attempt to fool users into taking irreparable actions while using your products. Even if an organization has the most expertly designed security system to protect itself, its own users may still be vulnerable. For example, blockchain applications often rely on cryptographic signatures that increase the likelihood of phishing attempts. Developers should consider making the signatures easily identifiable (for example, with EIP-712) and should create and promote guidance for their users to minimize the risk of abuse.

To avoid such attacks, an organization’s security strategy should include abusability testing, where your team considers how attackers can inflict social, psychological, and physical harm. Understanding the risks of significant financial or societal harms will help your team to evaluate necessary processes and mitigations. For example, if your protocol’s users include high-impact stakeholders, such as retirement funds, creating an assurance fund based on the protocol’s fees may help to make the users whole in case of compromise.

Don’t get rekt

These 12 controls are not the only actions that can determine your security posture, but we’re confident that they will enhance every developer’s software and operational security, even as blockchain technology rapidly innovates. This test should not serve as a one-time exercise; these questions have lasting value and should give organizations a roadmap as they continue to grow and develop new products. Answering “yes” to these questions doesn’t mean you will completely avoid a security incident, but it can empower you and your team to steer clear of the worst label in the industry: getting rekt.

Use our suite of eBPF libraries

By Artem Dinaburg

Trail of Bits has developed a suite of open-source libraries designed to streamline the creation and deployment of eBPF applications. These libraries facilitate efficient process and network event monitoring, function tracing, kernel debug symbol parsing, and eBPF code generation.

Previously, deploying portable, dependency-free eBPF applications posed significant challenges due to Linux kernel version disparities and the need for external tools for C-to-eBPF bytecode translation. We’ve addressed these issues with our innovative libraries, which use the latest eBPF and Linux kernel features to reduce external dependencies. These tools, ideal for creating on-machine agents and enabling cloud-native monitoring, are actively maintained and compatible with a variety of Linux distributions and kernel releases. Some are even integral to the functionality of osquery, the renowned endpoint visibility framework.

Our eBPF libraries

The libraries in this suite are linuxevents, ebpfpub, btfparse, and ebpf-common. Together they can be used to develop streamlined event monitoring with a high degree of accuracy and efficiency. Their applications range from network event monitoring, function tracing, and kernel debug symbol parsing to assisting in generating and using eBPF code.

linuxevents: A container-aware library for process monitoring with no runtime dependencies

The linuxevents library showcases how eBPF can monitor events without requiring accurate kernel headers or other external dependencies. No more shipping kernel headers, multiple copies of precompiled eBPF bytecode, or dependencies on BCC! The linuxevents library supports runtime code generation to create custom probes at runtime, not just during build. It is also much faster than traditional system-call-based hooking, an essential feature when monitoring events from multiple containers on a single machine. How does linuxevents do this?

First, linuxevents uses the Linux kernel’s BTF debugging data (via our btfparse library) to accurately identify function prototypes and kernel data structures. This allows linuxevents to automatically adjust to variances in data structure layout and to hook arbitrary non-public symbols in a way that greatly simplifies tracing.

This approach is faster than traditional system call based hooking not only because it has to hook fewer things (sched_process_exec vs execve, execveat, etc.) but also because it can avoid expensive correlations. For example, to trace which program on disk is executed via execve, one would normally have to correlate a file descriptor passed to execve with an open call and multiple chdir calls to get the full path of a program. The correlation is computationally expensive, especially on a machine with multiple active containers. The linuxevents library uses an accurate kernel data structure representation to hook just one function and simply extract the path from the kernel’s vfs layer.

A recording of the linuxevents library being used as a part of the execsnoop example that comes with the library

The linuxevents library is still a proof of concept; it is in use by osquery as a toggleable experiment. The library also has a canonical example of tracing executed processes with cross-container visibility.

ebpfpub: A function-tracing library for Linux

The ebpfpub library allows for monitoring system calls across multiple Linux kernel versions while relying on minimal external runtime dependencies. In ebpfpub, eBPF probes are autogenerated from function prototypes defined via a simple custom language, which can be created from tracepoint descriptors. This approach required proper headers for the running kernel and it came with performance penalties, such as the need to match file descriptors with system calls.

Depending on the desired target event, ebpfpub can use either kernel tracepoints, kprobes, or uprobes as the underlying tracing mechanism. The library includes the following examples:

  • execsnoop: Shows how to use Linux kernel tracepoints to detect program execution via execve()
  • kprobe_execsnoop: Like execsnoop, but uses a different hooking mechanism (kprobes instead of tracepoints)
  • readline_trace: Uses uprobes to hook the user-mode readline library, which can enable use cases such as monitoring whenever a live shell is used on a machine
  • sockevents: An example of how to trace sockets through a series of connect/accept/bind calls that establish connectivity to a remote machine
  • systemd_resolved: Shows how to use uprobes to hook into systemd’s DNS service (systemd-resolved), which will show in real time the domains being looked up by your local machine

The ebpfpub library is currently used by osquery to capture process and socket events by tracing executed system calls. While ebpfpub is still maintained and useful in specific circumstances (like the need to support older kernels and use runtime code generation), new projects should use the linuxevents approach instead.

btfparse: A C++ library that parses kernel debug symbols in BTF format

BTF, or the Binary Type Format, is a compact binary format for representing type information in the Linux kernel. BTF stores data such as structures, unions, enumerations, and typedefs. Debuggers and other tools can use BTF data to enable richer debugging features by understanding complex C types and expressions. BTF was introduced in Linux 4.20 and is generated from source code and traditional debugging information like DWARF. BTF is more compact than DWARF, and it improves the debugging experience by conveying more semantic type information than was previously available. The standardized BTF format also allows new debugging tools to leverage type data across compilers, enabling more consistent quality of introspection across languages.

The btfparse library allows you to read BTF data in your C++ projects and generate header files directly in memory without any external application. The library also comes with a tool, called btf-dump, that serves both as an example of using btfparse and as a standalone tool that can dump BTF data present in a Linux kernel image.

ebpf-common: A C++ library to help write new eBPF-based tools

The ebpf-common library is a set of utilities that assist with generating, loading, and using eBPF code. It is the common substrate that underpins all of our eBPF-related tooling. Use epbf-common to create your own runtime, eBPF-based tools!

The ebpf-common library’s main job is to compile C code to eBPF bytecode and to provide helpful abstractions that make writing eBPF hooks easier. Here are some of the features ebpf-common provides:

  • It uses LLVM and clang as libraries that write to in-memory scratch buffers.
  • It includes abstractions to make accessing eBPF data structures (hash maps, arrays, ring buffers, etc.) simple. These data structures are used to exchange data between eBPF and your application.
  • It includes abstractions to create and read perf outputs, another way eBPF can communicate with the tracing application.
  • It allows for the management of events (like kprobes, uprobes, and tracepoints) that trigger the execution of eBPF programs.
  • Finally, it includes functions that implement eBPF helpers and related functionality via LLVM.

The ebpf-common library is used as the core of all of our other eBPF tools, which serve as library clients and examples of use cases for ebpf-common for your applications. Refer to our blog post All your tracing are belong to BPF for additional guidance and examples for how to use ebpf-common.

Our eBPF tools

ebpfault: A Linux system call fault injector built on top of eBPF

ebpfault is a system-wide fault injector that does not require risky kernel drivers that could crash the system. It can start a specific program, target running processes, or target all processes except those on a specific list. A simple configuration file in JSON format lets you configure faults by using the syscall name, the probability of injecting a fault, and the error code that should be returned.

A recording of ebpfault running against the htop process and causing faults via a specific configuration

BPF deep-dives and talks

Most of the material available online is geared toward using the command-line sample tools that demonstrate eBPF, which work mostly as standalone demonstrations and not as reusable libraries. We wanted to fill in the gaps for developers and provide a step-by-step guide on how to actually integrate eBPF from scratch from the point of view of a developer writing a tracing tool. The documentation focuses on runtime code generation using the LLVM libraries.

The All your tracing belong to BPF blog post and our companion code guide show how to use epbf-common to create a tool that uses eBPF to count system calls, with each example increasing in complexity, starting from simple counting, to using maps to store data, to finally using perf events for outputs.

Monitoring Linux events is a talk by Alessandro Gario about using eBPF for event monitoring. Alessandro describes how to dynamically decide on what to monitor and to generate your own eBPF bytecode directly from C++. He touches on some of the intricacies of eBPF maps, perf events, and practical considerations for using our eBPF tools and libraries.

eBPF public contributions

The world of eBPF continues to expand and find applications in various domains. We have explored eBPF as it relates to interesting tasks beyond tracing and performance monitoring, such as improving CI/CD for eBPF bytecode, writing an eBPF-to-ARM64 JIT compiler for the Solana platform, and improving the experience of building eBPF projects on Windows.

ebpf-verifier: Sometimes it is necessary to bundle prebuilt eBPF programs. The Linux kernel “verifies” eBPF programs at load time and rejects any that it deems unsafe. Bundling eBPF bytecode is a CI/CD nightmare because every kernel’s verification is ever so slightly different. ebpf-verifier aims to eliminate that nightmare by executing the eBPF verifier outside of the running kernel and opens the door to the possibility of testing eBPF programs across different kernel versions.

Solana eBPF-to-ARM64 JIT compiler: eBPF makes an appearance in many surprising places! The Solana blockchain uses an eBPF virtual machine to run its smart contracts, and it uses a JIT compiler to compile the eBPF bytecode to native architectures. Trail of Bits ported the Solana eBPF JIT compiler to ARM64 to allow Solana applications to natively run on the now very popular ARM64 platforms like Apple Silicon.

Adding CMake support to eBPF for Windows: eBPF also works on Windows! To make Windows development easier, we ported the prior Visual Studio–based build system to CMake. The improvements include better handling of transitive dependencies and properties, better packaging, and enhanced build settings for a more efficient development experience.


We’ve used eBPF to provide rapid, high-quality monitoring data for system instrumentation agents like osquery. Our intention is that the frameworks and tools we’ve created will assist developers in integrating eBPF into their applications more seamlessly. eBPF is a useful technology with a bright future in a variety of fields, including increasingly in cloud-native monitoring and observation.

We plan to share more of our lessons learned from eBPF tool development in the near future, and we hope to apply some of these lessons to the problems of cloud-native monitoring and observability.

A mistake in the bulletproofs paper could have led to the theft of millions of dollars

By Jim Miller

We discovered a critical vulnerability in Incognito Chain that would allow an attacker to mint arbitrary tokens and drain user funds. Incognito offers confidential transactions through zero-knowledge proofs, so an attacker could have stolen millions of dollars of shielded funds without ever being detected or identified. The vulnerability stemmed from an insecure implementation of the Fiat-Shamir transformation of Incognito’s bulletproofs. 

This is not the first instance of this vulnerability; last year we disclosed a series of these same types of issues, which we dubbed Frozen Heart vulnerabilities. These vulnerabilities, as we detailed in our earlier blog series, resulted from a mistake in the original bulletproofs paper. The mistake was in the paper for over four years before it was corrected last year in response to our disclosure.

Since posting that blog series, Trail of Bits has continued research with Paul Grubbs (University of Michigan) and Quang Dao (Carnegie Mellon University) to review more codebases and proof systems for this issue, resulting in a paper recently accepted for publication at IEEE S&P 2023. The vulnerability in Incognito Chain was identified during this research.

After discovering the issue, we informed multiple members of the Incognito Chain team, who patched the vulnerability and released a new version of their privacy protocol. This new version has since been adopted by their validators, effectively patching the vulnerability and, to our knowledge, securing all funds.

Understanding bulletproofs, Fiat-Shamir, Frozen Heart, and confidential transactions

Bulletproofs are a special kind of zero-knowledge proof, also known as a range proof. They permit a prover to validate that an encrypted value lies within a specific range. These proofs serve as a crucial foundation for confidential transactions.

Fiat-Shamir is a transformation we detailed in a series of previous blog posts. These posts delve into how it can be incorrectly implemented and exploited across several zero-knowledge proof systems.

Frozen Heart vulnerabilities break the security of zero-knowledge proofs. When exploited, an attacker can forge zero-knowledge proofs, tricking the verifier into accepting incorrect proofs. This compromise can have severe implications for the protocol’s security.

Confidential transactions are a significant feature of Incognito Chain. Similar to Monero, these transactions hide the amount being transferred and the identities of both the sender and receiver.

For a more detailed examination of the Incognito Chain protocol and the cryptographic primitives it contains, you can check out their forum. A brief explanation is that coins are the encryption of their underlying value (technically, a commitment scheme instead of an encryption scheme). The identities of the sender and receiver are concealed by using a ring signature, which proves the transaction came from a group (or ring) of keys, one of which is controlled by the sender. To hide the receiver’s identity, the protocol relies on stealth or one-time addresses; the sender in the transaction generates one-time public keys that only the receiver can access.

A transaction includes a set of input coins, output coins, and a transaction fee. For a transaction to be valid, the sum of the inputs should equal the sum of the outputs plus the transaction fee. However, the values of the coins are encrypted. This challenge is overcome through the protocol’s reliance on homomorphic encryption, which allows the values of inputs and outputs to be added and subtracted while remaining encrypted. This balance check can be performed homomorphically, allowing the verified transaction to validate that the sums are equal without knowing their actual values.

There is a caveat to this balance check. Homomorphic encryption schemes ensure sums are equal modulo some value, typically a 256-bit or larger prime value (the group order). An adept attacker could manipulate this fact to execute an attack that functions essentially as an integer overflow, where the overflow occurs modulo the large prime. Consequently, instead of the transaction not generating additional funds (as a secure protocol requires), an attacker could mint extra funds—a substantial amount equivalent to a 256-bit prime number.

The protocol designers anticipated this issue, employing bulletproofs as a safeguard. Bulletproofs, as range proofs, confirm that all input and output values are less than a specific maximum. Thus, if the group order is a 256-bit prime, we can avert this overflow attack by using bulletproofs that validate that the values of each coin are at most 264. An overflow attack would then be unattainable when using a reasonable number of inputs and outputs.

Vulnerability details

Bulletproofs are an essential part of confidential transactions. They limit the underlying value of the privacy coins, thus safeguarding the system from attackers minting money illicitly. Like most zero-knowledge proofs, bulletproofs use the Fiat-Shamir transformation to be noninteractive. Our prior blog post highlights how an error in the original bulletproofs paper led to insecure implementations of the Fiat-Shamir protocol. If this mistake is implemented as instructed, an attacker can forge bulletproofs for values outside the range.

The original bulletproofs forgery results in a coin that has a uniformly random value. Although this could complicate exploitation since you can’t control the value exactly, it’s not entirely impossible. As we elaborate in the Attacking Mimblewimble section of our recently published paper, Wagner’s k-sum algorithm could be employed to engineer such an exploit.

However, Incognito Chain uses a variant of bulletproofs known as aggregate bulletproofs. As the name suggests, this variant aggregates multiple bulletproofs into a single proof, allowing more efficient verification. When this variant of bulletproofs is vulnerable to Frozen Heart, the severity escalates notably because it grants an attacker the liberty to select arbitrary values for the coins instead of being confined to random values. With this level of control over these values, an attacker can effortlessly solve the balance equation, thereby generating free money. What amplifies the concern surrounding this vulnerability is its target: confidential transactions, which inherently hide most information from external observers, make practical detection of exploitation a formidable challenge.

Coordinated disclosure

We notified multiple members of the Incognito Chain team of this vulnerability on April 25, 2023. They responded swiftly, confirmed the issue, and started working on a fix. On April 26, 2023, Incognito Chain submitted this patch and other commits that fixed the bulletproofs implementation. 

The Incognito Chain team released this patch as part of a new version (v3) of their privacy protocol to prevent future exploits. However, this initial patch inadvertently introduced a bug that caused a temporary network outage. The team detailed this issue on their forum, the problem has been resolved, and the network is operational once again.

We appreciate the quick and efficient response of the Incognito Chain team in addressing this issue.

How AI will affect cybersecurity: What we told the CFTC

Dan Guido, CEO

The second meeting of the Commodity Futures Trading Commission’s Technology Advisory Committee (TAC) on July 18 focused on the effects of AI on the financial sector. During the meeting, I explained that AI has the potential to fundamentally change the balance between cyber offense and defense, and that we need security-focused benchmarks and taxonomies to properly assess AI capabilities and risks.

  • The widespread availability of capable AI models presents new offensive opportunities that defenders must now account for. AI will make certain attacks dramatically easier, upsetting the equilibrium of offense and defense. We must reevaluate our defenses given this new reality.
  • Many think AI is either magical or useless, but the truth lies between these extremes. AI augments human capabilities; it does not wholly replace human judgment and expertise. One key question is: can a mid-level practitioner operate at an expert level with the help of AI? Our experience suggests yes.
    • AI models can do many helpful things: decompile code into high-level languages, identify and trigger bugs, and write scripts to launch exploits. But to leverage it effectively, we must ask the right questions (e.g., with knowledge of the subject matter and prompt engineering techniques) and evaluate progress correctly (is AI better than state-of-the-art techniques)?
    • It’s also necessary to choose the right problems. AI is better for problems that require breadth of knowledge and where mistakes are acceptable (e.g., document this function, write a phishing email). It’s not great at problems that require mastery and correctness (e.g., find and exploit this iOS 0-day).
  • Bug bounties, phishing defenses, antivirus, IDS, and attribution will be among the first fields impacted as AI confers a greater advantage to attackers in the near term. For example, AI can mass produce tailored phishing messages, for every target, in their native language, and without errors. We can’t just regulate these problems away; alignment and attempts to restrict model availability won’t work, since impressively capable open-source models are already here.
  • What’s needed now is a systematic measurement of these models’ capabilities that focuses on cybersecurity, not programming. We need benchmarks that let us compare AI versus existing state-of-the-art tools and human experts, and taxonomies that map advancements to opportunities and risks.

The full video is available here:

Finally, I am honored to have been named the co-chair of the Subcommittee on Cybersecurity. I look forward to continuing our work with the committee. We will continue studying the risks and opportunities of AI, supply chain security, and authentication technology in the finance industry.

Read our prior coverage of the CFTC TAC’s first meeting, which focused on blockchain risks. For our work on AI-enabled cybersecurity, see the links below:

The future of Clang-based tooling

By Peter Goodman

Clang is a marvelous compiler; it’s a compiler’s compiler! But it isn’t a toolsmith’s compiler. As a toolsmith, my ideal compiler would be an open book, allowing me to get to everywhere from anywhere. The data on which my ideal compiler would operate (files, macros, tokens), their eventual interpretation (declarations, statements, types), and their relations (data flow, control flow) would all be connected.

On its own, Clang does not do these things. libClang looks like an off-the-shelf, ready-to-use solution to your C, C++, and Objective-C parsing problems, but it’s not. In this post, I’ll investigate the factors that drive Clang’s popularity, why its tooling capabilities are surprisingly lacking despite those factors, and the new solutions that make Clang’s future bright.

What lies behind Clang’s success?

Clang is the name of the “compiler front end” that generates an intermediate representation (IR) from your C, C++, and Objective-C source code. That generated IR is subsequently taken as input by the LLVM compiler back end, which converts the IR into machine code. Readers of this blog will know LLVM by the trail of our lifting tools.

I adopted Clang as my primary compiler over a decade ago because of its actionable (and pretty!) diagnostic messages. However, Clang has only recently become one of the most popular production-quality compilers. I believe this because it has, over time, accumulated the following factors that drive compiler popularity:

  1. Fast compile times: Developers don’t want to wait ages for their code to compile.
  2. Generated machine code runs quickly: Everyone wants their code to run faster, and for some users, a small-percentage performance improvement can translate to millions of dollars in cost savings (so cloud spend can go further!).
  3. End-to-end correctness: Developers need to trust that the compiler will almost always (because bugs do happen) translate their source code into semantically equivalent machine code.
  4. Quality of diagnostic messages: Developers want actionable messages that point to errors in their code, and ideally recommend solutions.
  5. Generates debuggable machine code: The machine code must work with yesterday’s debugger formats.
  6. Backing and momentum: People with lots of time (those in academia) or money (those in the industry) need to push forward the compiler’s development so that it is always improving on the above metrics.

However, one important factor is missing from this list: tooling. Despite many improvements over the past few years, Clang’s tooling story still has a long way to go. The goal of this blog post is to present a reality check about the current state of Clang-based tooling, so let’s dive in!

The Clang AST is a lie

Clang’s abstract syntax tree (AST) is the primary abstraction upon which all tooling is based. ASTs capture essential information from source code and act as scaffolding for semantic analysis (e.g., type checking) and code generation.

But what about when things aren’t in the source code? In C++, for example, one generally does not explicitly invoke class destructor methods. Instead, those methods are implicitly invoked at the end of an object’s lifetime. C++ is full of these implicit behaviors, and almost none of them are actually explicitly represented in the Clang AST. This is a big blind spot for tools operating on the Clang AST.

The Clang CFG is a (pretty good) lie

I complained above that it was a shame that the wealth of information available to compilers is basically left on the table in favor of ad-hoc solutions. To be fair, this is simplistic; Clang is not ideally engineered for interactivity within an IDE, for example. But also, there are some really fantastic Clang-based tools out there that are actively used and developed, such as the Clang Static Analyzer.

Because the Clang Static Analyzer is “built on Clang,” one might assume that its analyses are performed on a representation that is faithful to both the Clang AST and the generated LLVM IR. Yet just above, I revealed to you that the Clang AST is a lie—it’s missing quite a bit, such as implicit C++ destructor calls. The Clang Static Analyzer apparently side-steps this issue by operating on a data structure called the CFG.

The Clang CFG, short for control-flow graph, represents how a theoretical computer would execute the statements encoded in the AST. The accuracy of analysis results hinges on the accuracy of the CFG. Yet the CFG isn’t actually used during Clang’s codegen process, which produces LLVM IR containing—you guessed it—control-flow information. The Clang CFG is actually just a very good approximation of the implementation that actually matters. As a toolsmith, I care about accuracy; I don’t want to have to guess about where the abstraction leaks.

LLVM IR as the one true IR is a lie

Clang’s intermediate representation, LLVM IR, is produced directly from the Clang AST. LLVM IR is superficially machine code independent. The closer you look, the easier it is to spot the machine-dependent parts, such as intrinsics, target triples, and data layouts. However, these parts are not expected to be retargetable because they are explicitly specific to the target architecture.

What makes LLVM IR fall short of being a practically retargetable IR actually has very little to do with LLVM IR itself, and more to do with how it is produced by Clang. Clang doesn’t produce identical-looking LLVM IR when compiling the same code for different architectures. Trivial examples of this are that LLVM IR contains constant values where the source code contained expressions like sizeof(void *). But those are the known knowns; the things that developers can reasonably predict will differ. The unreasonable differences happen when Clang over-eagerly chooses type, function parameter, and function return value representations that will “fit” well with the target application binary interface (ABI). In practice, this means that your std::pair<int, int> function parameter might be represented as a single i64, two i32s, an array of two i32s, or even as a pointer to a structure… but never a structure. Hilariously, LLVM’s back end handles structure-typed parameters just fine and correctly performs target-specific ABI lowering. I bet there are bugs lurking between these two completely different systems for ABI lowering. Reminds you of the CFG situation a bit, right?

The takeaway here is that the Clang AST is missing information that is invented by the LLVM IR code generator, but LLVM IR is also missing information that is destroyed by said code generator. And if you want to bridge that gap, you need to rely on an approximation: the Clang CFG.

Encore: the lib in libClang is a lie

Libraries are meant to be embedded into larger programs; therefore, they should strive not to trigger aborts that would tear down those program processes! Especially not when performing read-only, non-state-mutating operations. I say the “lib” in libClang is a lie because the “Clang API” isn’t really intended as an external API; it’s an internal API for the rest of Clang. When Clang is using itself incorrectly, it makes sense to trigger an assertion and abort execution—it’s probably a sign of a bug. But it just so happens that a significant portion of Clang’s API is exposed in library form, so here we are today with libClang, which pretends to be a library but is not engineered as such.

Encore the second: compile_commands.json is a lie

The accepted way to run Clang-based tooling on a whole program or project is a JSON format aptly named compile_commands.json. This JSON format embeds the invocation of compilers in command form (either as a string – yuck!, or as a list of arguments), the directory in which the compiler operated, and the primary source file being compiled.

Unfortunately, this format is missing environment variables (those pesky things!). Yes, environment variables materially affect the operation and behavior of compilers. Better-known variables like CPATH, C_INCLUDE_PATH, and CPLUS_INCLUDE_PATH affect how the compiler resolves #include directives. But did you know about CCC_OVERRIDE_OPTIONS? If not, guess what: neither does compile_commands.json!

Okay, so maybe these environment variables are not that frequently used. Another environment variable, PATH, is always used. When one types clang at the command line, the PATH variable is partially responsible for figuring out to which Clang binary the variable will be executed. Depending on your system and setup, this might mean Apple Clang, Homebrew Clang, vcpkg Clang, one of the many Clangs available in Debian’s package manager, or maybe a custom-built one. This matters because the clang executable is introspective. Clang uses its own binary’s path to discover, among other things, the location of the resource directory containing header files like stdarg.h.

As a toolsmith, I want to be able to faithfully reproduce the original build, but I can’t do that with the compile_commands.json format as it exists today.

Final encore: Compilers textbooks are lying to you (sort of)

I promise this is my last rant, but this one cuts to the crux of the problem. Compilers neatly fit the pipeline architecture: Source code files are lexed into tokens, which are then structured into AST by parsers. The ASTs are then analyzed for semantic correctness by type checkers before being converted into IRs for generic optimizations. Finally, the IR is targeted and lowered into a specific machine code by the back end.

This theoretical pipeline architecture has many nice properties. Pipeline architectures potentially enable third-party tools to be introduced between any two stages, so long as the tool consumes the right input format and produces the right output format. In fact, it is this pipeline nature that makes the LLVM back end excel at optimization. LLVM optimizers are “passes” that logically consume and produce LLVM IR.

The truth is that in Clang, lexing, parsing, and semantic analysis are a fractal of colluding components that cannot easily be teased apart. The semantic analyzer drives the pre-processor, which co-routines with the lexer to identify, annotate, and then discard tokens as soon as they aren’t needed. Clang keeps just enough information around to report pretty diagnostics and to handle parsing ambiguities in languages like C++, and throws away the rest in order to be as fast and memory-efficient as possible.

What this means in practice is that, surprisingly, Clang’s preprocessor can’t actually operate correctly on a pre-lexed token stream. And there are more subtle consequences; for example, interposing on the preprocessor to capture macro expansions appears to be supported, but is barely usable in practice. This support is implemented via a callback mechanism. Unfortunately, the callbacks often lack sufficient context or are called at the wrong time. From the stream of callbacks alone, one can’t distinguish between scenarios like macro expansion of macro arguments vs. expansion that occurs before a function-like macro invocation, or macro expansions before vs. inside of a conditional directive. This matters for tools that want to present both the source and the macro expansion tree. There’s a reason why Clang-based tools like the excellent Woboq Code Browser invoke a second preprocessor inside of the callbacks; there’s just no other way to see what actually happens.

At the end of the day, the mental model of a traditional compiler pipeline neatly described by compiler textbooks is simplistic and does not represent the way Clang actually works. Preprocessing is a remarkably complex problem, and reality often demands complex solutions to such problems.

The future of Clang-based tooling is on its way

If you agree with my rant, check out PASTA, a C++ and Python wrapper around a large percentage of Clang’s API surface area. It does things big and small. Among small things, it provides a disciplined and consistent naming scheme for all API methods, automatic memory management of all underlying data structures, and proper management of compile commands. Among the big, it provides bi-directional mappings between lexed tokens from files and AST nodes, and it makes API methods conventionally safe to use even if you shouldn’t use them (because Clang doesn’t document when things assert and tear down your process).

PASTA isn’t a panacea for all of my complaints. But—lucky for you, aspiring Clang toolsmith or reader—DARPA is generously funding the future of compiler research. As part of the DARPA V-SPELLS program, Trail of Bits is developing VAST, a new MLIR-based middle-end to Clang which we introduced in our VAST-checker blog post. VAST converts Clang ASTs into a high-level, information-rich MLIR dialect that simultaneously maintains provenance with the AST and contains explicit control- and data-flow information. VAST progressively lowers this MLIR, eventually reaching all the way down to LLVM IR. Maybe those textbooks weren’t lying after all, because this sounds like a pipeline connecting Clang’s AST to LLVM IR.

That’s right: we’re not throwing the baby out with the bathwater. Despite my long rant, Clang is still a great C, C++, and Objective-C front end, and LLVM is a great optimizer and back end. The needs of the time conspired to fit these two gems together in a less-than-ideal setting, and we’re working to develop the crown jewel. Watch this spot because we will be releasing a tool combining PASTA and VAST in the near future under a permissive open-source license.

This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA). The views, opinions and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.

Distribution Statement A – Approved for Public Release, Distribution Unlimited