It pays to be Circomspect

By Fredrik Dahlgren, Staff Security Engineer

In October 2019, a security researcher found a devastating vulnerability in Tornado.cash, a decentralized, non-custodial mixer on the Ethereum network. Tornado.cash uses zero-knowledge proofs (ZKPs) to allow its users to privately deposit and withdraw funds. The proofs are supposed to guarantee that each withdrawal can be matched against a corresponding deposit to the mixer. However, because of an issue in one of the ZKPs, anyone could forge a proof of deposit and withdraw funds from the system.

At the time, the Tornado.cash team saved its users’ funds by exploiting the vulnerability to drain the funds from the mixer before the issue was discovered by someone else. Then they patched the ZKPs and migrated all user funds to a new version of the contract. Considering the severity of the underlying vulnerability, it is almost ironic that the fix consisted of just two characters.

The fix: Simply replace = by <== and all is well (obviously!).

This bug would have been caught using Circomspect, a new static analyzer for ZKPs that we are open-sourcing today. Circomspect finds potential vulnerabilities and code smells in ZKPs developed using Circom, the language used for the ZKPs deployed by Tornado.cash. It can identify a wide range of issues that can occur in Circom programs. In particular, it would have found the vulnerability in Tornado.cash early in the development process, before the contract was deployed on-chain.

How Circom works

Tornado.cash was developed using Circom, a domain-specific language (DSL) and a compiler that can be used to generate and verify ZKPs. ZKPs are powerful cryptographic tools that allow you to make proofs about a statement without revealing any private information. For complex systems like a full computer program, the difficult part in using ZKPs becomes representing the statement in a format that the proof system can understand. Circom and other DSLs are used to describe a computation, together with a set of constraints on the program inputs and outputs (known as signals). The Circom compiler takes a program and generates a prover and a verifier. The prover can be used to run the computation described by the DSL on a set of public and private inputs to produce an output, together with a proof that the computation was run correctly. The verifier can then take the public inputs and the computed output and verify them against the proof generated by the prover. If the public inputs do not correspond to the provided output, this is detected by the verifier.

The following figure shows a small toy example of a Circom program allowing the user to prove that they know a private input x such that x5 - 2x4 + 5x - 4 = 0:

A toy Circom program where the private variable x is a solution to a polynomial equation

The line y <== x5 - 2 * x4 + 5 * x - 4 tells the compiler two things: that the prover should assign the value of the right-hand side to y during the proof generation phase (denoted y <-- x5 - 2 * x4 + 5 * x - 4 in Circom), and that the verifier should ensure that y is equal to the right-hand side during the proof verification phase (which is denoted y === x5 - 2 * x4 + 5 * x - 4 in Circom). This type of duality is often present in zero-knowledge DSLs like Circom. The prover performs a computation, and the verifier has to ensure that the computation is correct. Sometimes these two sides of the same coin can be described using the same code path, but sometimes (for example, due to restrictions on how constraints may be specified in R1CS-based systems like Circom) we need to use different code to describe computation and verification. If we forget to add instructions describing the verification steps corresponding to the computation performed by the prover, it may become possible to forge proofs.

The Tornado.cash vulnerability

In the case of Tornado.cash, it turned out that the MIMC hash function used to compute the Merkle tree root in the proof used only the assignment operator <-- when defining the output. (Actually, it uses =, as demonstrated in the GitHub diff above. However, in the previous version of the Circom compiler, this was interpreted in the same way as <--. Today, this code would generate a compilation error.) As we have seen, this only assigned a value to the output during proof generation, but did not constrain the output during proof verification, leaving the verifying contract vulnerable.

Our new Circom bug finder, Circomspect

Circomspect is a static-analyzer and linter for programs written in the Circom DSL. Its main use is as a tool for reviewing the security and correctness of Circom templates and functions. The implementation is based on the Circom compiler and uses the same parser as the compiler does. This ensures that any program that the compiler can parse can also be parsed using Circomspect. The abstract syntax tree generated by the parser is converted to static single-assignment form, which allows us to perform simple data flow analyses on the input program.

The current version implements a number of analysis passes, checking Circom programs for potential issues like unconstrained signals, unused variables, and shadowing variable declarations. It warns the user about each use of the signal assignment operator <--, and can often detect if a circuit uses <-- to assign a quadratic expression to a signal, indicating that the signal constraint assignment operator <== could be used instead. This analysis pass would have found the vulnerability in the Tornado.cash described above. All issues flagged by Circomspect do not represent vulnerabilities, but rather locations that should be reviewed to make sure that the code does what is expected.

As an example of the types of issues Circomspect can find, consider the following function from the circom-pairing repository:

An example function from the circom-pairing repository

This function may look a bit daunting at first sight. It implements inversion modulo p using the extended Euclidean algorithm. Running Circomspect on the containing file yields a number of warnings telling us that the assignments to the arrays y, v, and newv do not contribute to the return value of the function, which means that they cannot influence either witness or constraint generation.

Running Circomspect on the function find_Fp_inverse produces a number of warnings.

A closer look at the implementation reveals that the variable y is used only to compute newv, while newv is used only to update v and v is used only to update y. It follows that none of the variables y, v, and newv contribute to the return value of the function find_Fp_inverse, and all can safely be removed. (As an aside, this makes complete sense since running the extended Euclidean algorithm on two coprime integers num and p computes two integers x and y such that x * num + * p = 1. This means that if we’re interested in the inverse of num modulo p, it is given by x, and the value of y is not needed. Since x and y are computed independently, the code used to compute y can safely be removed.)

Improving the state of ZKP tooling

Zero-knowledge DSLs like Circom have democratized ZKPs. They allow developers without a background in mathematics or cryptography to build and deploy systems that use zero-knowledge technology to protect their users. However, since ZKPs are often used to protect user privacy or assure computational integrity, any vulnerability in a ZPK typically has serious ramifications for the security and privacy guarantees of the entire system. In addition, since these DSLs are new and emerging pieces of technology, there is very little tooling support available for developers.

At Trail of Bits, we are actively working to fill that void. Earlier this year we released Amarna, our static-analyzer for ZKPs written in the Cairo programming language, and today we are open sourcing Circomspect, our static analyzer and linter for Circom programs. Circomspect is under active development and can be installed from crates.io or downloaded from the Circomspect GitHub repository. Please try it out and let us know what you think! We welcome all comments, bug reports, and ideas for new analysis passes.

Magnifier: An Experiment with Interactive Decompilation

By Alan Chang

Today, we are releasing Magnifier, an experimental reverse engineering user interface I developed during my internship. Magnifier asks, “What if, as an alternative to taking handwritten notes, reverse engineering researchers could interactively reshape a decompiled program to reflect what they would normally record?” With Magnifier, the decompiled C code isn’t the end—it’s the beginning.

Decompilers are essential tools for researchers. They transform program binaries from assembly code into source code, typically represented as C-like code. A researcher’s job starts where decompilers leave off. They must make sense of a decompiled program’s logic, and the best way to drill down on specific program paths or values of interest is often pen and paper. This is obviously tedious and cumbersome, so we chose to prototype an alternative method.

The Magnifier UI in action

Decompilation at Trail of Bits

Trail of Bits is working on multiple open-source projects related to program decompilation: Remill, Anvill, Rellic, and now Magnifier. The Trail of Bits strategy for decompilation is to progressively lift compiled programs through a tower of intermediate representations (IRs); Remill, Anvill, and Rellic work together to achieve this. This multi-stage approach helps break down the problem into smaller components:

  1. Remill represents machine instructions in terms of LLVM IR.
  2. Anvill transforms machine code functions into LLVM functions.
  3. Rellic transforms the LLVM IR into C code via the Clang AST.

Theoretically, a program may be transformed at any pipeline stage, and Magnifier proves this theory. Using Magnifier, researchers can interactively transform Anvill’s LLVM IR and view the C code produced by Rellic instantaneously.

It started as a REPL

Magnifier started its life as a command-line read-eval-print-loop (REPL) that lets users perform a variety of LLVM IR transformations using concise commands. Here is an example of one of these REPL sessions. The key transformations exposed were:

  • Function optimization using LLVM
  • Function inlining
  • Value substitution with/without constant folding
  • Function pointer devirtualization

Magnifier’s first goal was to describe the targets being transformed; depending on the type of transformation, these targets could be instructions, functions, or other objects. To describe these targets consistently and hide some implementation details, Magnifier assigns a unique, opaque ID to all functions, function parameters, basic blocks, and IR instructions.

Magnifier’s next important goal was to track instruction provenance across transformations and understand how instructions are affected by operations. To accomplish this, it introduces an additional source ID. (For unmodified functions, source IDs are the same as current IDs.) Then during each transformation, a new function is created that propagates the source IDs but generates new, unique current IDs. This solution ensures that no function is mutated in place, facilitating before-and-after comparisons of transformations while tracking their provenance.

Lastly, for transformations such as value substitution, Magnifier enables the performance of additional transformations in the form of constant folding. These extra transformations are often desirable. To accommodate different use cases, Magnifier provides granular control over each transformation in the form of a universal substitution interface. This interface allows users to monitor all the transformations and selectively allow, reject, or modify substitution steps as they see fit.

Here’s an example of transformations in action using Magnifier REPL.

First, a few functions are defined as follows:

Here’s the same “mul2” function in annotated LLVM IR:

The opaque IDs and the provenance IDs are shown. “XX|YY” means “XX” is the current ID, and “YY” is the source ID. The IDs in this example are:

Function: 44
Parameter “a”: 45
Basic block (entry): 51
Instruction “ret i32”: 50

Now, substitution takes place that sets the parameter “a” to 10:

The “perform substitution” message at the top shows that a value substitution has happened. Looking at the newly transformed function, each instruction has a new current ID, but the source IDs still track the original function and instructions. Also, a call to “@llvm.assume” is inserted to document the value substitution.

Next, the “b” parameter is substituted for 20, and the two calls to “addOne” are inlined:

The end result is surprisingly simple. We now have a function that calls “@llvm.assume” on “a” and “b” then returns just 231. The constant folding here shows Magnifier’s ability to evaluate simple functions.

MagnifierUI: A More Intuitive Interface

While the combination of a shared library plus REPL is a simple and flexible solution, it’s not the most ideal setup for researchers who just want to use Magnifier as a tool to reverse-engineer binaries. This is where the MagnifierUI comes in.

The MagnifierUI consists of a Vue.js front end and a C++ back end, and it uses multi-session WebSockets to facilitate communication between the two. The MagnifierUI not only exposes most of the features Magnifier has to offer, but it also integrates Rellic, the LLVM IR-to-C code decompiler, to show side-by-side C code decompilation results.

We can try performing the same set of actions as before using the MagnifierUI:

Use the Upload button to open a file.

The Terminal view exposes the same set of Magnifier commands, which we can use to substitute the value for the first argument.

The C code view and the IR view are automatically updated with the new value. We can do the same for the second parameter.

Clicking an IR instruction selects it and highlights the related C code. We can then inline the selected instruction using the Inline button. The same can be done for the other call instruction.

After inlining both function calls, we can now optimize the function using the Optimize button. This uses all the available LLVM optimizations.

Simplified the function down to returning a constant value

Compared to using the REPL, the MagnifierUI is more visual and intuitive. In particular, the side-by-side view and instruction highlighting make reading the code a lot easier.

Capturing the flag with LLVM optimizations

As briefly demonstrated above, we can leverage the LLVM library in various ways, including its fancy IR optimizations to simplify code. However, a new example is needed to fully demonstrate the power of Magnifier’s approach.

Here we have a “mystery” function that calls “fibIter(100)” to obtain a secret value:

It would be convenient to find this secret value without running the program dynamically (which could be difficult if anti-debugging methods are in place) or manually reverse-engineering the “fibIter” function (which can be time-consuming). Using the MagnifierUI, we can solve this problem in just two clicks!

Select the “fibIter” function call instruction and click the “Inline” button

With the function inlined, we can now “Optimize”! 

Here’s our answer: 3314859971, the “100th Fibonacci number” that Rellic has tried to fit into an unsigned integer. 

This example shows Magnifier’s great potential for simplifying the reverse-engineering process and making researchers’ lives easier. By leveraging all the engineering wisdom behind LLVM optimizations, Magnifier can reduce even a relatively complex function like “fibIter,” which contains loops and conditionals, down to a constant.

Looking toward the Future of Magnifier

I hope this blog post sheds some light on how Trail of Bits approaches the program decompilation challenge at a high level and provides a glimpse of what an interactive compiler can achieve with the Magnifier project.

Magnifier certainly needs additional work, from adding support for transformation types (with the hope of eventually expressing full patch sets) to integrating the MagnifierUI with tools like Anvill to directly ingest binary files. Still, I’m very proud of what I’ve accomplished with the project thus far, and I look forward to what the future holds for Magnifier.

I would like to thank my mentor Peter Goodman for all his help and support throughout my project as an intern. I learned a great deal from him, and in particular, my C++ skills improved a lot with the help of his informative and detailed code reviews. He has truly made this experience unique and memorable!

Using mutants to improve Slither

By Alex Groce, Northern Arizona University

Improving static analysis tools can be hard; once you’ve implemented a good tool based on a useful representation of a program and added a large number of rules to detect problems, how do you further enhance the tool’s bug-finding power?

One (necessary) approach to coming up with new rules and engine upgrades for static analyzers is to use “intelligence guided by experience”—deep knowledge of smart contracts and their flaws, experience in auditing, and a lot of deep thought. However, this approach is difficult and requires a certain level of expertise. And even the most experienced auditors who use it can miss things.

In our paper published at the 2021 IEEE International Conference on Software Quality, Reliability, and Security, we offer an alternative approach: using mutants to introduce bugs into a program and observing whether the static analyzer can detect them. This post describes this approach and how we used it to write new rules for Slither, a static analysis tool developed by Trail of Bits.

Using program mutants

The most common approach to finding ways to improve a static analysis tool is to find bugs in code that the tool should have been able to find, then determine the improvements that the tool needs to find such bugs.

This is where program mutants come into play. A mutation testing tool, such as universalmutator, takes a program as input and outputs a (possibly huge) set of slight variants of the program. These variants are called mutants. Most of them, assuming the original program was (mostly) correct, will add a bug to the program.

Mutants were originally designed to help determine whether the tests for a program were effective (see my post on mutation testing on my personal blog). Every mutant that a test suite is unable to detect suggests a possible defect in the test suite. It’s not hard to extend this idea specifically to static analysis tools.

Using mutants to improve static analysis tools

There are important differences between using mutants to improve an entire test suite and using them to improve static analysis tools in particular. First, while it’s reasonable to expect a good test suite to detect almost all the mutants added to a program, it isn’t reasonable to expect a static analysis tool to do so; many bugs cannot be detected statically. Second, many mutants will change the meaning of a smart contract, but not in a way that fits into a general pattern of good or bad code. A tool like Slither has no idea what exactly a contract should be doing.

These differences suggest that one has to laboriously examine every mutant that Slither doesn’t detect, which would be painful and only occasionally fruitful. Fortunately, this isn’t necessary. One must only look at the mutants that 1) Slither doesn’t detect and 2) another tool does detect. These mutants have two valuable properties. First, because they are mutants, we can be fairly confident that they are bugs. Second, they must be, in principle, detectable statically: some other tool detected them even if Slither didn’t! If another tool was able to find the bugs, we obviously want Slither to do so, too. The combination of the nature of mutants and the nature of differential comparison (here, between two static analysis tools) gives us what we want.

Even with this helpful method of identifying only the bugs we care about, there might still be too much to look at. For example, in our efforts to improve Slither, we compared the bugs it detected with the bugs that SmartCheck and Securify detected (at the time, the two plausible alternative static analysis tools). This is what the results looked like:

A comparison between the bugs that Slither, SmartCheck, and Securify found and how they overlap

A handful of really obvious problems were detected by all three tools, but these 18 mutants amount to less than 0.5% of all detected mutants. Additionally, every pair of tools had a significant overlap of 100-400 mutants. However, each tool detected at least 1,000 mutants uniquely. We’re proud that Slither detected both the most mutants overall and the most mutants that only it detected. In fact, Slither was the only tool to detect a majority (nearly 60%) of all mutants any tool detected. As we hoped, Slither is good at finding possible bugs, especially relative to the overall number of warnings it produced.

Still, there were 1,256 bugs detected by SmartCheck and 1,076 bugs detected by Securify that Slither didn’t detect! Now, these tools ran over a set of nearly 50,000 mutants across 100 smart contracts, which is only about 25 bugs per contract. Still, that’s a lot to look through!

However, a quick glance at the mutants that Slither missed shows that many are very similar to each other. Unlike in testing, we don’t care about each individual bug—we care about patterns that Slither is not detecting and about the reasons Slither misses patterns that it already knows about. With this in mind, we can sort the mutants by looking at those that are as different as possible from each other first.

First, we construct a distance metric to determine the level of similarity between two given mutants, based on their locations in the code, the kind of mutation they introduce, and, most importantly, the actual text of the mutated code. If two mutants change similar Solidity source code in similar ways, we consider them to be very similar. We then rank all the mutants by similarity, with all the very similar mutants at the bottom of the ranking. That way, the first 100 or so mutants represent most of the actual variance in code patterns!

So if there are 500 mutants that change msg.sender to tx.origin, and are detected by both SmartCheck and Slither, which tend to be overly paranoid about tx.origin and often flag even legitimate uses, we can just dismiss those mutants right off the bat; we know that a good deal of thought went into Slither’s rules for warning about uses of tx.origin. And that’s just what we did.

The new rules (and the mutants that inspired them)

Now let’s look at the mutants that helped us devise some new rules to add to Slither. Each of these mutants was detected by SmartCheck and/or Securify, but not by Slither. All three of these mutants represent a class of real bug that Slither could have detected, but didn’t:

Mutant showing Boolean constant misuse:

if (!p.recipient.send(p.amount)) { // Make the payment

    ==> !p.recipient.send(p.amount) ==> true

if (true) { // Make the payment

The first mutant shows where a branch is based on a Boolean constant. There’s no way for paths through this code to execute. This code is confusing and pointless at best; at worst, it’s a relic of a change made for testing or debugging that somehow made it into a final contract. While this bug seems easy to spot through a manual review, it can be hard to notice if the constant isn’t directly present in the condition but is referenced through a Boolean variable.

Mutant showing type-based tautology:

require(nextDiscountTTMTokenId6 >= 361 && ...);

    ==> ...361...==>...0…

require(nextDiscountTTMTokenId6 >= 0 && ...);

This mutant is similar to the first, but subtler; a Boolean expression appears to encode a real decision, but in fact, the result could be computed at compile time due to the types of the variables used (DiscountTTMTokenId6 is an unsigned value). It’s a case of a hidden Boolean constant, one that can be hard for a human to spot without keeping a model of the types in mind, even if the values are present in the condition itself.

Mutant showing loss of precision:

byte char = byte(bytes32(uint(x) * 2 ** (8 * j)));

    ==> ...*...==>.../…

byte char = byte(bytes32(uint(x) * 2 ** (8 / j)));

This last mutant is truly subtle. Solidity integer division can truncate a result (recall that Solidity doesn’t have a floating point type). This means that two mathematically equivalent expressions can yield different results when evaluated. For example, in mathematics, (5 / 10) * 2 and (5 * 2) / 10 have the same result; in Solidity, however, the first expression results in zero and the other results in one. When possible, it’s almost always best to multiply before dividing in Solidity to avoid losing precision (although there are exceptions, such as when the size limits of a type require division to come first).

After identifying these candidates, we wrote new Slither detectors for them. We then ran the detectors on a corpus that we use to internally vet new detectors, and we confirmed that they are able to find real bugs (and don’t report too many false positives). All three detectors have been available in the public version of Slither for a while now (as the boolean-cst, tautology, and divide-before-multiply rules, respectively), and the divide-before-multiplying rule has already claimed two trophies, one in December of 2020 and the other in January of 2021.

What’s next?

Our work proves that mutants can be a useful tool for improving static analyzers. We’d love to continue adding rules to Slither using this method, but unfortunately, to our knowledge, there are no other static analysis tools that compare to Slither and are seriously maintained.

Over the years, Slither has become a fundamental tool for academic researchers. Contact us if you want help with leveraging its capacities in your own research. Finally, check out our open positions (Security Consultant, Security Apprenticeship) if you would like to join our core team of researchers.

The road to the apprenticeship

By Josselin Feist, Principal Security Engineer

Finding talent is hard, especially in the blockchain security industry. The space is new, so you won’t find engineers with decades of experience with smart contracts. Training is difficult, as the technology evolves constantly, and online content quickly becomes outdated. There are also a lot of misconceptions about blockchain technology that make security engineers hesitant to enter the space. As a result, the pool of people who are able to both master blockchain technology and grasp the mindset of a security engineer is fairly small.

We have now been working on blockchain projects for more than half a decade, and we have always struggled to find qualified applicants. Last year, to alleviate this problem, we created an intensive apprenticeship program to give apprentices the equivalent of two years’ experience in only three months. The program has been a huge success, and we have offered full-time positions to all of our apprentices!

Read on for more information about the program and the apprentices we’ve hired so far, as well as pointers for future applicants.

The apprenticeship program

The main goal of the program is to train our apprentices to become highly technical security engineers. We set high standards for our employees, and we want to enable our apprentices to quickly meet our expectations. There are two key aspects of the program:

Mentorship

Every apprentice has a mentor from the blockchain team (someone of at least the senior level). Each mentor has one apprentice at a time, which ensures that the mentor can provide personalized feedback and support. The mentor is responsible for making sure that the apprentice understands our processes and techniques and is challenged technically. For example, the mentor might task the apprentice with reading a section of the Yellow Paper and answering related questions; the apprentice could also be asked to study a new attack happening in the DeFi ecosystem (and to master the underlying technique). We have also developed a set of in-house challenges and exercises to help our apprentices grow.

Mentorship is a key part of our apprenticeship program and makes the training process fast and efficient.

Audit shadowing

Our apprentices work full time and participate in our audits, though their hours are not billed to our audit clients. By shadowing audits, apprentices learn how we approach a codebase, practice using our tools, write reports, and have a chance to interact with the team and clients.

This is a hands-on experience for our apprentices, and we want to give them as much exposure as possible to different approaches and code review strategies. To do that, we have our apprentices switch auditing teams: they may work with their mentors, but they could also work with anyone else in our Assurance Practice.

Who we are looking for

While we’ve seen a lot of different kinds of applicants, from recently graduated engineers to more experienced professionals, this opportunity is intended for exceptional entry- to mid-level professionals with experience in blockchain development or auditing. Over the past year, we’ve had eight apprentices:

  • Four of them had about one year of blockchain experience.
  • Two had previous cybersecurity experience.
  • Two had completed the Secureum bootcamp.
  • One had graduated one year before starting the apprenticeship.
  • Coincidentally, three of them had founded a startup in the past.

We’ve found two kinds of applicants to be the best fit:

Blockchain experts / security enthusiasts

These are exceptional blockchain engineers / researchers without a professional security background. People who fall into this category already have in-depth knowledge of Solidity and the EVM but have never done an audit in a professional setting. We help them strengthen their understanding of how to conduct an audit and train them to think outside of the box and to use our tools.

For example, take Jaime Iglesias. When Jaime joined our apprenticeship program, he had been working in the blockchain space for a couple of years and already had expertise in smart contracts. (He was one of the winners of the 2020 Underhanded Solidity Contest.) During his apprenticeship, Jaime learned how to conduct a professional audit and how to approach a codebase from an attacker’s point of view. He also learned how to write and structure reports and how to effectively manage and work with clients.

Security experts / blockchain enthusiasts

These are experienced security researchers with a background in traditional InfoSec. They know how to perform an audit and have been learning about blockchain technology in their free time, but there may be some gaps in their understanding of edge cases.

For example, Anish Naik was an offensive security analyst before becoming an apprentice. He knew how to think like an attacker and to participate in an audit, but he was working on blockchain projects only in his free time. During his apprenticeship, Anish had the opportunity to work full time on blockchain projects and to perfect his understanding of Solidity and the EVM. He also learned various auditing strategies from our team members and gained exposure to the latest tools, threat intelligence, and development practices.

How to get accepted into the program

We recommend that candidates do the following:

  • Strengthen your understanding of real-world vulnerabilities and auditing.
    • Review the material offered by Secureum, which will be useful as you start your blockchain security journey. Watch Secureum’s YouTube videos to gain an understanding of the most common vulnerabilities and to test your knowledge through quizzes.
    • Read our audit reports to get a better picture of real-world vulnerabilities, including less common bugs. Pay special attention to the descriptions of vulnerabilities and the structure of those descriptions. Reading our reports will help you to write better reports yourself.
  • Increase your knowledge of advanced topics, including the use of tools.
    • Read our blog posts. In particular, master the concept of contract upgradeability and learn about how we used Echidna to fuzz a library and how we fuzzed the Solidity compiler. Our blog posts detail technical challenges and pitfalls of blockchain security and will help you gain in-depth technical expertise.
    • Complete the exercises in the “Program Analysis” section of building-secure-contracts. Our building-secure-contracts repository contains guidance on how to efficiently use our program analysis tools (specifically Slither, Echidna, and Manticore). We use these tools in our professional audits, and they significantly enhance our auditing capabilities. Mastering them is key to becoming an expert auditor.
  • Put your knowledge to the test.

We receive a lot of applications, but you can stand out from the pool of applicants by demonstrating your knowledge publicly, through blog posts or tool contributions.

For example, before applying, Simone Monica made direct contributions to Slither (PR850: “Add support of ERC1155 for slither-check-erc tool”). Troy Sargent created a tool based on Slither to solve an Ethernaut challenge (as he explains in his blog post “Slithering Through the Dark Forest”). He ended up expanding on this work after joining the company and has since built slither-read-storage, a general tool for reading on-chain variables. (See his recent blog post for more information.)

By contributing to our tools, Simone and Troy demonstrated their technical expertise and ability to make contributions to the community.

Frequently asked questions

  • Is the apprenticeship program remote?
    Yes. Trail of Bits is a remote-first company; most members of the blockchain team are in either the Eastern time zone or Europe. We can hire apprentices in time zones from Pacific time to Indian standard time. The one requirement is that their hours overlap with the morning of the Eastern time workday.
  • What happens if an apprentice is not ready for a full-time position after three months?
    We find that on average, we need three months to train someone. However, if an apprentice is ready for a full-time role early, we can hire the apprentice right away (as we’ve already done multiple times). If someone is not ready after three months but would likely be ready after a bit more training, we can extend the apprenticeship. Our goal is to help apprentices successfully join our team, and we will invest the resources necessary to reach that goal.
  • What tech will I work on?
    At Trail of Bits, we work on many different aspects of blockchain technology, including smart contracts, consensus mechanisms, and virtual machine architecture. However, the apprenticeship focuses only on smart contracts; this gives us the time we need to help our apprentices become highly technical experts and meet our expectations. Once the apprenticeship is done, our new employees will have the opportunity to gain exposure to other components.
  • Do apprentices work only with the Ethereum chain?
    No, we are also looking for candidates with backgrounds in chains including Algorand, Cairo, Cosmos, Solana, and Substrate. Candidates who have experience with these chains may receive dual training (in Ethereum and an additional chain).
  • How many candidates do you accept?
    We usually welcome a new apprentice every month.

Join our team

Our apprenticeship program has been a successful experiment for us, and we’ve gotten positive feedback from our former apprentices (all of whom we’ve hired). Here’s what a few of our apprentices had to say about the program.

Anish Naik, who was an offensive security analyst and developer prior to joining us:

The apprenticeship was an incredible opportunity for me to enter the blockchain security space and learn from some of the best auditors. You get to work on a research-oriented and collaborative team, increase your knowledge of a variety of tools and technologies, and make a positive impact in the industry!

Justin Jacob, who graduated in 2021 and was working in blockchain analytics before starting the apprenticeship:

The apprenticeship is one of the best learning opportunities I have had in my career. Spending the day working with some of the smartest professionals in the space was extremely helpful and drastically improved my skills as an auditor. Furthermore, since being hired full time, I’ve loved the opportunities I have had to do more research about up-and-coming blockchain technology, learn new skills and techniques, and improve my overall understanding of the industry. The flexibility of the company allows me to dive into anything I find interesting, which I really appreciate. This has been such a positive growth opportunity, and I would highly encourage anyone interested in the program to apply.

Robert Schneider, who joined us after demonstrating his skills through the Secureum bootcamp:

In the apprenticeship program, you’re not just an observer, watching the process unfold—you’re a full-fledged member of the team! In my first audit, I researched issues, contributed to bug reports, and interfaced with the client—all while learning the trade from some of the best smart contract auditors in the industry.

The next round of the program starts in October, so be sure to apply for an apprenticeship if you are interested in joining our team!

Shedding smart contract storage with Slither

By Troy Sargent, Blockchain Security Engineer

You think you’ve found a critical bug in a Solidity smart contract that, if exploited, could drain a widely used cryptocurrency exchange’s funds. To confirm that it’s really a bug, you need to figure out the value at an obscure storage slot that has no getter method. Adrenaline courses through your body, followed by the pang of devastation when you lay your eyes on the Solidity storage documentation:

Your momentum crashes as you try to interpret these hieroglyphics, knowing that every wasted second could be catastrophic. Fortunately, slither-read-storage, a tool that retrieves storage slots from source code, can save you from this nightmare.

What is slither-read-storage?

slither-read-storage retrieves storage slots from source code by using information from Slither’s underlying type analysis (e.g., whether an array is fixed or dynamic) to inform slot calculations.

The tool can retrieve the storage slot(s) of a single variable or of entire contracts, and storage values can be retrieved by providing an Ethereum RPC.

slither-read storage is included with the latest release of Slither, version 0.8.3, and can be installed with Python’s package manager, pip:

pip install slither-analyzer==0.8.3

Use cases for slither-read-storage

Let’s explore some use cases for this tool.

Moonlight auditing

To determine all the addresses that can mint FRAX, we would have to manually input indices of the frax_pools_array one-by-one on Etherscan. Since frax_pools_array is dynamic, it would be helpful to know the current length of the array, which is not readily available. Finding these addresses in this way would be very time consuming, and we might waste time inputting indices that are out-of-bounds.

Querying frax_pools_array on Etherscan

Instead, we can run slither-read-storage on the FRAXStablecoin contract’s address to find the length of frax_pools_array:

slither-read-storage 0x853d955aCEf822Db058eb8505911ED77F175b99e --variable-name frax_pools_array --rpc-url $RPC_URL --value

Great! The length of a dynamic array is stored in one of the contract’s slots—in this case, slot 18. Examining the value at FRAXStablecoin’s slot 18, we can see that the length of the array is 25.

INFO:Slither-read-storage:
Contract 'FRAXStablecoin'
FRAXStablecoin.frax_pools_array with type address[] is located at slot: 18
INFO:Slither-read-storage:
Name: frax_pools_array
Type: address[]
Slot: 18
INFO:Slither-read-storage:
Value: 25

Now, we can retrieve the entirety of the FRAXStablecoin storage and specify the --max-depth 25 flag to pass the maximum depth of the data structures that we want slither-read-storage to return:

slither-read-storage 0x853d955aCEf822Db058eb8505911ED77F175b99e --layout  --rpc-url $RPC_URL --value --max-depth 25

The tool will then produce a JSON file with the storage layout and values, but we’re only interested in frax_pools_array. As of this writing, the tool retrieves 25 elements indicating the addresses that can mint FRAX.

   "frax_pools_array": {
"type_string": "address[]",
"slot": 18,
"size": 256,
"offset": 0,
“value”: 25,
"elems": {
// snip
"23": {
"type_string": "address",
"slot": 84827061063453624289975705683721713058963870421084015214609271099009937454171,
"size": 160,
"offset": 0,
"value": "0x36a0B6a5F7b318A2B4Af75FFFb1b51a5C78dEB8C"
},
"24": {
"type_string": "address",
"slot": 84827061063453624289975705683721713058963870421084015214609271099009937454172,
"size": 160,
"offset": 0,
"value": "0xcf37B62109b537fa0Cb9A90Af4CA72f6fb85E241"
}
}

Arbitrage bots

Aside from moonlight auditing on Etherscan, slither-read-storage can also be used to improve a program’s speed. For example, we can use slither-read-storage to have the program directly access an Ethereum node’s database rather than processing RPC calls. This is especially useful for contracts that do not provide a view function to retrieve the desired variable.

For instance, let’s say an arbitrage bot frequently reads the member sqrtPriceX96 of the variable slot0 on the WETH/USDC Uniswap V3 pool (see the data structure below).

struct Slot0 {
// the current price
uint160 sqrtPriceX96;
// the current tick
int24 tick;
// the most-recently updated index of the observations array
uint16 observationIndex;
// the current maximum number of observations that are being stored
uint16 observationCardinality;
// the next maximum number of observations to store, triggered in observations.write
uint16 observationCardinalityNext;
// the current protocol fee as a percentage of the swap fee taken on withdrawal
// represented as an integer denominator (1/x)%
uint8 feeProtocol;
// whether the pool is locked
bool unlocked;
}

Instead of calling the provided view function, we can use slither-read-storage to compute the slot as follows:

slither-read-storage 0x8ad599c3a0ff1de082011efddc58f1908eb6e6d8 --layout

The file produced by the tool contains the slot, size, and offset of sqrtPriceX96, which can be easily retrieved from an Ethereum node’s key-value store and sliced based on the size and offset. It turns out that the Uniswap developers aptly named this variable slot0, but this is rarely available in practice.

{
"slot0": {
"type_string": "UniswapV3Pool.Slot0",
"slot": 0,
"size": 256,
"offset": 0,
"elems": {
"sqrtPriceX96": {
"type_string": "uint160",
"slot": 0,
"size": 160,
"offset": 0
},
/// snip

Additionally, one can modify storage values using the Ethereum node RPC, eth_call, by passing in the storage slot and desired value and simulating how transactions applied to the modified state are affected. Greater detail on how to accomplish this can be found in this tutorial.

Portfolio tracking

The balance slot of this account can be found using the following slither-read-storage command with the token address as the target and the account address as the key:

slither-read-storage 0x1f9840a85d5aF5bf1D1762F925BDADdC4201F984 --variable-name balances --key 0xab5801a7d398351b8be11c439e05c5b3259aec9b --rpc-url $RPC_URL --value

In this specific instance, the slot is the address of the account we intend to retrieve, 0xab5801a7d398351b8be11c439e05c5b3259aec9b, along with slot 9, padded to 32 bytes:

000000000000000000000000ab5801a7d398351b8be11c439e05c5b3259aec9b0000000000000000000000000000000000000000000000000000000000000009

This value is hashed with keccak256, and the balance is written at the resulting slot using SSTORE(slot, value).

Upgradeable ERC20 token

A normal smart contract has contract logic and storage at the same address. However, when the delegatecall proxy pattern is used (allowing the contract to be upgradeable), the proxy calls the implementation, and the storage values are written to the proxy contract. That is, the DELEGATECALL opcode writes to the storage at the caller’s address using the storage information of the logic contract. To accommodate this pattern, the --storage-address flag is required to retrieve the balance slot for the same address:

slither-read-storage 0xa2327a938Febf5FEC13baCFb16Ae10EcBc4cbDCF --variable-name balances --key 0xab5801a7d398351b8be11c439e05c5b3259aec9b --storage-address 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 --rpc-url $RPC_URL --value

Closing thoughts

I learned a lot about Solidity’s handling of storage and Slither’s API by building this tool, and its release completes work that I started before joining Trail of Bits as an apprentice. In fact, I can attribute my invaluable experience as an apprentice here to discussing Slither on Twitter and finding issues in Slither while working on the first iteration of this tool.

If you’re keen on doing the same, check out our GitHub and grab an open issue. We’d love to help new contributors.

libmagic: The Blathering

By Evan Sultanik, Principal Security Engineer

A couple of years ago we released PolyFile: a utility to identify and map the semantic structure of files, including polyglots, chimeras, and schizophrenic files. It’s a bit like file, binwalk, and Kaitai Struct all rolled into one. PolyFile initially used the TRiD definition database for file identification. However, this database was both too slow and prone to misclassification, so we decided to switch to libmagic, the ubiquitous library behind the file command.

What follows is a compendium of the oddities that we uncovered while developing our pure Python cleanroom implementation of libmagic.

Magical Mysteries

The libmagic library is older than over half of the human population of Earth, yet it is still in active development and is in the 99.9th percentile of most frequently installed Ubuntu packages. The library’s ongoing development is not strictly limited to bug fixes and support for matching new file formats; the library frequently receives breaking changes that add new core features to its matching engine.

libmagic has a custom domain specific language (DSL) for specifying file format patterns. Run `man 5 magic` to read its documentation. The program compiles its DSL database of file format patterns into a single definition file that is typically installed to /usr/share/file/magic.mgc. libmagic is written in C and includes several manually written parsers to identify various file types that would otherwise be difficult to represent in its DSL (for example, JSON and CSV). Unsurprisingly, these parsers have led to a number of memory safety bugs and numerous CVEs.

PolyFile is written in Python. While libmagic does have both official and independent Python wrappers, we chose to create a cleanroom implementation. Aside from the native library’s security issues, there are several additional reasons why we decided to create something new:

  1. PolyFile is already written in pure Python, and we did not want to introduce a native dependency if we could avoid it.
  2. PolyFile is intended to detect polyglots and other funky file formats that libmagic would otherwise miss, so we would have had to extend libmagic anyway.
  3. PolyFile preserves lexical information like input byte offsets throughout its parsing, in order to map semantics back to the original file locations. There was no straightforward way to do this with libmagic.

The idea of reimplementing libmagic in a language with more memory safety than C is not novel. An effort to do so in Ruby, called Arcana, occurred concurrently with PolyFile’s implementation, but it is still incomplete. PolyFile, on the other hand, correctly parses libmagic’s entire pattern database and passes all but two of libmagic’s unit tests, and correctly identifies at least as many MIME types as libmagic on Ange Albertini’s 900+ file Corkami corpus.

The Magical DSL

In order to appreciate the eldritch horrors we unearthed when reimplementing libmagic, we need to offer a brief overview of its esoteric DSL. Each DSL file contains a series of tests—one per line—that match the file’s subregions. These tests can be as simple as matching against magic byte sequences, or as complex as seemingly Turing-complete expressions. (Proving Turing-completeness is left as an exercise to the reader.)

The file command executes the DSL tests to classify the input file. The tests are organized in the DSL as a tree-like hierarchy. First, each top-level test is executed. If a test passes, then its children are each tested, in order. Tests at any level can optionally print out a message or associate the input file with a MIME type classification.

Each line in the DSL file is a test, which includes an offset, type, expected value, and message, delimited by whitespace. For example:

10    lelong    0x00000100    this is a test

This line will do the following:

  1. Start at byte offset 10 in the input file
  2. Read a signed little-endian long (4 bytes)
  3. If those bytes equal 0x100, then print “this is a test”

Now let’s add a child test, and associate it with a MIME type:

10        lelong    0x00000100    this is a test
>20       ubyte 0xFF       test two
!:mime    application/x-foo

The “>” before the “20” offset in the second test means that it is a child of the previously defined test at the higher level.
This new version will do the following:

  1. If, and only if, the first test matches, then attempt the second test.
  2. If the byte at file offset 20 equals 0xFF, then print out “test two” and also associate the entire file with the MIME type application/x-foo.

Note that the message for a parent test will be printed even if its children do not match. A child test will only be executed if its parent is matched. Children can be arbitrarily nested with additional “>” prefixes:

10        lelong    0x00000100    this is a test
>20       ubyte     0xFF          test two
!:mime    application/x-foo
>>30      ubyte     0x01          this is a child of test 2
>20       ubyte     0x0F          this is a child of the first test that will be tested if the first test passes, regardless of whether the second child passes
!:mime    application/x-bar

If a test passes, then all of its children will be tested.

So far, all of the offsets in these examples have been absolute, but the libmagic DSL also allows relative offsets:

10      lelong    0x00000100    this is a test
>&20    lelong    0x00000200    this will test 20 bytes after its parent match offset, equivalent to absolute offset 10 + 20 = 30

as well as indirect offsets:

(20.s)      lelong    0x00000100    indirect offset!

The (20.s) here means: read a little-endian short at absolute byte offset 20 in the file and use that value as the offset to read the signed little-endian long (lelong) that will be tested. Indirect offsets can also include arithmetic modifiers:

(20.s+10)   read the offset from the little-endian short at absolute byte offset 20 and add 10
(0.L*0x20)   read the offset from the big-endian long at absolute byte offset zero and multiply by 0x20

Relative and indirect offsets can also be combined:

(&0x10.S)    read the offset from the big-endian short 0x10 bytes past the parent match
(&-4.l)      read the offset from the little-endian long four bytes before the parent
&(0.S-2)     read the first two bytes of the file, interpret them as a big-endian short, subtract two, and use that value as an offset relative to the parent match

Offsets are very complex!

Despite having existed for decades, the libmagic pattern DSL is still in active development.

Mischief, Unmanaged

In developing our independent implementation of libmagic—to the point where it can parse the file command’s entire collection of magic definitions and pass all of the official unit tests— we discovered many undocumented DSL features and apparent upstream bugs.

Poorly Documented Syntax

For example, the DSL patterns for matching MSDOS files contain a poorly documented use of parenthesis within indirect offsets:

(&0x10.l+(-4))

The semantics are ambiguous; this could mean, “Read the offset from the little-endian long 0x10 bytes past the parent match decremented by four,” or it could mean, “Read the offset from the little-endian long 0x10 bytes past the parent match and add the value read from the last four bytes in the file.” It turns out that it is the latter.

Undocumented Syntax

The elf pattern uses an undocumented ${x?true:false} ternary operator syntax. This syntax can also occur inside a !:mime directive!

Some specifications, like the CAD file format, use the undocumented regex /b modifier. It is unclear from the libmagic source code whether this modifier is simply ignored or if it has a purpose. PolyFile currently ignores it and allows regexes to be applied to both ASCII and binary data.

According to the documentation, the search keyword—which performs a literal string search from a given offset—is supposed to be followed by an integer search range. But this search range is apparently optional.

Some specifications, like BER, use “search/b64”, which is undocumented syntax. PolyFile treats this as equivalent to the compliant search/b/64.

The regex keyword has an undocumented T modifier. What is a T modifier? Judging from libmagic’s code, it appears to trim whitespace from the resulting match.

Bugs

The libmagic DSL has a type specifically for matching globally unique identifiers (GUIDs) that follows a standardized structure as defined by RFC 4122. One of the definitions in the DSL for Microsoft’s Advanced Systems Format (ASF) multimedia container does not conform to RFC 4122—it is two bytes short. Presumably libmagic silently ignores invalid GUIDs. We caught it because PolyFile validates all GUIDs against RFC 4122. This bug was present in libmagic from December of 2019 until we reported it to the libmagic maintainers in April 2022. In the meantime, PolyFile has a workaround for the bug and has always used the correct GUID.

Metagame

PolyFile is a safer alternative to libmagic that is nearly feature-compatible.

$ polyfile -I suss.png
image/png………………………………………………………..PNG image data
application/pdf…………………………………………………..Malformed PDF
application/zip…………………………………………………..ZIP end of central directory record Java JAR archive
application/java-archive…………………………………………..ZIP end of central directory record Java JAR archive
application/x-brainfuck……………………………………………Brainf*** Program

PolyFile even has an interactive debugger, modeled after gdb, to debug DSL patterns during matching. (See the -db option.) This is useful for DSL developers both for libmagic and PolyFile. But PolyFile can do so much more! For example, it can optionally output an interactive HTML hex viewer that maps out the structure of a file. It’s free and open source. You can install it right now by running pip3 install polyfile or clone its GitHub repository.

A Typical Day as a Trail of Bits Engineer-Consultant

Wherever you are in the world, a typical day as a Trail of Bits Engineer-Consultant means easing into your work.

Here’s a short video showing some of our European colleagues describing a typical day as a Trail of Bits Engineer-Consultant:

You generally set your own hours, to provide at least a couple of hours of overlap with colleagues around the world) by checking messages or comments received since you last checked in, and thinking about any requests for collaborative help. Then, depending on whether you’re on an audit or on non-client “bench-time”, your day could mean diving into code, or working on internal research and development, continuing education, or personal projects, etc.

Remote-First

One thing to know about Trail of Bits is that we have always been, and always will be, a remote-first company — “remote” is not a skill we added for the pandemic. That means that we are global in nature, and asynchronous by design. We’ve fostered a collegial atmosphere, one with close, intimate collaboration among colleagues.

Those of us who work here wouldn’t have it any other way.

At its heart, the art of asynchronous collaboration is about understanding the work, understanding our tasks, and asking clear questions that request actionable replies from our expert coworkers. It works. We believe that, according to the criteria described in “The Five Levels of Remote Work”, we are somewhere between levels four and five.

For example, we consider carefully when we need face-to-face meetings, to avoid the “this-meeting-could-have-been-handled-in-a-Slack-conversation” problem that plagues a lot of companies. When we do meet face-to-face, we use Google Meet; the meetings all have a written agenda, are recorded, and have notes taken and distributed to all attendees.

We have a minimal reliance on email for internal conversations, preferring the more secure and archived Slack as the primary chat and discussion forum. We strongly recommend that Slack is used with Notifications Off. We also do not require Slack to be installed on your mobile phone – in fact we suggest that you don’t – so you’re not tempted or compelled to check Slack during your time off (also, all personal mobile devices are required to run MDM if they handle any work data). Each project – whether formal or ad-hoc – has a dedicated Slack channel. Slack communications are written with the expectation that people have limited time, hence our focus on well-considered (and considerate) messages that come quickly to the point and make actionable requests for collaboration.

We use Trello and GitHub to visually collaborate on projects and a range of other purpose-built tools to reduce toil and encourage meaningful collaboration without getting all up in your grill.

Work Hours and Work-Life Balance

We expect you to maintain a good, healthy, and enjoyable balance between your personal time and work time — see that example above: we don’t want you using Slack as you lie in bed! Since you already have a desk in your house, wait until you get to your desk to turn on Slack and start work. You’ll find that we’re quite insistent that you turn off during your time off — recharge, refresh, and hit the ground running when you are back.

To that end, we have generous programs to set yourself up at home, like a $1,000 stipend to set up your home office, a $500 a year personal learning and development budget, a co-working space option, 20 days of paid time off and 15 company holidays per year, and more. See this page for more information.

Set Your Hours: A Typical Day

We are a results-oriented company, and we are less concerned with when you work than with the impact your work has on the company. So a typical day can look like this:

Morning (9am-noon)

We recommend certain practices to begin your day, to draw a distinction between home-life and work. For example, we recommend establishing a commute even if you work in your own home. You can pull up recordings of any meetings from earlier in the week, read some messages a colleague left you overnight, and check for next-best priorities on Github Issues. You meet on Google Meet for a quick standup and see how things are going. You can see it on their face as clear as day — everyone at Trail of Bits has high-end audio/video equipment — they’re excited about a monster new attack surface they found yesterday.

Afternoon (noon-3pm)

Maybe you’d visited a doctor in the beginning of the week, so today you’re working a couple of extra hours to time-shift. You’ve got a lunchtime invite to attend a lunch-and-learn, find it on the company-wide team meetings calendar and pop in. One of our Team Leads is reviewing an academic paper on such and such, and Sam Moelius is absolutely destroying him with extremely simple and polite questions. You file an issue you discovered by hand this morning into Github Issues. Document a few more security properties, these will be great for our fuzzer later this afternoon. It’s easy to focus because you followed the company-recommended advice to disable all but the most essential Slack notifications. But since your collaborator is a few timezones ahead, you pop out to run an errand before the stores close.

Evening (3-7pm)

You take a walk outside for coffee (“stupid little mental health walk”). Exercise and sun are good for the mind. Those properties you found this morning could result in excellent bugs if they break the project. You spend the rest of the evening writing them up into dylint/echidna/libFuzzer … whatever. You login to your dev machine over Tailscale/locally in VM/on DigitalOcean, and start a batch fuzzer job that will complete in the morning. You write a brief note on Slack to let your coworker know where things are and that you’re signing off for the night. You close the lid on your laptop, and you don’t have Slack installed on your phone. Time to raid a dungeon in Elden Ring!

Next week, you have IRAD planned to take the lessons learned from this project and incorporate them into the company’s new Automated Testing Handbook.

More questions?

Get in touch. Visit https://trailofbits.com/careers, or use our contact form.

The Trail of Bits Hiring Process

When engineers apply to Trail of Bits, they’re often surprised by how straightforward and streamlined our hiring process is. After years of experience, we’ve cut the process to its bedrock, so that it’s candidate focused, quick, and effective.

Here’s a short video showing some of our European colleagues discussing some cool things they’re working on now:

Our Interview Process

The process from interview to offer is in four parts, and the whole thing can take three weeks or less. We want to be respectful of your time—we won’t advance you in the process unless we think there’s a good reason to continue, and we ask you to do the same.

Here’s a short video showing some of our European colleagues describing the Trail of Bits interview process:

In a Nutshell

  1. Initial screen (~30 minutes, one-on-one)
  2. Assessment (2 hours, on your own)
  3. Final interview (~2 hours, with two engineers and a team or practice lead)
  4. Decision (within five business days) and offer letter or pass with explanation (we often recommend that candidates reapply in the future)

Initial Screen

We start our process with a 30-minute screening call, designed to assure a rough match of mission, skill, and capability. These calls are typically with a Trail of Bits recruiter, or the hiring manager for the position.

Assessment

Those making it through the screen are given a brief, take-home assessment, on which we want you to spend two hours or less. Interviews are only ever so good, and there’s a limit to what you can get across on a phone or video call. We want to see what you can do! Some people have a work portfolio, but even in many of those cases, we want to watch you work. So we prepared short assessments that we’ve benchmarked to only take approximately two hours. The assessment is technically focused and allows us to see your skills in practice. The assessment is reviewed by a lead engineer in the appropriate practice—cryptography, blockchain, application security, research, or engineering. In some cases, in place of the assessment, we are happy to accept a work sample you have already put together.

Final Interview

Those making it through the assessment are invited to the final interview, where the real matchmaking is done. Now, whether you provided a work sample or completed an assessment assignment we send you, if it got you past that hurdle and into this final interview, that’s something to talk about! We find that the best way to start our final interviews is right there, because in fact this is something you are rightfully proud of. Tell us about it, and how you approached the issue, the problems you faced while doing it and go ahead and brag a little! This is a perfect way to start a conversation about what it would be like if you worked here.

Our final interviews are about two hours in length—some are shorter, some longer, but it’s around there on average. You can expect a conversation with two to three peers, about a range of deeply technical subjects to assess whether this is a good alignment for us all. There are no trick questions, just a collegial approach to solving technical problems similar to those we face every day.

Your turn

A healthy portion of the final interview—about 20%—is dedicated to answering your questions about us. We’re very up-front about what it’s like to work with us.

Decision

Within five business days, you should receive either an offer letter, or an email explaining why we decided not to move forward with your candidacy. In many cases, we recommend that you reapply in the future. For example, when you get more experience in an area we felt you needed more depth in, or after you’ve developed some specific skills we mentioned during the process. But in all cases we will be open and communicative.

A sample of a Trail of Bits rejection letter from our Blockchain team

 

A sample of a rejection email from our cryptography team

Negotiating and acceptance

Our offer will be well-considered and based on the conditions and criteria we know to exist. If there are other factors you feel we have not taken into account, please do feel free to reply and negotiate. You’ll find us reasonable—at this point we want you to work with us as much as you want to work with us, so we make efforts to meet your expectations wherever we can.

During the interview process we’ll ask—or you’ll tell us—what your availability will be, so at the offer stage we will propose a start date. If you’re planning to accept an offer with us, we always recommend you take some extra time off between jobs. We advocate for it. If you need a different start date—for example, if you need more time to give notice, to finish some personal business prior to joining us, or even if you want to start sooner than we’d thought—we will do our best to accommodate your needs.

All paperwork is sent digitally and once it’s all signed, we work with you to get you all the equipment that you need. We also provide a virtual Ramp credit card for other onboarding expenses that we’re happy to cover (more on that in Onboarding, below).

Starting at Trail of Bits

Once you accept and sign your offer letter, we’ll provide you with the documentation you’ll need to have the most successful (and enjoyable!) onboarding experience. From items like payroll and benefits to our operating practices and procedures, you’ll find our documentation and resources are quite comprehensive. The first things you can expect to find are:

  • Onboarding checklist
  • Payroll and benefits enrollment steps
  • Employee handbook
  • Handbook for the practice you are joining (e.g., Assurance, Engineering, Project Management Organization, Operations, etc.)
  • Handbook for the team you are joining (e.g., Application Security, Technical Editing, etc.)
  • Compensation philosophy
  • Learning & development resources

Our Learning & Development Resources document, for example, contains a detailed and actionable list of resources that each Trail of Bits engineer and non-engineer can use to further their career and personal development goals. From books we think you should read to presentations—ours and those of others—you should watch, to references on specific courses we think are great for all of us. Managers will find guides to better leadership, all employees will find access to online classes and courses, and interns can benefit from tips and tricks lists. And engineers will love that we regularly schedule software development and skills-based training for the team and send individuals to targeted training for areas we intend to grow and specialize in.

Equipment

We outfit every employee with the top kit they need to work remotely (and securely!). Engineers at Trail of Bits currently receive the latest generation 14 or 16″ Macbook Pro with 64 GB of RAM, and Operations team members receive the latest generation 13″ Macbook Pro with 24 GB of RAM. Depending on your home country, this will either be ordered to arrive before your start date, or we will send you a Ramp card to buy the items in your home country. We will also order a YubiKey 5C and 5Ci, a high-end Logitech C925e or Brio webcam, and one of our standard headsets (typically, a Sennheiser Game One) to arrive before your first day. Your Ramp card is also loaded with extra cash to upgrade your home office (we recommend Dell U2723QE or Dell U3223QE monitors, a CalDigit TS4 Plus dock, or even a new router like the Eero Pro 6E). There’s lots more information in our onboarding guide, which you’ll receive when you join!

More questions?

Get in touch. Visit https://trailofbits.com/careers, or use our contact form.

Managing risk in blockchain deployments

Do you need a blockchain? And if so, what kind?

Trail of Bits has released an operational risk assessment report on blockchain technology. As more businesses consider the innovative advantages of blockchains and, more generally, distributed ledger technologies (DLT), executives must decide whether and how to adopt them. Organizations adopting these systems must understand and mitigate the risks associated with operating a blockchain service organization, managing wallets and encryption keys, relying on external providers of APIs, and many other related topics. This report is intended to provide decision-makers with the context necessary to assess these risks and plan to mitigate them.

In the report, we cover the current state, use cases, and deficiencies of blockchains. We survey the common pitfalls, failures, and vulnerabilities that we’ve observed as leaders in the field of blockchain assessment, security tooling, and formal verification.

Blockchains have significantly different constraints, security properties, and resource requirements than traditional data storage alternatives. The diversity of blockchain types and features can make it challenging to decide whether a blockchain is an appropriate technical solution for a given problem and, if so, which type of blockchain to use. To help readers make such decisions, the report contains written and graphical resources, including a decision tree, comparison tables, and a risk/impact matrix.

Should you use a blockchain?

A decision tree from the Trail of Bits operational risk assessment on blockchains

Goldman Sachs partnered with Trail of Bits in 2018 to create a Cryptocurrency Risk Framework. This report applies and updates some of the results from that study. It also includes information included in a project that Trail of Bits completed for the Defense Advanced Research Projects Agency (DARPA) to examine the fundamental properties of blockchains and the cybersecurity risks associated with them.

Key insights

Here are some of the key insights from our research:

  • Proof-of-work technology and its risks are relatively well understood compared to newer consensus mechanisms like proof of stake, proof of authority, and proof of burn.
  • The foremost risk is “the storage problem.” It is not the storage of cryptocurrency, but rather the storage of the cryptographic private keys that control the ownership of an address (account). Disclosure of, or even momentary loss of control over, the keys can result in the complete and immediate loss of that address’s funds.
    • Specialized key-storing hardware, either a hardware security module (HSM) or hardware wallet, is an effective security control when designed and used properly, but current hardware solutions are less than perfect.
    • Compartmentalization of funds and multisignature wallets are also effective security controls and complement the use of HSMs.
  • Security breaches or outages at third-party API providers represent a secondary risk, which is best mitigated by contingency planning.
  • Centralization of mining power is a systemic risk whose impact is less clear but important to monitor; it represents a potential for blockchain manipulation and, therefore, currency manipulation.
  • Most blockchain software, though open source, has not been formally assessed by reputable application-security teams. Commission regular security reviews to assess blockchain software for traditional vulnerabilities. Use network segmentation to prevent blockchain software from being exposed to potentially exploitable vulnerabilities.

It is our hope that this report can be used as a community resource to inform and encourage organizations pursuing blockchain strategies to do so in a manner that is effective and safe.

This research was conducted by Trail of Bits based upon work supported by DARPA under Contract No. HR001120C0084 (Distribution Statement A, Approved for Public Release: Distribution Unlimited). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Government or DARPA.

Are blockchains decentralized?

A new Trail of Bits research report examines unintended centralities in distributed ledgers

Blockchains can help push the boundaries of current technology in useful ways. However, to make good risk decisions involving exciting and innovative technologies, people need demonstrable facts that are arrived at through reproducible methods and open data.

We believe the risks inherent in blockchains and cryptocurrencies have been poorly described and are often ignored—or even mocked—by those seeking to cash in on this decade’s gold rush.

In response to recent market turmoil and plummeting prices, proponents of cryptocurrency point to the technology’s fundamentals as sound. Are they?

Over the past year, Trail of Bits was engaged by the Defense Advanced Research Projects Agency (DARPA) to examine the fundamental properties of blockchains and the cybersecurity risks associated with them. DARPA wanted to understand those security assumptions and determine to what degree blockchains are actually decentralized.

To answer DARPA’s question, Trail of Bits researchers performed analyses and meta-analyses of prior academic work and of real-world findings that had never before been aggregated, updating prior research with new data in some cases. They also did novel work, building new tools and pursuing original research.

The resulting report is a 30-thousand-foot view of what’s currently known about blockchain technology. Whether these findings affect financial markets is out of the scope of the report: our work at Trail of Bits is entirely about understanding and mitigating security risk.

The report also contains links to the substantial supporting and analytical materials. Our findings are reproducible, and our research is open-source and freely distributable. So you can dig in for yourself.

Key findings

  • Blockchain immutability can be broken not by exploiting cryptographic vulnerabilities, but instead by subverting the properties of a blockchain’s implementations, networking, and consensus protocols. We show that a subset of participants can garner undue, centralized control over the entire system:
    • While the encryption used within cryptocurrencies is for all intents and purposes secure, it does not guarantee security, as touted by proponents.
    • Bitcoin traffic is unencrypted; any third party on the network route between nodes (e.g., internet service providers, Wi-Fi access point operators, or governments) can observe and choose to drop any messages they wish.
    • Tor is now the largest network provider in Bitcoin; just about 55% of Bitcoin nodes were addressable only via Tor (as of March 2022). A malicious Tor exit node can modify or drop traffic.
  • More than one in five Bitcoin nodes are running an old version of the Bitcoin core client that is known to be vulnerable.
  • The number of entities sufficient to disrupt a blockchain is relatively low: four for Bitcoin, two for Ethereum, and less than a dozen for most proof-of-stake networks.
  • When nodes have an out-of-date or incorrect view of the network, this lowers the percentage of the hashrate necessary to execute a standard 51% attack. During the first half of 2021, the actual cost of a 51% attack on Bitcoin was closer to 49% of the hashrate—and this can be lowered substantially through network delays.
  • For a blockchain to be optimally distributed, there must be a so-called Sybil cost. There is currently no known way to implement Sybil costs in a permissionless blockchain like Bitcoin or Ethereum without employing a centralized trusted third party (TTP). Until a mechanism for enforcing Sybil costs without a TTP is discovered, it will be almost impossible for permissionless blockchains to achieve satisfactory decentralization.

Novel research within the report

  • Analysis of the Bitcoin consensus network and network topology
  • Updated analysis of the effect of software delays on the hashrate required to exploit blockchains (we did not devise the theory, but we applied it to the latest data)
  • Calculation of the Nakamoto coefficient for proof-of-stake blockchains (once again, the theory was already known, but we applied it to the latest data)
  • Analysis of software centrality
  • Analysis of Ethereum smart contract similarity
  • Analysis of mining pool protocols, software, and authentication
  • Combining the survey of sources (both academic and anecdotal) that support our thesis that there is a lack of decentralization in blockchains

The research to which this blog post refers was conducted by Trail of Bits based upon work supported by DARPA under Contract No. HR001120C0084 (Distribution Statement A, Approved for Public Release: Distribution Unlimited). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Government or DARPA.