Announcing the Trail of Bits osquery extension repository

Today, we are releasing access to our maintained repository of osquery extensions. Our first extension takes advantage of the Duo Labs EFIgy API to determine if the EFI firmware on your Mac fleet is up to date.

There are very few examples of publicly released osquery extensions. Very little documentation exists on the topic. This post aims to help future developers in navigating through the process of writing an extension for osquery. The rest of this post describes how we implemented the EFIgy extension for osquery.

About EFIgy

At this year’s Ekoparty, Duo Labs presented the results of its research on the state of support and security in EFI firmwares. These software components are really interesting for attackers. They operate on a privilege level that is out of reach even from operating systems and hypervisors. Duo Labs gathered and analyzed all the publicly released Apple updates from the last three years and verified the information by looking at more than 73,000 Macs across different organizations.

The researchers found that many of these computers were running on outdated firmware, even though the required EFI updates were supposed to be bundled in the same operating system patches that the hosts had installed correctly. Duo Labs followed this finding by creating the EFIgy service, a REST endpoint that can access the latest OS and EFI versions for any known Apple product through the use of details such as logic board id and product name.

Programmatically querying EFIgy

EFIgy expects a JSON object containing the details for the system we wish to query. The JSON request doesn’t require many keys; it boils down to hardware model and software versions:

{
  "board_id": "Mac-66E35819EE2D0D05",
  "smc_ver": "2.37f21",
  "build_num": "16G07",
  "rom_ver": "MBP132.0226.B20",
  "hw_ver": "MacBookPro12,1",
  "os_ver": "10.12.4",
  "hashed_uuid": ""
}

The rom_ver key is kind of tricky. It doesn’t include the full EFI version because (according to the author) it includes timestamps that are not necessarily useful or easy to keep track of. Separated by the dot characters, the only fields you need are the first, third and fourth.

As the name suggests, the hashed_uuid field is a SHA256 digest. To compute it correctly, the MAC address of the primary network interface must be prefixed to the system UUID like this: “0x001122334455” + “12345678-1234-1234-1234-1234567890AB”.

The remaining keys are self-explanatory, but keep in mind that they will not be reported correctly when running from a virtual machine. board_id, hw_ver and rom_ver will report information about the virtual components offered by your hypervisor.

Querying the service is simple. The JSON data is sent via an HTTP POST request to the following REST endpoint: https://api.efigy.io/apple/oneshot.

The server response is made of three JSON objects. Compare them with your original values to understand whether your system is fully up to date or not.

{
  "latest_efi_version" : {
    "msg" : "<version>"
  },

  "latest_os_version" : {
    "msg" : "<version>"
  },

  "latest_build_number" : {
    "msg" : "<error message>",
    "error" : "1"
  }
}

Developing osquery extensions

The utilities provided by Duo Labs are easy and straightforward to use, but manually running them all on each system in our fleet was not an easy task. We decided to implement an osquery extension that queries EFIgy, an idea we got from Chris Long.

Why write an extension and not a virtual table? We’ve decided to keep native operating system functions in core and convert everything else into an extension. If a new feature uses an external service or a non-native component, we will default to writing an extension.

The only toolset you have available is the standard library and what you manually import into your project. osquery links everything statically. You have to take special care of the libraries you use. Don’t rely on loading dynamic libraries at runtime from your program.

Once your environment is ready, table extensions do not require much work to be implemented. You just have to inherit from the osquery::TablePlugin class and override the two methods used to define the columns and generate the table rows.

class MyTable final : public osquery::TablePlugin {
  private:
    osquery::TableColumns columns() const override {
      return {
        std::make_tuple(
          “column_name”,
          osquery::TEXT_TYPE,
          osquery::ColumnOptions::DEFAULT
        )
      }
    }

    osquery::QueryData generate(osquery::QueryContext& request) override {
      osquery::Row row;
      row[“column_name”] = “value”;

      return { row };
    }
};

The source files must then be placed in a dedicated folder inside osquery/external. Note that you must add the “extension_” prefix to the folder name. Otherwise, the CMake project will ignore it.

For more complex projects, it is also possible to add a CMakeLists.txt file, creating the target using the following helper function:

ADD_OSQUERY_EXTENSION(${PROJECT_NAME} source1.cpp source2.cpp)

You will have access to some of the libraries in osquery such as Boost, but not some other utilities (e.g. the really useful http_client class).

There is no list of recommended steps to take when developing an extension, but if you plan on writing more than one I recommend you bundle your utility functions in headers that can then be easily imported and reused. Keeping all external libraries statically linked is also a good idea, as it will make redistribution easier.

Using our osquery extensions repo

Without anywhere to submit our new feature, we created a new repository for our extensions. The EFIgy extension is the first item available. Expect more to follow.

Using the repository is simple. However, you will have to clone the full source code of osquery first since the SDK is not part of the distributable package. Building the extension is easy. You only have to create a symbolic link of the source folders you want to compile inside the osquery/external folder, taking care to name the link according to the following scheme: extension_<name>. You can then follow the usual build process for your platform, as the default ALL target will also build all extensions.

cd /src/osquery-extensions
ln -s efigy /src/osquery/external/extension_efigy

cd /src/osquery
make sysprep
make deps

make -j `nproc`
make externals

Extensions are easy to use. You can test them (both with the shell and the daemon) by specifying their path with the –extension parameter. Since they are normal executables, you can also start them after osquery. They will automatically connect via Thrift and expose the new functions. The official documentation explains the process very well.

To quickly test the extension, you can either start it from the osqueryi shell, or launch it manually and wait for it to connect to the running osquery instance.

osqueryi --extension /path/to/extension

Take action

If you have a Mac fleet, you can now monitor it with osquery and the EFIgy extension, and ensure all your endpoints have received the required software and firmware updates.

If you’re reading this post some time in the future, you have even more reason to visit our osquery extension repository. We’ll keep it maintained and add to it over time.

Do you have an idea for an osquery extension? Please file an issue on our Github repo for it. Do you need osquery development? Contact us.

Securing Ethereum at Empire Hacking

If you’re building real applications with blockchain technology and are worried about security, consider this meetup essential. Join us on December 12th for a special edition of Empire Hacking focused entirely on the security of Ethereum.

Why attend?

Four blockchain security experts will be sharing how to write secure smart contracts, and hack them. Two speakers come from our team.

We’ve become one of the industry’s most trusted providers of audits, tools, and best practices for securing smart contracts and their adjacent technologies. We’ve secured token launches, decentralized apps, and entire blockchain platforms. As with past Empire Hacking events, we’re excited to share the way forward with the development community.

Who will be speaking?

  • Sophia D’Antoine of Trail of Bits will discuss Solidity and Ethereum challenges in recent CTF competitions and the tools required to exploit them.
  • John Maurelian of Consensys Diligence will discuss his highlights from Devcon3 about the latest developments in Ethereum security.
  • Mark Mossberg will be sharing how Trail of Bits finds bugs in EVM bytecode with our symbolic execution engine, Manticore.
  • Paul Makowski will share his upcoming security-focused Ethereum token, PolySwarm, which uses blockchain technology to address deficiencies in the threat intelligence industry.
  • Amber Baldet and Brian Schroeder of the Enterprise Ethereum Alliance will discuss the threat modeling, confidential transactions, and zero-knowledge proofs that went into the Quorum blockchain.

When and where?

We’ll be meeting December 12th at 6pm. This month’s meetup will take place at the DataDog offices in the New York Times building. RSVP is required. As per usual, light food and beer will be served.

To find out more about Empire Hacking:

How are teams currently using osquery?

In the year since we ported osquery to Windows, the operating system instrumentation and endpoint monitoring agent has attracted a great deal of attention in the open-source community and beyond. In fact, it recently received the 2017 O’Reilly Defender Award for best project.

Many large and leading tech firms have deployed osquery to do totally customizable and cost-effective endpoint monitoring. Their choice and subsequent satisfaction fuels others’ curiosity about making the switch.

But deploying new software to your company’s entire fleet is not a decision to be made lightly. That’s why we sought to take the pulse of the osquery community – to help current and potential users know what to expect. This marks the start of a four-part blog series that sheds light on the current state of osquery, its shortcomings and opportunities for improvement.

Hopefully, the series will help those of you who are sitting on the fence decide if and how to deploy the platform in your companies.

For our research, we interviewed teams of osquery users at five major tech firms. We asked them:

  • How is osquery deployed and used currently?
  • What benefits are your team seeing?
  • What have been your biggest pain points about using osquery?
  • What new features would you most like to see added?

This post will focus on current use of osquery and its benefits.

How are companies using osquery today?

Market Penetration

osquery’s affordability, flexibility, and cross-platform compatibility has quickly established its place in the endpoint monitoring toolkits of top tech firms. Since its debut in October, 2014, over 1,000 users from more than 70 companies have engaged with the development community through its Slack channel and GitHub repo. In August, osquery developers at Facebook began offering bi-weekly office hours to discuss issues, new features, and design direction.

Users have increased due to a number of recent developments. Since contributors like Trail of Bits and Facebook have transformed osquery to support more operating systems (Windows and FreeBSD), a broader number of organizations are now able to install osquery on a greater portion of their endpoints. Multiple supplementary tools, such as Doorman, Kolide, and Uptycs, have emerged to help users deploy and manage the technology. Monitoring of event-based logs (e.g. process auditing and file integrity monitoring) has further enhanced its utility for incident response. Each of these developments has spurred more organizations with unique data and infrastructure needs to use osquery, sometimes in favor of competing commercial products.

Current Use

All the companies surveyed leveraged osquery for high performance and flexible monitoring of their fleets. Interviewees expressed particular interest in just-in-time incident response including initial malware detection and identifying propagation.

Many teams used osquery in conjunction with other open source and commercial technologies. Some used collection and aggregation services such as Splunk to mine data collected by osquery. One innovative team built incident alerting with osquery by piping log data into ElasticSearch and auto-generated Jira tickets through ElastAlert upon anomaly detection. Most of the companies interviewed expected to phase out some paid services, especially costly suites (e.g. Carbon Black, Tripwire, Red Cloak), in favor of the current osquery build or upon addition of new features.

Deployment Maturity

Deployment maturity for osquery varied widely. One company reported being at the phase of testing and setting up infrastructure. Other companies had osquery deployed on most or all endpoints in their fleets, including one team who reported plans to roll out to 17,500 machines. Three out of the five companies we interviewed had osquery deployed on production servers. However, one of these companies reported having installed osquery on production machines but rarely querying these endpoints due to concerns about osquery’s reliability and scalability. Runaway queries on production fleets was a major concern for all companies interviewed though no production performance incidents were reported.

Strategies for Deployment

Most companies used Chef or Puppet to deploy, configure, and manage osquery installations on their endpoints. One company used the fleet management tool Doorman to maintain their fleet of remote endpoints and bypass the need for separate aggregation tools. Many teams leveraged osquery’s TLS documentation to author their own custom deployment tools that granted them both independence from third party applications and freedom to fully customize features/configurations to their native environments.

Multiple teams took precautions while rolling out osquery by deploying in stages. One team avoided potential performance issues by assigning osquery tasks to CGroups with limits on CPU and memory usage.

osquery Governance

Security teams were responsible for initiating the installation of osquery in the fleet. While most teams did so with buy-in and collaboration from other teams, some executed the installation covertly. One team reported that a performance incident had mildly tarnished the osquery reputation within their organization. Some security teams we interviewed collaborated with other internal teams such as Data Analytics and Machine Learning to mine log data and generate actionable insights.

Benefits of osquery

Teams reported that they liked osquery better than other fleet management tools for a variety of reasons, including:

  • simpler to use,
  • more customizable, and
  • exposed new endpoint data that they had never before had access to.

For teams exploring alternatives to their current tools, the open-source technology helped them avoid the bureaucratic friction of buying new commercial security solutions. For one team, osquery also fit into a growing preference for home-built software within their company.

In its current state, osquery appeared to be most powerful when leveraged as a flexible building block within a suite of tools. Where other endpoint monitoring tools expose users to select log data, osquery provided simple, portable access to a far richer variety of endpoint data. For teams who want to roll their own solutions, or who can’t afford expensive commercial comprehensive suites, osquery was the best option.

How it compares to other endpoint monitoring solutions

Our interviewees mentioned having used or evaluated some alternative endpoint monitoring solutions in addition to osquery. We list the highlights of their comments below. While osquery did present a more flexible, affordable solution overall, some paid commercial solutions still offer distinct advantages, especially in integrating automated prevention and incident response. However, as the development community continues to build features in osquery, the capability gap appears to be closing.

OSSEC

OSSEC is an open source system monitoring and management platform. It features essential incident response tools such as file integrity checking, log monitoring, rootkit detection, and automatic incident response. However, OSSEC lacks osquery’s ability to query multiple hosts (Windows, BSD, etc) with a universal syntax. It’s also not as flexible; users of osquery can quickly form new queries with the usability of SQL syntax, while OSSEC requires cumbersome log file decoders and deliberate ahead-of-time configuration. Both the overall simplicity and the on-going development for community contributed tables have often been cited as advantages osquery has over OSSEC.

SysDig

SysDig provides a commercial container performance monitoring tool and an open source container troubleshooting tool. While osquery is used for security and malicious incident detection, SysDig tools work with real-time data streams (network or file I/O, or tracking errors in running processes) and are best suited for monitoring performance. However, despite significant recent gains in container support including new Docker tables that allow instrumentation at the host level, SysDig maintains the advantage over osquery for performance-sensitive container introspection. Though osquery is capable of running within containers, our respondents indicated that the current version isn’t yet built to support all deployments cleanly. One user reported avoiding deployment of osquery on their Docker-based production fleet for this reason.

Carbon Black

Carbon Black is one of the industry’s leading malware detection, defense, and response packages. In contrast, osquery by itself only provides detection capabilities. However, when combined with alerting systems such as PagerDuty or ElastAlert, osquery can transform into a powerful incident response tool. Finally, interviewees considering Carbon Black remarked on its high price tag and voiced a desire to minimize its use.

Bromium vSentry

Bromium vSentry provides impact containment and introspection powered by micro-virtualization and supported by comprehensive dashboards. While companies can leverage tools like Kolide and Uptycs to access data visualizations similar to osquery, Bromium’s micro-virtualization isolation functionality to quarantine attacks remains an advantage. However, Bromium’s introspection is significantly less flexible and expansive. It can only access data about targeted isolated applications. osquery can be configured to gather data from a growing number of operating-level logs, events, and processes.

Red Cloak

Red Cloak provides automated threat detection as part of a service offering from Dell SecureWorks. It has two advantages over osquery: first, it provides an expert team to help with analysis and response; second, it aggregates endpoint information from all customers to inform and improve its detection and response. For organizations focused solely on breach response, Red Cloak may be worth its cost. However, for IT teams who want direct access to a variety of endpoint data, osquery is a better and cheaper solution.

Conclusion

osquery fills a need in many corporate security teams; its transparency and flexibility make it a great option for rolling bespoke endpoint monitoring solutions. Without any modification, it exposes all the endpoint data an analysis engine needs. We expect (and hope) to hear from more security teams multiplying osquery’s power with their incident response toolkit.

That will happen faster if teams would share their deployment techniques and lessons learned. Much of the Slack and Github discussions focus on codebase issues. Too few users openly discuss innovative implementation strategies. But that isn’t the only reason holding back osquery’s adoption.

The second post in this series will focus on users’ pain points. If you use osquery today and have pain points you’d like to add to our research, please let us know! We’d love to hear from you.

How does your experience with osquery compare to that of the teams mentioned in this post? Do you have other workarounds, deployment strategies, or features you’d like to see built in future releases? Tell us! Help us lead the way in improving osquery’s development and implementation.

Hands on the Ethernaut CTF

Last week Zeppelin released their Ethereum CTF, Ethernaut.

This CTF is a good introduction to discover how to interact with a blockchain and learn the basics of the smart contract vulnerabilities. The CTF is hosted on the ropsten blockchain, and you can receive free ethers for it. The browser developer console is used to interact with the CTF, as well as the metamask plugin.

I was fortunate enough to be the first one to finish the challenges. The following is how I did it.

1. Fallback

This challenge is the first one, and I think it is more of an introduction, to be sure that everyone is able to play with the API. Let us see in detail what a Solidity smart contract looks like.

Challenge description

The contract is composed of one constructor and four functions. The goal is to become the owner of the contract and to withdraw all the money.

The first function is the constructor of the contract, because it has the same name as the contract. The constructor is a specific function which is called only once when the contract is first deployed, and cannot be called later. This function is usually used to set up some parameters (here an initial contribution from the owner).

function Fallback() {
  contributions[msg.sender] = 1000 * (1 ether);
}

The second function, contribute(), stores the number of ethers (msg.value) sent by the caller in the contributions map. If this value is greater than the contributions of the contract owner, then the caller becomes the owner of the contract.

function contribute() public payable {
  require(msg.value < 0.001 ether);
  contributions[msg.sender] += msg.value;
  if(contributions[msg.sender] > contributions[owner]) {
    owner = msg.sender;
  }
}

getContribution() is a simple getter:

function getContribution() public constant returns (uint) {
  return contributions[msg.sender];
}

withdraw() allows the owner of the contract to withdraw all the money. Notice the onlyOwner keyword after the signature of the function. This is a modifier that ensures that this function is only called by the owner.

function withdraw() onlyOwner {
  owner.transfer(this.balance);
}

Finally, the last function is the fallback function of the contract. This function can be executed if the caller has previously made a contribution.

function() payable {
  require(msg.value > 0 && contributions[msg.sender] > 0);
  owner = msg.sender;
}

Fallback function

To understand what a fallback is, we have to understand the function selector and arguments mechanism in ethereum. When you call a function in ethereum, you are in fact sending a transaction to the network. This transaction contains, among other things, the amount of ether sent (msg.value) and a so-called data, which is an array of bytes. This array of bytes holds the id of the function to be called, and the function’s arguments. They choose to use the first four bytes of the keccak256 of the function signature as the function id. For example, if the function signature is transfer(address,uint256), the function id is 0xa9059cbb.

If you want to call transfer(0x41414141, 0x42), the data will be:

0xa9059cbb00000000000000000000000000000000000000000000000000000000414141410000000000000000000000000000000000000000000000000000000000000042

During its execution, the first thing that a smart contract does is to check the function id, using a dispatcher. If there is no match, the fallback function is called, if it exists.

You can visualize this dispatcher using our open source disassembler Ethersplay:

ethersplay_largeEthersplay shows the EVM dispatcher structure(*)

(*) For simplification, the Owner inheritance was removed from the source code. You can find the solidity file and the runtime bytecode here: fallback

Solution

If we put everything together we have to:

  1. Call contribution to put some initial value inside contributions
  2. Call the fallback function to become the owner of the contract
  3. Call withdraw to get all the money

(1) is easily done by calling contract.contribution({value:1}) in the browser’s developer tool console. A simple way to call the fallback function (2) is just to send to ether directly to the contract using the metamask plugin. Then (3) is achieved by calling contract.withdraw().

2. Fallout

Challenge description

The goal here is to become the owner of the contract.

At first, this contract appears to have one constructor and four functions. But if we look closer at the constructor, we realize that the name of the function is slightly different than the contract’s name:

contract Fallout is Ownable {

  mapping (address => uint) allocations;

  /* constructor */
  function Fal1out() payable {
    owner = msg.sender;
    allocations[owner] = msg.value;
  }

As a result, this function is not a constructor, but a classic public function. Anyone can call it once the contract is deployed!

This may look too simple to be a real vulnerability, but it is real. Using our internal static analyzer Slither, we have found several contracts where this mistake was made (for example, ZiberCrowdsale or PeerBudsToken)!

Solution

We only need to call contract.Fal1out() to become the owner, that’s it!

3. Token

Challenge description

Here we are given 20 free tokens in the contract. Our goal is to find a way to hold a very large amount of tokens.

The function called  transfer allows the transfer of tokens between users:

function transfer(address _to, uint _value) public returns (bool) {
  require(balances[msg.sender] - _value >= 0);
  balances[msg.sender] -= _value;
  balances[_to] += _value;
  return true;
}

At first, the function looks fine, as it seems to check for overflow

require(balances[msg.sender] - _value >= 0);

Usually, when I have to deal with arithmetic computations, I go for the easy way and use our open source symbolic executor manticore to check if I can abuse the contract. We recently added support for evm, a really powerful tool for auditing integer related issues. But here, with a closer look, we realize that _value and balances are unsigned integers, meaning that balances[msg.sender] - _value >= 0 is always true!

So we can produce an underflow in

balances[msg.sender] -= _value;

As a result, balances[msg.sender] will contain a very large number!

Solution

To trigger the underflow, we can simply call contract.transfer(0x0, 21).  balances[msg.sender] will then contain 2**256 – 1.

4.Delegation

Challenge description

The goal here is to become the owner of the contract Delegation.

There is no direct way to change the owner in this contract. However, it holds another contract, Delegate with this function:

function pwn() {
  owner = msg.sender;
}

A particularity of Delegation is the use of delegatecall in the fallback function.

function() {
  if(delegate.delegatecall(msg.data)) {
    this;
  }
}

The hint of the challenge is pretty explicit about it:

Usage of delegatecall is particularly risky and has been used as an attack vector on multiple historic hacks. With it, you contract is practically saying “here, -other contract- or -other library-, do whatever you want with my state”. Delegates have complete access to your contract’s state. The delegatecall function is a powerful feature, but a dangerous one, and must be used with extreme care.

Please refer to the The Parity Wallet Hack Explained article for an accurate explanation of how this idea was used to steal 30M USD.

So here we need to call the fallback function, and to put in the msg.data the signature of pwn(), so that the delegatecall will execute the function pwn() within the state of Delegation and change the owner of the contract.

Solution

As we saw in “Fallback”, we have to put in msg.data the function id of pwn(); which is 0xdd365b8b. As a result, Delegate.pwn() will be called within the state of Delegation, and we will become the owner of the contract.

5. Force

Challenge description

Here we have to send ethers to an empty contract. As there is no payable fallback function, a direct send of ether to the contract will fail.

There are other ways to send ether to a contract without executing its code:

  • Calling selfdestruct(address)
  • Specifying the address as the reward mining destination
  • Sending ethers to the address before the creation of the contract

Solution

We can create a contract that will simply call selfdestruct(address) to the targeted contract.

contract Selfdestruct{
  function Selfdestruct() payable{}
  function attack(){
    selfdestruct(0x..);
  }
}

Note that we use a payable constructor. Doing so we can directly put some value inside the contract at its construction. This value will then be sent through selfdestruct.

You can easily test and deploy this contract on ropsten using Remix browser.

6. Re-entrancy

Challenge description

The last challenge! You have to send one ether to the contract during its creation and then get your money back.

The contract has four functions. We are interested in two of them.

donate lets you donate ethers to the contract, and the number of ethers sent is stored in balances.

function donate(address _to) public payable {
  balances[_to] += msg.value;
}

withdraw, the second function allows users to retrieve the ethers that were stored in balances.

function withdraw(uint _amount) public {
  if(balances[msg.sender] >= _amount) {
    if(msg.sender.call.value(_amount)()) {
      _amount;
    }
    balances[msg.sender] -= _amount;
  }
}

The ethers are sent through the call msg.sender.call.value(_amount)().

At first, everything seems fine here, as the value sent decreased in  balances[msg.sender] -= _amount; and there is no way to increase balances without sending ethers.

Now recall the fallback function mechanism explained in “Fallback.” If you send ethers to a contract containing a fallback function, this function will be executed. What is the problem here? You can have a fallback function which calls back withdraw, and thus msg.sender.call.value(_amount)() can be executed twice before balances[msg.sender] -= _amount is executed!

This vulnerability is called a re-entrancy vulnerability and was used during the unfamous DAO hack.

Solution

To exploit a re-entrancy vulnerability you have to use another contract as a proxy. This contract will need to:

  1. Have a fallback function which will call withdraw
  2. Call donate to deposit ethers in the vulnerable contract
  3. Call withdraw

In our not-so-smart-contracts database, you will find an example of a generic skeleton to exploit this vulnerability. I’ll leave the exercise of adapting this skeleton to the reader.

Similar to the previous challenge, you can test and deploy the contract on ropsten using Remix browser.

Conclusion

This CTF was really cool. The interface makes it easy to get into smart contract security. Zeppelin did a good job, so thanks to them!

If you are interested in the tools mentioned in the article, or you need a smart contract security assessment, do not hesitate to contact us!

Trail of Bits joins the Enterprise Ethereum Alliance

We’re proud to announce that Trail of Bits has joined the Enterprise Ethereum Alliance (EEA), the world’s largest open source blockchain initiative. As the first information security company to join, and currently one of the industry’s top smart contract auditors, we’re excited to contribute our unparalleled expertise to the EEA.

As companies begin to re-architect their critical systems with blockchain technology, they will need a strong software engineering model that ensures the safety of confidential data and integration with existing security best practices. We already work with many of the world’s largest companies to secure critical systems and products. As businesses rush to make use of this emerging technology, we look forward to designing innovative and pragmatic security solutions for the enterprise Ethereum community.

Ethereum Enterprise Alliance

We’re helping to secure Ethereum with our expertise, tools, and results.

Preparing Ethereum for production enterprise use will take a lot of work. Collaboration with other motivated researchers, developers, and users is the best way to build a secure and useful Enterprise Ethereum ecosystem. By contributing the tools we’re building to help secure public Ethereum applications, and participating in the EEA Technical Steering Committee’s working groups, we will help the EEA to ensure the security model for Enterprise Ethereum meets enterprise requirements.

How we will contribute

  • Novel​ ​research.​ We’ll bring our industry-leading security expertise to help discover, formalize and secure the unexpected behaviors in DApps and Smart Contracts. We’re already accumulating discoveries from the security audits we’ve conducted.
  • Foundational​ ​tools.​ We’ll help other members reduce risk when building on this technology. As our audits uncover fundamental gaps in Ethereum’s tooling, we’ll fill them in with tools like our symbolic executor, Manticore, and others in development.
  • Sharing​ ​attitude.​ We’ll help define a secure development process for smart contracts, share the tools that we create, and warn the community about pitfalls we encounter. Our results will help smart contract developers and auditors find vulnerabilities and decrease risk.

Soon, we’ll release the internal tools and guidance we have adapted and refined over the course of many recent smart contract audits. In the weeks ahead, you can expect posts about:

  • Manticore, a symbolic emulator capable of simulating complex multi-contract and multi-transaction attacks against EVM bytecode.
  • Not So Smart Contracts, a collection of example Ethereum smart contract vulnerabilities, including code from real smart contracts, useful as a reference and a benchmark for security tools.
  • Ethersplay, a graphical Binary Ninja-based EVM disassembler capable of method recovery, dynamic jump computation, source code matching, and bytecode diffing.
  • Slither, a static analyzer for the Solidity AST that detects common security issues in reentrancy, constructors, method access, and more.
  • Echidna, a property-based tester for EVM bytecode with integrated shrinking that can rapidly find bugs in smart contracts in a manner similar to fuzzing.

We’ll also begin publishing case studies of our smart contract audits, how we used those tools, and the results we found.

Get help auditing your smart contracts

Contact us for a demonstration of how we can help your enterprise make the most of Ethereum and blockchain.

Our team is growing

We’ve added five more to our ranks in the last two months, bringing our total size to 32 employees. Their resumes feature words and acronyms like ‘CTO,’ ‘Co-founder’ and ‘Editor.’ You might recognize their names from publications and presentations that advance the field.

We’re excited to offer them a place where they can dig deeper into security research.

Please help us welcome Mike Myers, Chris Evans, Evan Teitelman, Evan Sultanik, and Alessandro Gario to Trail of Bits!

Mike Myers

Leads our L.A. office (of one). Mike brings 15 years of experience in embedded software development, exploitation, malware analysis, code audits, threat modeling, and reverse engineering. Prior to joining Trail of Bits, Mike was CTO at a small security firm that specializes in cyber security research for government agencies and Fortune 500 companies. Before that, as Principal Security Researcher at Crucial Security (acquired by Harris Corporation), he was the most senior engineer in a division providing cyber capabilities and on-site operations support. Mike contributed the Forensics chapter of the CTF Field Guide, and co-authored “Exploiting Weak Shellcode Hashes to Thwart Module Discovery” in POC || GTFO. He’s NOP certified.

Chris Evans

Chris’s background is in operating systems and software development, but he’s also interested in vulnerability research, reverse engineering and security tooling. Prior to joining Trail of Bits, Chris architected the toolchain and core functionality of a host-based embedded defense platform, reversed and exploited ARM TrustZone secure boot vulnerabilities, and recovered data out of a GPIO peripheral with Van Eck phreaking. Chris has experience in traditional frontend and backend development through work at Mic and Foursquare. He activates Vim key binding mode more frequently than could possibly be warranted.

Evan Teitelman

After solving the world’s largest case of lottery fraud, Evan spent the last year leading code audits for over a dozen state lotteries, including their random number generators (RNGs). Prior to that, Evan focused on security tooling for the Android Kernel and SELinux as a vulnerability researcher with Raytheon SI. Also, he brings experience as a circuit board designer and embedded programmer. He founded BlackArch Linux. In his free time he enjoys hiking and climbing in Seattle, Washington.

Evan Sultanik

A computer scientist with extensive experience in both industry and academia, Evan is particularly interested in computer security, AI/ML/NLP, combinatorial optimization, and distributed systems. Evan is an editor and frequent contributor to PoC || GTFO, a journal of offensive security and reverse engineering that follows in the tradition of Phrack and Uninformed. Prior to Trail of Bits, Evan was the Chief Scientist for a small security firm that specializes in providing cyber security research to government agencies and Fortune 500 companies. He obtained his PhD in Computer Science from Drexel University, where he still occasionally moonlights as an adjunct faculty member. A Pennsylvania native, Evan contributes time to Code for Philly and has published research on zoning density changes in Philadelphia over the last five years.

Alessandro Gario

Hailing out of Italy, Alessandro brings ten years of systems software development experience to the team. Prior to Trail of Bits, he worked on building and optimizing networked storage systems and reverse engineering file formats. In his free time, he enjoys playing CTFs and reverse engineering video games. Alessandro’s mustache has made him a local celebrity. He is our second European employee.

We are very excited for the contributions these people will make to the discipline of information security. If their areas of expertise overlap with the challenges your organization faces, please contact us for help.

iOS jailbreak detection toolkit now available

We now offer a library for developers to check if their apps are running on jailbroken phones. It includes the most comprehensive checks in the industry and it is App Store compatible. Contact us now to license the iVerify security library for your app.

Jailbreaks threaten your work

Users like to install jailbreaks on their phones for extra functionality, unaware that they’re increasing their exposure to risks. Jailbreaks disable many of iOS’s security features such as mandatory code signing and application sandboxing. Apps and code found outside the App Store can contain malware and jailbreaks themselves have included backdoors in the past.

Moreover, running an app on a jailbroken phone may indicate a user is attempting to manipulate the app. App developers deserve to know when their apps are installed on such untrustworthy phones.

Developing the security library to do so requires time and knowledge outside the core competency of many development teams. Establishing and maintaining expertise in the depths of iOS security internals and keeping up with new developments in jailbreak tools and techniques requires more time than many developers can afford. Ineffective jailbreak detection can be worse than no jailbreak detection at all.

Why you should use iVerify

Trail of Bits employs some of the world’s best experts in the field of iOS security internals. Our engineers have reviewed the security model of iOS, jailbreak techniques, and the tools that exist to run them today and developed the best checks possible. The resulting library, iVerify, includes checks for known jailbreaks, like Pangu, and checks for anomalies that may indicate unknown or custom jailbreaks.

iVerify easily integrates into your app as an iOS Framework. Your app can read a simple pass/fail result or it can inspect the results of each check individually.

As an optional feature, the raw checks are aggregated into a JSON message that includes the results of each individual check. A helper function adds other identifying information from the device. Developers can match this information with user information from their app, send it to a centralized logging facility for collection and analysis, and then use it to take action.

We continuously explore new versions of iOS for more effective checks capable of finding known and unknown jailbreaks. iVerify detects jailbreaks on iOS 10 and 11, and our expert team will update the library as new versions are released and new checks are developed.

Start protecting your work with superb jailbreak detection

iVerify delivers an easy-to-use solution without heavy-weight dependencies or obligations to a SaaS service. The checks are the best available and are maintained by our team of experts. The iVerify library is suitable for App Store deployment and will integrate into your app easily.

Contact us now to discuss licensing options.

Tracking a stolen code-signing certificate with osquery

Oct. 14th, 2017: This post has been updated to reflect a change in the osquery table’s final name, from ‘signature’ to ‘authenticode.’

Recently, 2.27 million computers running Windows were infected with malware signed with a stolen certificate from the creators of a popular app called CCleaner, and inserted into its software update mechanism. Fortunately, signed malware is now simple to detect with osquery thanks to a pull request submitted by our colleague Alessandro Gario that adds Windows executable code signature verification (also known as Authenticode). This post explains the importance of code signatures in incident response, and demonstrates a use case for this new osquery feature by using it to detect the recent CCleaner malware.

If you are unfamiliar with osquery, take a moment to read our previous blog post in which we explain why we are osquery evangelists, and how we extended it to run on the Windows platform. Part of osquery’s appeal is its flexibility and open-source model – if there’s another feature you need built, let us know!

Code-signed malware

Code signing was intended to be an effective deterrent against maliciously modified executables, and to allow a user (or platform owner) to choose whether to run executables from untrusted sources. Unfortunately, on general-purpose computing platforms like Windows, third-party software vendors are individually responsible for protecting their code-signing certificates. Malicious actors realized that they only needed to steal one of these certificates in order to sign malware and make it appear to be from a legitimate software vendor. This realization (and the high-profile Stuxnet incident) began a trend of malware signed with stolen code-signing certificates. It has become a routine feature of criminal and nation-state malware attacks in the past few years, and most recently happened again with an infected software update to the popular app CCleaner.

So, defenders already know that a trust model based on an assumption that all third-party software vendors can protect their code-signing certificates is untenable, and that on platforms like Windows, code-signing is only a weak trust marker or application whitelisting mechanism. But, there’s another use for code signatures: incident response. Once a particular signing certificate is known to be stolen, it also works as a telltale indicator of compromise. As the defender you can make lemonade out of these lemons: search for other systems on your network with executables that were also signed with this stolen certificate. The malware might have successfully evaded antivirus-type protections, but any code signed with a known-stolen certificate is an easy red flag: signing can be checked with a 0% chance of any false-positives. osquery offers an ideal method for performing such a search.

Verifying Authenticode signatures with osquery

New sensors are added to osquery with the addition of “tables,” maintaining the abstraction of all system information as SQL tables.

To add a table to osquery, you first define its spec, or schema. An osquery table spec is just a short description of the table’s columns, their data types, and short descriptions, as well as a reference to the implementation. In Alessandro’s pull request, he added an ‘authenticode’ virtual table for Windows, containing the following columns: path, original_program_name (from the publisher), serial_number, issuer_name, subject_name, and result.

Alessandro implemented the code to read code signature and certificate information from the system in osquery/tables/system/windows/authenticode.cpp. The verification of signatures is done using a call to the system API, WinVerifyTrust().

Here’s a simplified example of using osquery to check a Windows executable’s code signature:

osquery> SELECT serial_number, issuer_name, subject_name,
    ...> result FROM authenticode
    ...> WHERE path = 'C:\Windows\explorer.exe';

1

Most of the columns are self-explanatory. The result values aren’t. “Result” could mean:

State Explanation
missing Missing signature.
invalid Invalid signature, caused by missing or broken files.
untrusted Signature that could not be validated.
distrusted Valid signature, explicitly distrusted by the user.
valid Valid signature, but which is not explicitly trusted by the user.
trusted Valid signature, trusted by the user.

Getting focused results with SQL in osquery

To make the most out of this new functionality, perform JOIN queries with other system tables within osquery. We will demonstrate how using SQL queries enhances system monitoring by reducing the amount of noise when listing processes:

osquery> SELECT process.pid, process.path, authenticode.result
    ...> FROM processes as process
    ...> LEFT JOIN authenticode
    ...> ON process.path = authenticode.path
    ...> WHERE result = 'missing';

+------+-----------------------------------------------------------+---------+
| pid  | path                                                      | result  |
+------+-----------------------------------------------------------+---------+
| 3752 | c:\windows\system32\sihost.exe                            | missing |
| 3872 | C:\Windows\system32\notepad.exe                           | missing |
| 4860 | C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe | missing |
| 5200 | C:\Windows\system32\conhost.exe                           | missing |
| 6040 | C:\Windows\osqueryi.exe                                   | missing |
+------+-----------------------------------------------------------+---------+

Tracking a stolen signing certificate

Assume that you have just learned of a malware campaign. The malware authors code-signed their executables using a code-signing certificate that they stole from a legitimate software vendor. The vendor has responded to the incident by acquiring a new code-signing certificate and redistributing their application signed with the new certificate. In this example, we will use CCleaner. How can you search a machine for any software signed with this stolen certificate, but filter out software signed with the vendor’s new certificate?

Example 1: Find executables signed with the stolen certificate

osquery> SELECT files.path, authenticode.subject_name,
    ...>        authenticode.serial_number,
    ...>        authenticode.result AS status
    ...> FROM (
    ...>   SELECT * FROM file
    ...>   WHERE directory = "C:\Program Files\CCleaner"
    ...> ) AS files
    ...> LEFT JOIN authenticode
    ...> ON authenticode.path = files.path
    ...> WHERE authenticode.serial_number == "4b48b27c8224fe37b17a6a2ed7a81c9f";

3

Example 2: Find executables signed by the affected vendor, but not with their new certificate

osquery> SELECT files.path, authenticode.subject_name,
    ...>        authenticode.serial_number,
    ...>        authenticode.result AS status
    ...> FROM (
    ...>   SELECT * FROM file
    ...>   WHERE directory = "C:\Program Files\CCleaner"
    ...> ) AS files
    ...> LEFT JOIN authenticode
    ...> ON authenticode.path = files.path
    ...> WHERE authenticode.subject_name LIKE "%Piriform%"
    ...> AND authenticode.serial_number != "52b6a81474e8048920f1909e454d7fc0";

4

Example 3: Code signatures and file hashing

Perhaps you would also like to keep a log of hashes, to keep track of what has been installed:

SELECT files.path AS path,
    ...>        authenticode.subject_name AS subject_name,
    ...>        authenticode.serial_number AS serial_number,
    ...>        authenticode.result AS status,
    ...>        hashes.sha256 AS sha256
    ...> FROM (
    ...>   SELECT * FROM file
    ...>   WHERE directory = "C:\Program Files\CCleaner"
    ...> ) AS files
    ...> LEFT JOIN authenticode
    ...> ON authenticode.path = files.path
    ...> LEFT JOIN hash AS hashes
    ...> ON hashes.path = files.path
    ...> WHERE authenticode.subject_name LIKE "%Piriform%"
    ...> AND authenticode.serial_number != "52b6a81474e8048920f1909e454d7fc0"

5

For the purposes of our examples here, notice that we have restricted the searches to “C:\Program Files\CCleaner”. You could tailor the scope of your search as desired.

The queries we’ve shown have been run in osquery’s interactive shell mode, which is more appropriate for incident response. You could run any of these queries on a schedule – using osquery for detection rather than response. For this, you would install osqueryd (the osquery daemon) on the hosts you wish to monitor, and configure logging infrastructure to collect the output of these queries (feeding the osquery output to, for example, LogStash / ElasticSearch for later analysis).

Future osquery Work

In this post we demonstrated the flexibility of osquery as a system information retrieval tool: using familiar SQL syntax, you can quickly craft custom queries that return only the information relevant to your current objective. The ability to check Authenticode signatures is just one use of osquery as a response tool to search for potential indicators of compromise. Many IT and security teams are using osquery for just-in-time incidence response including initial malware detection and identifying propagation.

Trail of Bits was early to recognize osquery’s potential. For over a year we have been adding various features like this one in response to requests from our clients. If you are already using osquery or considering using it and there’s a feature you need built, let us know! We’re ready to help you tailor osquery to your needs.

Microsoft didn’t sandbox Windows Defender, so I did

Microsoft exposed their users to a lot of risks when they released Windows Defender without a sandbox. This surprised me. Sandboxing is one of the most effective security-hardening techniques. Why did Microsoft sandbox other high-value attack surfaces such as the JIT code in Microsoft Edge, but leave Windows Defender undefended?

As a proof of concept, I sandboxed Windows Defender for them and, am now open sourcing my code as the Flying Sandbox Monster. The core of Flying Sandbox Monster is AppJailLauncher-rs, a Rust-based framework to contain untrustworthy apps in AppContainers. It also allows you to wrap the I/O of an application behind a TCP server, allowing the sandboxed application to run on a completely different machine, for an additional layer of isolation.

In this blog post, I describe the process and results of creating this tool, as well as thoughts about Rust on Windows.

Flying Sandbox Monster running Defender in a sandbox to scan a WannaCry binary.

The Plan

Windows Defender’s unencumbered access to its host machine and wide-scale acceptance of hazardous file formats make it an ideal target for malicious hackers. The core Windows Defender process, MsMpEng, runs as a service with SYSTEM privileges. The scanning component, MpEngine, supports parsing an astronomical number of file formats. It also bundles full-system emulators for various architectures and interpreters for various languages. All of this, performed with the highest level of privilege on a Windows system. Yikes.

This got me thinking. How difficult would it be to sandbox MpEngine with the same set of tools that I had used to sandbox challenges for the CTF community two years ago?

The first step towards a sandboxed Windows Defender is the ability to launch AppContainers. I wanted to re-use AppJailLauncher, but there was a problem. The original AppJailLauncher was written as a proof-of-concept example. If I had any sense back then, I would’ve written it in C++ Core rather than deal with the pains of memory management. Over the past two years, I’ve attempted rewriting it in C++ but ended up with false starts (why are dependencies always such a pain?).

But then inspiration struck. Why not rewrite the AppContainer launching code in Rust?

Building The Sandbox

A few months later, after crash coursing through Rust tutorials and writing a novel of example Rust code, I had the three pillars of support for launching AppContainers in Rust: SimpleDacl, Profile, and WinFFI.

  • SimpleDacl is a generalized class that handles adding and removing simple discretionary access control entries (ACE) on Windows. While SimpleDacl can target both files and directories, it has a few setbacks. First, it completely overwrites the existing ACL with a new ACL and converts inherited ACEs to “normal” ACEs. Also, it disregards any ACEs that it cannot parse (i.e. anything other than AccessAllowedAce and AccessDeniedAce. Note: we don’t support mandatory and audit access control entries.).
  • Profile implements creation of AppContainer profiles and processes. From the profile, we can obtain a SID that can be used to create ACE on resources the AppContainer needs to access.
  • WinFFI contains the brunt of the functions and structures winapi-rs didn’t implement as well as useful utility classes/functions. I made a strong effort to wrap every raw HANDLE and pointer in Rust objects to manage their lifetimes.

Next, I needed to understand how to interface with the scanning component of Windows Defender. Tavis Ormandy’s loadlibrary repository already offered an example C implementation and instructions for starting an MsMpEng scan. Porting the structures and function prototypes to Rust was a simple affair to automate, though I initially forgot about array fields and function pointers, which caused all sorts of issues; however, with Rust’s built-in testing functionality, I quickly resolved all my porting errors and had a minimum test case that would scan an EICAR test file.

The basic architecture of Flying Sandbox Monster.

Our proof-of-concept, Flying Sandbox Monster, consists of a sandbox wrapper and the Malware Protection Engine (MpEngine). The single executable has two modes: parent process and child process. The mode is determined by the presence of an environment variable that contains the HANDLEs for the file to be scanned and child/parent communication. The parent process populates these two HANDLE values prior to creating an AppContainer’d child process. The now-sandboxed child process loads the malware protection engine library and scans the input file for malicious software.

This was not enough to get the proof-of-concept working. The Malware Protection Engine refused to initialize inside an AppContainer. Initially, I thought this was an access control issue. After extensive differential debugging in ProcMon (comparing AppContainer vs non-AppContainer execution), I realized the issue might actually be with the detected Windows version. Tavis’s code always self-reported the Windows version as Windows XP. My code was reporting the real underlying operating system; Windows 10 in my case. Verification via WinDbg proved that this was indeed the one and only issue causing the initialization failures. I needed to lie to MpEngine about the underlying Windows version. When using C/C++, I would whip up a bit of function hooking code with Detours. Unfortunately, there was no equivalent function hooking library for Rust on Windows (the few hooking libraries available seemed a lot more “heavyweight” than what I needed). Naturally, I implemented a simple IAT hooking library in Rust (32-bit Windows PE only).

Introducing AppJailLauncher-rs

Since I had already implemented the core components of AppJailLauncher in Rust, why not just finish the job and wrap it all in a Rust TCP server? I did, and now I’m happy to announce “version 2” of AppJailLauncher, AppJailLauncher-rs.

AppJailLauncher was a TCP server that listened on a specified port and launched an AppContainer process for every accepted TCP connection. I tried not to reinvent the wheel, but mio, the lightweight IO library for Rust, just didn’t work out. First, mio’s TcpClient did not provide access to raw “socket HANDLEs” on Windows. Second, these raw “socket HANDLEs” were not inheritable by the child AppContainer process. Because of these issues, I had to introduce another “pillar” to support appjaillauncher-rs: TcpServer.

TcpServer is responsible for instantiating an asynchronous TCP server with a client socket that is compatible with STDIN/STDOUT/STDERR redirection. Sockets created by the socket call cannot redirect a process’s standard input/output streams. Properly working standard input/output redirection requires “native” sockets (as constructed via WSASocket). To allow the redirection, TcpServer creates these “native” sockets and does not explicitly disable inheritance on them.

My Experience with Rust

My overall experience with Rust was very positive, despite the minor setbacks. Let me describe some key features that really stood out during AppJailLauncher’s development.

Cargo. Dependency management with C++ on Windows is tedious and complex, especially when linking against third-party libraries. Rust neatly solves dependency management with the cargo package management system. Cargo has a wide breadth of packages that solve many common-place problems such as argument parsing (clap-rs), Windows FFI (winapi-rs et. al.), and handling wide strings (widestring).

Built-in Testing. Unit tests for C++ applications require a third-party library and laborious, manual effort. That’s why unit test are rarely written for smaller projects, like the original AppJailLauncher. In Rust, unit test capability is built into the cargo system and unit tests co-exist with core functionality.

The Macro System. Rust’s macro system works at the abstract syntax tree (AST) level, unlike the simple text substitution engine in C/C++. While there is a bit of a learning curve, Rust macros completely eliminate annoyances of C/C++ macros like naming and scope collisions.

Debugging. Debugging Rust on Windows just works. Rust generates WinDbg compatible debugging symbols (PDB files) that provide seamless source-level debugging.

Foreign Function Interface. The Windows API is written in, and meant to be called from, C/C++ code. Other languages, like Rust, must use a foreign function interface (FFI) to invoke Windows APIs. Rust’s FFI to Windows (the winapi-rs crate) is mostly complete. It has the core APIs, but it is missing some lesser used subsystems like access control list modification APIs.

Attributes. Setting attributes is very cumbersome because they only apply to the next line. Squashing specific code format warnings necessitates a sprinkling of attributes throughout the program code.

The Borrow Checker. The concept of ownership is how Rust achieves memory safety. Understanding how the borrow checker works was fraught with cryptic, unique errors and took hours of reading documentation and tutorials. In the end it was worth it: once it “clicked,” my Rust programming dramatically improved.

Vectors. In C++, std::vector can expose its backing buffer to other code. The original vector is still valid, even if the backing buffer is modified. This is not the case for Rust’s Vec. Rust’s Vec requires the formation of a new Vec object from the “raw parts” of the old Vec.

Option and Result types. Native option and result types should make error checking easier, but instead error checking just seems more verbose. It’s possible to pretend errors will never exist and just call unwrap, but that will lead to runtime failure when an Error (or None) is inevitably returned.

Owned Types and Slices. Owned types and their complementary slices (e.g. String/str, PathBuf/Path) took a bit of getting used to. They come in pairs, have similar names, but behave differently. In Rust, an owned type represents a growable, mutable object (typically a string). A slice is a view of an immutable character buffer (also typically a string).

The Future

The Rust ecosystem for Windows is still maturing. There is plenty of room for new Rust libraries to simplify development of secure software on Windows. I’ve implemented initial versions of a few Rust libraries for Windows sandboxing, PE parsing, and IAT hooking. It is my hope that these are useful to the nascent Rust on Windows community.

I used Rust and AppJailLauncher to sandbox Windows Defender, Microsoft’s flagship anti-virus product. My accomplishment is both great and a bit shameful: it’s great that Windows’ robust sandboxing mechanism is exposed to third-party software. It’s shameful that Microsoft hasn’t sandboxed Defender on its own accord. Microsoft bought what eventually became Windows Defender in 2004. Back in 2004 these bugs and design decisions would be unacceptable, but understandable. During the past 13 years Microsoft has developed a great security engineering organization, advanced fuzzing and program testing, and sandboxed critical parts of Internet Explorer. Somehow Windows Defender got stuck back in 2004. Rather than taking Project Zero’s approach to the problem by continually pointing out the symptoms of this inherent flaw, let’s bring Windows Defender back to the future.

An extra bit of analysis for Clemency

This year’s DEF CON CTF used a unique hardware architecture, cLEMENCy, and only released a specification and reference tooling for it 24 hours before the final event began. cLEMENCy was purposefully designed to break existing tools and make writing new ones harder. This presented a formidable challenge given the timeboxed competition occurs over a single weekend.

Ryan, Sophia, and I wrote and used a Binary Ninja processor module for cLEMENCy during the event. This helped our team analyze challenges with Binary Ninja’s graph view and dataflow analyses faster than if we’d relied on the limited disassembler and debugger provided by the organizers. We are releasing this processor module today in the interest of helping others who want to try out the challenges on their own.

Binary Ninja in action during the competition

cLEMENCy creates a more equitable playing field in CTFs by degrading the ability to use advanced tools, like Manticore or a Cyber Reasoning System. It accomplishes this with architectural features such as:

  • 9-bit bytes instead of 8-bits. This makes parsing the binary difficult. The byte length of the architecture of the system parsing a challenge does not match that in cLEMENCy. The start of a byte on both systems would only match every 9th byte.
  • It’s Middle Endian. Every other architecture stores values in memory in one of two ways: from most significant byte to least significant (Big Endian), or least significant to most significant (Little Endian). Rather than storing a value like 0x123456 as 12 34 56 or 56 34 12, Middle Endian stores it as 34 56 12.
  • Instructions have variable length opcodes. Instructions were anywhere from 18 to 54 bits, with opcodes being anywhere from 4 bits to 18 bits.

This required creativity in a short timespan. With only 24 hours’ head start, we needed to work fast if we wanted something usable before the end of the four-day competition. This would have been hard to do even with an amenable architecture. Here’s how we solved these problems to write and use a disassembler during the CTF:

  • We expanded each 9-bit byte to a 16-bit short. Originally, I wrote some fancy bit masking and shifting to accomplish this, but then Ryan dropped a very simple script that did the same thing using the bitstream module. This had the side effect of doubling all memory offsets but that was trivial to correct.
  • We made liberal use of slicing in Python. Our disassembler first converted the bytes to a string of bits, then rearranged them to match the representation in the reference document. After that, we took the path of speed of implementation rather than brevity to compare the exact number of bits per opcode to identify and parse them.
  • We made instructions more verbose. The Load and Store instructions iterated over a specified number of registers from a starting point, copying each from or into a memory location. Rather than displaying the starting register and count alone, we expanded the entire list, making it much easier to understand the effects of the instruction in the disassembly at a glance.

With an implemented processor module, we could view and interact with the challenges, define functions with automated analyses, and control how assembly instructions were represented.

We also tried to write an LLIL lifter. This was not possible. You could either have consistent register math or consistent memory addresses, but not both. The weird three-byte register widths and the doubled memory addresses were incompatible. All was not lost, since enough instructions were liftable to locate strings with the dataflow analysis.

Binary Ninja’s graph view allowed us to rapidly analyze control flow structures

If you’d like to get started with our Binja module, you can find our Architecture and BinaryView plugins, as well as a script to pack and unpack the challenges, on our Github.

LegitBS has open-sourced their cLEMENCy tools. The challenges will be available shortly. We look forward to seeing how other teams dealt with cLEMENCy!

UPDATE: The challenges are now available. PPPChris Eagle, and Lab RATS released their processor modules for cLEMENCy.