When try, try, try again leads to out-of-order execution bugs

By Troy Sargent

Have you ever wondered how a rollup and its base chain—the chain that the rollup commits state checkpoints to—communicate and interact? How can a user with funds only on the base chain interact with contracts on the rollup?

In Arbitrum Nitro, one way to call a method on a contract deployed on the rollup from the base chain is by using retryable transactions (a.k.a. retryable tickets). While this feature enables these interactions, it does not come without its pitfalls. During our reviews of Arbitrum and contracts integrating with it, we identified footguns in the use of retryable tickets that are not widely known and should be considered when creating such transactions. In this post, we’ll share how using retryable tickets may allow unexpected race conditions and result in out-of-order execution bugs. What’s more, we’ve created a new Slither detector for this issue. Now you’ll be able to not only recognize these footguns in your code, but test for them too.

Retryable tickets

In Arbitrum Nitro, retryable tickets facilitate communication between the Ethereum mainnet, or Layer 1 (L1), and the Arbitrum Nitro rollup, or Layer 2 (L2). To create retryable tickets, users can call createRetryableTicket on the L1 Inbox contract of the Arbitrum rollup, as shown in the code snippet below. When retryable tickets are created and queued, ArbOS will attempt to automatically “redeem” them by executing them one after another on L2.

/**
 * @notice Put a message in the L2 inbox that can be reexecuted for some fixed amount of time if it reverts
 * @dev all msg.value will deposited to callValueRefundAddress on L2
 * @dev Gas limit and maxFeePerGas should not be set to 1 as that is used to trigger the RetryableData error
 * @param to destination L2 contract address
 * @param l2CallValue call value for retryable L2 message
 * @param maxSubmissionCost Max gas deducted from user's L2 balance to cover base submission fee
 * @param excessFeeRefundAddress gasLimit x maxFeePerGas - execution cost gets credited here on L2 balance
 * @param callValueRefundAddress l2Callvalue gets credited here on L2 if retryable txn times out or gets cancelled
 * @param gasLimit Max gas deducted from user's L2 balance to cover L2 execution. Should not be set to 1 (magic value used to trigger the RetryableData error)
 * @param maxFeePerGas price bid for L2 execution. Should not be set to 1 (magic value used to trigger the RetryableData error)
 * @param data ABI encoded data of L2 message
 * @return unique message number of the retryable transaction
 */
function createRetryableTicket(
    address to,
    uint256 l2CallValue,
    uint256 maxSubmissionCost,
    address excessFeeRefundAddress,
    address callValueRefundAddress,
    uint256 gasLimit,
    uint256 maxFeePerGas,
    bytes calldata data
) external payable returns (uint256);

The createRetryableTicket function interface

Assuming the gas costs are covered by the sender and no failures occur, the transactions will be executed sequentially, and the final state results from applying transaction B immediately following transaction A.

Figure 1: The happy path is when the transactions are all executed in order.

Wait, what does “retryable” mean?

Because any transaction may fail (e.g., the L2 gas price rises significantly following the creation of a transaction, and the user has insufficient gas to cover the new cost), Arbitrum created these types of transactions so that users can “retry” them by supplying additional gas. Failing retryable tickets will be persisted in memory and may be re-executed by any user who manually calls the redeem method of the ArbRetryableTx precompiled contract, sponsoring the gas costs. A retryable ticket that fails is different from a normal transaction that reverts, in that it does not require a new transaction to be signed to be executed again.

Additionally, retryable tickets in memory can be redeemed up to one week after they are created. A retryable ticket’s lifetime can be extended for another week by paying an additional fee for storing it; otherwise, it will be discarded after its expiration date.

Where things go wrong

While these types of transactions are useful—in that they facilitate L2-to-L1 communication and allow users to retry their transactions if failures occur—they come with pitfalls, risks that users and developers may not be aware of. Specifically, retryable tickets are expected to execute in the order they are submitted, but this is not always guaranteed to happen.

In scenario 1, both transactions A and B fail and enter the memory region. The state of the application is left unchanged.

Consider the three scenarios below in which two retryable tickets are created within the same transaction.

Figure 2: Two retryable tickets are created in the same transaction, but both fail and enter the memory region.

However, anyone can manually redeem transaction B before transaction A, which means that the transactions will be executed out of order unexpectedly.

Figure 3: Anyone can manually redeem transactions in the memory region out of order.

In scenario 2, transaction A fails and enters the memory region, but transaction B succeeds. Once again, the transactions are executed out of order (i.e., transaction A is not executed at all), and the final state is not what was expected.

Figure 4: Only transaction B is included in the final state.

In scenario 3, transaction A succeeds, but transaction B does not. That means transaction B must be re-executed manually. Transactions can be created more than once, which means that a second set of transactions A and B could be submitted before the first transaction B is re-executed. If developers of a protocol using the Arbitrum rollup system don’t account for the possibility that the protocol could receive a second transaction A prior to transaction B’s success, the protocol may not handle this case correctly.

Figure 5: Only transaction A is included in the final state.

The out-of-order execution vulnerability

In light of these scenarios, developers should consider that transactions may execute out of order. For instance, if the second transaction in a queue relies on completion of the first, but it executes before the first executes due to an insufficient gas failure, it may revert or not work correctly. It’s important that the callee, or message recipient, on the rollup can robustly handle situations such as the receipt of transactions in a different order than they were created and smaller subsets of transactions due to failures. If a protocol does not anticipate cases of reorderings and failures of retryable tickets, the protocol could break or be hacked.

Let’s consider the following L2 contract, which users can call to claim rewards based on some staked tokens. When they decide to unstake their tokens, any rewards that they haven’t yet claimed are lost:

function claim_rewards(address user) public onlyFromL1 {
    // rewards is computed based on balance and staking period
    uint unclaimed_rewards = _compute_and_update_rewards(user);
    token.safeTransfer(user, unclaimed_rewards);
}


// Call claim_rewards before unstaking, otherwise you lose your rewards
function unstake(address user) public onlyFromL1 {
    _free_rewards(user); // clean up rewards related variables
    balance = balance[user];
    balance[user] = 0;
    staked_token.safeTransfer(user, balance);
}

Users can submit retryable tickets for such operations with the following logic in the L1 handler:

// Retryable A
IInbox(inbox).createRetryableTicket({
    to: l2contract,
    l2CallValue: 0,
    maxSubmissionCost: maxSubmissionCost,
    excessFeeRefundAddress: msg.sender,
    callValueRefundAddress: msg.sender,
    gasLimit: gasLimit,
    maxFeePerGas: maxFeePerGas,
    data: abi.encodeCall(l2contract.claim_rewards, (msg.sender))
});
// Retryable B
IInbox(inbox).createRetryableTicket({
    to: l2contract,
    l2CallValue: 0,
    maxSubmissionCost: maxSubmissionCost,
    excessFeeRefundAddress: msg.sender,
    callValueRefundAddress: msg.sender,
    gasLimit: gasLimit,
    maxFeePerGas: maxFeePerGas,
    data: abi.encodeCall(l2contract.unstake, (msg.sender))
});

Here it is expected that claim_rewards will be called before unstake. However, as we’ve seen, the claim_rewards transaction is not guaranteed to execute before the unstake transaction. As covered in scenario 1 and shown in figure 3, an attacker can make it so that unstake is executed before claim_rewards if both transactions fail, causing the user to lose their rewards. It’s also possible that only the second transaction, unstake, succeeds, as shown in scenario 2.

To mitigate such risks, it’s essential to design protocols in a way that retryable tickets have an independent ordering, where the success of each transaction does not depend on the order or outcome of others. How independent ordering is implemented depends on the protocol and the given operations. In this example, claim_rewards could be called within unstake.

Slither to the rescue

As security researchers, we always try to find ways to automatically find these sorts of issues and flag them early in the development cycle, such as during code review. To that end, we’ve written a Slither detector that will flag functions that create multiple retryable tickets via the Arbitrum Nitro Inbox contract to alert developers of this pitfall. Following its release, you can use this detector by installing Slither and running the following command in the root of a Solidity project: python3 -m pip install slither-analzyer==0.10.1 && slither . –detect out-of-order-retryable. On our example contract, Slither provides the following diagnostic:

Multiple retryable tickets created in the same function:
         -IInbox(inbox).createRetryableTicket({to:address(l2contract),l2CallValue:0,maxSubmissionCost:maxSubmissionCost,excessFeeRefundAddress:msg.sender,callValueRefundAddress:msg.sender,gasLimit:gasLimit,maxFeePerGas:maxFeePerGas,data:abi.encodeCall(l2contract.claim_rewards,(msg.sender))}) (out_of_order_retryable.sol#25-34)
         -IInbox(inbox).createRetryableTicket({to:address(l2contract),l2CallValue:0,maxSubmissionCost:maxSubmissionCost,excessFeeRefundAddress:msg.sender,callValueRefundAddress:msg.sender,gasLimit:gasLimit,maxFeePerGas:maxFeePerGas,data:abi.encodeCall(l2contract.unstake,(msg.sender))}) (out_of_order_retryable.sol#36-45)
Reference: https://github.com/crytic/slither/wiki/Detector-Documentation#out-of-order-retryable-transactions
INFO:Slither:out_of_order_retryable.sol analyzed (3 contracts with 1 detectors), 1 result(s) found

Conclusion

If you are developing a protocol that uses retryable tickets, ensure that your protocol is equipped to handle the scenarios we’ve outlined here. Specifically, the use of retryable tickets shouldn’t rely on their order or on successful execution. You can spot potential out-of-order execution bugs using our new Slither detector!

If your application interacts with Arbitrum Nitro components or you’re building software that features rollup–base chain communication, contact us to see how we help.

Leave a Reply