Introduction to fuzzing

In this article, we will dive into the fascinating realm of fuzzing. In order to understand it and before setting up our tests in Foundry we will learn what is the core concept of fuzzing.

What is fuzzing?

Fuzzing, or fuzz testing, is a technique where invalid, unexpected, or random data is passed into a system to discover coding errors and security loopholes.

When applied to web applications, fuzzing targets common web vulnerabilities by determining data injection points to identify which entry points might be susceptible to such vulnerabilities.

What’s the idea of fuzzing?

Let’s give an example so we can clearly understand the idea of fuzzing.

Let’s say you have a program that asks for your age, and you’re supposed to type in a number. Fuzzing would be like having a disobedient robot that instead of typing in a number like “30”, types in all sorts of weird stuff like “banana”, a really long number like “9999999999”, or even nothing at all to see how the program reacts.

If the program gets confused and stops working when it gets “banana”, then the fuzzing has found a flaw.

And that idea applied to Web3?

In Web3, which uses blockchain and smart contracts, fuzzing is like that robot testing a digital vending machine (the smart contract) that’s supposed to only dispense a digital soda (like a cryptocurrency transaction) when it receives a digital coin. 

The fuzzing might try all sorts of wrong inputs to see if it can get the vending machine to give out a soda when it shouldn’t. 

If it ever does, a flaw must be fixed to ensure that the digital vending machine dispenses sodas when the correct digital coin is provided.

Difference between Web2 and Web3 fuzzing

Fuzzing was introduced in the era of Web2 as a way to test traditional software applications, including web applications, for vulnerabilities. 

In Web2, systems are typically centralized, meaning they are hosted and operated from central servers owned by companies (AFL, LibFuzzer, go-fuzz). Fuzzing in this context involves automatically sending lots of unexpected and malformed inputs to these web applications to find where they might break or act insecurely.

And the main goal, to be more specific, is to crash the program.

Key differences in Web3 

In the context of Web3 and smart contracts, fuzzing often takes on a property-based approach. This approach consists of rigorously testing these contracts' logic and rules (properties) against a wide range of inputs and scenarios. 

This method ensures that the contract will behave correctly and securely, even in edge cases, which is vital given the irreversible nature of blockchain transactions.

These properties are also known as invariants, and their main purpose is to define how the system is expected to work. On fuzz testing, the user is responsible for identifying and adding them to the tests.

A couple of examples of invariant/property to make it easier to understand would be:

• No one should be able to withdraw more of the protocol than they deposit

• The total supply of an ERC20 token does not change unless mint or burn are called

Benefits

Fuzz testing is particularly beneficial for smart contracts due to its ability to uncover hard-to-detect vulnerabilities, test under extreme conditions, and ensure the security and reliability of these contracts in handling complex, high-value transactions in the blockchain environment.

Example

Concepts are always better understood by going through examples. So, let’s see a clear example that will help us understand how important and beneficial it is to make use of fuzz tests.

Imagine you’re developing a smart contract for a DeFi platform that allows users to deposit cryptocurrency and earn interest over time. The smart contract is responsible for handling user deposits, calculating interest, and enabling withdrawals.

Potential Risks Without Fuzzing:

• Security Vulnerabilities: Without comprehensive testing, the contract might have vulnerabilities like reentrancy bugs.

• Logic Errors: There could be errors in interest calculation, leading to either excessive interest payouts or not paying enough.

• Handling Edge Cases: The contract might not handle edge cases well, such as extremely large deposits or withdrawals, which could lead to unexpected behavior or system crashes.

Benefits of Applying Fuzzing:

• Discovering Hidden Flaws: Fuzzing would aggressively test the smart contract with a wide range of inputs, including unexpected and malformed data, to uncover hidden flaws like the reentrancy vulnerability.

• Ensuring Accurate Logic: It helps ensure that the logic for interest calculations and fund withdrawals works correctly under all conditions, including edge cases.

---

Setup foundry fuzz tests

It’s time to learn and practice Foundry!

I propose to follow this process to get us up and running:

• Installation

• First steps with Foundry

• Fuzz Testing

• Invariant Testing

Installation

The first command to run on your terminal is:

curl -L https://foundry.paradigm.xyz | bash

Then, if you notice, the terminal will also point out to run:

foundryup

Now, you have to install cargo to have available the Rust compiler:

curl https://sh.rustup.rs -sSf | sh

And the last step is going to be installing Forge + Cast, Anvil, and Chisel:

cargo install --git https://github.com/foundry-rs/foundry --profile local --force foundry-cli anvil chisel

First steps with Foundry

Create a new project

Let’s create a new project with Foundry and play with it.

Run:

forge init safe-example

Now, get inside the newly created project cd safe-example and compile it:

forge build

How do you execute Foundry tests?

In order to run all test cases from the repository run:

forge test

However, let’s say you’re working on test cases for a specific contract (in this example let’s consider we’re working on Safe.t.sol test file) and you only want those to be executed. So what you can do here is:

forge test --match-path test/Safe.t.sol

Another great feature that Foundry offers while executing the tests is to introduce more logs on the results as per demand.

That is controlled by adding a flag with two to five -v‘s such as:

forge test --match-path test/Safe.t.sol -vvv

The more v’s you add the more verbosity you will get in the test results.

Verbosity levels:
- 2: Print logs for all tests
- 3: Print execution traces for failing tests
- 4: Print execution traces for all tests, and setup traces for failing tests
- 5: Print execution and setup traces for all tests

There’s a bonus feature that we get to have while executing tests with Foundry…

And it is the option to add a flag to get the gas report from the contract’s functions:

forge test --match-path test/Counter.t.sol --gas-report

The report will look like this:

Fuzz Testing

I’m going to use the fantastic example from Foundry’s documentation to go through this.

Here’s a simple contract:

contract Safe {
    receive() external payable {}

    function withdraw() external {
        payable(msg.sender).transfer(address(this).balance);
    }
}

Then, we want to write one unit test to make sure that the withdraw function works:

import "forge-std/Test.sol";

contract SafeTest is Test {
    Safe safe;

    // Needed so the test contract itself can receive ether
    // when withdrawing
    receive() external payable {}

    function setUp() public {
        safe = new Safe();
    }

    function test_Withdraw() public {
        payable(address(safe)).transfer(1 ether);
        uint256 preBalance = address(this).balance;
        safe.withdraw();
        uint256 postBalance = address(this).balance;
        assertEq(preBalance + 1 ether, postBalance);
    }
}

This test is simply checking that the balance from before withdrawing + the amount transferred is the same as the balance after withdrawing.

Now, who is to say that it works for all amounts, not just 1 ether?

Here’s when Stateless Fuzzing comes in

Forge will run any test that takes at least one parameter as a property-based test, so let’s rewrite:

contract SafeTest is Test {
    // ...

    function testFuzz_Withdraw(uint256 amount) public {
        payable(address(safe)).transfer(amount);
        uint256 preBalance = address(this).balance;
        safe.withdraw();
        uint256 postBalance = address(this).balance;
        assertEq(preBalance + amount, postBalance);
    }
}

By running this, we can see that Forge runs the property-based test, but it fails for high values of amount:

$ forge test
Compiling 1 files with 0.8.10
Solc 0.8.10 finished in 1.69s
Compiler run successful

Running 1 test for test/Safe.t.sol:SafeTest
[FAIL. Reason: EvmError: Revert Counterexample: calldata=0x215a2f200000000000000000000000000000000000000001000000000000000000000000, args=[79228162514264337593543950336]] testWithdraw(uint256) (runs: 47, μ: 19554, ~: 19554)
Test result: FAILED. 0 passed; 1 failed; finished in 8.75ms

Here we have the first example of fuzzing, instead of testing one scenario, by adding as a parameter the value that we need to test, it is going to use a vast amount of semi-random values.

And that might lead to finding a contract’s vulnerability.

Forge is printing as well some useful information when the test fails.

testWithdraw(uint256) (runs: 47, μ: 19554, ~: 19554)

• “runs” refers to the number of scenarios the fuzzer tested. By default, the fuzzer will generate 256 scenarios.

• “μ” (Greek letter mu) is the mean gas used across all fuzz runs

• “~” (tilde) is the median gas used across all fuzz runs

You might find this pretty useful about Foundry already, and I can agree with that. There’s, however, much more potential of Foundry to be explored.

Invariant Testing

An invariant test in Foundry is a stateful fuzz test, and this is another way you’re going to be hearing/reading from people who refer to the same type of test.

The key here is that in the same way, as mentioned above, tests in Foundry have to start with the prefix test_ and testFail_, in order to write this type of test you will have to use the prefix invariant.

And now is the time to see Stateful Fuzzing in action

Let’s see this in action by modifying a bit the contract we have used before:

contract Safe {
    address public seller = msg.sender;
    mapping(address => uint256) public balance;

    function deposit() external payable {
        balance[msg.sender] += msg.value;
    }

    function withdraw() external {
        uint256 amount = balance[msg.sender];
        balance[msg.sender] = 0;
        (bool s, ) = msg.sender.call{value: amount}("");
        require(s, "failed to send");
    }

    function sendSomeEther(address to, uint amount) public {
        (bool s, ) = to.call{value: amount}("");
        require(s, "failed to send");
    }
}

Here we have modified the withdraw function and added the chance to depositether.

The important point we want to test here is that the amount withdrawn must always be the same as the amount deposited.

contract InvariantSafeTest is Test {
    Safe safe;

    function setUp() external {
        safe = new Safe();
        vm.deal(address(safe), 100 ether); // Sets an address' balance, (who, newBalance)
    }

     function invariant_withdrawDepositedBalance() external payable {
        safe.deposit{value: 1 ether}();
        uint256 balanceBefore = safe.balance(address(this));
        assertEq(balanceBefore, 1 ether);
        safe.withdraw();
        uint256 balanceAfter = safe.balance(address(this));
        assertGt(balanceBefore, balanceAfter);
    }

    receive() external payable {}
}

If I first run this test I get the following output (you’re going to like this): 

% forge test
[⠢] Compiling...
[⠒] Compiling 2 files with 0.8.19
[⠆] Solc 0.8.19 finished in 819.69ms
Compiler run successful!

Running 1 test for test/Counter.t.sol:InvariantSafeTest
[PASS] invariant_withdrawDepositedBalance() (runs: 256, calls: 3840, reverts: 883)
Test result: ok. 1 passed; 0 failed; finished in 199.57ms

Yes, it passes.

But REMEMBER, always read the output. Because in this case, you can see that by running invariant_withdrawDepositedBalance()test, it gets this result:

(runs: 256, calls: 3840, reverts: 883)

And, that’s correct, it is in fact reverting a total of 883 times.

Why didn’t it fail?

Don’t worry, it is an expected result. In order to revert, we have to add the following to the foundry.toml file.

[invariant]
fail_on_revert = true

and if you run it again now, the result is different:

Running 1 test for test/InvariantSafeTest.t.sol:InvariantSafeTest
[FAIL. Reason: failed to send]
        [Sequence]
                sender=0x0000000000000000000000000000000000000003 addr=[src/Counter.sol:Counter]0x5615deb798bb3e4dfa0139dfa1b3d433cc23b72f 
                  calldata=withdraw(), args=[]
                sender=0x000000000000000000000000000000000000005b addr=[src/Counter.sol:Counter]0x5615deb798bb3e4dfa0139dfa1b3d433cc23b72f 
                  calldata=sendSomeEther(address,uint256), 
                  args=[0xa18255fD90ed742e6439A9b834A179E386DC0c1e, 115792089237316195423570985008687907853269984665640564039457584007913129639935]

 invariant_withdrawDepositedBalance() (runs: 1, calls: 2, reverts: 1)
Test result: FAILED. 0 passed; 1 failed; finished in 9.57ms

There’s a super interesting thing happening here. Please notice the part with

calldata=sendSomeEther(address,uint256)

That is how the output tells you the last function triggered that broke this contract’s invariant.

By adding a function to the contract that can be used to send ether to a different account, the invariant of withdrawing always the same amount of deposited ether is broken.

If I comment out the function sendSomeEther from the contract, then the results would be the following:

forge test
[⠆] Compiling...
[⠊] Compiling 2 files with 0.8.19
[⠢] Solc 0.8.19 finished in 724.85ms
Compiler run successful!

Running 1 test for test/InvariantSafeTest.t.sol:InvariantSafeTest
[PASS] invariant_withdrawDepositedBalance() (runs: 256, calls: 3840, reverts: 0)
Test result: ok. 1 passed; 0 failed; finished in 218.12ms

---

So, there you have it — fuzz testing is our secret weapon to keep smart contracts safe. Let’s not postpone it and use this introduction to start applying this knowledge in your projects or the projects you are auditing. Web3 world will increase its security with more fuzzing.