building raft in rust

I want to be able to open the election code and follow an election.

That's the starting point for Rafter, a Raft implementation in Rust. I want each part to have a clear job and its decisions to be easy to follow.

I'm using AI to help build it, which makes that goal more important to me. I still need to explain why a change works, what it assumes, and what would break it.

agreeing on what happened

Suppose three machines keep a copy of the same data. A client asks to change a value. The machines need to agree on which changes happened and in what order, even if one of them crashes or loses its connection.

Raft gives them a shared, ordered log. Each entry describes a change the application will apply. One machine acts as leader and coordinates copying entries to the others, called followers.

Raft numbers election periods as terms. When the leader creates a new entry in its current term, it can commit it once a majority has stored it durably: here, the leader and one follower.

Committing means the group has agreed to keep the entry. The replicas apply committed entries in order, using the same rules to update their local data.

Two connected members can keep making progress if the third goes away. A member on its own cannot commit new writes. If the leader disappears, the others can elect a replacement with a sufficiently up-to-date log.

The interesting part is what happens between those sentences: an old leader sends a late message, a follower restarts with an incomplete log, or a vote reaches the network just before a crash.

a file should answer a question

Raft separates elections, replication, and safety to make the algorithm easier to understand. I want that separation to survive in the implementation.

A smaller module helps only when it reduces what I have to understand to make a change.

Here's a selected part of Rafter's source tree:

node/
  election.rs
  lifecycle.rs
  replication/
    receive.rs
    response.rs
    send.rs
  commit/
    tracker.rs
    apply.rs

Sending log entries, receiving them, and handling an acknowledgment each involve a different decision. They get separate files. To see how a follower checks incoming entries, I start at replication/receive.rs.

I'm careful about extracting helpers here. A shorter function can be harder to follow if every meaningful step disappears behind a generic callback. A helper should name a decision well enough that the caller becomes easier to read.

State is grouped by lifetime. Persistent state holds things that must survive a restart, such as the voting record. Leader state holds things that only matter while leading. When a node steps down, that leader state is reset together.

More files mean more navigation. The reading guide maps the paths, and each production module describes what it owns. Automated checks restrict where I/O and state changes belong and flag modules that outgrow their size budget.

When I review a change, I still need to follow the affected path, see which state it touches, and find the relevant tests. A file can be short and still hide an assumption several modules away.

making the core easy to drive

The protocol core doesn't open sockets, write files, start tasks, or read a clock. It takes an event, changes its state, and returns an ordered list of things to do.

Here's the core interface, with one node configured as part of a three-member group:

use rafter::{Input, Node, NodeConfig, NodeId};

let config = NodeConfig::new(NodeId(1), vec![NodeId(2), NodeId(3)], 10)
    .expect("valid config");

let mut node = Node::new(config);
let outputs = node.step(Input::Tick);

A tick tells the node that a unit of logical time has passed; 10 sets the election timeout in ticks. Incoming messages and client proposals enter through the same step method. Outputs include messages to send and committed entries for the application to apply.

This is often called a sans-I/O design. It lets me read a transition without also tracing a socket or storage implementation. Given the same starting state and sequence of inputs, the core makes the same decisions.

It also leaves scheduling and transport choices with the application. A service can drive the core from its existing runtime; a simulator can drive it from a list of events.

The snippet only steps the protocol. A durable node must also save the required state before releasing dependent messages. Rafter supplies a runtime layer for that.

a reply is a promise

To become leader, a candidate needs votes from a majority. A node can grant a vote to at most one candidate in a term, and it has to remember that vote after restarting.

A request with a newer term first moves the node into that term. Then this part of the election code decides whether to grant the vote:

let vote_granted = request.term == self.current_term()
    && self.effective_membership().contains_voter(candidate_id)
    && self.candidate_log_is_up_to_date(
        request.last_log_term,
        request.last_log_index
    )
    && self
        .persistent
        .voted_for
        .is_none_or(|voted_for| voted_for == candidate_id);

if vote_granted {
    self.persistent.voted_for = Some(candidate_id);
    self.election.reset_timeout();
}

The decision is visible: the term must match, the candidate must be a voter, its log must be at least as up to date as this node's, and this node must not have voted for someone else in the term.

Updating voted_for changes memory. If the node sends a yes and crashes before saving that vote, it can forget and vote for someone else in the same term.

Rafter's durable runtime keeps that dependency explicit:

The function produces a reply, but the runtime withholds it until the required writes finish. The same ordering applies when a follower acknowledges new log entries. Keeping I/O outside the core means this responsibility has to live somewhere; the runtime handles it in one place.

agreement is only part of a write

A committed Raft entry doesn't define how the application recovers. In the reference ledger, account changes and the applied position—the last log entry processed—are saved in the same transaction. Recovery uses that position to know where to resume.

Suppose a transfer is committed, the application saves the new balances, and the reply never reaches the client. The client times out. That doesn't tell it whether the transfer happened.

Retrying as a new transfer could move the money twice. The ledger identifies each request by its client, session, and sequence number. Each client may have only one mutation outstanding, and only the next sequence may execute.

That constraint keeps retry state small: each session retains its latest completed request and result. An exact retry of that request returns the saved result without moving the money again. The session state and result are saved in the same transaction as the balances and applied position.

This puts a limit on what consensus can do for an application. Raft orders commands; the application still defines what a repeated request means and how its state survives a restart. Those decisions belong in the API and storage contract.

The other reference applications ask different questions. A fenced-lock service exercises authority: who is still allowed to act? A sharded counter runs many independent Raft groups, putting pressure on scheduling and isolation. They use public APIs and are also checked against packaged crates, so they exercise the interfaces an outside application receives.

trying to break it

The difficult bugs involve events arriving in an order I didn't expect. A straightforward test can show that an election succeeds. I also want to know what happens when a message from that election arrives after the next one has started.

The tests ask these questions at three levels: Rust transitions, a separate protocol model, and client-visible operations.

invariants and the implementation

An invariant is a rule that must keep holding as the system changes. For example:

• A node can vote for at most one candidate in a term.
• Two replicas cannot apply different commands at the same log position.
• A reply cannot escape before the write it depends on.

The invariant suite connects named rules to executable checks. For the voting example, runtime tests inject a storage failure and check that the operation fails without releasing its reply. Restart tests check that the stored vote is still honored.

The simulator drives the actual Rust core through delayed, duplicated, and dropped messages, partitions, and restarts. It checks the rules across those schedules. Because time and delivery are inputs, a failure can be replayed as a sequence of events.

The checkers have deliberately invalid cases too. If a checker claims to detect a node voting for two different candidates in the same term, we give it a history with that violation and require it to complain. I want to know that the test can recognize the mistake.

exploring the design with TLA+

TLA+ is a language for describing a system's states and the steps between them. Rafter has a separate model of the protocol. The TLC model checker explores possible sequences of steps and looks for a violation of the stated rules.

It can challenge the design before implementation details enter the picture. If the rules allow two leaders in one term, a counterexample gives a sequence of steps to examine.

The model has explicit bounds, such as the number of nodes, terms, and log entries. Some checks explore every reachable state within their configured bounds. The larger scheduled explorations have not finished, and the reports distinguish their partial coverage from completed checks.

Checking this model does not prove that the Rust implementation matches it, which is why testing the actual code remains necessary.

checking what clients observe

Maelstrom runs node programs as separate processes, routes messages through a simulated network, and records requests and responses. Rafter's test node exposes a key/value service through that harness. The scenarios cover failures such as partitions, leader restarts, and crashes around application persistence.

For this key/value workload, the checker asks whether the recorded operations could have taken effect one at a time, while respecting which finished before others began. This is linearizability. Once a write succeeds, a later read of that key must see it or a newer write.

The results cover the workloads and failures sampled in that run. They help test whether consensus, storage, and application behavior fit together.

measure the path you're optimizing

The same question comes up in benchmarks: what has happened when a write counts as finished?

Rafter's comparison benchmark measures a three-node protocol cluster in memory, without the cost of making a write durable.

The repository also has a durable-runtime benchmark with file-backed storage and group commit, where several entries can share a storage commit. Reducing work in the core and reducing the number of durable writes are different optimizations. A throughput number only helps me choose between them when I know what the benchmark waited for before counting an operation as complete.

room to keep building

Rafter is still pre-1.0, and its APIs and storage contracts can change. There is more work to do before I'd treat those boundaries as settled.

What I want to preserve as it grows is a clear route from a question to the code that answers it. An election leads to a durable vote. A committed entry leads to an application change. A test asks what happens when that path is interrupted. I want to be able to follow those connections and understand why I trusted them.