Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

10. Building a Program

Everything so far has been the language. This chapter is the rest of the job: how a project is laid out, how code is split into modules, how other people’s code gets in, how tests are written, and how the whole thing is documented. It ends with a complete program that uses all of it.

10.1 Projects

cargo new name makes a program and cargo new --lib name makes a library. The difference is one file: src/main.rs with a main function, or src/lib.rs without one. A project can have both, the usual arrangement for anything that deserves tests: the work lives in the library and main.rs is a thin wrapper around it.

Cargo.toml is the manifest. It names the package, sets the edition, and lists dependencies. The edition is Rust’s way of making breaking changes without splitting the language: editions interoperate, and a crate says which one it is written in.

Additional programs go in src/bin/, one file each. That is how the examples for this book are arranged. cargo run --bin name picks one.

Cargo builds into target/, which you do not commit. Cargo.lock records the version of every dependency actually used; commit it for a program, and for a library do whatever the project’s convention is.

10.2 Modules

A module is a namespace with a privacy boundary. Declaring one in a file:

mod geometry {
    pub struct Point {
        pub x: f64,
        pub y: f64,
    }

    pub fn distance(a: &Point, b: &Point) -> f64 {
        ((a.x - b.x).powi(2) + (a.y - b.y).powi(2)).sqrt()
    }
}

use geometry::{distance, Point};

fn main() {
    let a = Point { x: 0.0, y: 0.0 };
    let b = Point { x: 3.0, y: 4.0 };
    println!("{}", distance(&a, &b));
}

Everything is private to its module unless marked pub, and privacy is per item: a public struct may have private fields, which is how you keep an invariant. A module can see its parent’s private items, but not a sibling’s.

Modules usually live in their own files. mod counter; in lib.rs tells the compiler to look for src/counter.rs, or src/counter/mod.rs when the module has children of its own. There is no include and no header; the mod line is the declaration.

Paths start from the crate root with crate::, from the current module with self::, or from the parent with super::. A use line shortens a path and nothing more, and pub use re-exports, which lets a library present a tidy surface regardless of how its files are arranged.

10.3 Dependencies

cargo add rand adds a line to Cargo.toml and fetches the crate from crates.io on the next build. Versions follow semantic versioning, and "0.8" means any compatible 0.8 release. cargo update moves within those bounds; changing the major version is an edit you make deliberately.

A few crates are near-universal and worth knowing by name: serde for turning data into JSON and back, clap for command-line parsing, rand for random numbers, regex, chrono for dates, reqwest for HTTP, tokio for async, anyhow and thiserror for the error handling of Chapter 8. The standard library is deliberately small and does not include any of these.

Be a little careful. Every dependency is code you are shipping and a supply-chain risk you are taking, and the ease of cargo add makes it easy to end up with three hundred of them. cargo tree shows what you have actually pulled in.

10.4 Tests

A test is a function marked #[test] that panics when something is wrong. assert!, assert_eq!, and assert_ne! do the panicking, and assert_eq! prints both values when it fails.

Unit tests live in the file they test, in a module marked #[cfg(test)] so they are compiled only for cargo test:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn counts_are_case_insensitive() {
        let counts = count("The the THE cat");
        assert_eq!(counts["the"], 3);
        assert_eq!(counts["cat"], 1);
        assert_eq!(counts.len(), 2);
    }
}

use super::* brings in the module being tested, including its private items, and reaching them is the point of putting tests inside the file.

Integration tests live in tests/, one file per test binary, and can only use the public interface, which makes them a check on that interface as much as on the behavior.

cargo test runs everything, in parallel, capturing output from passing tests. cargo test name runs the tests whose names contain that string, and cargo test -- --nocapture lets the passing ones print.

A test that should panic is marked #[should_panic], and a test may return Result so ? can be used inside it.

10.5 Documentation

/// before an item documents it, in Markdown, and //! at the top of a file documents the file’s module. cargo doc --open builds the HTML and shows it, including the documentation for every dependency at the version you are actually using. It is the fastest way to answer a question about a library.

Code blocks in documentation are compiled and run by cargo test:

/// The `n` most frequent words, most frequent first, ties broken alphabetically.
///
/// ```
/// let top = wf::counter::top(&wf::counter::count("a b a c a b"), 2);
/// assert_eq!(top[0], ("a".to_string(), 3));
/// assert_eq!(top[1], ("b".to_string(), 2));
/// ```

Documentation that is wrong fails the build. This is the best feature in the toolchain and it is available in no other language’s standard workflow.

10.6 The Program

wf reads text from files or standard input and prints the most frequent words. It is small, but it has the shape of a real program: a library with tests, a binary that parses arguments and reports errors, and documentation that is checked.

src/counter.rs, the work:

//! Counting words in text.

use std::collections::HashMap;

/// A word is a run of letters, digits, and apostrophes, folded to lower case.
pub fn words(text: &str) -> impl Iterator<Item = String> + '_ {
    text.split(|c: char| !(c.is_alphanumeric() || c == '\''))
        .filter(|w| !w.is_empty())
        .map(|w| w.to_lowercase())
}

/// Count how often each word appears.
pub fn count(text: &str) -> HashMap<String, usize> {
    let mut counts = HashMap::new();
    for word in words(text) {
        *counts.entry(word).or_insert(0) += 1;
    }
    counts
}

/// The `n` most frequent words, most frequent first, ties broken alphabetically.
pub fn top(counts: &HashMap<String, usize>, n: usize) -> Vec<(String, usize)> {
    let mut pairs: Vec<(String, usize)> =
        counts.iter().map(|(w, c)| (w.clone(), *c)).collect();
    pairs.sort_by(|a, b| b.1.cmp(&a.1).then(a.0.cmp(&b.0)));
    pairs.truncate(n);
    pairs
}

The return type of words says a good deal. impl Iterator<Item = String> + '_ means “some iterator of strings, borrowing from the argument”. The '_ is the lifetime of text, and it is required because the iterator holds a reference into the string it is splitting. Nothing is collected, so a large file is processed as it is walked.

src/main.rs, the program:

//! wf - print the most frequent words in a file or on standard input.
//!
//!     wf [-n COUNT] [FILE ...]

use std::error::Error;
use std::io::{self, Read};
use std::process;

use wf::counter;

struct Options {
    count: usize,
    files: Vec<String>,
}

fn parse_args(args: Vec<String>) -> Result<Options, String> {
    let mut options = Options { count: 10, files: Vec::new() };
    let mut rest = args.into_iter();

    while let Some(arg) = rest.next() {
        match arg.as_str() {
            "-n" => {
                let value = rest.next().ok_or("-n needs a number")?;
                options.count = value.parse().map_err(|_| format!("{value:?} is not a number"))?;
            }
            "-h" | "--help" => return Err("usage: wf [-n COUNT] [FILE ...]".to_string()),
            _ => options.files.push(arg),
        }
    }

    Ok(options)
}

fn read_input(files: &[String]) -> Result<String, Box<dyn Error>> {
    if files.is_empty() {
        let mut text = String::new();
        io::stdin().read_to_string(&mut text)?;
        return Ok(text);
    }

    let mut text = String::new();
    for path in files {
        text.push_str(&std::fs::read_to_string(path).map_err(|e| format!("{path}: {e}"))?);
        text.push('\n');
    }
    Ok(text)
}

fn run() -> Result<(), Box<dyn Error>> {
    let options = parse_args(std::env::args().skip(1).collect())?;
    let text = read_input(&options.files)?;
    let counts = counter::count(&text);

    let width = counter::top(&counts, options.count)
        .iter()
        .map(|(_, n)| n.to_string().len())
        .max()
        .unwrap_or(1);

    for (word, n) in counter::top(&counts, options.count) {
        println!("{n:width$} {word}");
    }
    Ok(())
}

fn main() {
    if let Err(e) = run() {
        eprintln!("wf: {e}");
        process::exit(1);
    }
}
$ cargo run -q -p wf --bin wf -- -n 5 gettysburg.txt
3 and
3 nation
2 a
2 are
2 conceived

$ printf 'a b a c a b\n' | cargo run -q -p wf --bin wf -- -n 3
3 a
2 b
1 c

$ cargo run -q -p wf --bin wf -- nosuch.txt
wf: nosuch.txt: No such file or directory (os error 2)
$ echo $?
1

Three things in that program are worth pointing at.

main does nothing but call run and report. All the work is in a function that returns Result, so ? is available throughout and there is one place where errors turn into messages and an exit status. This is the standard shape for a Rust program.

Arguments are read from an iterator, and -n takes the next item from that same iterator inside the loop. std::env::args() yields the program’s own name first, which .skip(1) discards. Real programs graduate to clap at about the third option; doing it by hand once is worth the half hour.

{n:width$} takes the field width from a variable, so the counts line up whatever their magnitude. The format language has a good deal in it, and the documentation for std::fmt is where it lives.

cargo test on the finished crate runs the unit tests, the integration test, and the documentation example:

running 5 tests
test counter::tests::empty_input_has_no_words ... ok
test counter::tests::keeps_apostrophes ... ok
test counter::tests::counts_are_case_insensitive ... ok
test counter::tests::splits_on_punctuation ... ok
test counter::tests::ties_break_alphabetically ... ok

running 1 test
test counts_a_paragraph ... ok

   Doc-tests wf
running 1 test
test src/counter.rs - counter::top (line 23) ... ok

10.7 The Rest of the Toolchain

cargo fmt formats the code to the community standard. There is no style debate in Rust; run it and stop thinking about it.

cargo clippy is a lint that knows the idioms. It will tell you that a loop should be an iterator chain, that a match should be an if let, that a &String parameter should be &str. Run it before you ask anyone to read your code; early on, run it constantly, because it teaches.

cargo build --release turns on optimization. The difference against a debug build is often a factor of ten or more, so never benchmark a debug build and never ship one.

10.8 Where to Go Next

This book has covered the language a program is written in. Three areas remain, each with a reason to wait until now.

Threads are std::thread::spawn, and they are unusually approachable in Rust, because the ownership rules you already know are the rules that prevent data races. A value sent to another thread must be Send, a value shared between threads must be Sync, and the compiler checks both. Channels are in std::sync::mpsc, shared state is Arc<Mutex<T>>, and the guarantee is that the mistakes which make threaded C++ unmaintainable do not compile.

Async is a different model for a different problem: tens of thousands of tasks waiting on a network, rather than a few doing computation. async fn and .await are language features, but the runtime that drives them is a library, usually tokio. Learn threads first, and reach for async when you have a concrete reason.

unsafe marks code where you take responsibility for what the compiler would otherwise guarantee: dereferencing a raw pointer, calling into C, touching mutable global state. It does not turn off the type system, and it is not an escape hatch for arguments you lost with the borrow checker. It exists so the safe abstractions above it can be built, and most programmers never need it.

Macros you have been calling since the first page. Writing them, with macro_rules! or as procedural macros, is a real subject and can wait until you have wanted one.

For reference, the standard library documentation is the best-organized part of Rust and is available offline with rustup doc. The Rust Programming Language covers the language more slowly than this book; Programming Rust covers it in more depth; Rust for Rustaceans is what to read when the code you write starts being read by other people.

Exercise 10-1. Add a -u option to wf that counts distinct words rather than printing frequencies, and write the tests first.

Exercise 10-2. wf reads every file into one string before counting. Change it to count each file as it reads, and report per-file totals as well as the combined one.

Exercise 10-3. Add a --min LENGTH option that ignores short words. Decide what --min 0 and a negative value should do, and make the argument parser say so.

Exercise 10-4. Move parse_args into the library, give it tests, and note what had to change to make it testable.

Exercise 10-5. Write the same program in the scripting language you came from. Compare the two on length, on speed against a large file, and on what happens when the file does not exist.