Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

10. Building a Program

Everything so far has been one file. This chapter is about the rest: how code is split into packages, how packages are grouped into a module, how you test it, and how the go command ties it together. Then it builds a real command-line program that uses all of it.

10.1 Packages

One directory is one package. Every .go file in a directory must declare the same package name, and a package cannot span directories. The package name is conventionally the last element of the directory path, lower case, one word, no underscores.

A package is the unit of visibility. Capitalized names are exported to importers and everything else is private to the package, including across files, so two files in the same package see each other’s unexported names freely. This is why Go packages tend to be a handful of files rather than one file per type.

Import paths are the module path plus the directory. Inside the module goexamples, the package in ch10/wf/count is imported as goexamples/ch10/wf/count and referred to as count:

import "goexamples/ch10/wf/count"

counts, err := count.Words(r)

Name packages for what they provide, and remember the name will be read at every call site. count.Words reads well; utils.CountWords does not, and a package called util, common, helpers, or base is a sign that nobody decided what it was for.

A directory named internal is special to the toolchain: packages under it can only be imported by code rooted at its parent. That is how you publish a module with a private implementation.

10.2 Modules

A module is a tree of packages versioned together, defined by a go.mod at its root:

module goexamples

go 1.23

go mod init <path> writes it. The module path is what importers will type, so for anything you publish it is the repository URL, like github.com/anderix/wf. Dependencies are added by go get, which records them in go.mod, and go mod tidy adds anything imported and missing while removing anything recorded and unused. Run it before you commit.

go.sum records a checksum for every dependency and must be committed. It is what makes a build reproducible and what would notice if a version’s contents ever changed.

Versions are git tags of the form v1.4.2, and Go takes semantic versioning literally: a major version above 1 goes in the import path, so v2 of a module is imported as github.com/anderix/wf/v2. That is blunt, and it means an incompatible change cannot silently break anybody.

10.3 A Program

wf prints the most frequent words in its input. It is two packages: a library that does the work and a main that deals with the command line.

ch10/wf/
    main.go              package main
    count/
        count.go         package count
        count_test.go

The library knows nothing about files, flags, or standard output.

// Package count tallies word frequencies in a stream of text.
package count

import (
	"bufio"
	"cmp"
	"io"
	"slices"
	"strings"
	"unicode"
)

// Pair is one word and how often it occurred.
type Pair struct {
	Word string
	N    int
}

// Words reads r to the end and returns a count of the words in it. A word is a
// run of letters, digits, and apostrophes; everything else separates. Words are
// lowercased so that "The" and "the" count together.
func Words(r io.Reader) (map[string]int, error) {
	counts := map[string]int{}

	in := bufio.NewScanner(r)
	in.Split(bufio.ScanWords)
	for in.Scan() {
		w := strings.ToLower(strings.TrimFunc(in.Text(), isNotWord))
		if w != "" {
			counts[w]++
		}
	}
	return counts, in.Err()
}

func isNotWord(r rune) bool {
	return !unicode.IsLetter(r) && !unicode.IsDigit(r) && r != '\''
}

// Top returns the n most frequent words, most frequent first, with ties broken
// alphabetically so the result is stable. A n of zero or less returns all of
// them.
func Top(counts map[string]int, n int) []Pair {
	pairs := make([]Pair, 0, len(counts))
	for w, c := range counts {
		pairs = append(pairs, Pair{w, c})
	}

	slices.SortFunc(pairs, func(a, b Pair) int {
		if d := cmp.Compare(b.N, a.N); d != 0 {
			return d
		}
		return cmp.Compare(a.Word, b.Word)
	})

	if n > 0 && n < len(pairs) {
		pairs = pairs[:n]
	}
	return pairs
}

Words takes an io.Reader rather than a filename, section 9.1’s advice in practice: the caller can pass a file, a network connection, or a strings.Reader in a test. Top returns a sorted slice rather than printing, so the decision about output belongs to whoever called it. Both are exported and documented; isNotWord is neither.

Sorting deserves a note. slices.SortFunc takes a comparison returning a negative number, zero, or a positive one, and cmp.Compare produces those. Comparing b.N to a.N rather than the other way round is what puts the largest first, and falling through to compare the words makes the order total, so the output does not shuffle between runs on ties.

Now the command:

// Command wf prints the most frequent words in its input.
//
// Usage:
//
//	wf [-n count] [file...]
//
// With no files, wf reads standard input.
package main

import (
	"flag"
	"fmt"
	"io"
	"os"

	"goexamples/ch10/wf/count"
)

func main() {
	if err := run(); err != nil {
		fmt.Fprintln(os.Stderr, "wf:", err)
		os.Exit(1)
	}
}

func run() error {
	n := flag.Int("n", 10, "how many words to print")
	flag.Parse()

	readers, closeAll, err := open(flag.Args())
	if err != nil {
		return err
	}
	defer closeAll()

	counts, err := count.Words(io.MultiReader(readers...))
	if err != nil {
		return err
	}

	for _, p := range count.Top(counts, *n) {
		fmt.Printf("%7d %s\n", p.N, p.Word)
	}
	return nil
}

// open turns the named files into readers, or returns standard input when
// there are none. The returned function closes whatever was opened.
func open(names []string) ([]io.Reader, func(), error) {
	if len(names) == 0 {
		return []io.Reader{os.Stdin}, func() {}, nil
	}

	var readers []io.Reader
	var files []*os.File
	closeAll := func() {
		for _, f := range files {
			f.Close()
		}
	}

	for _, name := range names {
		f, err := os.Open(name)
		if err != nil {
			closeAll()
			return nil, nil, err
		}
		files = append(files, f)
		readers = append(readers, f)
	}
	return readers, closeAll, nil
}

$ go run ./ch10/wf -n 5 gettysburg.txt
      3 and
      3 nation
      2 a
      2 are
      2 conceived

Four things in that file are the standard shape of a Go program.

main does nothing but call run and report. run returns an error, main prints it to standard error with the program’s name and exits non-zero. Keeping main to four lines means everything else is testable and everything else can use defer, which os.Exit would skip.

Flags come from the flag package. flag.Int returns a *int that is filled in by flag.Parse, and the remaining arguments are flag.Args(). You get -n 5, -n=5, and a generated -h for free.

Files are opened through a helper that also returns its own cleanup, so run can defer closeAll() without knowing how many files there were, and a failure partway through closes the ones already opened.

io.MultiReader concatenates readers, so several files become one stream and count.Words never learns there was more than one. Composing small interfaces like this is what the standard library is for.

$ go run ./ch10/wf nope.txt
wf: open nope.txt: no such file or directory
$ echo $?
1

10.4 Testing

Tests live beside the code in files ending _test.go, which the compiler excludes from normal builds. A test is a function named TestXxx taking a *testing.T. There is no assertion library in the standard library and the convention is not to use one: you write the comparison yourself and call t.Errorf with a message saying what you did, what you got, and what you wanted.

package count

import (
	"fmt"
	"strings"
	"testing"
)

func TestWords(t *testing.T) {
	cases := []struct {
		name string
		in   string
		want map[string]int
	}{
		{"empty", "", map[string]int{}},
		{"case folded", "The the THE", map[string]int{"the": 3}},
		{"punctuation stripped", "hi, there. hi!", map[string]int{"hi": 2, "there": 1}},
		{"apostrophe kept", "don't don't", map[string]int{"don't": 2}},
		{"digits are words", "in 1863 in", map[string]int{"in": 2, "1863": 1}},
	}

	for _, c := range cases {
		t.Run(c.name, func(t *testing.T) {
			got, err := Words(strings.NewReader(c.in))
			if err != nil {
				t.Fatalf("Words(%q) returned error: %v", c.in, err)
			}
			if len(got) != len(c.want) {
				t.Fatalf("Words(%q) = %v, want %v", c.in, got, c.want)
			}
			for w, n := range c.want {
				if got[w] != n {
					t.Errorf("Words(%q)[%q] = %d, want %d", c.in, w, got[w], n)
				}
			}
		})
	}
}

func TestTopBreaksTiesAlphabetically(t *testing.T) {
	got := Top(map[string]int{"pear": 2, "apple": 2, "fig": 5}, 3)
	want := []Pair{{"fig", 5}, {"apple", 2}, {"pear", 2}}

	if len(got) != len(want) {
		t.Fatalf("Top() = %v, want %v", got, want)
	}
	for i := range want {
		if got[i] != want[i] {
			t.Errorf("Top()[%d] = %v, want %v", i, got[i], want[i])
		}
	}
}

func TestTopLimit(t *testing.T) {
	counts := map[string]int{"a": 3, "b": 2, "c": 1}
	if got := Top(counts, 2); len(got) != 2 {
		t.Errorf("Top(counts, 2) returned %d pairs, want 2", len(got))
	}
	if got := Top(counts, 0); len(got) != 3 {
		t.Errorf("Top(counts, 0) returned %d pairs, want all 3", len(got))
	}
}

func ExampleTop() {
	counts, _ := Words(strings.NewReader("go go go stop stop wait"))
	for _, p := range Top(counts, 2) {
		fmt.Println(p.N, p.Word)
	}
	// Output:
	// 3 go
	// 2 stop
}

func BenchmarkWords(b *testing.B) {
	text := strings.Repeat("the quick brown fox jumps over the lazy dog ", 100)
	b.ReportAllocs()
	for range b.N {
		Words(strings.NewReader(text))
	}
}

$ go test ./ch10/...
?   	goexamples/ch10/wf	[no test files]
ok  	goexamples/ch10/wf/count	0.002s

Table-driven tests are the Go idiom. A slice of anonymous structs holds the cases and one loop runs them, so adding a case is one line and the logic is written once. t.Run gives each case a name and its own subtest, so a failure points at the case and you can rerun one:

$ go test -run 'TestWords/case_folded' ./ch10/wf/count

t.Errorf records a failure and keeps going; t.Fatalf stops that subtest. Use Fatalf when continuing would panic or produce noise, as it does after an unexpected error, and Errorf otherwise so one run reports every problem.

A failure names the file, the line, and the subtest:

--- FAIL: TestWords (0.00s)
    --- FAIL: TestWords/case_folded (0.00s)
        count_test.go:29: Words("The the THE") = map[THE:1 The:1 the:1], want map[the:3]
FAIL

That is what dropping the strings.ToLower from Words produces, and the message tells you the input, the result, and the expectation without opening the test.

An Example function is a test and documentation at once. ExampleTop compiles, runs, and has its output compared against the // Output: comment, so a documented example cannot rot. It also appears in go doc output and on the generated documentation page, attached to Top because of its name.

A Benchmark function takes a *testing.B and runs the body b.N times, with the framework choosing b.N. It runs only when asked:

$ go test -bench=Words -benchtime=100x ./ch10/wf/count
BenchmarkWords-8   	     100	     48032 ns/op	    8432 B/op	     904 allocs/op

b.ReportAllocs adds the last two columns, which are usually more actionable than the timing. Go 1.24 added for b.Loop() as a replacement for for range b.N that is harder to get wrong; older code uses b.N and so does this book, which targets 1.23.

Four flags to know. -v lists every test as it runs. -run takes a regular expression selecting tests and subtests. -cover reports statement coverage, and -race builds with the race detector, Chapter 11’s subject and a thing that belongs in your CI.

A test that needs input files puts them in a directory named testdata, which the go tool ignores when building.

10.5 Documentation

A comment immediately above a declaration, starting with the declared name, is its documentation. go doc count.Words prints it; go doc -all count prints the package.

The comment above package documents the package itself, conventionally starting Package count ..., and for a main package it starts Command wf ... and describes the usage, as wf does. That convention is the reason go doc output reads consistently across the whole ecosystem.

Write the first sentence so it stands alone, because it is what appears in the package index next to the name.

10.6 The go Command

go run ./pkg          build and run
go build ./...        build everything, leaving binaries
go install ./cmd/wf   build and put the binary in ~/go/bin
go test ./...         run every test
go fmt ./...          format
go vet ./...          report likely mistakes the compiler allows
go doc strings.Cut    print documentation
go mod tidy           fix up dependencies
go clean -cache       when you suspect the build cache, which you will not

./... means this directory and everything under it, and it is how you address a whole project.

go vet deserves a habit. It catches Printf verbs that do not match their arguments, copied mutexes, unreachable code, and shadowed errors — real bugs that are legal Go. Run it alongside your tests, since go test already runs a subset of it.

Exercise 10-1. Add a -min flag to wf that ignores words shorter than some length. Test the library change, not the flag.

Exercise 10-2. Add a case to TestWords for input containing a hyphenated word and decide what the right answer is before you look at what the code does.

Exercise 10-3. Move count to internal/count and confirm the program still builds. Then try to import it from a package outside ch10/wf.

Exercise 10-4. Write a benchmark comparing Top on a map of 10 words against one of 10,000. Use b.Run to make it one benchmark with two sizes.