Our tests:

• Go “pkg” tests, that don’t use an external ddev binary
• Go “cmd” tests that rely on a matching external ddev binary
• Container tests written using bats (bash)

How many tests are there? How many lines of test code?

• According to cloc there are 22,322 lines of go code: cloc ./pkg ./cmd --include-lang=Go --not-match-f='.*_test\.go$’
• There are 14,853 lines of go test code (_test.go): cloc ./pkg ./cmd --include-lang=Go --match-f='._test\.go$’)
• (There is a modest amount of bats test code as well for the docker images)

Types of tests:

• Golang tests are in files named *_test.go and test function names start with Test
• Golang E2E tests, for example TestConfigFunctionality
• Golang Unit tests, for example TestCreateTmpDir
• Image tests are written in bats, for example require binaries installed correctly.

Flaky tests and fragile tests.

• Try to recreate the situation locally
• Add more output to figure out what’s going on.
• Don’t run the test on a platform if it doesn’t add value. Example: TestComposerVersionConfig.

Slow tests

• Figure out why (doing too many things? Too many starts and stops? Can it do less? Is the actual test valid?) Example: TestDdevXhprofEnabled see PR.

Big Infra Test Improvements (and problems)

• Could we run most tests only on Linux and macOS arm64?
• Really not all are needed, and some could be factored out/retired/consolidated
• Add Windows ARM64 testing?
• Spend more testing time on more important platforms, like macOS arm64.
• Carbon costs
• Could we run tests in parallel? What would be the implications.

Running Individual Tests

• export GOTEST_SHORT=true or export GOTEST_SHORT=12
• The built ddev binary should be in PATH if running a test in cmd
• make testpkg TESTARGS="-count=1 -run TestDdevAllDatabases"