Automated Testing is fundamental to modern software development. There are just too many complexities and too many permutations to rely on manual testing (although manual testing is absolutely essential!). But how do you test a product that has to run on two different hardware architectures, three or four different OSs, and many different Docker providers? Well, it’s hard! But to Make Developers Happy™ we have to have predictable behavior everywhere. It’s amazing that go lets us build binaries that run the same everywhere with no external dependencies and Docker and docker-compose let us build the system the same.

Bonus contribution opportunity before we start: You’ve probably seen the once-a-day messages that happen on ddev start these days. Those come from ddev/remote-config, and they can get stale. What can you think of to add there? When you think of anything, just do a PR to remote-config.jsonc.

Our testing environments:

• GitHub amd64 Linux
• GitHub amd64 macOS Ventura
• Windows 11 Docker Desktop/Mutagen (buildkite)
• Windows 11 WSL2 Docker Desktop (buildkite)
• Windows 11 WSL2 docker-ce (buildkite)
• macOS arm64 and amd64 (buildkite)
• CircleCI Linux arm64
• See photo of buildkite test runners

Test runner orchestration:

• Github Actions
• CircleCI
• Buildkite

Test Runner Access:

• Via normal Github techniques
• Via Chrome Remote Desktop

Test Runner Monitoring: Many test runners, especially Windows Docker Desktop, fail quite often. We have an icinga (nagios) monitoring system that notifies us when this happens, see https://newmonitor.thefays.us/icingaweb2/dashboard - that way we don’t have to wait until a test fails and wonder why when the runner was the actual problem.

Tour of recent tests: https://github.com/ddev/ddev/commit/d12fbb4eec7e6830645e73ee97ad9120134f4a2d (on master) or https://github.com/ddev/ddev/pull/5353 for a PR

Agenda for testing (contribution opportunities)

• Add testing for Orbstack and possibly Rancher Desktop (macOS)
• Limit some testing so we don’t waste so much. Move some testing out of PRs and do only on pull. Can we get by with just Linux + macOS arm64 on PRs?
• Try to stabilize Windows tests and WSLDD tests
• Improve individual flaky tests
  • TestDdevXhprofEnabled
  • **TestDdevDescribe fails regularly with wrong state for container**
  • As does TestDdevPause
• Review tests for the ones that are taking more time than they deserve.
• Try to find out why WSL2 instances crash during test runs. Affects Win11/DD and WSL2/docker-ce
• How can we move more testing out of Randy’s house (Windows/Mac)

Add-on Tests (bats): https://github.com/ddev/ddev-addon-template