Episode 218
218: Balancing test coverage with test costs - Nicole Tietz-Sokolskaya
April 18, 202427m 57s
Audio is streamed directly from the publisher (test-and-code.sfo3.cdn.digitaloceanspaces.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
Nicole is a software engineer and writer, and recently wrote about the trade-offs we make when deciding which tests to write and how much testing is enough.
We talk about:
- Balancing schedule vs testing
- How much testing is the right about of testing
- Should code coverage be measured and tracked
- Good refactoring can reduce code coverage
- Is it worth testing error conditions?
- Are rare error codes ok to just monitor?
- API drift and autospec
- Mitigating risk
- Deciding what to test and what not to test
- Focus testing on key money-making features
- If there's a bug in this part of the code, how much business impact is there?
- Performance testing needs to approximately match real world workloads
- Cost of a service breaking vs the cost of creating, maintaining, and running tests
- Keeping test suites quick to minimize getting distracted
Links:
- Too much of a good thing: the trade-off we make with tests
- Load testing is hard, and the tools are... not great. But why?
- Yet Another Rust Resource (YARR!)
- Goodhart's law - "When a measure becomes a target, it ceases to be a good measure"
Topics
pythonprogrammingsoftwaretesting