Testing only affected code in Python

I guess you are looking for a continuous testing tool?

I created a tool that sits in the background and runs only impacted tests: (You will need PyCharm plugin and pycrunch-engine from pip)

https://github.com/gleb-sevruk/pycrunch-engine

This will be particularly useful if you are using PyCharm.

More details are in this answer: https://stackoverflow.com/a/58136374/2377370

The idea of automating the selective testing of parts of your application definitely sounds interesting. However, it feels like this is something that would be much easier to achieve with a statically typed language, but given the dynamic nature of Python it would probably be a serious time investment to get something that can reliably detect all tests affected by a given commit.

When reading your problem, and putting aside the idea of selective testing, the approach that springs to mind is being able to group tests so that you can execute test suites in isolation, enabling a number of useful automated test execution strategies that can shorten the feedback loop such as:

• Parallel execution of separate test suites on different machines
• Running tests at different stages of the build pipeline
• Running some tests on each commit and others on nightly builds.

Therefore, I think your approach of using tags to partition tests into different 'groups' is a smart one, though as you say the management of these becomes difficult with a large test suite. Given this, it may be worth focussing time in building tools to aid in the management of your test suite, particularly the management of your tags. Such a system could be built by gathering information from:

• Test result output (pass/fail, execution time, logged output)
• Code coverage output
• Source code analysis

Good luck, its definitely an interesting problem you are trying to solve, and hope some of these ideas help you.