Performance testing in continuous integration?
> You can't run full load (or soak or capacity) tests in CI, it's not practical.
After the TISQA conference here in the States this week, I'm more inclined to say that we should confidently be automating more and more of the full, complex load testing with CI automation.
You might even consider having a separate CI instance running in the larger load testing lab, configured with more realistic infrastructure to support meaningful test results. The load testing process itself is not unlike a separate software development process (design, construct, deploy, execute, analyze, repeat). Most of every performance tool is now supporting more elegant and robust integrations to CI solutions including SOASTA, LoadRunner/PC, JMeter, Neotys, Blazemeter, Flood.io.
But here's three things to watch out for - similar to Oliver's comments: - there's a lot more nuances to performance results...not just clearly PASS or FAIL - don't forget script maintenance to keep in sync with app changes - synchronizing/scaling your load testing lab with production might also be automated
If you wish - review some of the slides from my own TISQA presentation here. That might be a start on how to use CI + Performance across the entire lifecycle. Such as, why not have a CI instance that just "watches the configuration" as it gets changed in PROD and sync those changes back to your load test environment?
Neither JMeter nor Selenium are tools for CI. JMeter is performance testing tool, Selenium is tool for automated functional testing. So, to have performance testing integrated into build process, you can use JMeter with any of CI servers:Jenkins, Bamdoo, etc.
AFAIK, there are two common solution of using JMeter with Jenkins nowadays:
Use Jenkins/Hudson with JMeter plugin for it, which allow to start performance testing task after finishing build process. In this case you need to have appropriate number of load generator with JMeter configured on it.
Another way - using JMeter testing cloud. This service provides Jenkins plugin, which allows to start remote test after building application. In this case you don't need to care about configuring test servers.
P.S. While I'm working for Blazemeter, I wanted to provide objective information. Hope, it's helpful.
First off, JMeter is a good choice for inclusion in CI because it can be run from the command line and you can pass in variables when you do this. I would recommend it for this task.
In general though, integrating Perf. testing into CI is difficult. You've already listed many of the reasons why this is so you're already half way there because you understand the limitations. And that's the rub: it IS possible to have Perf. tests in CI but only to a limited extent.
I think that a good approach follows some of these principles:
You can't run full load (or soak or capacity) tests in CI, it's not practical. The results are subjective and need human interpretation and it takes time to run the tests. But you can run a simpler, cut down set of tests that measure response times for requests and then you can evaluate these response times either:
- Against an NFR or expected range - Ie. Should be less than 1 sec.
- Against the previous results - Ie. Should not deviate more than 10% than the last build.
You can also run automated load / perf. tests - at full volume - outside of the build process. 'Semi CI'. So maybe you could automate a test to run overnight and then check the results in the morning?
Iterate. Just start doing it and getting results and fine tune the tests and how you interpret them over time. Keep it simple and focus on the areas that appear to be useful. Don't launch with a fanfare, keep it quiet until you have confidence with the process and then start failing builds and telling people about it - initially, you're likely to get lots of false negatives.
Instrument your results Do this. A lot. CI is all about failing early so if you take that as you end objective then the best way to achieve it is to run tests early and often but the problem with that is you risk getting buried in data. So an effective method to crunch the data and present the relevant information helps considerably.
You can't automate the whole process down to Red Flag Green Flag - but you should try to get as far down that path as possible.
Finally, there was a very good talk given by the lead Perf. tester at Google that covers this subject. It's a bit old now but the principles still stand. Plus, in a few weeks I'm going to a meetup where Channel4, a British media company, will be talking about how they approached this - maybe you can ask for some slides.