Drupal - Doing A/B testing
I would recommend you to consider testing http://visualwebsiteoptimizer.com/ for your use case. There is also a module http://drupal.org/project/visual_website_optimizer which makes the integration easier for you.
From the FAQ page of VWO you can see that
- Uses cookies to track visitors
- Visitors see same variation even on repeat visits
So that should solve your problem. To make the decision of which variate should be visible to which users, there is sleek UI for selecting those conditions.
However if you want to do the things, the Drupal way then you can use http://drupal.org/project/multivariate which gives you more control. Multivariate is more like Rules or Context module. You have a set of conditions(is the user using X browser, is his IP originating from Y country and so on) and when those conditions are met a set of actions can be performed(serve his multivariate A instead of B).
An important thing is that a user (based on IP or login) should see the template he saw for the first time forever not only for his session.
For making sure that above condition is met you can increase the life of the cookies set. If you want to track the users before loggin in(anonymous users) and after they have authenticated, then you might need to do some custom coding.
There are a few options for A/B testing a Drupal website, and all options will run the simple test requirements of this question (splitting traffic with specified percentages and keeping visitors in their initial experiences). The two basic approaches are using a paid external service to run the A/B tests or using a fully Drupal-integrated module. Here are the options for each approach:
Drupal Modules
- A/B test: Available for D7, this module is described on the project page as a "low-level API for implementing A/B tests in code". The tests are implemented with custom PHP, which requires knowledge of Drupal data structures and hooks on the PHP side. Also note that an HTML cacheing layer will complicate matters (if not all site visitors are hitting backend PHP). This module has not been updated since April 2015 and has not had a stable release, but there are 30+ sites that currently report using it.
- Multivariate: As Gokul mentioned, Multivariate allows tests to be configured through in the admin UI, without needing any custom code. It is available for D7, but has not had a stable release and hasn't been updated since October, 2013. It does have the highest current usage of any non-external A/B testing module with 100+ sites currently reporting that they use it. As it is a backend solution, an HTML cacheing layer could prevent usage of this module.
- A/B Test JS: This is a new module for D7 and D8 that runs tests in JavaScript and exposes an admin UI for test creation. It does not currently have sites reporting usage, but is actively being developed and maintained. Custom JavaScript snippets are required to create tests, and evaluation of the tests must be done in a separate analytics platform like Google Analytics.
External services
The external options run all testing logic on the frontend (in JavaScript) and can be implemented with either direct inclusion of the scripts, or through Drupal modules when available.
- Optimizely: I have not used Optimizely, but I do know that is widely used and considered state-of-the art for A/B testing. It includes a WYSIWYG UI for creating the test modifications, so tests can be created without any code. The Optimizely Drupal module is available in D7 and D8, and has the most usage of any of the A/B testing options.
- Visual Website Optimizer: I don't have experience with this either, but it does include a WYSIWYG interface for creating tests and interface for viewing results. A Drupal module that includes the VWO javascript in your site's pages is available for D6, D7, and D8, and also has a significant amount of usage on Drupal.
- Adobe Test and Target: From personal experience with Adobe T&T, it has the WYSIWYG interface for creating tests, but I found the UI quite buggy, and worst of all, many of our tests caused "flicker" for site visitors, where the original page flashes before test modifications are implemented. The Test&Target Drupal module also has very little usage, and is only available in D7.
- Google Content Experiments: As a part of Google Analytics, Content Experiments are not as fully featured as the options above, but I believe the service is free and would satisfy the requirements of this question. The Content Experiments Drupal module is available in D7.
Conclusion
Long story short, this use case can be satisfied with any of these solutions so long as they function as intended. More specifics would be needed to find the best solution. In general, I prefer to use Drupal A/B Test JS, but that's also because I made it :)