HtmlUnitDriver (HtmlUnit) vs GhostDriver (PhantomJS)?
I've used phantomJS in a few projects over the last couple of years, but have often had issues with it. For example javascript on pages behaving different to chrome, firefox, internet explorer. Some pages simply not loading, possibly due to redirects but I'm not positive (e.g. keycloak log in pages).
I've not used HtmlUnit as much, but as I type this it is avoiding some of the above phantom issues for me on tests with keycloak login pages.
From my experience with a number of headless browsers, I'd say:
HtmlUnitDriver: the fastest of all implementations I've come across, and perfect for simple, static pages, especially those without JavaScript. Any remotely complex page seems to produce problems - that's my practical experience even if I can't justify in detail. Perfect for testing Selenium features on demo pages, scraping status pages etc.
PhantomJSDriver (PhantomJS + GhostDriver): not as much faster as you might hope vs the desktop browsers, however, much easier to set up than Firefox + xvfb. By default screenshots can look a bit odd, but that usually turns out to be because PhantomJS defaults to a narrow window unless explicitly set (read below for why).
Update: a bit more detail on PhantomJS versions, from my other answer.
Like Safari, PhantomJS uses WebKit for rendering (e.g. Firefox uses Gecko)
Different PhantomJS versions are built against different WebKit versions. PhantomJS 2.x uses WebKit 538.x, which makes it equivalent to Safari 7 or 8. whereas PhantomJS 1.9.8 uses WebKit 534.34, which is equivalent to Safari 5.
This may be an issue for you, since Google determines Safari 5 to be an "old" browser and will therefore potentially render its search pages differently.
So ensuring you use PhantomJS 2.x can reduce the rendering differences that a lot of people report vs. desktop browsers.
Another interesting possibility is SlimerJS. However, I've not got it to work reliably enough yet.
I've never had reliability issues with either HtmlUnitDriver or PhantomJSDriver (the only annoyance one was a HttpClient 4.5 / HtmlUnit 2.17 incompatibility issue).
(In answer to the comment about modifying HTTP requests, I'd personally recommend sticking to the WebDriver API and use a proxy like BrowserMob to mutate requests and responses rather than taking advantage of browser-specific features.)
All in all, I'd advise against creating a tool or process that forces users to choose one browser over another. If possible, allow them to configure or override. For the majority of cases I'd plump for PhantomJS, as it won't let you down. However, the performance of HtmlUnit should be considered for the simplest pages.
See also (perhaps): http://www.guru99.com/selenium-with-htmlunit-driver-phantomjs.html and https://www.quora.com/Software-Testing/How-does-PhantomJS-compare-to-Selenium