How to use non-default browser?

First, you should also save the previous values of both browser option and R_BROWSER so that you can restore the previous state of the session:

old_R_BROWSER <- Sys.getenv("R_BROWSER")
old_browser <- options()$browser

Then you can achieve desired behaviour by re-running the command @Hack-R posted after setting R_BROWSER.

Sys.setenv("R_BROWSER" = "/usr/bin/open -a '/Applications/Safari.app'")
options(browser = as.vector(Sys.getenv("R_BROWSER")))
browseURL("http://www.google.com") # opens in Safari, though my default is Chrome

(You can also just directly set options(browser = "/usr/bin/open -a '/Applications/Safari.app'") and browseURL works.)

Finally, restore the system state

Sys.setenv("R_BROWSER" = old_R_BROWSER)
options(browser = old_browser)

R looks for the browser specified at configure time.

The default setting of options("browser") is set to be

options(browser = as.vector(Sys.getenv("R_BROWSER")))

and the browser found at configure time is set in the R_BROWSER variable in the file etc/Renviron. So edit that file.

Finally, if this is a shared system, you can have the following in your ~/.Renviron file

R_BROWSER=${R_BROWSER-'FireFox'}

See ?Startup. (You could also use the Rprofile files, but people normally only have one .Renviron file.)

I base this on this.