How to use browsermob with python-selenium?
You need to configure the driver to use BMP as a proxy so it can record the network activity. Here is an example....
from browsermobproxy import Server
from selenium import webdriver
server = Server('/path/to/bmp/bin/browsermob-proxy') #Local path to BMP
server.start()
proxy = server.create_proxy() #Proxy is used to generate a HAR file containing the connection URLS that the MP3s are loaded from.
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy)) #Configure chrome options
driver = webdriver.Chrome(chrome_options=chrome_options)
proxy.new_har('filename')
Then all the activity will be recorded to that file.
In my case, the path for the binary file was C:\Python27\Lib\site-packages\browsermobproxy\browsermob-proxy-2.1.0-beta-3\bin\browsermob-proxy
on Windows with Python 2.7
This problem is related to the fact, that installation of browsermob-proxy is not just doing:
pip install browsermob-proxy
After you executed the code above, you need to go to https://bmp.lightbody.net, download the zip, unzip it and then when calling Server() in your python script, identify path to the executable from the zip you just created. It is located in "bin" folder and is called browsermob-proxy.
In my case it was:
server = Server("/anaconda3/lib/python3.7/site-packages/browsermobproxy/browsermob-proxy-2.1.4/bin/browsermob-proxy")
You can try the code below:
from browsermobproxy import Server
import psutil
import time
for proc in psutil.process_iter():
# check whether the process name matches
if proc.name() == "browsermob-proxy":
proc.kill()
dict = {'port': 8090}
server = Server(path="./BrowserMobProxy/bin/browsermob-proxy", options=dict)
server.start()
time.sleep(1)
proxy = server.create_proxy()
time.sleep(1)
from selenium import webdriver
profile = webdriver.FirefoxProfile()
selenium_proxy = proxy.selenium_proxy()
profile.set_proxy(selenium_proxy)
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("google")
driver.get("http://www.google.co.uk")
print (proxy.har) # returns a HAR JSON blob
server.stop()
driver.quit()
Two things, if your code fails the process could be left open sometimes. So I added the code below for closing the duplicate instances.
import psutil
import time
for proc in psutil.process_iter():
# check whether the process name matches
if proc.name() == "browsermob-proxy":
proc.kill()
Also a sleep of 1 second before and after creating the proxy.
server.start()
time.sleep(1)
proxy = server.create_proxy()
time.sleep(1)
This helps in getting rid of some intermittent issues which can be faced due to the server taking some time to start.
BrowserMob Proxy
BrowserMob Proxy
is an open source tools which is used to capture performance data for a web applications in an HAR format. It also allows to manipulate browser behavior and traffic, such as simulating network traffic, rewriting HTTP requests and responses etc and manipulate network traffic from their AJAX applications. In short, BrowserMob proxy
helps us to capture client side performance data for a web application using Selenium WebDriver automated tests.
You can find more details about BrowserMob Proxy
from the Python Documentation and this tutorial.
Demonstration of BrowserMob Proxy 2.0 with Python client on Windows
Install browsermob-proxy through the CLI :
C:\Users\your_user>pip install browsermob-proxy Collecting browsermob-proxy Downloading browsermob-proxy-0.8.0.tar.gz Collecting requests>=2.9.1 (from browsermob-proxy) Downloading requests-2.18.4-py2.py3-none-any.whl (88kB) 100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 92kB 148kB/s Collecting idna<2.7,>=2.5 (from requests>=2.9.1->browsermob-proxy) Downloading idna-2.6-py2.py3-none-any.whl (56kB) 100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 61kB 639kB/s Collecting urllib3<1.23,>=1.21.1 (from requests>=2.9.1->browsermob-proxy) Downloading urllib3-1.22-py2.py3-none-any.whl (132kB) 100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 133kB 250kB/s Collecting certifi>=2017.4.17 (from requests>=2.9.1->browsermob-proxy) Downloading certifi-2017.11.5-py2.py3-none-any.whl (330kB) 100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 337kB 401kB/s Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->browsermob-proxy) Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB) 100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 143kB 296kB/s Installing collected packages: idna, urllib3, certifi, chardet, requests, browse rmob-proxy Running setup.py install for browsermob-proxy ... done Successfully installed browsermob-proxy-0.8.0 certifi-2017.11.5 chardet-3.0.4 id na-2.6 requests-2.18.4 urllib3-1.22
Download
the browsermob-proxy binaries browsermob-proxy-2.1.4-bin form the following url :https://bmp.lightbody.net/
Extract
andSave
the directory withinC:\Utility
Launch
the Browsermobproxy Server manually through the CLI command :C:\Utility\browsermob-proxy-2.1.4\lib>java -jar browsermob-dist-2.1.4.jar --port 9090 Running BrowserMob Proxy using LittleProxy implementation. To revert to the legacy implementation, run the proxy with the command-line option '--use-littleproxy false'. [INFO 2018-01-17T19:01:30,276 net.lightbody.bmp.proxy.Main] (main) Starting BrowserMob Proxy version 2.1.4 [INFO 2018-01-17T19:01:30,388 org.eclipse.jetty.util.log] (main) jetty-7.x.y-SNAPSHOT [INFO 2018-01-17T19:01:30,471 org.eclipse.jetty.util.log] (main) started o.e.j.s.ServletContextHandler{/,null} [INFO 2018-01-17T19:01:30,871 org.eclipse.jetty.util.log] (main) Started [email protected]:9090
You can also launch the Browsermobproxy Server through your code as below.
Create a new
PyDev
module (if usingEclipse
) and write a basic program through yourIDE
as follows :from browsermobproxy import Server server = Server("C:\\Utility\\browsermob-proxy-2.1.4\\bin\\browsermob-proxy") server.start() proxy = server.create_proxy() from selenium import webdriver profile = webdriver.FirefoxProfile() profile.set_proxy(proxy.selenium_proxy()) driver = webdriver.Firefox(firefox_profile=profile) proxy.new_har("google") driver.get("http://www.google.co.in") proxy.har # returns a HAR JSON blob server.stop() driver.quit()
Snapshot :
- Execute your
Test
as a Python Run - While your
Program
executes you will observe Firefox Quantum Browser gets initialized and theurl
http://www.google.co.in opens up and gets closed at the end of the test. On completion of the
Test Execution
you will find the following files within your work space which will give you all the details of theTest Execution
:bmp.log geckodriver.log server.log
Snapshot :