how to combine selenium and beautifulsoup code example

Example 1: beautifulsoup(driver.page_source 'html.parser')

In [8]: from bs4 import BeautifulSoup

In [9]: from selenium import webdriver

In [10]: driver = webdriver.Firefox()

In [11]: driver.get('http://news.ycombinator.com')

In [12]: html = driver.page_source

In [13]: soup = BeautifulSoup(html)

In [14]: for tag in soup.find_all('title'):
   ....:     print tag.text
   ....:     
   ....:     
Hacker News

Example 2: beautifulsoup(driver.page_source 'html.parser')

# We will try and render what requests returns from https://www.webscraper.io/test-sites/e-commerce/allinone
# without running javascript first
import requests
from IPython.display import HTML

url = 'https://www.webscraper.io/test-sites/e-commerce/ajax/computers/laptops'

r = requests.get(url)

HTML(r.text)

Tags:

Html Example

Related