How to call JavaScript function using BeautifulSoup and Python

You can't do this using BeautifulSoup alone. This module was created to scrape HTML (Hyper Text Markup Language) not JavaScript, CSS or any other web language.

It can extract between <script></script> tags (which will be quite useful) but beyond this BeautifulSoup is not what you need.

To call a JavaScript functions you will need a headless browser such as PhantomJS or Selenium. There have also been attempts to parse JavaScript as well as using regex (which is not a good idea) and using other methods (recommended) some methods are described in this question and may be useful.


You can use https://pypi.org/project/requests-html/ this library to render JavaScript content and then use beautiful soup to parse it.

Example:

from requests_html import HTMLSession  
  
def render_JS(URL):
    session = HTMLSession()
    r = session.get(URL)
    r.html.render()
    return r.html.text

You can't. If you want to run JavaScript, you'll need to use a headless browser. Otherwise, you'll have to disassemble the JavaScript and see what it does.

Click on the element while your browser's developer tools are open in the Network tab:

enter image description here

You can now see that the JavaScript downloads new HTML from that URL. You can easily send the same request with urllib.