Python selectolax tutorial

Selectolax is Python binding to Modest engine - fast HTML5 parser with CSS selectors. It is really fast, it can be used to parse HTML just like BeautifulSoup4 but again, it is much much faster than Beautifulsoup4

Installation #

From PyPI using pip:

pip install selectolax

Or you can install the development version from github:

git clone --recursive  https://github.com/rushter/selectolax
cd selectolax
pip install -r requirements_dev.txt
python setup.py install

API usage #

Import selectolax

from selectolax.parser import HTMLParser

Say we have html document like this:

html = """
<body>
    <span id="vspan"></span>
    <h1>Welcome to selectolax tutorial</h1>
    <div id="text">
        <p class='p3' style='display:none;'>Excepteur <i>sint</i> occaecat cupidatat non proident</p>
        <p class='p3' vid>Lorem ipsum</p>
    </div>
    <div>
        <p id='stext'>Lorem ipsum dolor sit amet, ea quo modus meliore platonem.</p>
    </div>
</body>
"""

Select all p tags with class p3 # 

selector = "p.p3"

for node in HTMLParser(html).css(selector):
    print('---------------------')
    print('Node: %s' % node.html)
    print('attributes: %s' % node.attributes)
    print('node text: %s' % node.text(deep=True, separator='', strip=False))
    print('tag: %s' % node.tag)
    print('parent tag: %s' % node.parent.tag)
    if node.last_child:
        print('last child inside current node: %s' % node.last_child.html)
    print('---------------------\n')

Output:

---------------------
Node: <p class="p3" style="display:none;">Excepteur <i>sint</i> occaecat cupidatat non proident</p>
attributes: {'class': 'p3', 'style': 'display:none;'}
node text: Excepteur sint occaecat cupidatat non proident
tag: p
parent tag: div
last child inside current node:  occaecat cupidatat non proident
---------------------

---------------------
Node: <p class="p3" vid="">Lorem ipsum</p>
attributes: {'class': 'p3', 'vid': None}
node text: Lorem ipsum
tag: p
parent tag: div
last child inside current node: Lorem ipsum
---------------------

Select first match # 

print("H1: %s" % HTMLParser(html).css_first('h1').text())

Output:

H1: Welcome to selectolax tutorial

Default return value if there is no matches # 

print("Title: %s" % HTMLParser(html).css_first('title', default='not-found'))

Output:

Title: not-found

Strictly one match # 

HTMLParser(html).css_first("p.p3", default='not-found', strict=True)

Error will be raised:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-7a26563add76> in <module>
----> 1 HTMLParser(html).css_first("p.p3", default='not-found', strict=True)

~/Projects/python/selectolax/selectolax/parser.pyx in selectolax.parser.HTMLParser.css_first()

~/Projects/python/selectolax/selectolax/node.pxi in selectolax.parser.Node.css_first()

ValueError: Expected 1 match, but found 2 matches

Nested selectors # 

HTMLParser(html).css_first('div#text').css_first('p:nth-child(2)').html

Output:

<p class="p3" vid="">Lorem ipsum</p>'

Iterate over all nodes on the current level # 

for node in HTMLParser(html).css("div#text"):
    for cnode in node.iter():
        print(cnode.tag, cnode.html)

Output:

-text 
        
p <p class="p3" style="display:none;">Excepteur <i>sint</i> occaecat cupidatat non proident</p>
-text 
        
p <p class="p3" vid="">Lorem ipsum</p>
-text

Tag removal # 

html_parser = HTMLParser(html)
for node in html_parser.tags('p'):
    node.decompose()
print(html_parser.body.html)

Output:

<body>
    <span id="vspan"></span>
    <h1>Welcome to selectolax tutorial</h1>
    <div id="text">
        
        
    </div>
    <div>
        
    </div>

</body>

Tag unwrapping # 

print(html)

Output:

<body>
    <span id="vspan"></span>
    <h1>Welcome to selectolax tutorial</h1>
    <div id="text">
        <p class='p3' style='display:none;'>Excepteur <i>sint</i> occaecat cupidatat non proident</p>
        <p class='p3' vid>Lorem ipsum</p>
    </div>
    <div>
        <p id='stext'>Lorem ipsum dolor sit amet, ea quo modus meliore platonem.</p>
    </div>
</body>

html_parser = HTMLParser(html)
html_parser.unwrap_tags(['p', 'i'])
print(html_parser.body.html)


<body>
    <span id="vspan"></span>
    <h1>Welcome to selectolax tutorial</h1>
    <div id="text">
        Excepteur sint occaecat cupidatat non proident
        Lorem ipsum
    </div>
    <div>
        Lorem ipsum dolor sit amet, ea quo modus meliore platonem.
    </div>

</body>
Attribute manip

Attribute manipulation # 

html_parser = HTMLParser(html)
node = html_parser.css_first('div#text')
node.attrs['data'] = 'secrect data'
node.attrs['id'] = 'new_id'
print(node.attributes)
del node.attrs['id']
print(node.attributes)
print(node.html)


{'data': 'secrect data', 'id': 'new_id'}
{'data': 'secrect data'}
<div data="secrect data">
        <p class="p3" style="display:none;">Excepteur <i>sint</i> occaecat cupidatat non proident</p>
        <p class="p3" vid="">Lorem ipsum</p>
    </div>

Tree traversal # 

html_parser = HTMLParser(html)
for node in html_parser.root.traverse():

    if node.tag == '-text':
        text = node.text(deep=True).strip()
        if text:
            print(text)
    else:
        print(node.tag)

Output:


-undef
html
head
body
span
h1
Welcome to selectolax tutorial
div
p
i
sint
occaecat cupidatat non proident
p
Lorem ipsum
div
p
Lorem ipsum dolor sit amet, ea quo modus meliore platonem.

Tags:

Python

Recent Posts