Depth First Traversal on BeautifulSoup Parse Tree
mytag.find_all()
already does that:
If you call mytag.find_all(), Beautiful Soup will examine all the descendants of mytag: its children, its children’s children, and so on
from bs4 import BeautifulSoup # pip install beautifulsoup4
soup = BeautifulSoup("""<!doctype html>
<div id=a>A
<div id=1>A1</div>
<div id=2>A2</div>
</div>
<div id=b>B
<div id=I>BI</div>
<div id=II>BII</div>
</div>
""")
for div in soup.find_all("div", recursive=True):
print(div.get('id'))
Output
a
1
2
b
I
II
The output confirms that, it is a depth first traversal.
Old Beautiful Soup 3 answer:
recursiveChildGenerator()
already does that:
soup = BeautifulSoup.BeautifulSoup(html)
for child in soup.recursiveChildGenerator():
name = getattr(child, "name", None)
if name is not None:
print name
elif not child.isspace(): # leaf node, don't print spaces
print child
Output
For the html from @msalvadores's answer:
html
ul
li
Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
li
Aliquam tincidunt mauris eu risus.
li
Vestibulum auctor dapibus neque.
html
NOTE: html
is printed twice due to the example contains two opening <html>
tags.
I think you can use the method "childGenerator" and recursively use this one to parse the tree in a DFT fashion.
def recursiveChildren(x):
if "childGenerator" in dir(x):
for child in x.childGenerator():
name = getattr(child, "name", None)
if name is not None:
print "[Container Node]",child.name
recursiveChildren(child)
else:
if not x.isspace(): #Just to avoid printing "\n" parsed from document.
print "[Terminal Node]",x
if __name__ == "__main__":
soup = BeautifulSoup(your_data)
for child in soup.childGenerator():
recursiveChildren(child)
With "childGenerator" in dir(x)
we make sure that an element is a container, terminal nodes such as NavigableStrings
are not containers and do not contain children.
For some example HTML like:
<html>
<ul>
<li>Lorem ipsum dolor sit amet, consectetuer adipiscing elit.</li>
<li>Aliquam tincidunt mauris eu risus.</li>
<li>Vestibulum auctor dapibus neque.</li>
</ul>
</html>
This scripts prints ...
[Container Node] ul
[Container Node] li
[Terminal Node] Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
[Container Node] li
[Terminal Node] Aliquam tincidunt mauris eu risus.
[Container Node] li
[Terminal Node] Vestibulum auctor dapibus neque.