Extracting contents from specific meta tags that are not closed using BeautifulSoup
Edited: Added regex for case sensitivity as suggested by @Albert Chen.
Python 3 Edit:
from bs4 import BeautifulSoup
import re
import urllib.request
page3 = urllib.request.urlopen("https://angel.co/uber").read()
soup3 = BeautifulSoup(page3)
desc = soup3.findAll(attrs={"name": re.compile(r"description", re.I)})
print(desc[0]['content'])
Although I'm not sure it will work for every page:
from bs4 import BeautifulSoup
import re
import urllib
page3 = urllib.urlopen("https://angel.co/uber").read()
soup3 = BeautifulSoup(page3)
desc = soup3.findAll(attrs={"name": re.compile(r"description", re.I)})
print(desc[0]['content'].encode('utf-8'))
Yields:
Learn about Uber's product, founders, investors and team. Everyone's Private Dri
ver - Request a car from any mobile phoneΓÇötext message, iPhone and Android app
s. Within minutes, a professional driver in a sleek black car will arrive curbsi
de. Automatically charged to your credit card on file, tip included.
Description is Case-Sensitive.So, we need to look for both 'Description' and 'description'.
Case1: 'Description' in Flipkart.com
Case2: 'description' in Snapdeal.com
from bs4 import BeautifulSoup
import requests
url= 'https://www.flipkart.com'
page3= requests.get(url)
soup3= BeautifulSoup(page3.text)
desc= soup3.find(attrs={'name':'Description'})
if desc == None:
desc= soup3.find(attrs={'name':'description'})
try:
print desc['content']
except Exception as e:
print '%s (%s)' % (e.message, type(e))
soup3 = BeautifulSoup(page3, 'html5lib')
xhtml requires the meta tag to be closed properly, html5 does not. The html5lib parser is more "permissive".