Want to scrape table using Puppeteer. How can I get all rows, iterate through rows, and then get "td's" for each row?
One way to achieve this is to use evaluate that first gets an array of all the TD's then returns the textContent of each TD
const puppeteer = require('puppeteer');
const html = `
<html>
<body>
<table>
<tr><td>One</td><td>Two</td></tr>
<tr><td>Three</td><td>Four</td></tr>
</table>
</body>
</html>`;
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(`data:text/html,${html}`);
const data = await page.evaluate(() => {
const tds = Array.from(document.querySelectorAll('table tr td'))
return tds.map(td => td.innerText)
});
//You will now have an array of strings
//[ 'One', 'Two', 'Three', 'Four' ]
console.log(data);
//One
console.log(data[0]);
await browser.close();
})();
You could also use something like:-
const data = await page.$$eval('table tr td', tds => tds.map((td) => {
return td.innerText;
}));
//[ 'One', 'Two', 'Three', 'Four' ]
console.log(data);
Two-Dimensional Array Method
You can also scrape the innerText into a two-dimensional array representing your table.
[
['A1', 'B1', 'C1'], // Row 1
['A2', 'B2', 'C2'], // Row 2
['A3', 'B3', 'C3'] // Row 3
]
page.$$eval()
const result = await page.$$eval('#example-table tr', rows => {
return Array.from(rows, row => {
const columns = row.querySelectorAll('td');
return Array.from(columns, column => column.innerText);
});
});
console.log(result[1][2]); // "C2"
page.evaluate()
const result = await page.evaluate(() => {
const rows = document.querySelectorAll('#example-table tr');
return Array.from(rows, row => {
const columns = row.querySelectorAll('td');
return Array.from(columns, column => column.innerText);
});
});
console.log(result[1][2]); // "C2"