user2728808 user2728808 - 1 year ago 181
R Question

Missing elements when using `read_html` using `rvest` in R

I'm trying to use the

function in the
package, but have come across a problem I am struggling with.

For example, if I were trying to read in the bottom table that appears on this page, I would use the following code:

html_content <- read_html("")

By inspecting the HTML code in the browser, I can see that the content I would like is contained in a
tag (specifically, it is all contained within
<table class="t-calc">
). But when I try to extract this using:

tables <- html_nodes(html_content, xpath = '//table')

I retrieve the following:

> tables
{xml_nodeset (4)}
[1] <table class="tippingpointroi unexpanded">\n <tbody>\n <tr data-state="FL" class=" "> ...
[2] <table class="tippingpointroi unexpanded">\n <tbody>\n <tr data-state="NV" class=" "> ...
[3] <table class="scenarios">\n <tbody/>\n <tr data-id="1">\n <td class="description">El ...
[4] <table class="t-desktop t-polls">\n <thead>\n <tr class="th-row">\n <th class="t ...

Which includes some of the table elements on the page, but not the one I am interested in.

Any suggestions on where I am going wrong would be most appreciated!

Answer Source

The table is built dynamically from data in JavaScript variables on the page itself. Either use RSelenium to grab the text of the page after it's rendered and pass the page into rvest OR grab a treasure trove of all the data by using V8:


URL <- ""

pg <- read_html(URL)

js <- html_nodes(pg, xpath=".//script[contains(., 'race.model')]") %>%  html_text()

ctx <- v8()

race <- ctx$get("race", simplifyVector=FALSE)

str(race) ## output too large to paste here

If they ever change the formatting of the JavaScript (it's an automated process so it's unlikely but you never know) then the RSelenium approach will be better provided they don't change the format of the table structure (again, unlikely, but you never know).

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download