Dr Xorile Dr Xorile - 1 month ago 7x
Python Question

python lxml etree applet information from yahoo

Yahoo finance updated their website. I had an lxml/etree script that used to extract the analyst recommendations. Now, however, the analyst recommendations are there, but only as a graphic. You can see an example on this page. The graph called Recommendation Trends on the right hand column shows the number of analyst reports showing strong buy, buy, hold, underperform, and sell.

My guess is that yahoo will make a few adjustments to the page over the coming little while, but it got me wondering whether such data was extractable in any reasonable way?

  1. I mean, is there a way to get the graphic to work with that?

  2. Even if one were successful, would there be a reasonable way to extract the data from the graphic?

I used to get the source like this:

url = 'https://finance.yahoo.com/quote/'+code+'/analyst?p='+code
tree = etree.HTML(urllib.request.urlopen(url).read())

and then find the data in the html tree. But obviously that's impossible now.


The page is quite dynamic and involves a lot of javascript executed in a browser. To follow the @Padraic's advice about switching to selenium, here is a complete sample working code that produces a month-to-trend dictionary at the end. The values of each bar are calculated as proportions of bar heights:

from pprint import pprint

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

driver = webdriver.Chrome()

# wait for the chart to be visible
wait = WebDriverWait(driver, 10)
trends = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "section[data-reactid$=trends]")))
chart = trends.find_element_by_css_selector("svg.ratings-chart")

# get labels
month_names = [month.text for month in chart.find_elements_by_css_selector("g.x-axis g.tick")]
trend_names = [trend.text for trend in trends.find_elements_by_css_selector("table tr > td:nth-of-type(2)")]

# construct month-to-trend dictionary
data = {}
months = chart.find_elements_by_css_selector("g[transform]:not([class])")
for month_name, month_data in zip(month_names, months):
    total = month_data.find_element_by_css_selector("text.total").text
    data[month_name] = {'total': total}

    bars = month_data.find_elements_by_css_selector("g.bar rect")

    # let's calculate the values of bars as proportions of a bar height
    heights = {trend_name: int(bar.get_attribute("height")) for trend_name, bar in zip(trend_names[::-1], bars)}
    total_height = sum(heights.values())
    for trend_name, bar in zip(trend_names, bars):
        data[month_name][trend_name] = heights[trend_name] * 100 / total_height




{u'Aug': {u'Buy': 19,
          u'Hold': 45,
          u'Sell': 3,
          u'Strong Buy': 22,
          u'Underperform': 8,
          'total': u'26'},
 u'Jul': {u'Buy': 18,
          u'Hold': 44,
          u'Sell': 3,
          u'Strong Buy': 25,
          u'Underperform': 7,
          'total': u'27'},
 u'Jun': {u'Buy': 21,
          u'Hold': 38,
          u'Sell': 3,
          u'Strong Buy': 28,
          u'Underperform': 7,
          'total': u'28'},
 u'May': {u'Buy': 21,
          u'Hold': 38,
          u'Sell': 3,
          u'Strong Buy': 28,
          u'Underperform': 7,
          'total': u'28'}}

The total values are labels that you see on top of each bar.

Hope this would at least be a good start for you. Let me know if you want me to elaborate on any part of the code or require any additional information.