testerjoe2 testerjoe2 - 9 months ago 100
Java Question

Why is selenium getAttribute("href") not working?

Summary: My code goes to a craigslist ad url. It extracts the hidden phone numbers in the ad body. The code works fine for many urls except the one which I have included in the code. (Btw, you can copy and run my code without writing any additional code.)

Problem: The

is returning a
only for this url. Why ? How do I fix this ?


import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;

import java.util.ArrayList;
import java.util.List;

public class Temp {
private static final WebDriver browser = new ChromeDriver();
private static WebDriver temp_browser = new ChromeDriver();

/*The code fails only for this url.*/
private static String url = "https://sfbay.craigslist.org/pen/apa/5764613878.html";

public static String phone_btns_xpath = "//section[@id='postingbody']//*[contains(.,'show contact info')]";
public static By phone_btns_loc = By.xpath(phone_btns_xpath);

public static void main(String[] args) {
List<String> phones = reveal_hidden_phone_numbers(temp_browser);

public static List<String> reveal_hidden_phone_numbers(WebDriver temp_browser) {
List<WebElement> phone_btns = browser.findElements(phone_btns_loc);
List<String> phones = null;
String text = null;

if (phone_btns.size() > 0) {
WebElement phone_btn_0 = phone_btns.get(0);

String url = phone_btn_0.getAttribute("href");
text = temp_browser.findElement(By.tagName("body")).getText();

for (WebElement phone_btn : phone_btns) {

phones = extract_phone_numbers(text);
return phones;

public static List<String> extract_phone_numbers(String text) {
List<String> output = new ArrayList<String>();
output.add("PHONE ;)");
return output;


Stack trace:

<a href="/fb/sfo/apa/5764613878" class="showcontact" title="click to show contact info" rel="nofollow">show contact info</a>

Exception in thread "main" java.lang.NullPointerException: null value in entry: url=null
at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33)
at com.google.common.collect.SingletonImmutableBiMap.<init>(SingletonImmutableBiMap.java:39)
at com.google.common.collect.ImmutableBiMap.of(ImmutableBiMap.java:49)
at com.google.common.collect.ImmutableMap.of(ImmutableMap.java:70)
at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:316)
at com.craigslist.Temp.reveal_hidden_phone_numbers(Temp.java:38)
at com.craigslist.Temp.main(Temp.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)


As I'm seeing in your provided stacktrace this line System.out.println(phone_btn_0.getAttribute("innerHTML")); from your code printed the inner HTML of phone_btn_0 element as :-

<a href="/fb/sfo/apa/5764613878" class="showcontact" title="click to show contact info" rel="nofollow">show contact info</a>

That means you're trying to get href attribute on wrong element. It's on parent element instead of actual link element where href attribute does not exist, that's why you're getting null.

Assuming you want to get href attribute value from this printed link element HTML, So you should try to get href attribute value on the child element of phone_btn_0 as below :-

WebElement phone_btn_0 = phone_btns.get(0);

String url = phone_btn_0.findElement(By.tagName("a")).getAttribute("href");