Paul85 Paul85 - 1 year ago 189
HTML Question

Extract value hidden under DIV id

Hi Java expert,
I am trying to extract data from given URL address where information is hidden under "div id". My URL query page looks like this:

enter image description here

I am giving peptide sequence as my query and then clicking on "Search Dataset" button to view the result as a table.

But when I am trying to do "View page source" to view the result as HTML and I didn't see that table.

After using 'firebug' I can see that table in HTML and that looks like this:

enter image description here

In order to get data for my query I have written simple JAVA script:

package retrieve.information;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

public class DemoExtractHidenHtml {
public static void main(String[] args) {
Document document;
try {
document = Jsoup.connect("{\"searched_button\":\"datasets\",\"peptide\":\"NLAVSQVVHK\"}").userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21").get();
Element dataset ="td.table[datasets]_row[0]_column[1]").first();
} catch (IOException e) {
// TODO Auto-generated catch block



And of course it not working for me and I am getting following error:

Exception in thread "main"$SelectorParseException: Could not parse query 'td.table[datasets]_row[0]_column[1]': unexpected token at '_row[0]_column[1]'
at retrieve.information.DemoExtractHidenHtml.main(

Anyone has any idea how to overcome this problem and I am a newbie in JAVA.

Answer Source

Hi I have solved that problem using selenium. So solution for my problem:

import java.util.Scanner;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

public class ExtractDataDynamic {
private static Scanner kb;

public static void main(String[] args) {
    // TODO Auto-generated method stub
    kb = new Scanner(;
    String userpepseq;
    userpepseq = kb.nextLine();
    if (userpepseq.trim().isEmpty()){
        System.out.println("User didn't input any value!");
    } else {
        if (Pattern.matches("[a-zA-Z]+", userpepseq) == true) {
            WebDriver drivermassid = new FirefoxDriver();
            //Here we are storing the value from the cell in to the string variable
            String sCellValuemassid = drivermassid.findElement(By.xpath(".//*[@class='result']/tbody")).getText();
            if (sCellValuemassid.length() > 0){
                String massiveid="";
                String status="";
                Pattern pattern = Pattern.compile("MSV\\d+\\s+\\d+\\s+");
                Matcher macther= pattern.matcher(sCellValuemassid);
                while (macther.find()){
                    massiveid=(("\\ "))[0];
                    status=(("\\ "))[1];
                if (massiveid.length() > 0 ){
                    WebDriver drivermasspro = new FirefoxDriver();
                    String sCellValuemasspro = drivermasspro.findElement(By.xpath(".//*[@class='result']/tbody")).getText();
                    if (sCellValuemasspro.length() > 0){
                        String [] proteinifo = sCellValuemasspro.split("\\n");
                        for (int i=0;i<proteinifo.length;i++) {
                            String [] subproteinifo = proteinifo[i].split("\\ ");
                            System.out.println(massiveid+" "+status+" "+subproteinifo[1]);
                } else {
                    System.out.println("Massive ID doesn't exist for "+userpepseq +".");
            } else {
                System.out.println(userpepseq+" doesn't exist in Massive database.");

        } else {
            System.out.println(userpepseq+" should not contain any number!");

Becuase that table is dynamic and they are using javascript to populate data into the table so I have found this is one of way to solve my porblem. Thanks

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download