user2923535 user2923535 - 25 days ago 7
Java Question

Jsoup select doesn't seem to be working on Android

So I first wrote this code in just Java to test if it works and it does:

Document doc = Jsoup.connect("https://myanimelist.net/anime/31646/").get();
Elements studio = doc.select("span:contains(Studios) + a[href][title]");
System.out.println(studio.text());
String studioA = "";
for(Element link : studio){
studioA = link.text();
}
System.out.println(studioA);


When I run it it prints:

Shaft
Shaft


Which is what I want. When I copy and refactor the code into an AsyncTask in Android Studio, the code doesn't seem to work:

public void getStudioAndRank(String animeURL) {
try {
Log.e("URL", animeURL);
Document doc = Jsoup.connect(animeURL).get();
Log.e("GET", "JSOUP \n"+ doc.text());
Elements studios = doc.select("span:contains(Studios) + a[href][title]");
Log.e("studio", studios.text()+" ");
for(Element link : studios){
Log.e("studios", link.text()+" ");
}
studioAnime = studios.text();
Log.e("STUDIO", studioAnime+" ");

} catch (IOException e) {
e.printStackTrace();
}
}


Where animeURL is the clicked on show. The url will be similar to the one I'm testing in the first method. However, the output is this:

E/URL: https://myanimelist.net/anime/31646/
12-16 10:51:05.679 16333-16706/ E/GET: JSOUP
3-gatsu no Lion (March comes in like a lion) - MyAnimeList.net
My Anime ListAd FreeSign UpSearch menu 3-gatsu no Lion March comes in like a lion TV (23eps) Aired Fall 2016 Studios Shaft More Information 7.84 6,788 users Ranked #773 0 Reviews Synopsis...
12-16 10:51:05.706 16333-16706 E/studio:
12-16 10:51:05.710 16333-16706/ E/STUDIO:


The doc.select() is basically returning "". I don't understand why as the same doc.select() works in the first method.

Answer

Evidently the website detects usage of a mobile device and returns a different response, had such cases before. The fix is simple: just add a user-agent for a desktop version:

Example Code

public void getStudioAndRank(String animeURL) {

    String userAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.87 Safari/537.36";

    try {
        Log.e("URL", animeURL);
        Document doc = Jsoup.connect(animeURL).userAgent(userAgent).get();
        Log.e("GET", "JSOUP \n"+ doc.text());
        Elements studios  = doc.select("span:contains(Studios) + a[href][title]");
        Log.e("studio", studios.text()+" ");
        for(Element link : studios){
            Log.e("studios", link.text()+" ");
        }
        studioAnime = studios.text();
        Log.e("STUDIO", studioAnime+" ");

    } catch (IOException e) {
        e.printStackTrace();
    }
}

Output

11-07 20:47:56.474 22199-24058/de.fredericklein.sojsoup E/GET: JSOUP 
                                                               3-gatsu no Lion (March comes in like a lion) - MyAnimeList.net
                                                                MyAnimeList.net Support MALLoginSign Up AllAnimeMangaCharactersPeopleNewsFeatured ArticlesForumClubsUsers View all results for ${ keyword } Anime Anime Search Top Anime Seasonal Anime Videos Reviews Recommendations Manga Manga Search Top Manga Reviews Recommendations Community Forums Clubs Blogs Users Industry News Featured Articles People Characters Watch Episode Videos Promotional Videos Help About Support Advertising FAQ Report Staff MAL Supporter Edit Anime Information What would you like to edit? SynopsisBackgroundAlternative TitlesPictureAiring DatesProducersRelationsRatingDurationSourceBroadcast   3-gatsu no Lion Add to My List Status: WatchingCompletedOn-HoldDroppedPlan to Watch Eps Seen: / 23 Your Score: Select(10) Masterpiece(9) Great(8) Very Good(7) Good(6) Fine(5) Average(4) Bad(3) Very Bad(2) Horrible(1) Appalling   Add Detailed Info Add to Favorites Alternative Titles English: March comes in like a lion Synonyms: Sangatsu no Lion Japanese: 3月のライオン Information Type: TV Episodes: 23 Status: Currently Airing Aired: Oct 8, 2016 to ? Premiered: Fall 2016 Broadcast: Saturdays at 23:00 (JST) Producers: Aniplex, Dentsu, NHK, Hakusensha, Asmik Ace Entertainment, Toy's Factory Licensors: Aniplex of America Studios: Shaft Source: Manga Genres: Drama, Game, Seinen, Slice of Life Duration: 25 min. per ep. Rating: PG-13 - Teens 13 or older Statistics Score: 7.841 (scored by 6,788 users) 1 indicates a weighted score. Please note that 'Not yet aired' titles are excluded. Ranked: #7732 2 based on the top anime page. Please note that 'Not yet aired' and 'R18+' titles are excluded. Popularity: #1119 Members: 49,845 Favorites: 82 Details Videos Episodes Reviews Recommendations Stats Characters & Staff News Forum Featured Clubs Pictures Top   >   Anime   >   3-gatsu no Lion 7.84 Ranked #773Popularity #1119Members 49,845 Fall 2016TVShaft Add to ListSelect(10) Masterpiece(9) Great(8) Very Good(7) Good(6) Fine(5) Average(4) Bad(3) Very Bad(2) Horrible(1) Appalling Episodes: /23 Rightstuf Trailer Aniplex US versionplay More videos EditSynopsisRei Kiriyama is a 17-year-old boy who recently started living alone, financed by his salary as a professional Shogi player. Despite his independence, however, he's yet to mature emotionally, and his problems continue to haunt him in his daily life. His relationship with his adoptive family is strained, and he has difficulties interacting with his fellow high school students. Meanwhile, his professional career has entered a slump. Burdened with the heavy expectations placed on him as the fifth to become pro in middle school, his wins and losses are fluctuating as his record and progression into the ranks begin to stagnate. Acquainted with Rei are the three Kawamoto sisters: Akari, Hinata, and Momo. Unlike Rei, they live happily in their modest home, which they warmly welcome Rei into as if he were one of their own. Despite his reservations about becoming too close to the family, he frequently visits, interacting with them and receiving the kind of care and affection he never quite had while under his foster home. This is the story of Rei's triumphs and failures, relationships new and old, and his growth as a person. (Source: TV Tropes) EditBackgroundNo background information has been added to this title. Help improve our database by adding background information here. EditRelated Anime Adaptation: 3-gatsu no Lion Other: 3-gatsu no Lion meets Bump of Chicken More charactersCharacters & Voice Actors Kiriyama, Rei Main Kawanishi, Kengo Japanese Uchiyama, Yumi Japanese Kawamoto, Akari Main Kayano, Ai Japanese Kawamoto, Hinata Main Hanazawa, Kana Japanese Kawamoto, Momo Main Kuno, Misaki Japanese More staffStaff Shinbou, Akiyuki Director Okada, Kenjirou Director, Episode Director Asano, Naoyuki Director, Storyboard, Animation Director Kameyama, Toshiki Sound Director Edit Opening Theme "Answer (アンサー)" by BUMP OF CHICKEN Edit Ending Theme "Fighter (ファイター)" by BUMP OF 
11-07 20:47:56.476 22199-24058/de.fredericklein.sojsoup E/studio: Shaft 
11-07 20:47:56.476 22199-24058/de.fredericklein.sojsoup E/studios: Shaft 
11-07 20:47:56.476 22199-24058/de.fredericklein.sojsoup E/STUDIO: Shaft 
Comments