Annapoornima Koppad Annapoornima Koppad - 2 months ago 14
Python Question

Webscraping the Indian Patent Website for patent data

I am trying to write a webscraper for the Indian patent search website to get data about patents. Here is the code that I have so far.

#import the necessary modules
import urllib2
#import the beautifulsoup functions to parse the data
from bs4 import BeautifulSoup

#mention the website that you are trying to scrape
patentsite="http://ipindiaservices.gov.in/publicsearch/"

#Query the website and return the html to the variable 'page'
page = urllib2.urlopen(patentsite)

#Parse the html in the 'page' variable, and store it in Beautiful Soup format
soup = BeautifulSoup(page)

print soup


Unfortunately, the Indian patent website is not robust or I am not sure how to proceed further in this regard.

This is the output for the above code.

<!--
###################################################################
## ##
## ##
## SIDDHAST.COM ##
## ##
## ##
###################################################################
--><!DOCTYPE HTML>
<html>
<head>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta charset="utf-8"/>
<title>:: InPASS - Indian Patent Advanced Search System ::</title>
<link href="resources/ipats-all.css" rel="stylesheet"/>
<script src="app.js" type="text/javascript"></script>
<link href="resources/app.css" rel="stylesheet"/>
</head>
<body></body>
</html>


The thing that I want to give is, suppose I provide a company name, the scraper should get all the patents for that particular company. I want to do other things if I can get this part right, like providing a set of inputs that the scraper will use to find patents. But I am stuck at the part where I am unable to proceed further.

Any pointers on how to get this data will be greatly appreciated.

Answer

You can do this with just requests. The post is to http://ipindiaservices.gov.in/publicsearch/resources/webservices/search.php with one paqram which is a timestamp that we create with time.time, each value in "field[]" should match up to each in "fieldvalue[]" and in turn match to "operator[]" whether you chose to *AND* *OR* or *NOT*, the [] after each key specifies that we are passing an array of value, without that nothing would work.:

data = {
    "publication_type_published": "on",
    "publication_type_granted": "on",
    "fieldDate": "APD",
    "datefieldfrom": "19120101",
    "datefieldto": "20160906",
    "operatordate": " AND ",
    "field[]": ["PA"], # claims,.description, patent-number codes go here
    "fieldvalue[]": ["chris*"], # matching values for ^^ go here
    "operator[]": [" AND "], # matching sql logic for ^^ goes here
    "page": "1", #  gives you next page results
    "start": "0", # not sure what effect this actually has.
    "limit": "25"} # not sure how this relates as  len(r.json()[u'record']) stays 25 regardless

import requests
from time import time

post = "http://ipindiaservices.gov.in/publicsearch/resources/webservices/search.php?_dc={}".format(
    str(time()).replace(".", ""))

with requests.Session() as s:
    s.get("http://ipindiaservices.gov.in/publicsearch/")
    s.headers.update({"X-Requested-With": "XMLHttpRequest"})
    r = s.post(post, data=data)
    print(r.json())

Output will look like the following, I cannot add it all as there is too much data to post:

{u'success': True, u'record': [{u'Publication_Status': u'Published', u'appDate': u'2016/06/16', u'pubDate': u'2016/08/31', u'title': u'ACTUATOR FOR DEPLOYABLE IMPLANT', u'sourceID': u'inpat', u'abstract': u'\n    Systems and methods are provided for usin.............

If you use the record key you get a list of dicts like:

{u'Publication_Status': u'Published', u'appDate': u'2015/01/27', u'pubDate': u'2015/06/26', u'title': u'CORRUGATED PALLET', u'sourceID': u'inpat', u'abstract': u'\n    A corrugated paperboard pallet is produced from two flat blanks which comprise a pallet top and a pallet bottom. The two blanks are each folded to produce only two parallel vertically extending double thickness ribs&nbsp;three horizontal panels&nbsp;two vertical side walls and two horizontal flaps. The ribs of the pallet top and pallet bottom lock each other from opening in the center of the pallet by intersecting perpendicularly with notches in the ribs. The horizontal flaps lock the ribs from opening at the edges of the pallet by intersecting perpendicularly with notches&nbsp;and the vertical sidewalls include vertical flaps that open inward defining fork passages whereby the vertical flaps lock said horizontal flaps from opening.\n  ', u'Assignee': u'OLVEY Douglas A., SKETO James L., GUMBERT Sean G., DANKO Joseph J., GABRYS Christopher W., ', u'field_of_invention': u'FI10', u'publication_no': u'26/2015', u'patent_no': u'', u'application_no': u'642/DELNP/2015', u'UCID': u'WVJ4NVVIYzFLcUQvVnJsZGczcVRmSS96Vkh3NWsrS1h3Qk43S2xHczJ2WT0%3D', u'Publication_Type': u'A'}

which is your patent info.

You can see if we choose a few values in our browser, the values in all fieldvalue, field and operator line up, ANDis the default so you see that for every option:

enter image description here

enter image description here

So figure out the code, pick what you want and post.

Comments