MNedelko MNedelko - 5 months ago 175
Python Question

Bing Congitive Web Search API with Python 3

Hi Stackoverflow Community

I am attempting to access the new Bing Cognitive Search API through a Python 3 script. I was able to find threats on using Bing Search 2.0 (since depreciated) but wasn't able to identify an example for the new API with Python 3. I used the following code:

import urllib.parse
import urllib.request
import json
import base64


def bing_search(query):
key = 'mysubscription_key'
query = urllib.parse.quote(query)

#Create credentials for authentication
user_agent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; FDM; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 1.1.4322)'
encoded = base64.b64encode(bytes(':%s' % key, 'utf-8'))
credentials = encoded[:-1] # the "-1" is to remove the trailing "\n" which encode adds
print(credentials)
auth = 'Basic %s' % credentials
print(auth)
url = 'https://api.cognitive.microsoft.com/bing/v5.0/search?q=' + query + '&mkt=en-us'
print(url)

#Create the API request
urlrequest = urllib.request.urlopen(url) # in Python3 urllib.request(...) becomes urllib.request.open(...)
urlrequest.add_header('Authentication', auth)
urlrequest.add_header('User Agent', user_agent)
request_opener = urllib.request.build_opener()

# Handle the response
response = request_opener.open(urlrequest)
results = json.load(response)

result_list = results['webPages']['values']
print(result_list)

bing_search('good news')


Unfortunately I am receiving the following 'Access denied' error. Would anyone be able to point me in the right direction?

C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\python.exe C:/Users/Admin/PycharmProjects/momely/placementarchitect/bingtest.py
b'OmZhNTlmNTBlMmFmMjQyZjhhYmE5MTZlNmZkYThhMDM'
Basic b'OmZhNTlmNTBlMmFmMjQyZjhhYmE5MTZlNmZkYThhMDM'
https://api.cognitive.microsoft.com/bing/v5.0/search?q=good%20news&mkt=en-us
Traceback (most recent call last):
File "C:/Users/Admin/PycharmProjects/momely/placementarchitect/bingtest.py", line 34, in <module>
bing_search('good news')
File "C:/Users/Admin/PycharmProjects/momely/placementarchitect/bingtest.py", line 22, in bing_search
urlrequest = urllib.request.urlopen(url) # in Python3 urllib.request(...) becomes urllib.request.open(...)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 162, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 471, in open
response = meth(req, response)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 581, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 509, in error
return self._call_chain(*args)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 443, in _call_chain
result = func(*args)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 589, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Access Denied


Thank you kindly
M.

Answer

The authentication is not through basic auth but through custom header Ocp-Apim-Subscription-Key:

Subscription key which provides access to this API.

Which can be found in your subscriptions

See full details at https://www.microsoft.com/cognitive-services/en-us/bing-web-search-api

Assuming the key variable holds the subscription key, what you need to do is simply urlrequest.add_header('Ocp-Apim-Subscription-Key', key)

As a minor suggestion, your code can be greatly simplified by using python-requests. A minimum example can be:

import requests

def bing_search(query):
    url = 'https://api.cognitive.microsoft.com/bing/v5.0/search'
    # query string parameters
    payload = {'q': query}
    # custom headers
    headers = {'Ocp-Apim-Subscription-Key': 'xxxxxxxxxxxxx'}
    # make GET request
    r = requests.get(url, params=payload, headers=headers)
    # get JSON response
    return r.json()

j = bing_search('good news')
print(j.get('webPages', {}).get('value', {}))