Here is my for loop. It takes very long to run over 100,000 iterations.
for urlt in xrange(0, len(clean_url_data)):
Rather than extending the
website_title list using
website_title.append(urlt) and then immediately overwriting that
urlt item with your title data, you should just append the title data directly to the list.
Also, it's better to iterate directly over the
clean_url_data list, rather than using an integer loop to index
clean_url_data. If you need an index you can use
enumerate. like this:
website_title =  for i, url in enumerate(clean_url_data): website_title.append(g.extract(url).title) print i
If you don't really need that index, you can make things simpler. Here's an approach using a list comprehension, which is a little faster than using append in a loop.
website_title = [g.extract(url).title for url in clean_url_data]
The list comprehension uses a special LIST_APPEND bytecode, which is basically an operator, so it's faster than looking up the list's
.append method then calling it; calling a Python function is relatively slow compared to executing an operator.