displayname displayname - 1 year ago 92
Python Question

POS-Tagger is incredibly slow

I am using

to generate n-grams from sentences by first removing given stop words. However,
is extremely slow taking up to 0.6 sec on my CPU (Intel i7).

The output:

['The first time I went, and was completely taken by the live jazz band and atmosphere, I ordered the Lobster Cobb Salad.']
["It's simply the best meal in NYC."]
['You cannot go wrong at the Red Eye Grill.']

The code:

for sentence in source:

nltk_ngrams = None

if stop_words is not None:
start = time.time()
sentence_pos = nltk.pos_tag(word_tokenize(sentence))
print time.time() - start

filtered_words = [word for (word, pos) in sentence_pos if pos not in stop_words]
filtered_words = ngrams(sentence.split(), n)

Is this really that slow or am I doing something wrong here?

Answer Source

Use pos_tag_sents for tagging multiple sentences:

>>> import time
>>> from nltk.corpus import brown
>>> from nltk import pos_tag
>>> from nltk import pos_tag_sents
>>> sents = brown.sents()[:10]
>>> start = time.time(); pos_tag(sents[0]); print time.time() - start
>>> start = time.time(); [pos_tag(s) for s in sents]; print time.time() - start
>>> start = time.time(); pos_tag_sents(sents); print time.time() - start 
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download