I'm hoping someone has experience with this as I'm unable to find any comments online besides a bug report from 2015 regarding the NERtagger which is probably the same.
Anyway, I'm trying to batch process text to get around the poor performing base tagger. From what I understand, tag_sents should help.
from nltk.tag.stanford import StanfordPOSTagger
from nltk import word_tokenize
stanford_model = 'stanford-postagger/models/english-bidirectional-distsim.tagger'
stanford_jar = 'stanford-postagger/stanford-postagger.jar'
tagger = StanfordPOSTagger(stanford_model, stanford_jar)
tagger.java_options = '-mx4096m'
text = "The quick brown fox jumps over the lazy dog."
tag_sents function takes a list of list of strings.
tagger.tag_sents(word_tokenize("The quick brown fox jumps over the lazy dog."))
Here's a useful idiom:
tagger.tag_sents(word_tokenize(sent) for sent in sent_tokenize(text))
text is a string.