Edward Edward - 2 months ago 28
Python Question

How to extract nouns from dataframe

I want to extract nouns from dataframe. Only nouns.
I do as below

import pandas as pd
import nltk
from nltk.tag import pos_tag
from nltk import word_tokenize
df = pd.DataFrame({'noun': ['good day', 'good night']})


I want to get

noun
0 day
1 night


My code

df['noun'] = df.apply(lambda row: nltk.word_tokenize(row['noun']), axis=1)
noun=[]
for index, row in df.iterrows():
noun.append([word for word,pos in pos_tag(row) if pos == 'NN'])
df['noun'] = noun



---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-194-688cfbb21ec5> in <module>()
1 noun=[]
2 for index, row in df.iterrows():
----> 3 noun.append([word for word,pos in pos_tag(row) if pos == 'NN'])
4 df['noun'] = noun

C:\Users\Edward\Anaconda3\lib\site-packages\nltk\tag\__init__.py in pos_tag(tokens, tagset)
109 """
110 tagger = PerceptronTagger()
--> 111 return _pos_tag(tokens, tagset, tagger)
112
113

C:\Users\Edward\Anaconda3\lib\site-packages\nltk\tag\__init__.py in _pos_tag(tokens, tagset, tagger)
80
81 def _pos_tag(tokens, tagset, tagger):
---> 82 tagged_tokens = tagger.tag(tokens)
83 if tagset:
84 tagged_tokens = [(token, map_tag('en-ptb', tagset, tag)) for (token, tag) in tagged_tokens]

C:\Users\Edward\Anaconda3\lib\site-packages\nltk\tag\perceptron.py in tag(self, tokens)
150 output = []
151
--> 152 context = self.START + [self.normalize(w) for w in tokens] + self.END
153 for i, word in enumerate(tokens):
154 tag = self.tagdict.get(word)

C:\Users\Edward\Anaconda3\lib\site-packages\nltk\tag\perceptron.py in <listcomp>(.0)
150 output = []
151
--> 152 context = self.START + [self.normalize(w) for w in tokens] + self.END
153 for i, word in enumerate(tokens):
154 tag = self.tagdict.get(word)

C:\Users\Edward\Anaconda3\lib\site-packages\nltk\tag\perceptron.py in normalize(self, word)
222 if '-' in word and word[0] != '-':
223 return '!HYPHEN'
--> 224 elif word.isdigit() and len(word) == 4:
225 return '!YEAR'
226 elif word[0].isdigit():

AttributeError: 'list' object has no attribute 'isdigit'


Please, help, how to improve it?
* Sorry, i have ro write some text so that i can insert all traceback
I guess thr problem is that i cann't convert list to needed format?

Answer

The problem is that in your loop, row is a pandas Series rather than a list. You can access the list of words by writing row[0] instead:

>>> for  index, row in df.iterrows():
>>>     noun.append([word for word,pos in pos_tag(row[0]) if pos == 'NN'])
>>> print(noun)
[['day'], ['night']]

Here you're getting a list of lists, with each list containing the nouns from one sentence. If you really want a flat list (as your question says), write noun.extend(...) instead of noun.append.

Comments