Snoozer Snoozer - 7 months ago 56
Python Question

How can I create new rows in a pandas data frame containing the words in a string of an existing row?

I have a

with a column called
with strings of text. I would like to get the individual words of those strings on their own rows with identical values for the other columns. For example if I have 3 strings (and an unrelated column, Time):

Strings --- Time
"The dog" --- 4Pm
"lazy dog"--- 2Pm
"The fox" --- 1Pm

I want new rows containing the words from the string, but with otherwise identical columns

Strings --- Words ---Time
"The dog" --- "The" --- 4Pm
"The dog" --- "dog" --- 4Pm
"lazy dog"--- "lazy"--- 2Pm
"lazy dog"--- "dog" --- 2Pm
"The fox" --- "The" --- 1Pm
"The fox" --- "fox" --- 1Pm

I know how to split the words up from the strings:

string_list = '\n'.join(
word_list = re.findall('[a-z]+', Strings)

But how can I get these into the dataframe while preserving the index & other variables? I'm using Python 2.7 and pandas 0.10.1.

I now understand how to expand rows using groupby found in this question:

def f(group):
row = group.irow(0)
return DataFrame({'words': re.findall('[a-z]+',row['Strings'])})
df.groupby('class', group_keys=False).apply(f)

I would still like to preserve the other columns. Is this possible?


Here is my code that doesn't use groupby(), I think it's faster.

import pandas as pd
import numpy as np
import itertools

df = pd.DataFrame({
"strings":["the dog", "lazy dog", "The fox jump"], 

w = df.strings.str.split()
c =
idx = np.repeat(c.index, c.values)
#words = np.concatenate(w.values)
words = list(itertools.chain.from_iterable(w.values))
s = pd.Series(words, index=idx) = "words"
print df.join(s)

Thre result:

        strings value words
0       the dog     a   the
0       the dog     a   dog
1      lazy dog     b  lazy
1      lazy dog     b   dog
2  The fox jump     c   The
2  The fox jump     c   fox
2  The fox jump     c  jump