Thomas Matthew Thomas Matthew - 6 months ago 35
Python Question

T-Test on one DataFrame with Groups from another DataFrame

The Goal:

Perform t-test on DataFrame (df_rna) using group found in another DataFrame (df_cnv). Reduce the test DataFrame (df_rna) the row indices with the most significant scores from the t-test.

Code Sample:

# Dataframe (df_cnv) that forms groups of columns (cells) either\ belonging to True or False for t-test.
cnv = {'gene': ['x','y','z','n'],
'cell_a': [0,-1,0,-1],
'cell_b': [0,-1,-1,-1],
'cell_c': [-1,0,-1,0],
'cell_d': [-1,0,-1,0],
'cell_e': [-1,0,0,0]
}
df_cnv = pd.DataFrame(cnv)
df_cnv.set_index('gene', inplace=True)
cnv_mask = df_cnv < 0
cnv_mask # True values are negative (gene loss is True)


cnv_mask masked for negative values

# DataFrame for t-test and subsequent reduction to most significant rows
rna = {'gene': ['x','y','z','n'],
'cell_a': [1, 5, 8,9],
'cell_b': [8, 5, 4,9],
'cell_c': [8, 6, 1,1],
'cell_d': [1, 2, 7,1],
'cell_e': [5, 7, 9,1],
}
df_rna = pd.DataFrame(rna)
df_rna.set_index('gene')


df_rna

# Manually computed T-Tests, save results in DataFrame df_report
x = scipy.stats.ttest_ind([8,1,5],[1,8])
y = scipy.stats.ttest_ind([5,5], [6,2,7])
z = scipy.stats.ttest_ind([4,1,7], [8,9])
n = scipy.stats.ttest_ind([9,9], [1,1,1])

tStat = [gene.statistic for gene in [x,y,z,n]]
pVal = [gene.pvalue for gene in [x,y,z,n]]

report = {'gene':['x','y','z','n'],
't_stat':tStat,
'p_val':pVal}
df_report = pd.DataFrame(report)
df_report.set_index('gene', inplace=True)


df_report from t-test

# Create reduced version of test DataFrame (df_rna) to contain only rows (genes
df_pass = df_report.loc[df_report['p_val'] < 0.05]
passed_genes = set(df_pass.index)
passed_genes

df_rna_pass = df_rna.loc[df_rna['gene'].isin(passed_genes)]
df_rna_pass.set_index('gene')


df_rna_pass reduced df_rna DataFrame with only rows with p_val < 0.05

The Question:

Manually setting the t-test groups is not feasible for my large dataset. How do I compute all the t-test statistics across the whole DataFrame
df_rna
when the groups of cells being either True and False changes for every row?

Mystery Hang: (happens if your don't cache the results of
rnadf_all[~cnv_mask]
)

C:\Users\test\Anaconda3\lib\site-packages\numpy\core\_methods.py:82: RuntimeWarning: Degrees of freedom <= 0 for slice
warnings.warn("Degrees of freedom <= 0 for slice", RuntimeWarning)
---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
<ipython-input-16-ccabe33b2612> in <module>()
34
35 for r in rnadf_all[cnv_mask].iterrows():
---> 36 df_report.at[r[0], 't_stat'], df_report.at[r[0], 'p_val'] = scipy.stats.ttest_ind(r[1].dropna(), rnadf_all[~cnv_mask].loc[r[0]].dropna())
37
38 df_pass = df_report.loc[df_report['p_val'] < 0.05]

C:\Users\test\Anaconda3\lib\site-packages\pandas\core\frame.py in __getitem__(self, key)
1963 return self._getitem_array(key)
1964 elif isinstance(key, DataFrame):
-> 1965 return self._getitem_frame(key)
1966 elif is_mi_columns:
1967 return self._getitem_multilevel(key)

C:\Users\test\Anaconda3\lib\site-packages\pandas\core\frame.py in _getitem_frame(self, key)
2036 if key.values.dtype != np.bool_:
2037 raise ValueError('Must pass DataFrame with boolean values only')
-> 2038 return self.where(key)
2039
2040 def query(self, expr, **kwargs):

C:\Users\test\Anaconda3\lib\site-packages\pandas\core\generic.py in where(self, cond, other, inplace, axis, level, try_cast, raise_on_error)
3931 # try to align
3932 try_quick = True
-> 3933 if hasattr(other, 'align'):
3934
3935 # align with me

KeyboardInterrupt:

Answer
 from scipy import stats

 # Create empty DF for t-test results
 df_report = pd.DataFrame(index=df_rna.index, columns=['p_val', 't_stat'])

 not_df_rna = df_rna[~cnv_mask]

 # Iterate through df_rna rows, apply mask, drop NaN values, run ttest_ind and save result to df_report
 for r in df_rna[cnv_mask].iterrows():
     df_report.at[r[0], 't_stat'], df_report.at[r[0], 'p_val'] = stats.ttest_ind(r[1].dropna(), not_df_rna.loc[r[0]].dropna())

Result:

df_report

         p_val     t_stat
gene                     
x     0.966863  0.0450988
y            1          0
z     0.141358   -1.98508
n            0        inf
Comments