Huey Huey - 3 months ago 41
Python Question

Pandas still getting SettingWithCopyWarning even after using .loc

At first, I tried writing some code that looked like this:

import numpy as np
import pandas as pd
np.random.seed(2016)
train = pd.DataFrame(np.random.choice([np.nan, 1, 2], size=(10, 3)),
columns=['Age', 'SibSp', 'Parch'])

complete = train.dropna()
complete['AgeGt15'] = complete['Age'] > 15


After getting SettingWithCopyWarning, I tried using.loc:

complete.loc[:, 'AgeGt15'] = complete['Age'] > 15
complete.loc[:, 'WithFamily'] = complete['SibSp'] + complete['Parch'] > 0


However, I still get the same warning. What gives?

Answer

When complete = train.dropna() is executed, dropna might return a copy, so out of an abundance of caution, Pandas sets complete.is_copy to a Truthy value:

In [220]: complete.is_copy
Out[220]: <weakref at 0x7f7f0b295b38; to 'DataFrame' at 0x7f7eee6fe668>

This allows Pandas to warn you later, when complete['AgeGt15'] = complete['Age'] > 15 is executed that you may be modifying a copy which will have no effect on train. For beginners this may be a useful warning. In your case, it appears you have no intention of modifying train indirectly by modifying complete. Therefore the warning is just a meaningless annoyance in your case.

You can silence the warning by setting

complete.is_copy = False

This is quicker than making an actual copy, and nips the SettingWithCopyWarning in the bud (at the point where _check_setitem_copy is called):

def _check_setitem_copy(self, stacklevel=4, t='setting', force=False):
    if force or self.is_copy:
        ...

If you are really confident you know what you are doing, you can shut off the SettingWithCopyWarning globally with

pd.options.mode.chained_assignment = None # None|'warn'|'raise'
Comments