ChrisB - 1 year ago 72

Python Question

From the pandas documentation, I've gathered that unique-valued indices make certain operations efficient, and that non-unique indices are occasionally tolerated.

From the outside, it doesn't look like non-unique indices are taken advantage of in any way. For example, the following

`ix`

`In [23]: import numpy as np`

In [24]: import pandas as pd

In [25]: x = np.random.randint(0, 10**7, 10**7)

In [26]: df1 = pd.DataFrame({'x':x})

In [27]: df2 = df1.set_index('x', drop=False)

In [28]: %timeit df2.ix[0]

1 loops, best of 3: 402 ms per loop

In [29]: %timeit df1.ix[0]

10000 loops, best of 3: 123 us per loop

(I realize the two

`ix`

`ix`

Is there any way to coax pandas into using faster lookup methods like binary search on non-unique and/or sorted indices?

Recommended for you: Get network issues from **WhatsUp Gold**. **Not end users.**

Answer Source

When index is unique, pandas use a hashtable to map key to value O(1). When index is non-unique and sorted, pandas use binary search O(logN), when index is random ordered pandas need to check all the keys in the index O(N).

You can call `sort_index`

method:

```
import numpy as np
import pandas as pd
x = np.random.randint(0, 200, 10**6)
df1 = pd.DataFrame({'x':x})
df2 = df1.set_index('x', drop=False)
df3 = df2.sort_index()
%timeit df1.loc[100]
%timeit df2.loc[100]
%timeit df3.loc[100]
```

result:

```
10000 loops, best of 3: 71.2 µs per loop
10 loops, best of 3: 38.9 ms per loop
10000 loops, best of 3: 134 µs per loop
```

Recommended from our users: **Dynamic Network Monitoring from WhatsUp Gold from IPSwitch**. ** Free Download**