Pandas and scikit-learn: KeyError: [....] not in index

Question

I do not understand why do I get the error KeyError: '[ 1351 1352 1353 ... 13500 13501 13502] not in index' when I run this code:

cv = KFold(n_splits=10)

for train_index, test_index in cv.split(X):
    f_train_X, f_valid_X = X[train_index], X[test_index]
    f_train_y, f_valid_y = y[train_index], y[test_index]

I use X (a Pandas dataframe) to split I cv.split(X).

X.shape
y.shape
Out: (13503, 17)
Out: (13503,)

seralouk · Accepted Answer · 2019-01-22 15:05:59Z

29

The problem is the way you are trying to index the X using X[train_index]. You need to use .loc or .iloc since you have pandas dataframe.

Use this

cv = KFold(n_splits=10)

for train_index, test_index in cv.split(X):
    f_train_X, f_valid_X = X.iloc[train_index], X.iloc[test_index]
    f_train_y, f_valid_y = y.iloc[train_index], y.iloc[test_index]

1st way: Example using `iloc`

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))

df[[1,2]]
#KeyError: '[1 2] not in index'

df.iloc[[1,2]]
#    A   B   C   D
#1  25  97  78  74
#2   6  84  16  21

2nd way: Example by converting pandas to numpy in advance

df = df.values

#now this should work fine
df[[1,2]]
#array([[25, 97, 78, 74],
#      [ 6, 84, 16, 21]])

edited Jan 22, 2019 at 15:05

answered Jun 28, 2018 at 21:02

seralouk

33.6k10 gold badges127 silver badges141 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ScalaBoy Over a year ago

Thank you so much! Does it mean that X[train_index] requires numpy array?

seralouk Over a year ago

The default indexing of scikit-learn functions are expecting numpy arrays. Use .iloc or use X = X.values in advance. Please consider accepting my answer

Collectives™ on Stack Overflow

Pandas and scikit-learn: KeyError: [....] not in index

1 Answer 1

Use this

1st way: Example using `iloc`

2nd way: Example by converting pandas to numpy in advance

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Use this

1st way: Example using iloc

2nd way: Example by converting pandas to numpy in advance

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related

1st way: Example using `iloc`