Pfthb

I am facing a simple error with the code below.

My objective is to use simpleimputer to plug missing values of different datatypes in one shot.

When i try to do that, the fit_transform seems to be not work as expected.
When dtype argument is not used, the code works just fine, but the resulting dataframe loses its data type information. When i include the dtype list in the arguments, i am seeing the below error. You should be able to simulate the error by just copying and pasting in your local.

import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer

import sklearn
print(sklearn.__version__)

0.21.dev0

data = [['Alex','NJ',21,5.10],['Mary','NY',20,np.nan],['Sam',np.nan,np.nan,6.3]]
df = pd.DataFrame(data,columns=['Name','State','Age','Height'])

df.dtypes
Name object
State object
Age float64
Height float64
dtype: object 

imp = SimpleImputer(strategy="most_frequent")

#df = pd.DataFrame(imp.fit_transform(df),columns=df.columns) <<<<----- This works just fine
#df
#Name State Age Height
#0 Alex NJ 21 5.1
#1 Mary NY 20 5.1
#2 Sam NJ 20 6.3
#df.dtypes
#Name object
#State object
#Age object
#Height object
#dtype: object

The below statement fails - with the error listed below ( I am trying to preserve dtypes during imputing process)

df = pd.DataFrame(imp.fit_transform(df),columns=df.columns,dtype=df.dtypes)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-23-e9780979921f> in <module>()
 7 
 8 imp = SimpleImputer(strategy="most_frequent")
----> 9 df = pd.DataFrame(imp.fit_transform(df),columns=df.columns,dtype=df.dtypes)

~AppDataLocalContinuumanaconda3libsite-packagespandascoreframe.py in __init__(self, data, index, columns, dtype, copy)
 337 data = 
 338 if dtype is not None:
--> 339 dtype = self._validate_dtype(dtype)
 340 
 341 if isinstance(data, DataFrame):

~AppDataLocalContinuumanaconda3libsite-packagespandascoregeneric.py in _validate_dtype(self, dtype)
 166 
 167 if dtype is not None:
--> 168 dtype = pandas_dtype(dtype)
 169 
 170 # a compound dtype

~AppDataLocalContinuumanaconda3libsite-packagespandascoredtypescommon.py in pandas_dtype(dtype)
 2020 # which we safeguard against by catching them earlier and returning
 2021 # np.dtype(valid_dtype) before this condition is evaluated.
-> 2022 if dtype in [object, np.object_, 'object', 'O']:
 2023 return npdtype
 2024 elif npdtype.kind == 'O':

~AppDataLocalContinuumanaconda3libsite-packagespandascoregeneric.py in __nonzero__(self)
 1574 raise ValueError("The truth value of a 0 is ambiguous. "
 1575 "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
-> 1576 .format(self.__class__.__name__))
 1577 
 1578 __bool__ = __nonzero__

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

If you want to preserve the dtype, I recommend using pandas to find the mode and then call fillna:

df = df.fillna(df.agg(lambda x: pd.Series.mode(x)[0], axis=0))
print(df)
 Name State Age Height
0 Alex NJ 21.0 5.1
1 Mary NY 20.0 5.1
2 Sam NJ 20.0 6.3

print(df.dtypes)
Name object
State object
Age float64
Height float64
dtype: object

Alternatively, use astype and pass a dictionary:

df = pd.DataFrame(
 imp.fit_transform(df), columns=df.columns
).astype(df.dtypes.to_dict())

print(df)
 Name State Age Height
0 Alex NJ 21.0 5.1
1 Mary NY 20.0 5.1
2 Sam NJ 20.0 6.3

print(df.dtypes)
Name object
State object
Age float64
Height float64
dtype: object

Explicit astype call is needed because, as per the documentation, only a single dtype can be passed to the pd.DataFrame constructor.

?pd.DataFrame
...
dtype : dtype, default None
 | Data type to force. Only a single dtype is allowed.