Pfthb

I have a pandas dataframe with a categorical column containing NaN values, e.g.:

g = pd.Series(["A", "B", "C", np.nan], dtype="category")
g

0 A
1 B
2 C
3 NaN
dtype: category
Categories (3, object): [A, B, C]

In pandas NaN is not a category but you can have NaN values in your categorical data. I want to pass this dataframe through to R using %%R in a Jupyter notebook . The categorical column is successfully recognised by R as a factor, but the factor is malformed, presumably because of the Nan values:

%%R -i g
str(g)
Factor w/ 3 levels "A","B","C": 1 2 3 0
 - attr(*, "names")= chr [1:4] "0" "1" "2" "3" 

print(g)
Error in as.character.factor(x) : malformed factor

Is there any way to make sure that the factor is not malformed - e.g. to have an
NA factor level created automatically?

R: 3.5.1, rpy2: 2.9.4, Python - 3

At the time of writing this is a bug with rpy2's conversion of pandas categories, that is fixed and will be included in rpy2 starting with release 2.9.5: https://bitbucket.org/rpy2/rpy2/issues/493/rpy2-conversion-of-categorical-data

A workaround is rather trivial: don't use a NaN in a pandas Category.

g = pd.Series(["A", "B", "C", np.nan], dtype="category")
# Prepare alternative representation to pass it to R
g_r = g.replace(np.nan, 'Missing')

When converting it is now looking like:

%%R -i g_r
str(g_r)

Factor w/ 4 levels "A","B","C","Missing": 1 2 3 4
- attr(*, "names")= chr [1:4] "0" "1" "2" "3"

Translating back into an R NA is only a matter of dropping that added level:

%%R -i g_r
str(droplevels(g_r, exclude = "Missing")) 

Factor w/ 3 levels "A","B","C": 1 2 3 NA
- attr(*, "names")= chr [1:4] "0" "1" "2" "3"