Pfthb

I've recently been trying to train some models on an AWS SageMaker jupyter notebook instance.

Everything is worked very well until I tried to load in some custom dataset (REDD) through files.

I have the dataframes stored in Pickle (.pkl) files on an S3 bucket. I couldn't manage to read them into sagemaker so I decided to convert them to csv's as this seemed to work but I ran into a problem. This data has an index of type datetime64 and when using .to_csv() this index gets converted to pure text and it loses it's data structure (and I need to keep this specific index for correct plotting.)

So I decided to try the Pickle files again but I can't get it to work and have no idea why.

The following code for csv's works but I can't use it due to the index problem:

bucket = 'sagemaker-peno'
houses_dfs = 
data_key = 'compressed_data/'
data_location = 's3:///'.format(bucket, data_key)
for file in range(6):
 houses_dfs[file+1] = pd.read_csv(data_location+'house_'+str(file+1)+'.csv', index_col='Unnamed: 0')

But this code does NOT work even though it uses almost the exact same syntax:

bucket = 'sagemaker-peno'
houses_dfs = 
data_key = 'compressed_data/'
data_location = 's3:///'.format(bucket, data_key)
for file in range(6):
 houses_dfs[file+1] = pd.read_pickle(data_location+'house_'+str(file+1)+'.pkl')

Yes it's 100% the correct path, because the csv and pkl files are stored in the same directory (compressed_data).

It throws me this error while using the Pickle method:

FileNotFoundError: [Errno 2] No such file or directory: 's3://sagemaker-peno/compressed_data/house_1.pkl'

I hope to find someone who has dealt with this before and can solve the read_pickle() issue or as an alternative fix my datetime64 type issue with csv's.

Thanks in advance!

read_pickle() likes the full path more than a relative path from where it was run. This fixed my issue.