When running profile_report in ydata_profiling (used to be pandas_profiling) how do I fix the 'to_pydatetime' error?

Question

I am trying to profile_report in ydata_profiling and I get the following error:

TypeError: descriptor 'to_pydatetime' for 'pandas._libs.tslibs.timestamps._Timestamp' objects doesn't apply to a 'datetime.date' object.

I figured that the pandas data frame I am using had some date fields and indeed those cause this error. I used print(df['effective_date'].apply(type)) and I got datetime.date as the data type in this field. I tried to convert it to the timestamp data type but I faced some challenges.

Why does the data type need to be timestamp? The to_pydatetime seems to convert timestamp into datetime which my data already is. Is there a way around this to opt out from profile_report trying to convert my data into something it already is?

could you provide some example code that reproduces the problem? — FObersteiner
– FObersteiner, Commented Aug 19, 2023 at 13:21

jarson · Accepted Answer · 2023-12-28 23:53:43Z

0

I managed to get around this by forcing datetime64 for any columns where the source database (snowflake in my case) had the data stored as DATE or TIMESTAMP (any variant).

However, I'm running into the same error when I try against a different database (oracle), even after trying to force all columns to datetime64. Tried using errors='coerce' and 'ignore' in pd.to_datetime().

Debugging is made difficult because of the way the ydata-profiling library works. It does not start the computations that result in the error until you actually try to write the profile report file. When a break is encountered, the debugger doesn't know to go into the profiler code and instead keeps dropping me back to the line where the file creation is occurring.

answered Dec 28, 2023 at 23:53

jarson

1

Sign up to request clarification or add additional context in comments.

Comments

Amarpreet Singh · Accepted Answer · 2025-02-07 10:45:27Z

0

The problem arises when your date columns are not in datetime64[ns] format and your non-datetime columns are in datetime64[ns] format. So make sure you correct your columns data types before passing it into the ydata_profiling module:

for i in ['effective_date']:
    df[i] = pd.to_datetime(df[i])

answered Feb 7 at 10:45

Amarpreet Singh

2774 silver badges13 bronze badges

Collectives™ on Stack Overflow

When running profile_report in ydata_profiling (used to be pandas_profiling) how do I fix the 'to_pydatetime' error?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related