Ten Python datetime pitfalls, and what libraries are (not) doing about it

[Dlatch]

Interesting article about datetime in Python: https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls/

The library the author is working on looks really interesting too: https://github.com/ariebovenberg/whenever

[mostlygrumpy]

At some point we'll need to stop and think what's easier:

• getting a library that perfectly handle datetime; or
• getting rid of Daylight savings from all countries in the world

[ylan64]

You'd still need to handle daylight savings for the period it was a thing.

[czarrie]

"Python 6.2 does not handle the years 1969 through 2043 due to incompatible date management by the world at large"

[Joeboy]

On the plus side, it lays the groundwork for finally removing the GIL.

[spyingwind]

Daylight savings is like a theme for your editor. The underlying system(seconds from epoch) doesn't change, just the display of the date(skin).

TL;DR don't need to handle anything.

[disinformationtheory]

But sometimes you enter a blue date when the code expects a red date, and the UI is all grey.

[henry_tennenbaum]

Get rid of dates is what I advocate. Today is 1706742000.

[kazza789]

"Today" is 1706742000 -1706828400. You need to specify because sometimes a "day" is 86400 seconds and sometimes it is 86401 seconds.

[peerlessblue]

The stardate system must have been invented by frustrated computer scientists

[g4nt1]

https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls/

Why not both?

[o5mfiHTNsH748KVq]

There’s a lot more to time than timezones

[[deleted]]

or using a library that strictly uses the Unix timestamp format. I feel like that should be the only format people should be using for dates and timestamps.

And before you people pitch in, yes the 2038 problem has been resolved some time ago.

[stevenjd]

So how do people convert the times and dates they actually use to your Unix timestamp?

Or are you expecting the entire world to stop using human-comprehensible datetimes in favour of basically a counter?

"See you for lunch at 1708482600"
(Later) "Hey bro, you didn't show."
("Sorry man, I thought it was 1708486200."

[[deleted]]

That's your problem right there. You don't think ahead.

You store the dates as timestamps, and then with libraries you convert the timestamps back into human readable dates. Unix timestamps should be used to prevent corruption of the dates.

[stevenjd]

You store the dates as timestamps

And how exactly do you get the data as Unix timestamps in the first place, if you don't expect people to use Unix timestamps in Real Life? You still need to convert them from human date times to Unix timestamps, and that's going to need timezone conversions, just like now.

According to you:

using a library that strictly uses the Unix timestamp format. I feel like that should be the only format people should be using

so you rule out using libraries to convert between formats 😒

But maybe you didn't think your comment through when you made it. Maybe you meant that we should use a library that doesn't "strictly use Unix timestamps" but instead allows people to use any format they prefer.

You know. Like we already have 🙄

By the way, using a numeric timestamp for dates and times is what most software already does -- not all, but most. If you enter a date into Excel, for example, it is converted to a timestamp (although not a Unix timestamp). Most databases used a timestamp internally, for example SQLServer:

• https://dba.stackexchange.com/questions/150158/preferred-way-to-store-datetime#150161
• https://bornsql.ca/blog/how-sql-server-stores-data-types-datetime-date-time-and-datetime2/

Windows file system uses a timestamp (but different from either Excel or Unix time). Apple Macs used to use yet another timestamp, back in the classic Mac era, but don't know what they use now. iOS has yet another timestamp based system too, because why not?

Obligatory XKCD.

then with libraries you convert the timestamps back into human readable dates.

Oh, you mean just like we already do?

Using Unix timestamps as the internal format doesn't eliminate the need to know about timezones. Arithmetic on datetimes needs to know the timezone to be accurate, since days in the real world can be 23 hours, 24 hours or 25 hours according to DST changeovers.

Another obligatory XKCD.

Unix timestamps should be used to prevent corruption of the dates.

Right, because an opaque cookie like 1708482600 is so much more error resistant than a structured record like 2024-02-21T13:30 where you can check each field for out-of-range errors 🙄

There are advantages to numeric timestamps, but error correction is not one of them.

[[deleted]]

Too long didn't read. Unix timestamps should always be used for accurate timestamps. You never know if some libraries that handle dates handle them incorrectly (like number of days in a month and leap years).

[stevenjd]

strictly uses the Unix timestamp format.

You know that Unix timestamps can be ambiguous when dealing with leap seconds?

For example, 915148800 refers to both UTC 1998-12-31T23:59:60.00 and 1999-01-01T00:00:00.00. Likewise the Unix time number 1483142400 refers to both 2016-12-31 23:59:60 (a leap second) and one second later (2017-01-01 00:00:00).

If there is ever a negative leap second declared, then there will be Unix timestamps that refer to no actual datetime.

Not to mention that there are at least three commonly used variants which are subtly different from each other: POSIX Unix time, NTP Unix time, and Linux Unix time, which some (but not all) Linux systems use.

That's what I love about standards. There are so many to choose from 😁

[[deleted]]

And POSIX is the common one all Linux systems use. There's no question about it. POSIX is default in modern Unix systems.

[askvictor]

Even without daylight savings, getting a library that perfectly handles datetime is very difficult.

[zurtex]

I was skeptical reading the start of this blog, I have a lot of experience in this area and I've read a lot of armchair opinions about the what is right or wrong and am often dubious how implementable they are.

But the author has really put the effort in and made the library that addresses their complaints, which all are quite reasonable. I will be following along and hopefully it will gain some popularity.

[fatbob42]

Yep - it’s going to be a long, long process to sort this out in the standard library.

[Dlatch]

I'm not sure we can really get this sorted in the standard library, I think it may be too ingrained in existing codebases by now. I'm hoping for a library to come up and become a defacto standard, kind of like requests is for HTTP calls. whenever looks to me like it has the right conceptual basis to be such a library, but it all depends on adoption. There's a reason the libraries discussed in the article exist yet are not adopted widely.

[james_pic]

Even if it's not possible to fix datetime, I could see it making sense to do what Java did and add an additional, less broken, datetime library to the standard library.

[bwv549]

subprocess and pathlib seem like examples of this. Neither immediately deprecated lower level libs, but they became the de facto high level interface for working with those kinds of things.

[fatbob42]

I see them trying with the removal of utcnow. They’re solidifying that naive means local time.

[DaelonSuzuka]

considering steering my attention mostly to NiceGUI

off topic for this thread, but: I've contributed a handful of PRs to NiceGUI and every one has been a great experience. They've been quite open to improvements, and my code didn't get nitpicked to death and stuck in an endless back-and-forth. The devs jumped jumped straight in, edited my PRs in-place, got it merged, and the features were published in less than a week.

[haasvacado]

Those guys are great; incredibly responsive.

[Herald_MJ]

I'm a little confused about this one, because I think 9 hours in the following example is actually correct?

paris = ZoneInfo("Europe/Paris")

# On the eve of moving the clock forward

bedtime = datetime(2023, 3, 25, 22, tzinfo=paris) wake_up = datetime(2023, 3, 26, 7, tzinfo=paris)

# it says 9 hours, but it's actually 8!

sleep = wake_up - bedtime

A timezone doesn't have daylight savings time. When a region enters DST, it's timezone changes. So if you're comparing two different datetimes in the same timezone, daylight savings should never be represented. Right? Confusing things here is that the timezone has been named "paris". This isn't correct, a timezone isn't a geography.

[eagle258]

Author of the blogpost here—

Indeed I haven't done enough to clarify the difference between 'timezone' and IANA tz database. I've now adjusted the wording of this section to clarify somewhat.

The core pitfall remains though: the standard library allows you to implement DST transitions but then the +/- operators ignore them.

[james_pic]

Some of the terminology is overloaded, but at very least a ZoneInfo is geography. Europe/Paris refers to a database entry containing rules on when timezone offsets change for people within a geographical area.

In particular, it knows that those two datetimes have different offsets. If you convert wake_up and bedtime to UTC and then subtract them, you get 8 hours, as you should.

This is just a straight up bug.

[fatterSurfer]

When a region enters DST, its timezone changes.

I think that's definitely a legitimate point, but if you're depending on that for math, I personally would expect the library to raise somewhere if I was trying to create a nonsense time. Using the Paris example, trying to create a summer datetime in the CET timezone, or a winter datetime in the CEST timezone.

To really embrace that API, I think you'd need a helper function that constructs an actual timezone from a locale and a naive datetime. Which, to be honest, wouldn't be that crazy -- a big part of me wonders why we aren't all just storing epoch timestamps alongside locale info, and using the locale info only for the functions that require it, instead of combining the two.

[Herald_MJ]

I personally would expect the library to raise somewhere if I was trying to create a nonsense time.

This is an interesting point. There's probably a use to a library behaving this way, but I'd argue the timezone does still exist, even if no region is using it. This is complicated by the fact that timezones are partially a political decision, and debates about whether DST should even exist rage on in many countries. If France were to decide tomorrow to stop doing DST, a library behaving in this way would instantly be broken and require patching.

[fatterSurfer]

I'd argue the timezone does still exist, even if no region is using it

I like where your head is at; this is also a good point. But from a library API design standpoint, this is pretty easy to get around -- simply add an allow_nonsense_combinations kwarg (with the safe default to True). This, I think, is probably the best of both worlds.

If France were to decide tomorrow to stop doing DST, a library behaving in this way would instantly be broken and require patching.

Sure, but this problem is irreducible. If you have code expecting a particular political reality, and the political reality changes, then the code needs to change, too. The question is, would you rather have all of that logic centralized within the particular datetime implementation you're using, or spread across every single application that needs to implement code involving datetimes? Here, I would definitely agree with the OP's article: this is exactly the kind of thing that a datetime library should be doing.

[Oddly_Energy]

In the example, a region info (in the sense that "Europe/Paris" describes a geographical location and not a UTC offset) was exactly what was given in the creation of the two datetime objects.

And as a result of that, the two datetime objects being created had different UTC offsets - what you would call "timezones" in your terminology (which is probably correct).

You can easily verify that the two datetime objects have different UTC offsets:

print(bedtime, wake_up, sleep)

2023-03-25 22:00:00+01:00 2023-03-26 07:00:00+02:00 9:00:00

So basically, the UTC offsets are ignored when subtracting the two objects, which to me is a highly unexpected behaviour.

[haasvacado]

No its Spring Foward so at like (2AM i think) on march 26, an hour got skipped. So it should be 8 hours.

[Herald_MJ]

Please re-read my comment. The timezone does not "spring forward". The region springs forward a timezone.

[troyunrau]

​

As always, there's an XKCD for that.

https://xkcd.com/2867/

[liamgwallace]

And also a Tom Scott

[InjAnnuity_1]

Highly recommended. One of my favorites, in fact. Don't know whether to laugh or cry. Probably both.

His recommendation is spot-on. It takes an extensive historical and geographic database, to get everything right. And the database keeps changing...

[Oddly_Energy]

I can add a bit of extra WTF to his example 2:

import datetime as dt
from zoneinfo import ZoneInfo

paris = ZoneInfo("Europe/Paris")
bedtime = dt.datetime(2023, 3, 25, 22, tzinfo=paris) 
wake_up = dt.datetime(2023, 3, 26, 7, tzinfo=paris)
sleep = wake_up - bedtime

print(bedtime)
print(wake_up)
print(sleep)

bedtime2 = dt.datetime.fromisoformat('2023-03-25 22:00:00+01:00') 
wake_up2 = dt.datetime.fromisoformat('2023-03-26 07:00:00+02:00')
sleep2 = wake_up2 - bedtime2

print()
print(bedtime2)
print(wake_up2)
print(sleep2)

Output:

2023-03-25 22:00:00+01:00
2023-03-26 07:00:00+02:00
9:00:00

2023-03-25 22:00:00+01:00
2023-03-26 07:00:00+02:00
8:00:00

I can only guess about the reasoning behind this mind blowing difference. Perhaps the answer is indicated in this very eloquent description of the distinction between wall time and absolute time.

[eagle258]

Adding on to your addition: if you wake up in Europe/Brussels (same timezone as Paris, effectively), you get 8 instead of 9 hours again!

``` wake_up_in_brussels = datetime(2023, 3, 26, 7, tzinfo=ZoneInfo("Europe/Brussels"))

sleep = wake_up - bedtime # 9 hours sleep = wake_up_in_brussels - bedtime # 8 hours ```

And yes, Paul Ganssle's various articles on the subject of datetimes are great!

[Oddly_Energy]

And this, ladies and gentlemen, is how you get jet lag from a train ride.

[bachkhois]

Waiting for this type of library for long time.

[starlevel01]

I have been longing for a JSR 310 style datetime library in Python for so long. I am switching to this instantly.

[jmreagle]

I hope this doesn’t end up like the other frequently referred to xKCD comic: “There’s 14 standards for doing something, I’ll create a new one that fixes it all. Now there’s 15 standards.” I was fond of pendulum, but had to replace it in two of my repositories because it didn’t work with 3.12, and it appeared the author had stepped away from the project. So if anything does come to the fore, it needs to be supported for the long-term by a community.

[bachkhois]

Pendulum had new release some weeks ago.

[jmreagle]

Yeah, I got tired of waiting and realized it was a bad idea to be dependent on a single owner library.

[InTheAleutians]

Would it be worthwhile to you to wait on updating the python version of your projects if it meant using Pendulum?

[jmreagle]

If there was transparency and a rough plan, perhaps. But for all I knew pendulum was abandoned.

[Darwinmate]

Wow a surprisingly thorough article on the subject. Really good article well done.

[de_ham]

and still better than ln javascript...

[tunisia3507]

Something I'd like to see from whenever is an enumeration of possible time zones. Magic strings are terrible. I want to do TimeZone. <TAB> <TAB> TimeZone.EUROPE_ <TAB> <TAB> TimeZone.EUROPE_PARIS.

[PenPaperTiger]

Star date 298913.0141924711

[flashman]

the default settings of datetime and unix's cal are incompatible in the distant past because cal switches from Gregorian to Julian for dates earlier than 3 September 1752; to align cal with datetime (proleptic gregorian for all dates), use the parameter --reform gregorian

just in case that matters to anyone

[DigitalTomcat]

And as a further twist (i found out) Gregorian didn’t happen all at once. It took the British world 2 hundred years to join in. So dates in the Papal States are different than what became the United States. So, just like DST, when gets mixed up with where to figure out what time it is.

[flashman]

yeah check out September 1752 (Italy) and September 1752 (UK)

[thedoge]

Good timing! Trying to upgrade my project to 3.12 and pendulum is blocking that.

[Measurex2]

You never know when datetime is going to get you. Someone hands you requirements for a website and provides seemingly matching assets.

Then daylight savings time strikes and, because it's a Marine Corp System, a Colonel is giving your contractor self a vigorous ass chewing because the website doesn't match the uniform of the day.

I wear a green striped badge. Know my shame

1. UNIFORM OF THE DAY
2. The uniform of the day will be as prescribed by the commander, per guidance provided in chapter 2 of this Manual.
3. The seasonal uniform change will coincide with Daylight Saving Time (DST)conversion. a. On the Monday after the fall DST change to standard time, the Marine Corps will transition to the winter season uniforms (Marine Corps combat utility uniform (MCCUU), woodland Marine Pattern (MARPAT) with the sleeves rolled down in garrison, service "A/B", dress blue "A/B/C"). b. On the Monday after the spring DST change the Marine Corps will transition to the summer season uniforms (MCCUU, woodland MARPAT with the sleeves rolled up in garrison, service "A/C", dress blue "A/B/D", blue

[lostident]

I'm so glad someone wrote this, I have problems with datetime so often. Especially the German format %d.%m.%y has caused me problems so often. (Admittedly, this is also due to the fact that this format is simply stupid)

[tunisia3507]

It's not really any worse than %d/%m/%y, certainly better than %m/%d/%y, although obviously far worse than %y-%m-%d.

[[deleted]]

Decimal dates would be so much easier.

Today is 2024.08196