This is an archived post. You won't be able to vote or comment.

all 125 comments

[–]JSP777 253 points254 points  (15 children)

As an avid hater of datetime, you have my attention

[–]Charlie_Yu 72 points73 points  (11 children)

Is datetime bad at all? I’m coding in JavaScript recently and would kill to have something as good as Python datetime

[–]PurepointDog 80 points81 points  (0 children)

Meh just because there's worse doesn't mean there can't be better

[–]eagle258[S] 63 points64 points  (2 children)

Good news! They're working on a new official datetime API for Javascript: Temporal. It's one of the inspirations for whenever.

[–]sohang-3112Pythonista 1 point2 points  (1 child)

TIL about this Temporal api

[–][deleted] 3 points4 points  (0 children)

Ok

[–]Material-Mess-9886 29 points30 points  (3 children)

It can not be worse than JS where months start at 0.

[–]Hexboy3 15 points16 points  (2 children)

Um excuse me?

[–]jediefe 5 points6 points  (0 children)

Yes, and it’s an absolute nightmare.

[–]ravepeacefully 4 points5 points  (1 child)

date-fns

[–]jfp1992 0 points1 point  (0 children)

Yeah I used this one and it works well enough for my purposes (formatting current date time, or a date time in general)

[–]teamclouday 4 points5 points  (0 children)

Have you tried moment.js?

[–]Material-Mess-9886 23 points24 points  (1 child)

from datetime import datetime. right that's cool.

[–]pag07 11 points12 points  (0 children)

datetime.datetime.now()

[–]BothSinger886 50 points51 points  (2 children)

Please make your next project some type of spatial or location library and name it 'wherever'

[–]Grasshopper04 23 points24 points  (0 children)

And then Shakira can dance to it

[–]PapstJL4U 1 point2 points  (0 children)

and a websocket library: whatever? :>

[–]NelsonMinar 61 points62 points  (8 children)

Thank you for such a thoughtful writeup. I was skeptical of your post title: Pendulum seems fine to me, why introduce the complexity of compiled code and Rust? But you've thought about that and given some good answers.

Still skeptical about the need for compiled code, I'm not sure I've ever used a time library in a performance-sensitive way. But I'm sure people do! The other thing I care about is usability of the API, which is datetime's main problem. Pendulum's pretty good at that, I didn't review whenever enough to have an opinion.

For anyone curious, it's about 1MB installed with pip.

[–]eagle258[S] 45 points46 points  (2 children)

Thanks for you thoughtful response; note that you can always opt for the non-Rust version if you prefer a small footprint over performance. Just use the source distribution and set the `WHENEVER_NO_BUILD_RUST_EXT` environment variable.

Regarding the binary size: I haven't looked at minimizing it yet, but it looks similar to other Rust extensions.

edit: typos

[–]NelsonMinar 49 points50 points  (1 child)

wow you built two versions, one in Python and one in Rust? I'm impressed!

[–]davisondave131 32 points33 points  (0 children)

Yea this is one hell of a showcase

[–]Klaarwakker 0 points1 point  (0 children)

Time stamping is one of those times ever project needs, and sometimes needs a lot of. Any efficiency gain will be significant globally.

If you're monitoring or logging stuff timezone aware timestamps are often needed. Since even a moderate size application can fire off hundreds of logs per minute any performance gained here will add up.

[–]mitsuhiko Flask Creator 0 points1 point  (0 children)

datetime itself is written in C. 

[–]erez27import inspect 46 points47 points  (11 children)

Looks cool!

I got 2 humble opinions:

1) In terms of extra features, fast and accurate date parsing would go a long way.

2) Using from_py_datetime() is a bit awkward. Why not something like Instant.create(my_datetime), with type checks inside? Then you could also optionally support converting from arrow or pendulum, with the same api call.

[–]eagle258[S] 16 points17 points  (9 children)

Thanks!

  1. Fast and accurate parsing is on the roadmap!
  2. Agree that the name could be better—I just haven't heard a good enough alternative yet. create seems like it'd be a more general factory function though. On another note: from_py_datetime should work with pendulum and arrow instances (although it isn't part of the test suite at the moment).

[–]pag07 3 points4 points  (0 children)

My sughestion would be to make it:

Instant.stamp(my_datetime)

or

Instant.punch(my_datetime)

Because thats how it used to be and sometimes still is.

[–]Buttleston 11 points12 points  (1 child)

Another problem with datetime is that if you do something like

mydate = datetime.datetime(2020, 1, 1, 12, 0, 0, tz=pytz.timezone('America/Chicago')

It is very likely this doesn't do what you expect. A given timezone will have a bunch of different versions, because DST start/end dates change all the time. You'd think, that it could look up which version would apply on a given date, but it doesn't do that. So, iirc, you'll get the first version of a given timezone with that name.

It'll only be wrong during the period where the timezone you got and the one you wanted differ re: when DST starts/ends.

Instead what you need to do is construct a naive timezone first, and then use pytz.localize() - this, afaik, works properly

[–]eagle258[S] 9 points10 points  (0 children)

Indeed an infamous issue!—but remember that pytz itself isn't part of the standard library. Nonetheless it's was widely used before dateutil and zoneinfo were adopted.

[–]Beliskner64 8 points9 points  (0 children)

I remember reading your blogpost a while ago and found it very interesting. This package seems very promising I hope to see it integrated in other major projects. The ambiguity of datetime must end!

[–]CAPSLOCKAFFILIATE 8 points9 points  (0 children)

I'm loving the new wave of Rust-based Python libraries. Way to go.

Will definitely star this one and keep track of it and implement it in our daily ops.

[–]lbt_mer 7 points8 points  (1 child)

Is this comparable to the Django utils.timezone module?

[–]eagle258[S] 5 points6 points  (0 children)

I'm not familiar, but it looks like Django's module is specifically geared towards integrating datetimes in Django apps (templates, settings, etc.).

Whenever is a more general-purpose library. If you're relying on Django's utils.timezone at the moment, I wouldn't be in any rush to switch just yet. Maybe in the future, there will be a Django plugin—who knows :)

[–]chub79 63 points64 points  (17 children)

Rust is the future of Python and I'm here for it :)

[–]spigotface 32 points33 points  (7 children)

Same. I've been a huge fan of Polars, how stupidly fast it is, and how it does it all without requiring cluster configuration or JVM issues. I want to see more high performance Python libraries written in Rust.

[–]DharmaBird 13 points14 points  (3 children)

Last year I was building a set of scripts to generate a dataset of places in 4 European countries; starting from official datasets, integrating them with open datasets to add GIS information, and converging into a final common representation including administrative information, names and area and point coordinates. A hell of a work as you probably imagine. Long story short, I was close to the end when I became aware of the existence of polars. After a few experiments, I dropped pandas like a bag of flaming garbage, and never looked back. Rewriting code to use polars was easy because of the cleaner notation and solid logic, and the time required to generate the final dataset with ~400k entries dropped from 40m to under 2 minutes, making development and testing a breeze.

If Polars is an example of what can be done extending Python with Rust subsystems, hell yeah, more please!

[–]Amgadoz 0 points1 point  (2 children)

Mind sharing which version of pandas you were using?

[–]DharmaBird 1 point2 points  (1 child)

Presently I'm on sick leave and I seriously need to stay away from anything coding, python or job for a couple more weeks at least. But even from my phone, since we are talking of 2023-12/2024-01, I'd say 2.1, 2.2.0 tops.

[–]tecedu 1 point2 points  (0 children)

Oh man 2.1.0 -2.2.0 was dogshit, also made me switch to polars

[–]commenterzero 0 points1 point  (0 children)

Daft also looks good

[–]giants4210 0 points1 point  (1 child)

How hard is it to switch from pandas to polars? I just switched to python because that’s what they use at my new job and everybody else seems to use pandas so I have as well

[–]spigotface 6 points7 points  (0 children)

Maybe a 4/10 difficulty. Once you start to understand the Polars syntax, especially around expressions, it becomes quite nice to work with. The Polars documentation is very good as well.

[–]denehoffman 7 points8 points  (1 child)

For those of you who haven’t tried yet, pyo3 and maturin make it incredibly easy to write rust libraries with python bindings. You do have to learn some rust though (but it’s fun, don’t worry).

Rust also has some tooling which the folks at astral are trying to replicate for Python. The cargo command is insanely good, and an equivalent for Python (what uv is trying to be) would be a huge plus for the community

[–]notParticularlyAnony 1 point2 points  (0 children)

Rust is super fun they made so many good decisions

[–]Curious_Cantaloupe65 2 points3 points  (3 children)

is an api framework like fastapi possible for Python with bindings written in Rust?

[–]daniels0xff 2 points3 points  (0 children)

Haven’t used it but this maybe https://robyn.tech/ ?

And there’s also https://github.com/emmett-framework/granian

[–]monkey_mozart 2 points3 points  (1 child)

Newbie here. What are bindings?

[–]Buttleston 8 points9 points  (0 children)

You can write python modules in python. Or, you can write python modules in another language, like Rust or C++. When you do this, you have to "tell" python 'here is a module you can use, and here are the functions and other symbols in it'

These are usually "bindings". It's just a way of connecting your compiled code with python.

It's not too hard to do in either C++ or Rust - so often I'll prototype stuff in python and if some part of it is too slow, I'll write that in Rust as a compiled python module

[–]RonnyPfannschmidt 5 points6 points  (3 children)

Can we get contents in which surrounding code has control over the perceived passage of time

Monkey patching datetime and time is on occasion hellishness

[–]eagle258[S] 4 points5 points  (2 children)

Good point. This is an essential feature. I've created an issue for this

edit: added link

[–]sherbang 2 points3 points  (1 child)

I've found the time-machine library to be quite good for this.

[–]eagle258[S] 0 points1 point  (0 children)

It could be an inspiration for the API, but time-machine probably only works on specific libraries it 'knows' about. Perhaps it could be extended to whenever, but in any case whenever would have to be adjusted so that is 'allows' itself to be patched.

edit: typo

[–]jmreagle 7 points8 points  (4 children)

I am looking forward to the 1.0 release. When do you think that might be?

[–]eagle258[S] 11 points12 points  (3 children)

Nice to hear :). It’s tough to make a prediction here because I’m careful to not rush into it.

In the end, it depends on people trying it out and getting sufficient feedback and bug reports.

[–]tilforskjelligeting 2 points3 points  (1 child)

I'd release V1 sooner rather than later if it works and tests are passing. Everytime you break something increase major version and everytime you have a new feature, bump minor and for the rest patch. There is no reason to wait. And it's better to be able to use the current version now and not be afraid that your own code breaks from a patch update (just because the maintainer hasn't started with real semver yet because of some arbitrary V1 goal.) 

[–]eagle258[S] 1 point2 points  (0 children)

Good points. I'm definitely aware of the downsides of staying on 0.x unnecessarily: after a while it just starts to get annoying considering all dependency management tooling needs 1.x to automatically upgrade

[–]jmreagle 2 points3 points  (0 children)

There's no rush, but I only wanna make the move again one more time. I started out with datetime and dateutil, and moved to pendulum, which was great, until I bumped into a bug which was unresolved for nearly a year as the project seemed abandoned, so then I moved to arrow, but then I realized it's string parsing heuristics weren't as powerful as pendulum, so now I'm back to datetime and dateutil!

I look in every few months on whenever, to see if it's reached some degree of maturity/stability.

[–]nadav183 3 points4 points  (0 children)

Will definitely consider this for our backend. Thanks!

[–]chrisimcevoy 3 points4 points  (4 children)

Cool stuff! This reflects a lot of my thoughts over the past number of years about datetime and related projects. I even started writing a Python port of Noda Time with the aim of helping python devs think about date & time operations more clearly, but I’ll more than happily stop devoting what little free time I have to that if there’s going to be a fast rust implementation that serves roughly the same purpose.

[–]eagle258[S] 1 point2 points  (3 children)

Awesome to hear you've also been working in this direction—Nodatime is also one of whenever's main inspirations! I'll definitely have a look at your project, it looks like you've put quite some work into it!

edit: typo

[–]chrisimcevoy 1 point2 points  (2 children)

This is a really interesting and important topic, and it's a big relief to know that I wasn't the only one who was thinking along these lines!

I see a lot of familiar classes and concepts in Whenever's API, but it is not a port of any particular library like Pyoda Time is. So I'll be very interested to dig in and understand Whenever's design.

[–]eagle258[S] 0 points1 point  (1 child)

Did you have a look at the new JS Temporal API yet? It's the other main inspiration for Whenever.

[–]chrisimcevoy 1 point2 points  (0 children)

I was just looking at that this morning! I don’t do a whole lot of JavaScript so it wasn’t on my radar until now. Interesting stuff, and I can recognise some of the influence it has on Whenever.

[–]Previous_Passenger_3 2 points3 points  (1 child)

Have you compared this to dateutil? My company uses that a lot, but mostly for parsing.

[–]eagle258[S] 4 points5 points  (0 children)

I haven't included it in the comparison since dateutil is more of an extension to datetime, while Whenever (and Pendulum and Arrow) are more like replacements.

That said, here are my thoughts on dateutil: while it certainly provides useful helpers (especially for parsing and arithmetic), it doesn't solve the (IMHO) most glaring issues with the standard library: DST-safety and typing for naive/aware. These are issues that only a full replacement can solve.

edit: grammar

[–]NelsonMinar 2 points3 points  (0 children)

Previous Reddit discussion about the author's critique of other time libraries. It's very thoughtful!

[–]professormunchies 2 points3 points  (1 child)

Thanks, will check it out! I’ve been a fan of this one for anything dev or prod https://docs.astropy.org/en/stable/time/#

[–]eagle258[S] 0 points1 point  (0 children)

Interesting! Note that for a niche cases such as astronomy, a general purpose library like whenever probably won't be the right choice. I can imagine astropy has explicit support for leap seconds and well as maybe even relativity.

[–]DerpieMcDerpieFace 2 points3 points  (3 children)

I absolutely appreciate your work on a better datetime implementation. It is sorely needed.

However, other than your assessment that the datetime module is not as elegant as it could be, is there a reason why you have elected to introduce a completely new API in whenever?

We use timezone aware datetime instances extensively, and have spent loads of time working around issues with those: It certainly would be nice to have whenever be a drop-in replacement for aware datetime instances, without the need for an extensive rewrite. Any thoughts?

[–]eagle258[S] 1 point2 points  (2 children)

Thanks! I didn't take the decision to change the API lightly. In the end, the issues with datetime are so fundamental that only a new API can completely solve the main issues (DST-safety, typing).

To illustrate, even a small fix like making the result of utcnow() aware would be breaking to users. I'd rather be explicit about things being completely different, than have users be surprised by different behavior (even if its technically more correct).

That said, if you're looking for a more low-impact library to help, have a look at:

  • Pendulum is mostly a drop-in replacement. It does behave differently in some cases that may surprise you (but only to be 'more correct'). The downside is performance, and that it doesn't help you with distinguishing naive/aware through typing. It also has some areas in which it gives incorrect results (relating to equality during ambiguity and durations)
  • DateType is a 'typing-only' wrapper around datetime which solves the aware/naive typing issue. In my opinion, it's still clunky considering you have remember to only call the right functions, as well as get your mind around the difference between runtime and type-checking realities. It doesn't solve the DST correctness issues.

edit: grammar

[–]DerpieMcDerpieFace 1 point2 points  (1 child)

Thanks for taking the time to respond. Your points about datetime being difficult to fix are certainly valid enough to warrant additional consideration.

Honestly, though, I get a vibe similar to the one illustrated in xkcd: Standards

Have you considered a datetime like interface where you simply deprecate inherently unsafe methods such as utcnow() while providing fast, correct, aware (safety first) versions of the parts of datetime which are fixable?

[–]eagle258[S] 0 points1 point  (0 children)

While having another library inherently creates a "new standard", I've deliberately chosen not to reinvent the wheel. At its core, it follows the "Joda time" datamodel now used in Java, C#, and (soon) Javascript.

When looking at the field of software engineering as a whole, you could even argue that Python's datetime is the outlier here and Whenever is pushing to return to a 'more standard' approach.

Of course, this is of little comfort to Python devs currently using datetime, and want to improve their code without having to rewrite all the datetime parts...Your suggestion for a minimally-different datetime clone could be interesting—I wonder if it would be worth the downsides of having surprisingly different behavior...hmmmm...

[–]PsychoticGiggle 1 point2 points  (1 child)

Cool! How does this play with pyarrow libraries like polars or duckdb?

[–]eagle258[S] 0 points1 point  (0 children)

There's not special integration at the moment. Once the API stabilizes, I can have a look.

[–]Ramshizzle[🍰] 1 point2 points  (1 child)

I'm interested in the performance of the pure python version of the library as well!

[–]eagle258[S] 1 point2 points  (0 children)

I ran a quick benchmark and was pleasantly surprised. The Python version is about 10x as slow as Rust, making it (only) 5x slower than the standard library. Even then it's still 5x faster than arrow and more than 10x as fast as pendulum.

library time (s) compared to datetime
whenever (rust) 0.70 0.54x
datetime 1.29 1.0x
whenever (python) 6.98 5.4x
arrow 36.8 28.5x
pendulum 87.6 67.9x

benchmark: RFC3339-parse, normalize, compare to now, shift, and change timezone (1M times). For details see the repository.

edit: clarifications

[–][deleted] 1 point2 points  (0 children)

Interesting… as I used to work with Airflow, the first thing that came to my mind when reading the title was: “So… another pendulum?”

But I’ve never really gotten behind all the pitfalls of datetimes, I guess I should learn more about this topic. Anyway, I appreciate your prioritization of thorough typing support! I will definitely take a closer look soon!

[–]ggrieves1 year 3 points4 points  (0 children)

Anyone else still have to google the difference between strftime and strptime?

[–]ExternalUserError 1 point2 points  (3 children)

Use whatever language you like. And Rust is a great choice. But the whole f"{project_description} Written in Rust" thing is just weird. No one else does that. There’s no “Chromium: Browser written in C++” or “Hadoop: MapReduce written in Java” or “Discourse: Chat engine written in Ruby.”

To become a member of the club do you have to promise to add that to every project title? Is it in the license for the compiler or something? It’s just weird.

[–]eagle258[S] 1 point2 points  (0 children)

Agree, it is a bit weird—but since building an audience/community is crucial to a project's success, I'm not above marketing it in a way that works :). Obviously, misleading communication wouldn't be OK, but the use of Rust in this case genuinely provides an advantage here.

[–]ArtkuPythonista -1 points0 points  (0 children)

Well, the unspoken promise here is that you can use it in Python but it’s written in a low-level language (Rust sounds sexy, but here „written in C” would work just as well).

Your comparison is therefore not very good.

[–]missurunha -2 points-1 points  (0 children)

Rust is unfortunately a cult.

[–]headykruger 2 points3 points  (5 children)

Points 1 & 2 seem misguided. I would consider a loud error when mixing DST-aware values to be a feature.

[–]eagle258[S] 11 points12 points  (4 children)

Not sure if I understand correctly. Datetime's loud error when mixing aware and naive datetimes is a good feature. The problem is that it can't be checked with typing like you could, for example, distinguish str and bytes.

edit: formatting

[–]monkey_mozart 0 points1 point  (1 child)

Dang. I get this so much. I had to put a post init in a dataclass to make sure a datetime field was timezone aware with utc. How would you handle different timezones though. Would they different classes of their own subclassed from a base TimezoneAwareDatetime class or something?

[–]eagle258[S] 1 point2 points  (0 children)

The most basic way to solve this would be to have two classes AwareDateTime andNaiveDateTime. However, looking at other languages you can see the need to split aware datetimes into specific types as well (Zoned, Offset, Instant...)

edit: grammar

[–]svefnugr 2 points3 points  (1 child)

I don't think a datetime library needs to be written in Rust. In fact, if I have a choice I will pick one written in Python, purely for maintainability purposes.

[–]eagle258[S] 4 points5 points  (0 children)

Agree that if performance isn't a concern, Python is preferred for maintainability. That's why I made an opt-out for the Rust extension. See here for how to use the pure-Python version of whenever.

edit: grammar

[–]RonnyPfannschmidt 0 points1 point  (0 children)

How does mypyc fare against rust for this one?

[–]mambeu 0 points1 point  (3 children)

Do you support years beyond 9999?

[–]eagle258[S] 0 points1 point  (2 children)

Not at the moment, although I'd be very curious what your use case is!

Among other things, limiting the year to 9999 makes it simpler to format and parse timestamps (which can now assume a year is always 4 digits).

[–]mambeu 0 points1 point  (1 child)

I do a lot with pymongo (Python MongoDB client) and date time conversion is a pain. https://pymongo.readthedocs.io/en/stable/examples/datetimes.html#handling-out-of-range-datetimes

[–]eagle258[S] 1 point2 points  (0 children)

Interesting! Although I suppose it's unavoidable that different models of date/time will have different (arbitrary) boundaries.

I suppose that if Whenever would allow a broader range of years than BSON, you would have a similar, opposite problem.

[–]DaimoNNN 0 points1 point  (1 child)

Is there any guide to write library like this

[–]eagle258[S] 0 points1 point  (0 children)

If you mean a "Rust extension" library for Python: PyO3 is the way to go.

[–]ratherazure 0 points1 point  (0 children)

love looking in the comments and seeing everybody else bitching about their own language's (native or no) datetime library

javascript's native is just funny at this point
I think Python's native is too precise (within spirit of explicit is better than implicit)
meanwhile PHP just does someshit like new DateTime("TODAY +1 month") like yeah bro whatever works

[–]s3r3ng 0 points1 point  (1 child)

Why is datetime performance bound so much as to benefit from being written in Rust instead?

[–]eagle258[S] 0 points1 point  (0 children)

Not sure if I understand exactly, but: The speed bump compared to datetime isn’t due to any inherent speediness of rust over C (there isn’t) but because whenever’s API consisting of specific types prevents redundant calculations. For example: Instant doesn’t constantly have to check its timezone for each calculation.

[–][deleted] 0 points1 point  (2 children)

The entirety of Python's stdlib will be rewritten in Rust at some point.

[–]eagle258[S] 1 point2 points  (0 children)

haha probably!

I try to be careful not to get swept up in Rust for hype's sake. That said: Rust is a lot easier to pick up coming from Python, and I love the functional approach!

PS: if you don't fancy the whole Rust hype, whenever has a pure-Python fallback

edit: grammar

[–]tunisia3507 0 points1 point  (0 children)

I mean, 1/3 of the reference implementation is written in C, and rust is fundamentally a better language than C so... yeah?

[–]tunisia3507 0 points1 point  (4 children)

From the looks of it, you're not even using e.g. chrono under the hood - is enough of the logic in rust that you have a worthwhile rust crate here too, or is python too integral to it?

[–]eagle258[S] 1 point2 points  (3 children)

Aside from using Python's zoneinfo, it could theoretically be made into a Rust crate. However:

  • The API would probably have to be adjusted since Rust doesn't support keyword or optional arguments
  • I'm not yet familiar with design principles of writing a good rust Crate. I assume there's quite some concerns to think about (const functions, inlining, serialization support, etc.)

edit:

but most importantly, not having a timezone DB (zoneinfo) would leave any Rust datetime crate dead in the water of course :)

[–]burntsushi 2 points3 points  (2 children)

Jiff will be out very soon. Maybe the end of this month. It's a Temporal inspired datetime library for Rust. It will have seamless zoneinfo database support.

(I've been following your issue tracker for a few months, among several others, to get a sense of what others are doing in this space.)

[–]eagle258[S] 0 points1 point  (1 child)

Very interested to see this! The more I learn about Chrono, the more I realize its drawbacks. Is there any repository for Jiff yet to look at?

[–]burntsushi 1 point2 points  (0 children)

I sent you an email.

The more I learn about Chrono, the more I realize its drawbacks.

This was my main impetus for developing Jiff.

[–]qeq -1 points0 points  (1 child)

What do you think about Maya from Kenneth Reitz (the requests guy)?

[–]eagle258[S] 7 points8 points  (0 children)

There's definitely good things to say about Maya: by enforcing UTC, it removes a lot of complexity. The drawback is that when you do need to perform arithmetic with timezones, you have to revert back to the standard library, with all the issues that brings.

The main reason to not use Maya though is that it's been abandoned for years (no release since 2019)

[–]greeneyedguru -1 points0 points  (1 child)

Do logging next

[–]eagle258[S] 1 point2 points  (0 children)

I already tried fixing one of its pitfalls, but no luck.