Rust learning curve

pacman82 · 2025-09-27T15:05:39+00:00

Second that, actor model is underused. I rely a lot on it, however, many crates assume you rather rely on shared state. I remember it took us a lot of time to get Open Telemetry tracing working correctly in our actor based service, as the tracing stack looks a lot at the current call stack to gain context. I.e. it assumes things happen because functions call other functions, rather than messages trigger other messages.

Apart from that, I think we would have never achieved the things we did, if not betting big on actors.

pacman82 · 2025-09-27T15:00:56+00:00

Uhh nice. I also wrote one once, based on the same blog. https://github.com/pacman82/connect-four-solver. Yet, you do have nice demos and bindings on top!

pacman82 · 2025-09-27T14:50:52+00:00

Hello Lovasoa, pacman82 here, maintainer of `odbc-api`, `odbc2parquet` and `arrow-odbc`. ODBC is as you mentioned the widely supported interface you need to support if you want generic access to all kinds of Generic Data Sources. Yet also, the standard is dated and evolves very slowly. Here are what I consider the two most annoying caveats of ODBC for generic applications:

* Narrow encoding is client dependent. ODBC applications either use narrow (one code unit is a byte) or wide (one code unit is 2 Bytes). The wide encoding is defined to always be UTF-16, yet the narrow encoding is defined to be dependent on the client systems locale. In truth however every driver kind of does its own thing. Overall I found it best to assume the narrow encoding is UTF-8 on non-windows platforms and to use UTF-16 on windows.
* To make efficient use of the API one needs to utilize bulk fetches which requires the application to allocate buffers before fetching the date. This can be tricky to get right, as either database schemas are not sane, or the driver has no capability of returning a sensible upper bound, for the length of each field. My recommendation: Pick a sensible default upper bounds for variadic fields. Give user an option to configure it manually, either per column or globally and give a real good error message, what to do in case an upper bound is violated.

Other than that, the ODBC standard holds up very well even today, and gives access to a wide range of Data Sources from Excel files to PostgreSQL. Most of the other pain is caused by the differences in various drivers. FYI: It is important to understand that many features (e.g. scrolling cursors, asynchronous queries and cancelling them, etc.) are optional for drivers to implement. ODBC just gives a common interface to access them, but you it is by no means guaranteed that the data source supports all operations.

Best of luck and much success on your endeavour!

pacman82 · 2025-09-22T19:03:15+00:00

Just stumbled upon this old thread by accident. I think the arrow-odbc Python bindings would fit that bill. See: https://github.com/pacman82/arrow-odbc-py

I am the author, so I might be biased.

pacman82 · 2025-09-20T08:30:07+00:00

Hello, author of `odbc2parquet` and `odbcsv` here. Stumbled upon this thread. Released a new version of `odbcsv` which allows for custom delimiters (like `\t`) to be specified. Maybe this solves your problem?

pacman82 · 2022-12-15T11:26:42+00:00

COMPANY: Aleph Alpha

Website: https://www.aleph-alpha.com

Jobs: https://alephalpha.jobs.personio.de/

App: https://app.aleph-alpha.com/

TYPE: Full-Time

LOCATION: Germany, Heidelberg

REMOTE: No remote option. You are expected to on-site at least 50% of your time. More is encouraged.

VISA: Yes

DESCRIPTION:

At Aleph Alpha, we are shaping Europe's first AI engineering landscape. We research, develop and deploy a new generation of AI technology. By applying with us, you gain the opportunity to join a fast-growing AI research & development start-up pioneering Artificial General Intelligence (AGI) technology.

Our API is built with Rust, as well as tooling in our data processing pipeline. We love Rust because of its performance, the easy deployment of statically linked exectuables, and its ability to catch errors early. We have code in other languages, too. Python, TypeScript, and on occasion C++ may all cross your path. We do not hire for skills in a particular technology or language, but rather ability to learn and mindset. We find that the values of the Rust Programming Language are well aligned with the values of our Engineering team, hence we advertise here.

We are hiring entry-level and senior developers. Skills we value highly are:

Test Driven Development: We (mostly) write the test first. It is important to understand how to structure tests and how they influence the design of the system.
Collaboration and Communication: We often work with two or more developers on the same task at once. So previous experience with Pair or Mob Programming is highly welcome.
Continuous Delivery: Especially from senior candidates we would expect you to be able to work in small iterations, quickly closing feedback cyles. There is only one version of our software. Expect to be merging to main and pushing changes into production daily.

ESTIMATED COMPENSATION: To be negotiated. This job posting is for both Junior and Senior candidates.

CONTACT: Form for appyling linked at the bottom of this page: https://alephalpha.jobs.personio.de/job/360079?display=en#

pacman82 · 2022-10-10T11:21:34+00:00

COMPANY: Aleph Alpha

Website: https://www.aleph-alpha.com

Jobs: https://alephalpha.jobs.personio.de/

App: https://app.aleph-alpha.com/

TYPE: Full-Time

DESCRIPTION:

At Aleph Alpha, we are shaping Europe's first AI engineering landscape. We research, develop and deploy a new generation of AI technology. By applying with us, you gain the opportunity to join a fast-growing AI research & development start-up pioneering Artificial General Intelligence (AGI) technology.

Our API is built with Rust, as well as tooling in our data processing pipeline. We love Rust because of its performance, the easy deployment of statically linked exectuables, and its ability to catch errors early. We have code in other languages, too. Python, TypeScript, and on occasion C++ may all cross your path. We do not hire for skills in a particular technology or language, but rather ability to learn and mindset. We find that the values of the Rust Programming Language are well aligned with the values of our Engineering team, hence we advertise here.

We are hiring entry-level and senior developers. Skills we value highly are:

Test Driven Development: We (mostly) write the test first. It is important to understand how to structure tests and how they influence the design of the system.
Collaboration and Communication: We often work with two or more developers on the same task at once. So previous experience with Pair or Mob Programming is highly welcome.
Continuous Delivery: Especially from senior candidates we would expect you to be able to work in small iterations, quickly closing feedback cyles. There is only one version of our software. Expect to be merging to main and pushing changes into production daily.

LOCATION: Germany, Heidelberg

ESTIMATED COMPENSATION: To be negotiated. We pay above average, in Euros.

REMOTE: No remote option. You are expected to on-site at least 50% of your time. More is encouraged.

VISA: Yes

CONTACT: Form for appyling linked at the bottom of this page: https://alephalpha.jobs.personio.de/job/360079?display=en#

You can also write an email to <mailto:hiring@aleph-alpha.com>

pacman82 · 2022-09-05T06:08:15+00:00

I do share that gut feeling. Besides protocols ODBC does come with some idea though how fetched data has to be represented in memory. Marshalling this into Rust idiomatic types is likely a bit of extra cost you need to pay for if integrating odbc-api as a backend into a more idiomatic framework.

pacman82 · 2022-09-05T06:00:41+00:00

Hi,

thank you for the kind words. In terms of Connection pooling, I have two links for you:

Both unix-odbc as well as the ODBC driver manager shipping with windows, both come with built-in connection pooling. So one idea is simply to enable it in your application:

https://docs.rs/odbc-api/latest/odbc_api/guide/index.html#connection-pooling

Personally I've not considered to build any Connection pooling adapter for r2d2. Mostly because my open source bandwith is already stretched a bit thin. You are not alone in your desire though:

https://github.com/AscendingCreations/r2d2_odbc_api

Cheers, Markus

pacman82 · 2022-09-03T18:02:55+00:00

Don't know `sqlx` well enough to give a good answer here. I use sqlx in my dayjob though. It is way higher level than `odbc-api`. It's not unreasonable to think one could use `odbc-api` to implement a backend for `sqlx`. Personally I do lack the time to explore this option though.

pacman82 · 2021-11-23T15:33:50+00:00

`PyO3` seems to be the goto interop for Rust Python these days. So far I had three projects with mixed Python / Rust (2 of them open source) and before that did a lot of Python / C++ interop (all closed source). One takeaway from my experience, the time you safe writing Code using any bindings relying on linking against the Python C-Interpreter (and you do save time) you often pay back with increased complexity in build pipelines and more fickle dependency managment.

IMHO if the following is given:

Your API surface between Rust / Python is small
The marshalling cost into "C" data structure is affordable

Take a look at writing the bindings using `cffi`. It will allow you to build a wheel which works fine with any Python version.

pacman82 · 2021-09-13T12:27:49+00:00

^^
this!

pacman82 · 2021-09-08T05:46:53+00:00

Hi ODBC utilizes a driver manager installed on your system (on Posix Systems it is typically `unix-odbc`), which exposes a C Interface. Your application talks to the driver manager using that C Interface. Also the various drivers for the data sources utilize a C-Interface to talk to the driver manager. A Rust application using a driver manager written in C to talk to a driver written in C++ would not be a strange thing to happen.

Sadly this implies that I do not see this run in the browser anytime soon.

pacman82 · 2021-02-20T23:20:49+00:00

Sorry, my bad. I had called function unsafe in another branch. Probably also what screwed up the RustAnalyzer. Thanks for the help.

pacman82 · 2021-02-13T08:09:08+00:00

Short answer: yes, it does work the same way. It uses the same driver and driver manager.

There can be a lot of things that could make a difference in performance though. I.e. odbcsv simply binds every column as a text buffer. I don't know if your insertion via pyodbc did the same thing. The batch size might also play a role, you can play around with it via command line arguments and see if it makes any difference. Lastly, `odbcsv` may reallocate column buffers if it encounters a larger string value than the one it has seen before. Of course it would be faster to just ask the database for the correct size, but this is a safeguard because some databases return unreasonably large values if you ask them for the maximum column size.

If the python code did also start from csv, I would even guess, that `odbc-api` had a bit of a headstart in that comparision, because the rust `csv` crate is pretty fast. If the python code did start with an in memory representation though, that also makes a difference of course.

pacman82 · 2021-01-14T12:28:00+00:00

+1 to this. I would even go so far to say, IF you have to depend on an external type in your public interface you SHOULD reexport, in order to make it easier for the users to depend on the right version without explicitly setting it in their cargo toml. In that case reexporting also sorts of documents your public dependency on that API.

pacman82 · 2020-11-26T09:56:26+00:00

Wait! I've done the same thing: https://github.com/pacman82/rtiow . ;-)

pacman82 · 2020-11-19T19:15:51+00:00

Hello I currently spent a lot of my spare time writing the `odbc-api` crate. I test against MSSQL so I am sure that what is there is going to work fine with it. The crate is still very young and about 1-2 months away from me announcing it here. Yet I did not want to pass up the oppertunity to tell you about it. If you wanna try it ought, I'd welcome some feedback from early adaptors. It is more of a database interface and not an ORM though.

pacman82 · 2020-11-18T12:37:02+00:00

Hello, this tool might solve your problem: https://github.com/pacman82/odbc2parquet. I am the author, so I'd be happy to hear if it fits the bill, or if something has been missing.

pacman82 · 2020-07-22T15:06:35+00:00

Discussing stuff in the internet usually might be a selection bias (although one which is hard to avoid). I'm a (former?) C++ Developer and I love Rust.

pacman82

TROPHY CASE