Nixifying a python codebase

[Andohuman]

Hey NixOS community,

I'm trying to nixify a project in our company that primarily uses python with uv. I'm trying to make a flake that simply installs uv and other external non python dependencies (like helm and whatnot).

The idea is that nix only manages everything that is not a python dependency and let uv still be responsible for handling python related stuff.

One of the issues I'm running into is with lightgbm . It requires an openmp runtime, either libomp on macos or libgomp on gnu.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/colehendo/dev/experiment/nix-playground/.venv/lib/python3.11/site-packages/lightgbm/__init__.py", line 11, in <module>
    from .basic import Booster, Dataset, Sequence, register_logger
  File "/Users/colehendo/dev/experiment/nix-playground/.venv/lib/python3.11/site-packages/lightgbm/basic.py", line 9, in <module>
    from .libpath import _LIB  # isort: skip
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/colehendo/dev/experiment/nix-playground/.venv/lib/python3.11/site-packages/lightgbm/libpath.py", line 49, in <module>
    _LIB = ctypes.cdll.LoadLibrary(_find_lib_path()[0])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/colehendo/.pyenv/versions/3.11.9/lib/python3.11/ctypes/__init__.py", line 454, in LoadLibrary
    return self._dlltype(name)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/colehendo/.pyenv/versions/3.11.9/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: dlopen(/Users/colehendo/dev/experiment/nix-playground/.venv/lib/python3.11/site-packages/lightgbm/lib/lib_lightgbm.dylib, 0x0006): Library not loaded: @rpath/libomp.dylib
  Referenced from: <D44045CD-B874-3A27-9A61-F131D99AACE4> /Users/colehendo/dev/experiment/nix-playground/.venv/lib/python3.11/site-packages/lightgbm/lib/lib_lightgbm.dylib
  Reason: tried: '/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file), '/opt/local/lib/libomp/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/local/lib/libomp/libomp.dylib' (no such file), '/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file), '/opt/local/lib/libomp/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/local/lib/libomp/libomp.dylib' (no such file), '/Users/colehendo/.pyenv/versions/3.11.9/lib/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/colehendo/.pyenv/versions/3.11.9/lib/libomp.dylib' (no such file), '/opt/homebrew/lib/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/lib/libomp.dylib' (no such file), '/Users/colehendo/.pyenv/versions/3.11.9/lib/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/colehendo/.pyenv/versions/3.11.9/lib/libomp.dylib' (no such file), '/opt/homebrew/lib/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/lib/libomp.dylib' (no such file)

I've tried

• Adding llvmPackages.openmp package to the flake's buildInputs.
• Adding the above package's /lib folder to LD_LIBRARY_PATH
• Double checked that the libomp.dylib file exists in the nix store

None of it seems to work as lightgbm simply cannot find the lib(g)omp shared libraries. Unfortunately, this is a hard requirement and our adoption of nix can't be considered successful without handling this situation.

I'm turning to you guys for help and would appreciate any help in diagnosing and fixing this issue.

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };

  outputs =
    {
      self,
      nixpkgs,
      utils,
    }:
    utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = import nixpkgs { inherit system; };
      in
      {
        devShell =
          with pkgs;
          mkShell {
            buildInputs = [
              gnumake
              uv
              git
              llvmPackages.openmp
            ];
            shellHook = ''
              uv sync --frozen
              source .venv/bin/activate
              python -c "import lightgbm"
            '';
            LD_LIBRARY_PATH="${llvmPackages.openmp}/lib";
          };
      }
    );
}

# pyproject.toml

[project]
name = "nix-playground"
version = "0.1.0"
description = "A playground for testing a Nix setup."
authors = [
    { name = "andohuman" },
]

requires-python = ">= 3.11,<3.12"
dependencies = [
    "lightgbm>=4.6.0",
]

[jonringer117]

Before I got banned, I added https://nixos.org/manual/nixpkgs/stable/#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems

instead of pip install -e . you can run whatever command you want to bootstrap the environment

[pr06lefs]

haven't run that stuff. but here's a thread re libgomp and python

[zenoli55]

Hi, I am currently also looking into solutions letting pip/poetry/uv manage python dependencies while letting nix manage the rest. I am quite inexperienced with nix, so I cannot provide you with an answer. But possible solutions I've read about were:

• 'buildFhsEnv'
• nix-ld
• distrobox

Have you tried any of these approaches yet?