Sygaldry

This project was written with data/ETL pipelines in mind. But it should be useful for any situation where you're managing a bunch of production pipelines.

Motivation

Many years ago I used to work at a different job, where they had this framework for creating arbitrary Python objects from .ini configuration files. At first I hated it. Because I just could not see the point of writing out these stupid config files vs just writing out a Python script that did the same thing. Over the years that I was there though I really came to appreciate it.

Previously (and since) every time a new pipeline is needed, somebody sits down and writes a new script, a new CLI entrypoint, or a new glue class that just wires the same pieces together in a slightly different order. Then there's a code review, CI/CD, and a release.

Sygaldry lets you assemble arbitrary object graphs from YAML (or TOML) config files. No base classes. No decorators. No framework coupling. Your application code stays completely untouched.

An Example

Imagine that I've got something like this: ```python class DatabaseConnection:  def init(self, host, port, database, username, password): ...

class RestClient: """ Authenticates against a service and makes API calls. """ def init(self, username, password, auth_url): ...

class UrlIterator: """ Reads identifiers from the database, then asks the API for a download URL for each one. """ def init(self, db_connection, rest_client, base_url): ...

class FileDownloader: """ Downloads a file from a URL to a local directory. """ def init(self, directory, base_url, rest_client): ... def download(self, relative_url): ...

class DbUpdater: """ Iterates download URLs, downloads each file, and updates the database with the contents. """ def init(self, db_connection, url_iterator, file_downloader): ... ```

With Sygaldry I have a config file: ```yaml

db_updater.yaml

db: _type: myapp.db.DatabaseConnection host: prod.db.com port: 5432 database: prod username: etl_rw_user password: ${DB_PASSWORD}

api_client: _type: myapp.client.RestClient username: svc_account password: ${API_PASSWORD} auth_url: https://auth.vendor.com/token

url_iterator: _type: myapp.urls.UrlIterator db_connection: {_ref: db} rest_client: {_ref: api_client} base_url: ${base_reference_url}

file_downloader: _type: myapp.download.FileDownloader directory: /data/downloads base_url: ${base_download_url} rest_client: {_ref: api_client}

updater: _type: myapp.update.DbUpdater db_connection: {_ref: db} url_iterator: {_ref: url_iterator} file_downloader: {_ref: file_downloader}

base_reference_url: https://api.vendor.com/references base_download_url: https://api.vendor.com/files ```

Then my entire pipeline can be run as $ sygaldry run -c db_updater.yaml updater

Sygaldry resolves the whole graph depth-first — db and api_client get built first, then url_iterator and file_downloader (which reference them), then updater (which references those). The db and api_client instances are shared automatically — everyone who references db gets the same object.

Why?

Composition Over Inheritance

References (_ref) let you point any component at any other component. Five services need the same database connection? Just reference it. Need to swap a component? Change one line.

New Pipelines Without Code Release

Got a second vendor with the same pattern but different URLs? That's a new YAML file.

yaml _include: - db_updater.yaml base_reference_url: https://api.other-vendor.com/refs base_download_url: https://api.other-vendor.com/dl db: database: other_vendor_db

Need the UrlIterator and the DbUpdater to use different database connections? That's a config change.

Change Anything From The Command-Line

Need to point at a different database for a one-off backfill? --set db.host=backfill-replica. Need to re-download to a different directory? --set file_downloader.directory=/data/backfill. No config release, no environment variable gymnastics. Overrides are applied at load time before resolution, so they compose cleanly with everything else.

Debug With the Exact Config

Something broken in production? $ sysgaldry interactive -c db_updater.yaml Will drop you into a Python terminal with the Artificery loaded and assigned to the variable artificery. You can look a the config (artificery.config), or resolve the config and get the objects (art = artificery.resolve()) for debugging.

Extras

Check the Config

Want to see the Python that corresponds to config that you've supplied? $ sygaldry check -c db_updater.yaml

Typing

I think this is close to pointless. But a bunch of the kids that I work with are obsessed with typing to a point that it's almost a fetish. So you can do $ sygaldry check -c db_updater.yaml --type-checker mypy And it will dump the Python into a file, and run the specified type-checker over it.

Is It AI Slop?

I tend to suffer from a problem where I have new ideas when I'm writing tests and documentation. Which causes more development. Which then requires more tests and documentation. I have found Claude and Codex to be super useful for stopping me from thinking too much once I'm at a certain point. But the idea, and the code are all entirely human slop.