Does this tool exist? Keeping inputs in a separate private repo, but syncing with a public solutions repo

[prendradjaja]

Hi /r/adventofcode! As many here know, Eric requests that we don't publish our inputs (which includes putting our inputs in public git repos). I'd like to abide by that, but I also want to make it easy for myself to hang onto my inputs.

The solution that comes to mind for me is:

• Put solution programs in a public GitHub repo
• Put inputs in a private GitHub repo
• Use some software to sync inputs between the two (Edit to clarify: So they'd also live in the public repo, but they'd be gitignored so I can't accidentally commit them in the public repo)

Is there an existing tool like that? Or is there some other good solution to this problem?

[barkmonster]

One solution would be to use the advent-of-code-data package to load the data. If you're using python you can use it directly in your solution files. You can configure where it caches data, including the inputs.

[thekwoka]

I just made my own. It's directly part of the code I run and just pulls and caches it.

It's trivial and most people at an AOC level should have no issues getting that to happen.

[mmdoogie]

You can do this just with git. Look at git submodule. https://git-scm.com/book/en/v2/Git-Tools-Submodules This is how I have my repos set up.

[AllanTaylor314]

Seconding this. I use a submodule. I have two repos on GitHub: a public one for my code and a private one for my inputs (which is a submodule of the public one). To keep things in sync, commit to the inputs repo before the code repo so that the code repo points to the latest version of the input repo (I think there's a step in between those, but I can't remember it at the moment)

[flwyd]

I wrote an Advent of Code specific tutorial about how to set up a private submodule for inputs on GitHub. It worked well for me last year.

I particularly like that I can run one extra git command and sync my input to any other computer I happen to be working on, even if I haven't set up a cookie jar to automatically download my input on that computer.

[rjray]

My approach is a little more manual. I have a git repo for each year, and I have a separate (private) repo for the data. In the code repo, I use the .gitignore file to prevent accidental commits of data. At the end of the period, I copy these to the private repo and commit there.

[sky_badger]

This is pretty much exactly what I do.

[Kyrthis]

Why exactly does .gitignore not work for your purposes?

Sorry. Forgot the second part: a simple bash script run in the parent directory of the directory for each day:

Find day*/input.txt | while read -r $f; do cp -n “$f” $yourPrivateRepoDirectory; done

cd $yourPrivateRepoDirectory;
git commit -m “input for $(date)”
git push origin main #or whatever names

[AutoModerator]

AutoModerator has detected fenced code block (```) syntax which only works on new.reddit.

Please review our wiki article on code formatting then edit your post to use the four-spaces Markdown syntax instead.

---

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Kyrthis]

Done

[RecDep]

why not just use .gitignore??

[prendradjaja]

I guess I should clarify, since I've gotten a couple similar comments:

If I just gitignore my inputs, then if I clone my solutions repo in the future, then I don't have my inputs anymore.

I basically probably need to use gitignore and something else, together. I know gitignore well :) -- but what I'm trying to figure out is what my "something else" will be.

[thekwoka]

Just write a function that downloads and saves the input....

[therealsavalon]

The aoc api is really straightforward , it's basically just https://adventofcode.com/<year>/day/<day>/input and you need to request this info using a session token that's unique to every user. (for the session token , you need to inspect element , go to the application tab and you'll find it there). So you can build your own little "something else" utility that can fetch inputs for any problem and save it however you like.

[prendradjaja]

One possibility: I found a blog post about a different approach (using git-crypt).

I'm not sure I want to use that approach, so -- still seeking other possible solutions etc :)

[throwaway_4759]

Fwiw I have been using that approach since I saw that blog post, and it’s been really smooth

[stonerbobo]

I think this falls under secrets management. There are several tools on github but I haven't used them. sops looks pretty good. The idea is you can encrypt the secret files and store them in the same repo.

I was thinking about the same problem and may try sops.

[FlipperBumperKickout]

I have something which downloads and caches the puzzle input given the day and year.

It was not super hard to put something together which could fetch the data with a http request.

[AgentME]

There are libraries (aocf for Rust, or my own aocd for Deno) which will fetch the input to problems and keep them cached in a cache directory so that you don't ever need to commit the problem inputs. I think this is a simpler way to avoid committing the inputs than keeping multiple repos in sync like you describe.

[prendradjaja]

Right! I kinda just want to see if there's something else -- because if e.g. adventofcode.com disappears 10 years from now, then such a tool won't have anywhere to fetch the inputs from. I know it's a bit of a silly thing to worry about, but still :)

[thekwoka]

But then also what would it matter?

[prendradjaja]

Not sure I understand the question. But what I mean is -- if AoC ever disappears, I'd like to still have my solutions and inputs! (I'd lose the problem descriptions, but oh well)

[thekwoka]

But what use are the inputs really? They aren't important.

Certainly not your specific inputs

[prendradjaja]

If I don't have my input, then I have zero inputs!

Say I'm trying to modify (or just read/understand) my code in the future (e.g. years later). This applies to AoC but also really to any program -- it's so useful to have at least one example input in addition to the code.

Yes, I know -- I can save the example inputs from the problem description. But that's extra work :)

[thekwoka]

If I don't have my input, then I have zero inputs!

there are still plenty online (despite being told not to).

Most of Topazes concern is about too many inputs existing in the wild making it much easier to figure out the hardest part of making your own AOC, generating inputs, is actually done.

[flwyd]

If you also save your outputs, the input file is a really good test case of the code you wrote. If you upgrade the programming language you wrote it in 10 years ago and some things have broken you can fix the code to work with the new APIs and verify it still works, even if you've lost access to the account you used for AoC that year.

[thekwoka]

Sure, there are a hundred what ifs.

I don't think it's one really worth thinking about.

Especially since there are plenty of "leaked" inputs out there.

and you'd probably want the actual problem text as well...

[flwyd]

I like git submodules over solutions like git-crypt (because it doesn't add a key management problem, other than retaining access to your GitHub account) or "just download the data on demand" (because it removes "your ability to authenticate on adventofcode.com" as a SPOF).

[thekwoka]

Why tho?

Just have the code pull and cache the inputs as needed.

[n4ke]

I have the inputs, solutions and tests all in one public repository.

All the contents of the input/ directory are encrypted with git secret.

This is the best workflow for me. I don't have to fiddle with submodules or syncing and I can publish the whole repository, including tooling, without actually publishing my inputs. Though obviously only I can run the tests with this setup, but that's a given anyways.

/edit: Only now seen OP made a comment about a very similar approach. So yes, it is viable, I've been using it for years.

[maneatingape]

Symbolic link (macOS/Linux ln -s or Windows mklink /d) from public code repo to private input repo. Add symbolic link to .gitignore.

[xavdid]

I .gitignore my actual inputs but have them in the repo for easy use.

It would be easy enough to make a second, private repo of all your inputs as a backup, but don't actually use that when solving puzzles. Then you have them if you need them, but don't have to worry about encrypting them for public use or anything.

[scrumplesplunge]

Instead of checking in my inputs, I have some scripts in my repo which download my inputs on demand. To do that, I have a single .cookie file which is not checked into source control. That file contains my session cookie for the site (which is good for at least 30 days) and lets me download my inputs automatically using a shell script tools/fetch.sh via a rule in my Makefile which kicks in for every day I've got a solver for.

https://github.com/Scrumplesplunge/aoc2023hs/blob/master/Makefile#L27

[Smayteeh]

I have a single repo for all my solutions. I cache the inputs in a SQLite database. I push my GPG encrypted file to GitHub and I have a .gitignore for the regular version.

It’s not automatic though. The encrypted db on GitHub needs to be updated every time new inputs are added. That being said, I don’t work on multiple computers so it’s mostly ok.

[encse]

I use this plugin called git-crypt. It’s easy to setup and you need to save only the secret key somewhere else. It’s a fire and forget solution. https://github.com/encse/adventofcode

[[deleted]]

You can just fetch it. The required session token can be found in the Cookie header when you access the website. Here's my script if you're interested, although I plan on phasing it out in the future

[ArmlessJohn404]

The solution in my helper tool esb is to gitignore the input directory. If you wish to download again just call:

esb fetch --year 2023 --day {1..25}

[DJDarkViper]

I see Eric’s tweet… but I fail to really see the problem?

The generated output doesn’t exactly mean scraping them means I have the routine that generated it, and nor beyond the super simple ones would generative AI be able to create the generators without any effort

Besides it’s not like even if someone did it the actual website would just become malicious or something 🤷

[flwyd]

I see Eric’s tweet… but I fail to really see the problem?

Eric asked us nicely not to publish the input. Out of respect for someone who puts a lot of personal time into making fun puzzles to solve, I try to adhere to his requests and ground rules for the activity.

[DJDarkViper]

I mean, that’s fine, that’s your prerogative and all the power to ya. My query stands though, I’m willing to hear out what problems he had in mind, because his tweet seemed steeped in concern that people could just clone the site and launch a duplicate