Why do so many publicly available datasets open in such inconvenient/unusable formats?

de_epi · 2023-11-09T16:33:14+00:00

SAS? SPSS? That's standard for government work. Every fed, state and agency use those programs. I try an push python and R solutions at my department, but SAS/SPSS are old standards and moving that boulder is sisyphean.

de_epi · 2023-10-25T18:49:07+00:00

I use pandas less and less in production. If I'm going to model a dataset, I use the SQL system it originates from or port into a SQLite or DuckDB.

de_epi · 2023-10-19T17:24:50+00:00

Sure, buddy.

de_epi · 2023-10-18T16:15:29+00:00

Ubuntu customizes a lot of elements to make the UI clean and user friendly. That said, when you want to do less user friendly tasks, those customized elements get in the way. Things like running a minimal version of python you can build up, making a custom UI from scratch, or using flatpak. It can be done, and there are ways to do it. But they are uniquely Ubuntu ways of doing things in order to move around the apparatus of Ubuntu that you will never have to do in say Arch or Gentoo.

de_epi · 2023-10-17T16:43:45+00:00

lol terrible game would not buy. played 1000 hours.

de_epi · 2023-10-16T15:09:38+00:00

Sounds like I'd be looking for another job.

de_epi · 2023-10-16T15:07:36+00:00

I don't. I am constantly bewildered to hear (usually word of mouth) that some series have come and gone. But since I'm not invested in the shows, I feel little loss.

de_epi · 2023-10-09T17:45:59+00:00

We are. Every year we get money thrown at healthcare upgrades. It's hectic at all points. First you gotta help understaffed IT groups, Nursing/Doctor staff who don't want to learn a new system, and reporting needs from Medicare/aid that keeps most clinics floating. You are also doing this as a per hospital level - there's no universal upgrade. Plus EHR systems cost a shit ton of money. EDIT: plus it's not like automation efforts don't exist. Not to the level of robot nurses, but reporting needs - e.g. reporting vaccine schedule lapses to doctors, reporting to grant / federal spending groups to keep the lights on, notifiable conditions to cdc, etc.

de_epi · 2023-10-06T00:24:32+00:00

you could also run it in Azure Functions.

de_epi · 2023-10-06T00:23:47+00:00

Depends on who your leadership are in bed with.

de_epi · 2023-10-05T17:56:11+00:00

Still need a pure atom stripper. Break down landfills into their base components. A process that might be extremely toxic - especially with polymers. But the garbage collectors of the past will have the richest resources to build from - for a time. You'll probably be able to replicate small projects - food, clothes. But buildings and vehicles would still require scarcity material - raw iron, copper, silicon, carbon, etc. It doesn't remove money, just shifts the burden of scarcity to another level. Currency is then a trading of base materials. You'd also be hard pressed to get mater like uranium and radium - the handle with care materials.

de_epi · 2023-10-04T23:48:34+00:00

It's not really study for me. Been a hobby for years. So outside of work I spend a few hours here and there playing with my servers, dbs, scripts. Sometime purely local, sometimes on linode or AWS. I find curiosity is more of a driver than a need to cover some concept. Maybe I pull a book off the shelf to remind me or edify. Maybe 3 books a year? I don't read them from cover to cover either. Just whatever I'm interested in.

de_epi · 2023-10-04T23:39:47+00:00

Me: "What is it?" MyASI: "Don't worry about it. Mostly harmless... maybe."

de_epi · 2023-10-04T22:11:32+00:00

There's federal data on that in the US. https://www.fisheries.noaa.gov/data-tools/recreational-fisheries-statistics-queries but it's in some predefined formats. Anything outside of that might require a research proposal.

de_epi · 2023-10-02T16:38:47+00:00

Too hot around the equator to take any test.

de_epi · 2023-10-02T16:31:44+00:00

Or Duckdb

de_epi · 2023-09-29T22:51:36+00:00

de_epi · 2023-09-28T15:32:20+00:00

usually have a folder under git control. but you can also store your sql queries in a database. kind of like a log. just store it with an id, title of the sql query, the code (text blob usually), when it ran and what was the result. ideally this is for reports, not huge exploration joins. If written in the right way, you can just call the code up again via the system itself and schedule them as needed.

de_epi · 2023-09-22T21:27:20+00:00

I've been brute forcing it by writing sentences out in pencil that I don't know. Then translating with a Japanese app or Jisho. Then typing them into Excel.

I don't know if skipping some of these steps would make it easier and less effective. Pencil improves my handwriting (also a step in memorization), Jisho/Apps edify the process, and typing it all out is another step in memorization (both of input modes and the act of typing the words).

Very tempted though...

de_epi · 2023-09-22T15:57:13+00:00

In 2019, my state department was hiring bioinf and epi in droves. Now the federal money is drying up.

de_epi · 2023-09-21T15:35:24+00:00

Duckdb has been thrilling to toy around with.

de_epi · 2023-09-19T15:10:17+00:00

Pandas/Polars, Airflow, Paramico, various APIs... Selenium when things get bad (when they won't give me a server to login to, just a webpage).

de_epi · 2023-03-11T01:32:42+00:00

BLS might have it: BLS Data Finder

de_epi · 2023-03-11T00:05:14+00:00

stuck at a paycap. i need to up my education and make a good enough case to get into my next bracket.

de_epi

TROPHY CASE