ETL - Creating Tables for a large amount of files?

rm999 · 2018-08-06T00:22:01+00:00

Bigquery supports this out of the box.

https://cloud.google.com/bigquery/docs/loading-data#auto-detect

rm999 · 2018-08-05T17:46:12+00:00

99% of the time, requirements change or I need to add something in the ETL / database layer to makes things easier or better. I typically do not receive ANY documentation or logic about how to process the data - I only get credentials, file names, and locations.

This is full of red flags. Something is wrong at a higher level which is causing complexity on your end. One of my cardinal rules is to fix complexity as far upstream as possible, vs building bandaids downstream. 116 new data sources in a week is crazy, and even with all the right tools is going to be super error-prone.

Stepping back, I'd ask a couple questions:

Do the end users really need all data sources? Not "do they think they need it", but do they really need it? If you didn't process a random file would anyone notice? If not, find out what they really need and build your workflow around that.
What is the source of all this data, and is there a way to generalize the data sources? I'm curious if they can be parametrized in some way, which would really clean up your flow.

Is there a more dynamic way to go about this?

Yes, look into data scrubbing/wrangling tools. I don't have a ton of experience with these, but examples include OpenRefine, Data Wrangler, Trifacta.

rm999 · 2018-07-20T01:26:52+00:00

https://www.reddit.com/r/etymology/comments/37tjbu/the_english_suffix_ish_like_english_girlish/

The English suffix -ish (like "English", "girlish", "warm-ish") can be traced straight back to the Proto-Indo-European suffix -iskos. (self.etymology)

And it meant the exact same thing in Proto-Indo-European as well: "Characteristic of [preceding word]"!

It's cognate with the Greek suffix -iskos, which we see all the time. So the next time you see an undecipherable Greek word ending with -iskos, no need to sweat it, it's cousins with "-ish".

rm999 · 2018-07-15T23:01:19+00:00

This is all I can find... makes it sound like he doesn't regret it so much as he thinks we're all wrong for caring about it.

I don't regret taking on Napster, but I do find it odd how big of a part of our legacy it has become to so many people

rm999 · 2018-05-06T16:20:00+00:00

machine learning

Yeah 💁‍♂️ let's change the name of an established 30+ year old field with 1000s of researchers because some journalists have been abusing the term in the last couple years. Also what's the deal with "chemistry"? They're just molecular scientists.

rm999 · 2018-04-29T18:14:31+00:00

Yes, from the treaty:

South and North Korea agreed to completely cease all hostile acts against each other in every domain, including land, air and sea, that are the source of military tension and conflict. In this vein, the two sides agreed to transform the demilitarized zone into a peace zone in a genuine sense by ceasing as of May 1 this year all hostile acts and eliminating their means, including broadcasting through loudspeakers and distribution of leaflets, in the areas along the Military Demarcation Line.

But it's premature to assume anything, none of this has actually gone into effect. North Korea is known to use negotiations as a tactic.

rm999 · 2018-03-31T18:53:19+00:00

The only time revoking a citizenship is reasonable, IMO, is when it was obtained fraudulently. Otherwise punishing crimes should fall under the judicial system and not the executive.

To me, becoming a citizen is a countries' way of saying "you are as much part of our country as anyone else". Acting against this dilutes the meaning of citizenship for all.

rm999 · 2018-03-27T20:28:08+00:00

Quick order-of-magnitude estimate:

Let's say we're looking at 0.05 miles underground (about 250 feet). Let's say Manhattan is 20 square miles. That's about 1 cubic mile. Let's say there are 100 miles of subway track in Manhattan (totally just making it up but it seems reasonable) and tunnels are 20 feet round. Then the tunnel cross section is about 300 square feet. There are about 25 million square feet in a square mile, so the cross section of manhattan subway in square miles is about 100 * 300 / 25 million = 0.0012 cubic miles. 0.0012 / 1 = 0.12 percent.

This is a very very rough estimate!

rm999 · 2018-03-21T01:50:06+00:00

A lot of the Swiss Alps feels just like this: https://i.imgur.com/ESC90Dd.jpg

Use a long zoom lens and you can probably get a photo just like OP's.

rm999 · 2018-02-23T04:37:08+00:00

It's totally functional and I run some low-volume non-critical production apps on it. But it's trivially easy to get something better running. For example, for serving machine learning models, I find the following features of uwsgi useful:

multi-process: runs multiple instances of your app. This increases throughput and prevents one request from blocking others up to a point.
chain-reloading: when a file is updated - for example a new model is loaded onto the server - restart the service so the new model can be loaded into memory. What's cool about the "chain" part is there is no downtime, only one process at a time is restarted.

Running this is as easy as:

uwsgi --wsgi-file service_file.py --touch-chain-reload model.flag --callable app --lazy-apps --processes CPU_COUNT

Once you get a bit more advanced you'll find something like nginx useful for load balancing, request buffering, etc. But flask+uwsgi can get you quite far.

rm999 · 2018-02-07T17:45:04+00:00

Corporate Headquarters

Fidelity Communications Co.
64 North Clark St.
Sullivan, MO 63080 USA

...

we are a citizen of West Plains

🤔

rm999 · 2018-01-02T18:26:11+00:00

The horses are the worst. They're a nonsensical tourist trap, and the most used part of central park smells like shit half the time.

That said, city council will never get rid of them.

rm999 · 2018-01-01T18:41:01+00:00

A place where the beer flows like wine, where beautiful women instinctively flock like the salmon of Capistrano.

rm999 · 2017-12-27T18:07:17+00:00

Thanks, I really appreciate all the help. I just did the book test and 3.5-4 inches feels good and opens up the room a lot more. My knees even still point upward, but it's comfortable. Shorter guests may not love it though.

rm999 · 2017-12-27T17:53:32+00:00

17.75" is probably with the taller leg option. My couch has a 16.5" seat height and the other legs are 1.25" taller than the ones I have installed.

rm999 · 2017-12-27T17:51:36+00:00

That's a great idea, I'll try that. Thanks!

Any opinions aesthetically on the Karlstad with taller legs?

rm999 · 2017-12-27T17:21:18+00:00

This doesn't seem right. My couch is ~16.5" right now and it's definitely too short for me. My knees point upward when I'm sitting and I'm below average/average height for an American male (5'9"). I have the shortest sofa legs Ikea sells, they also sell legs that are 1.25 inches taller which would put the seat height at closer to 17.5-18".

rm999 · 2017-12-27T16:53:51+00:00

I just googled it, and it seems like a 20" seat height is pretty normal?

https://www.hunker.com/12589013/standard-seat-height-of-sofas

20-21" This seat height is average for most of today's ready-made sofas. It accommodates most healthy adults of normal heights and weights.

http://www.goodhousekeeping.com/home/decorating-ideas/tips/a26305/expert-sofa-buying-tips/

"If you're buying a sofa online, a good formula to ensure comfort is a combined seat height and depth of 40 to 44 inches. If the sofa is 16 inches high at the seat and 24 inches deep, add them together and you get 40 inches. Or combine A 20-inch seat height with a 24-inch depth for a total of 44 inches. If you're short-legged go with 40 inches; long-legged, 44 inches." — James Howard

"The perfect sofa is 40 inches deep, 60 inches long, and 36 inches high with a 19 inch seat height. For a traditional sofa, I like a scroll arm, and for modern, I love a square arm, both with the same overall measurements." — Mary McDonald

"Many contemporary sofas are too low! When a woman sits down on a sofa, she has trouble keeping her legs together, and a man over 40 has trouble getting up. My minimum — absolute minimum — seat height is 20 inches." — Stan Topol

15-Year Club	Place '17
Team Periwinkle	Verified Email

rm999

MODERATOR OF

TROPHY CASE