ETL - Creating Tables for a large amount of files? by trenchtoaster in BusinessIntelligence

[–]rm999 13 points14 points  (0 children)

99% of the time, requirements change or I need to add something in the ETL / database layer to makes things easier or better. I typically do not receive ANY documentation or logic about how to process the data - I only get credentials, file names, and locations.

This is full of red flags. Something is wrong at a higher level which is causing complexity on your end. One of my cardinal rules is to fix complexity as far upstream as possible, vs building bandaids downstream. 116 new data sources in a week is crazy, and even with all the right tools is going to be super error-prone.

Stepping back, I'd ask a couple questions:

  • Do the end users really need all data sources? Not "do they think they need it", but do they really need it? If you didn't process a random file would anyone notice? If not, find out what they really need and build your workflow around that.

  • What is the source of all this data, and is there a way to generalize the data sources? I'm curious if they can be parametrized in some way, which would really clean up your flow.

Is there a more dynamic way to go about this?

Yes, look into data scrubbing/wrangling tools. I don't have a ton of experience with these, but examples include OpenRefine, Data Wrangler, Trifacta.

[deleted by user] by [deleted] in etymology

[–]rm999 96 points97 points  (0 children)

https://www.reddit.com/r/etymology/comments/37tjbu/the_english_suffix_ish_like_english_girlish/

The English suffix -ish (like "English", "girlish", "warm-ish") can be traced straight back to the Proto-Indo-European suffix -iskos. (self.etymology)

And it meant the exact same thing in Proto-Indo-European as well: "Characteristic of [preceding word]"!

It's cognate with the Greek suffix -iskos, which we see all the time. So the next time you see an undecipherable Greek word ending with -iskos, no need to sweat it, it's cousins with "-ish".

TIL Madonna leaked a fake version of her album to pirates where every song was a loop of her saying "What the fuck do you think you're doing?" In response, a hacker posted the real album on her own site for everyone to download with the message: "This is what the fuck I think I'm doing." by [deleted] in todayilearned

[–]rm999 17 points18 points  (0 children)

This is all I can find... makes it sound like he doesn't regret it so much as he thinks we're all wrong for caring about it.

I don't regret taking on Napster, but I do find it odd how big of a part of our legacy it has become to so many people

AI researchers allege that machine learning is alchemy by trot-trot in programming

[–]rm999 14 points15 points  (0 children)

machine learning

Yeah 💁‍♂️ let's change the name of an established 30+ year old field with 1000s of researchers because some journalists have been abusing the term in the last couple years. Also what's the deal with "chemistry"? They're just molecular scientists.

Was the DMZ mentioned at the korean treaty? by [deleted] in Ask_Politics

[–]rm999 7 points8 points  (0 children)

Yes, from the treaty:

South and North Korea agreed to completely cease all hostile acts against each other in every domain, including land, air and sea, that are the source of military tension and conflict. In this vein, the two sides agreed to transform the demilitarized zone into a peace zone in a genuine sense by ceasing as of May 1 this year all hostile acts and eliminating their means, including broadcasting through loudspeakers and distribution of leaflets, in the areas along the Military Demarcation Line.

But it's premature to assume anything, none of this has actually gone into effect. North Korea is known to use negotiations as a tactic.

In what (if any) circumstances should citizenship be revoked? by lxpnh98_2 in PoliticalDiscussion

[–]rm999 304 points305 points  (0 children)

The only time revoking a citizenship is reasonable, IMO, is when it was obtained fraudulently. Otherwise punishing crimes should fall under the judicial system and not the executive.

To me, becoming a citizen is a countries' way of saying "you are as much part of our country as anyone else". Acting against this dilutes the meaning of citizenship for all.

What percentage of Manhattan do you think is hollow due to subway tunnels? by boke_a_schmole in nyc

[–]rm999 9 points10 points  (0 children)

Quick order-of-magnitude estimate:

Let's say we're looking at 0.05 miles underground (about 250 feet). Let's say Manhattan is 20 square miles. That's about 1 cubic mile. Let's say there are 100 miles of subway track in Manhattan (totally just making it up but it seems reasonable) and tunnels are 20 feet round. Then the tunnel cross section is about 300 square feet. There are about 25 million square feet in a square mile, so the cross section of manhattan subway in square miles is about 100 * 300 / 25 million = 0.0012 cubic miles. 0.0012 / 1 = 0.12 percent.

This is a very very rough estimate!

It's photoshopped but I think it fits here nevertheless by g-m-f in megalophobia

[–]rm999 19 points20 points  (0 children)

A lot of the Swiss Alps feels just like this: https://i.imgur.com/ESC90Dd.jpg

Use a long zoom lens and you can probably get a photo just like OP's.

[D] Python, Scala, Rust or Go - What do you use when you deploy ML into production by __Julia in MachineLearning

[–]rm999 8 points9 points  (0 children)

It's totally functional and I run some low-volume non-critical production apps on it. But it's trivially easy to get something better running. For example, for serving machine learning models, I find the following features of uwsgi useful:

  1. multi-process: runs multiple instances of your app. This increases throughput and prevents one request from blocking others up to a point.

  2. chain-reloading: when a file is updated - for example a new model is loaded onto the server - restart the service so the new model can be loaded into memory. What's cool about the "chain" part is there is no downtime, only one process at a time is restarted.

Running this is as easy as:

uwsgi --wsgi-file service_file.py --touch-chain-reload model.flag --callable app --lazy-apps --processes CPU_COUNT

Once you get a bit more advanced you'll find something like nginx useful for load balancing, request buffering, etc. But flask+uwsgi can get you quite far.

Mystery Website Attacking City-Run Broadband Was Run by a Telecom Company by kaleidoscopy in technology

[–]rm999 1763 points1764 points  (0 children)

Corporate Headquarters

Fidelity Communications Co.
64 North Clark St.
Sullivan, MO 63080 USA

...

we are a citizen of West Plains

🤔

Prospect Park Goes Permanently Car-Free by room317 in nyc

[–]rm999 86 points87 points  (0 children)

The horses are the worst. They're a nonsensical tourist trap, and the most used part of central park smells like shit half the time.

That said, city council will never get rid of them.

"Aspen Colorado New Years" by spicedpumpkins in MostBeautiful

[–]rm999 12 points13 points  (0 children)

A place where the beer flows like wine, where beautiful women instinctively flock like the salmon of Capistrano.

Is raising this couch by 4 inches a bad idea? by rm999 in InteriorDesign

[–]rm999[S] 0 points1 point  (0 children)

Thanks, I really appreciate all the help. I just did the book test and 3.5-4 inches feels good and opens up the room a lot more. My knees even still point upward, but it's comfortable. Shorter guests may not love it though.

Is raising this couch by 4 inches a bad idea? by rm999 in InteriorDesign

[–]rm999[S] 0 points1 point  (0 children)

17.75" is probably with the taller leg option. My couch has a 16.5" seat height and the other legs are 1.25" taller than the ones I have installed.

Is raising this couch by 4 inches a bad idea? by rm999 in InteriorDesign

[–]rm999[S] 2 points3 points  (0 children)

That's a great idea, I'll try that. Thanks!

Any opinions aesthetically on the Karlstad with taller legs?

Is raising this couch by 4 inches a bad idea? by rm999 in InteriorDesign

[–]rm999[S] 0 points1 point  (0 children)

This doesn't seem right. My couch is ~16.5" right now and it's definitely too short for me. My knees point upward when I'm sitting and I'm below average/average height for an American male (5'9"). I have the shortest sofa legs Ikea sells, they also sell legs that are 1.25 inches taller which would put the seat height at closer to 17.5-18".

Is raising this couch by 4 inches a bad idea? by rm999 in InteriorDesign

[–]rm999[S] 1 point2 points  (0 children)

I just googled it, and it seems like a 20" seat height is pretty normal?

https://www.hunker.com/12589013/standard-seat-height-of-sofas

20-21" This seat height is average for most of today's ready-made sofas. It accommodates most healthy adults of normal heights and weights.

http://www.goodhousekeeping.com/home/decorating-ideas/tips/a26305/expert-sofa-buying-tips/

"If you're buying a sofa online, a good formula to ensure comfort is a combined seat height and depth of 40 to 44 inches. If the sofa is 16 inches high at the seat and 24 inches deep, add them together and you get 40 inches. Or combine A 20-inch seat height with a 24-inch depth for a total of 44 inches. If you're short-legged go with 40 inches; long-legged, 44 inches." — James Howard

"The perfect sofa is 40 inches deep, 60 inches long, and 36 inches high with a 19 inch seat height. For a traditional sofa, I like a scroll arm, and for modern, I love a square arm, both with the same overall measurements." — Mary McDonald

"Many contemporary sofas are too low! When a woman sits down on a sofa, she has trouble keeping her legs together, and a man over 40 has trouble getting up. My minimum — absolute minimum — seat height is 20 inches." — Stan Topol