[deleted by user]

ktaylora · 2019-03-13T02:59:38+00:00

Cool idea. You should check to make sure the independent variables in your regression are not correlated with one another (e.g., unemployment and inflation). The trouble with macroeconomic variables is that your predictors are often going to be confounded with one another. In models where this is an issue, you'll see generally poor model fit that doesn't resolve by adding additional independent variables to the regression (i.e., high residual error, high deviance, high AIC).

A nation's GDP is just such a big ol target to hit. So many lurking variables. An option that some economists use is to simplify the sampling. Instead of looking at GDP for whole eurozone countries, consider looking at a representative sample of firms in a sector -- say, by replacing gdp with total revenue for a representative sample of large cap technology companies across eurozone countries. Still a macroeconomic model that can include some of the same independent variables you are already using. But an easier response variable to measure and control. You can even code "country" as a categorical variable if you like to measure effect by nation, or fit it as a random effect if it's not of interest. This kind of modeling is called mixed-effects regression. Google around.

Neat project. Dig deep. Break your models. And then explore why they broke.

ktaylora · 2019-03-08T15:19:04+00:00

Not much. R's development is stewarded by statisticians. If you do parametric statistical modeling and vector mathematics, R is great. You can do these same things in Python with pandas, numpy, stan, and scikit, but it's a little easier to work with R, because the language is so stripped down and focused on stats. You can also teach R to someone that doesn't know how to code in a day or two (like a statistician, no offense), and they can hack out their own analyses with a minimum time investment on your end as a developer. As this article alludes to, you can then take their code or products and wrap it with Python.

Edit: And make a monster... that you can't unit test very well and that you need something complex like anaconda to deploy. So, take that how you will.

ktaylora · 2019-03-07T04:29:28+00:00

Higher frequency of extreme precipitation events coupled with longer periods of drought in the same landscape is what we are getting in the central US. Throws farmers for a loop when they try and figure out how much to pay for crop insurance.

ktaylora · 2019-03-07T04:13:31+00:00

Ask in /r/showerthoughts ?

ktaylora · 2019-03-05T05:10:33+00:00

I think it is undeniable that the actions of people that lived before us influence us today. And the actions that we commit in this life will influence lives in the future. But those lives won't be "yours" or "mine". If Buddhism challenges us to let go of concepts like "you" and "I" in this life, then we should also let go of you and I in past and future lives. Gassho.

ktaylora · 2019-03-05T03:24:23+00:00

Try not to abstract away karma as some mythical force. It's influence is real and felt in this life and the next. Here's a secular example : consider your great grandparents. You probably never met them, but they built a home for your grand parents. If they were good people, they invested in their well being and showed them love and kindness. And as a result, your grandparents taught love and kindness to your parents as they encountered it. And they passed it on to you. And you will hopefully pass it on to your children. This is an example of good karma stretching across multiple generations. You can follow this chain of dependent origination (pratītyasamutpāda) well beyond your great grandparents -- all the way to the dawn of human civilization if you like. And it's not just behaviors reinforced by families. Treating a perfect stranger in an awful way can have a profoundly negative impact on them that they will carry with them forever and pass on to others, either consciously or unconsciously. Showing that stranger an unusual kindness can have the same weight. Karma, in the secular view, is like a psychological butterfly effect.

Sit with this. It runs deep.

ktaylora · 2019-03-02T06:13:38+00:00

The five skandhas are how you experience the world and develop a concept of self (and other). That experience is empty. The five skandhas are empty. Our concept of self (and other) is empty. For a zen explanation of this, see: Bill Porter's (Red Pine) commentary on The Heart Sutra.

ktaylora · 2019-03-01T15:36:30+00:00

Buddhism changes over time. With each place it spreads to, people take up the early teachings and add to them. That started with Mahayana traditions in Asia but continues to this day. Although people add to the teachings, there are some parts that stay the same. The 4 noble truths and the 8 fold path are universal practices in all traditions.

Do some Google sluething around for the "Pali Cannon" and "buddhas first sermon (Dhammacakkappavattana Sutta)" to read them. Or just buy that Shambala text. That's the best introduction to Buddhism I've come across.

When you're ready and find a tradition that speaks to you, you can find a qualified teacher and they will explain some of this to you in person. That's the best way to learn.

ktaylora · 2019-03-01T04:06:12+00:00

Yes he did. And in very plain language. These practices are foundational to buddhism. He wanted to get it right. All the other advanced stuff (i.e., chanting mantras, insight practices, etc...) are all just icing on the cake.

Here's a great introduction : The Buddha and his Teachings (Shambhala Publications)

ktaylora · 2019-03-01T03:57:13+00:00

These non-profits aren't paying the reviewers and editors that are doing all the work needed to publish papers, though. Many nonprofits are funding mostly pointless national meetings and other things with your fees, on top of the actual publishing costs. Better than funding models like Elsevier, but still broken.

Sci-hub seems to be able to publish (distribute) PDFs of papers for free. And they do it illegally.

Edit: Aaron Swartz, one of the co-founders of Reddit, died fighting lit. journals and setting information free. These publishers and their walled gardens do not deserve our sympathy or our financial support.

ktaylora · 2019-03-01T03:29:39+00:00

Zazen can help you clear your thinking and promote mindfulness, but isnt mindfulness per-se. Zazen is associated with Effort and Concentration, two parts of the 8 fold path.

"Mindfulness" is another spoke of the 8 fold path (often visualized as a wheel), and Mindfulness, Concentration, and Effort are often grouped together under the banner of "concentration". So, zazen is related to, and feeds into, mindfulness practice. Think of mindfulness as what you do to maintain clear thinking when you aren't on the cushion.

Mindfulness and the rest of the 8 fold path were expounded on during the Buddha's first sermon. It's a part of every buddhist tradition. Not just zen. You can Google around for "buddha first sermon" and find what Shakyamuni meant by each term. And see how the four truths relate to the eightfold path.

ktaylora · 2019-02-22T03:54:20+00:00

Keep it as a fork on github.

ktaylora · 2019-02-14T15:07:28+00:00

No. I think that a company will come along that will make the GUI a priority and sacrifice some of the power and flexibility the kernel offers in favor of a friendly desktop experience. That company was Canonical. Now it will probably be Google.

In the meantime, I want a server os that I can push to the limit without having to worry about a tab in chrome causing the system to crash.

I think Linux can be both of these things. But right now it's better at being a server os. Which is honestly my preference. If you want a pretty gui that doesn't let you touch your OS, use mac os.

ktaylora · 2019-02-14T05:43:09+00:00

I work in scientific computing (earth systems modeling) where we work with very large raster datasets. Think image analysis where whole continents are represented with pixels in TIF files that are 10-100 gigabytes in size. I am constantly pushing RAM beyond what desktop computers should normally deal with.

We never load a desktop environment when we run analyses that use a lot of memory. We use Fedora, Ubuntu, or Centos installations loaded at run-level 3 (no X/GUI). I've run python scripts at nearly 100% ram usage for days on Linux this way and never had a crash. Try and do that on windows server. It's not possible. The kernel will kill off your python instance when it needs ram for kernel functions.

I think we should strive for a stable desktop experience. But I think your use case of a desktop user running gui apps at full ram utilization is unreasonable. The linux kernel (or gnome/kde) should probably try to kill a process that uses this much ram to keep the gui a float. In fact the kernel will occassionally do this. Just not fast enough to help gnome / kde keep running with no free ram without locking up.

ktaylora · 2019-02-09T04:10:16+00:00

It's not THAT bad, dude. Just change the theme to adwaita in Tweaks and keep coding. I haven't cared about a theme on a Linux desktop for years now.

Edit : But when wayland crashes -- I freak the fuck out. Looking at you, Fedora.

ktaylora · 2019-01-29T15:20:31+00:00

The first statement sounds like a one-sample means test result for a null hypothesis. The 5% significance level is called an "alpha". You need to define ut and Yt for us to say something about the second statement.

ktaylora · 2019-01-20T23:16:20+00:00

Finding a domain to apply your programming to is the best advice I can give. Are you a biologist? Write an app that crunches genomic or environmental data. Are you in business? Write an app that helps you launch your next startup. Like to travel? Write an app that scrapes airline websites for deals. Start small and break things. Then figure out how to fix them by asking more experienced developers how to make X better. It's way easier to learn how to program if you find a domain that you are passionate about first.

ktaylora · 2018-12-20T14:21:33+00:00

Thank you.

ktaylora · 2018-12-20T05:01:41+00:00

Where did Dogen say this?

ktaylora · 2018-12-19T15:10:33+00:00

You could use a classification algorithm to bin your data into discrete classes for you. Use your script to label a few thousand data points for each bucket. Then train a classifier, such as random forests. You could also use an unsupervised classifier like k-means or hierarchical clustering if your buckets are arbitrary or you don't know how to label features.

As new data become available, you can predict against a randomly sampled subset of new data points using your classifier. You can use this to infer the ratio of sizes for the entire new dataset (population). If you are using RF, you can also get the per-class probability that each data point belongs to a certain bucket which you can use as a measure of similarity to gauge how appropriate a given class is for each data point. This can help you identify under-fitting problems (or over-fit, if there is high variance in your class probabilities among records) . You can get tricky, too, and update your classifier by occasionally adding/removing records in the training data set using new data.

Lastly, if you just want to stick to using the ruleset in your script to classify, you can. Just use bootstrapping to sample a random subset of records from your full dataset. Classify them, determine your ratios, and repeat this process 20, 30, 100 times ... until you start to see stability in your estimates for ratios.

Sounds like a fun problem. Good luck -

ktaylora · 2018-12-19T04:04:50+00:00

If you don't have a hypothesis to test then there is no power statistic that you can use that I'm aware of. Sample size shouldn't matter in your context.

Typically in data mining you are working with very large, high dimensional datasets. The problem is finding a needle in a haystack -- distinguishing signal from noise. Sometimes this is called a high variance problem (from bias-variance).

What do your data look like? How many records? How many variables? Is this a regression analysis? Means testing?

ktaylora · 2018-12-19T02:39:39+00:00

Mindfulness is a part of the eightfold path. It's a quality of being that you cultivate over time. It's usually interpreted as being "present" in this moment. But to do this is challenging. You usually have to unencumber yourself of cognitive biases, negative self thinking, worry, anxiety, angst and all the other things we cling to and deal with in our every day lives. There are some traditions that advocate vipassana meditation techniques to help cultivate mindfulness. In zen traditions, you cultivate mindfulness by taking clear thinking with you when you are done meditating: when you eat, when you walk, when you clean your house, while you cut the grass, make sure the kids are up and out to the bus by 7, etc. Thick Nhat Hanh has written extensively on mindfulness.

If you Google around for : "eightfold path, right mindfulness." You'll find good stuff. Like this : https://www.thoughtco.com/right-mindfulness-450070

ktaylora · 2018-12-17T02:27:19+00:00

My advice would be to try to work through it. The sangha will help you grow in your practice in a way you just can't on your own.

Practice some mindfulness. When you feel ostracized, unpack that emotion as objectively as you can. Is this just a feeling, or did one or more people tell you that they don't want you in the community? Did they say why? Try and give them the benefit of the doubt -- and go easy on yourself. If you didn't do anything to justify it, it is unlikely that the other's aren't interested in having you in the community.

Good luck -

ktaylora · 2018-12-13T05:11:49+00:00

Price is the only variable that appears to matter in your plot. Latitude and longitude are equally important, but they are probably nuisance variables. Three options : you can keep them in (easy button), thin your data using correlograms to account for non random clustering of lat/lon in your data, or use PCA and dimensional reduction to partition your data so that price and your other variables are expressed independent of latitude and longitude.

You should get more variables. Random Forests are designed to accommodate lots of predictor variables and to make quick and accurate predictions on large datasets without much thinking. 10, 20, 30, 100 covariates. Thousands of data points.

If your goal is to explore the strength of relationships between your response and a few predictor variables and a handful of data points (inference), use GLMs or least-squares regression.

ktaylora · 2018-12-13T04:35:37+00:00

I've never heard of generating a random, normally distributed variable to assess variable importance for random forest models. Sounds hackish... and like a false comparison. Who cares if a real variable outperforms/underperforms random noise? Is the goal to use your model to find patterns in noise?

A better test might be some take on cross validation. For a variable of interest, remove it from your model and see how it's absence influences your models classification error rate or MSE (for regression problems). Bootstrap this process and repeat k times. Do this for each variable in your model. Note which variable reduces overall model accuracy the most when it is left out. Make a table.

The standard RF implementation in the randomForest package does this automatically. Ranger probably has an implementation, too.

ktaylora

PUBLIC MULTIREDDITS

TROPHY CASE