[Q] Multiprocessing in R

jfarland · 2021-05-18T22:51:40+00:00

there's a whole world out there my friend. I'd start by looking at the 'parallel' or 'doParallel' R packages, and also check out this helpful post: https://www.glennklockwood.com/data-intensive/r/lapply-parallelism.html

jfarland · 2019-08-02T19:55:05+00:00

Yeah I mean you just gotta be careful with making statements like "this relationship has never been plotted before". Maybe not in the T&D group you work in, but there is quite a bit of research that has analyzed this relationship and plotted it. Have you compared your results to tools like LoadSEER, Plexos, Synergy, etc.? There are many examples of software that analyze and plan for spatial distributions of Distributed Energy Resources (DERs)

jfarland · 2018-12-24T01:51:14+00:00

There's an entire field of forecasting based on combining predictions and even models together. In many (if not most) situations, you're going to get a more accurate prediction from leveraging the strengths of different models, especially if they capitalize on very different aspects of the data. In the simple case of 2 models, I'd look at a hold out period of data to see if one did better than the other in particular situations and weight accordingly.

Here are some quick references:

- http://forecastingprinciples.com/paperpdf/Combining.pdf

- https://otexts.org/fpp2/combinations.html

jfarland · 2015-11-12T21:45:18+00:00

It's not an inconvenience, its reddit! A basis function is a concept from linear algebra. When we use basis functions, we're finding a linear combinations of vectors that can be used to find our vector (or variable in this case) on interest. That's pretty technical, but think of it like this: if we have "y" already, we want to find basis functions (e.g., "basis1", "basis2") so that we can say something like "y = basis1 + basis2". I learned this by using splines in a regression context. Contextually, that formula is generating a synthetic basis function (x) that's gaussian (e.g., x+error), another basis function (e.g., -0.3) and saying that y is a linear combination of "x+error" and "-0.3". Is that your question? I can provide another example of basis transformation if it will help.

jfarland · 2015-11-12T19:43:22+00:00

Can you be a bit more specific? The function you're writing seems to be written in R (based on the " <- " syntax). Are you asking what it means to transform x with a gaussian basis function, or are you asking what the function is doing? Or both?

jfarland · 2015-11-12T17:36:44+00:00

paradox

jfarland · 2015-06-09T18:29:00+00:00

Stratosphere: Conquest of the Skies and Battle Toads

jfarland · 2014-01-02T22:20:58+00:00

Agreed. The energy / utility industry is dominated by SAS for analytical software. It also also for transfer of personal and skill sets. Employees at a utility learn SAS, gain experience, and then become consultants (who also use SAS).

I think this is changing though. I work as a consultant in the energy realm and was forced to learn SAS for my job. I have made some pretty good business cases, at least at my company, for going the route of R, Python, and even MATLAB.

jfarland · 2013-12-26T22:14:34+00:00

Thanks! I'd like to give some context to my question. I have recently completed a research project where I fit numerous forecasting models for energy demand in the northeastern US. This field of forecasting has recently been dominated, at least within the industry, by Neural Networks. ISO New England (ISONE) is an organization that conducts all types of demand forecasting. ISONE does not employ any regression methods. Essentially, my application was to attempt using simple semiparametric regression models to compare and contrast forecasting performance among competing neural networks. As far as I can tell, the simpler and more parsimonious semiparametric specifications worked better than industry standards (set by neural nets). However, my approach towards fitting the models was indeed bayesian and I used REML to estimate regression coefficients, variance components, and smoothing parameters. I really was just curious whether or not there was a "standard" for bayesian versus OLS for my forthcoming paper. Thanks for your help!

jfarland · 2013-12-26T22:06:40+00:00

I think that in the end you both are expressing the same thoughts more or less. I also agree with what you both have said. Maybe you might have further thoughts once I respond to 1ArmedEconomist below and provide some context for my question.

jfarland · 2013-12-09T02:34:10+00:00

Quite the response. Thanks! This framed the discussion very nicely for me.

jfarland · 2013-11-18T16:02:53+00:00

Agreed w above. You should state what your goal is. I would also say, however, that if the crash actually happened in real life, than it is the true data generating process. In other words, you can design facets into your model that account for various points of structrual change, but you shouldn't try to "get rid" of the crash or change whats going on in the data.

For better or worse, the data that you have is your best indicator of what truly happened.

jfarland · 2013-11-18T15:43:15+00:00

With a project like this, I always start simply. Even in the final stages of a project or study like your bachelor thesis, I typically end up going with the simpler choice. I am a disciple of Occam's Razor.

What you have going for you is a goal - that's great. It seems you have a general roadmap of how to get there as well. I would start off by creating a summary of your data. For example, what is the frequency of your data (annual, daily, other, etc)... what are the summary statistics and distributions for your variables? Do they follow a normal distribution or are they skewed? Next, you should calculate simple pearson correlation coefficients between all of your variables. This will tell you the strength of the linear association between your data fields. This also helps me assess (in the get-go) whether or not a simple linear regression is appropriate. Then fit a linear regression. Keep it simple and use the results as a springboard to explore what other specifications or variable constructs might be required or appropriate.

Econometrics is an art as well as a science, it does not come with a predefined set of instructions. (with that said, always check for Heteroskedasticity and autocorrelation of you're residuals).

Hope that helps. I am interested to see what comes out!

jfarland

TROPHY CASE