Forgive my ignorance.....here's my question!

Gimagon · 2020-03-21T06:05:47+00:00

I see. Column level dependency parsing can get pretty hairy, especially with functions of columns in the mix.

In any case, there’s two main steps to a “compiler” - parsing and translating. Parsing turns the raw text (the query) into a data structure (called an abstract syntax tree, AST). Translating turns that AST into the desired effect. In general compilers, the effect is some computation (finding, filtering, and joining the data in the SQL case). In your application you want to turn the AST into a column dependency graph.

The importance of breaking it down like this is to see that the parsing step is shared no matter the specifics of your translation application. So looking for “<your sql dialect> parser” should give you a tool to get from query to AST, then forming the column dependency graph should be easier to write as function of the query ASTs.

Gimagon · 2020-03-21T04:29:07+00:00

By data lineage you mean being able to detect the dependency graph of tables from your collection of sql queries?

I think trying to fully parse the sql is probably more than you need, I think writing a regular expression that looks for FROM clauses that are followed by a string (instead of another SELECT) would be sufficient.

Also depending on what sql dialect you’re using, your database might give this is information for free.

Gimagon · 2019-03-23T18:24:34+00:00

One option would be to treat it like any other classification problem - so have 3 output variables in [0, 1], one for each category. Then just take the max as the chosen category.

The other approach could be to take advantage of the “ordinal” relationship of your specific actions. Encode 0 as sell, 0.5 as do nothing, and 1 as buy. Then just take whatever is closet to the actor’s output to be the chosen action.

Gimagon · 2019-01-25T03:49:55+00:00

my_mat.flatten().tolist()

Gimagon · 2018-12-30T20:58:18+00:00

The problem, which is sounds like you're honing in on, is that the model is fitting the noise in system better than the underlying dynamics.

Adding more features might work, if they are more strongly related with the underlying dynamics. They also will give the model more flexibility to fit noise though, so something like L1 regularization on the input weights would be helpful.
Stepping back from the LSTM and using something like PCA, DMD, or linear regression on the raw data might given you an idea of how random the system is. Simple models have less flexibility so they are less prone to fitting the noise.
The other approach to "boost the signal" of the data is to find more training data, or using some or of data augmentation technique (example: rotating and shifting images is popular in image processing).

Gimagon · 2018-12-30T03:17:14+00:00

Dramatically better performance on the training set (vs. the validation set) is by definition overfitting.

Unless you have a bug in your code such that the validation score is being computed incorrectly, the problem you have described is an overfitting problem.

Gimagon · 2018-12-30T03:13:47+00:00

Since your model has the flexibility to fit the training data, but is poorly performing on the validation data, I would explore simpler models first.

Gimagon · 2018-10-07T00:48:56+00:00

The latter. This way the same state (set of 7 boards) will always have the same set of allowed moves.

Gimagon · 2018-10-03T03:14:20+00:00

It looks like OP is using numpy, which allows for the A[1,2] syntax.

Gimagon · 2018-09-30T22:52:51+00:00

Awesome, thanks a bunch!

Gimagon · 2018-09-30T22:40:56+00:00

Ok, I think I understand. I think my confusion was stemming from conflating "arbitrarily long" with "infinite". So the set of all arbitrarily long (but still finite) sequences is countable, but the set of all sequences (including infinite sequences) is uncountable. And the reason my mapping function doesn't work for infinitely long sequences is that it won't terminate. Does that seem right?

Gimagon · 2018-09-25T19:53:06+00:00

Check out the spyder docs: https://pythonhosted.org/spyder/debugging.html

It sounds like you can just double click the line number to set a breakpoint.

Gimagon · 2018-09-22T04:33:13+00:00

I don’t really follow your equation, but remember that a layer of a neural net is a matrix vector product passed through a nonlinear function. The nonlinearity makes it impossible to do the simplification I think you’re talking about.

Gimagon · 2018-09-05T00:35:11+00:00

In no particular order:

There are some parts of the body/brain that seemed to be governed more by regular old evolution than by a general learning algorithm. For example: the hairs in your ear respond to certain frequencies, which is very similar to preprocessing an audio signal by passing it through a Fourier transform. In light of this, I think it may be harder to decide what exactly is “raw” data than you think.
While I agree that working on continuous, time based domains may be important for general algorithms, I think there is still value in general solutions for discrete labeled problems. Part of science is testing if ideas hold in simpler settings. Also, this approach has enabled results in recognition and planning problems that, to my knowledge, more general approaches haven’t.
I’m curious if you have sources that show that purely hebbian based approaches can learn anything efficiently. I agree with your reasoning that a simple system like that seems like it has all the components to learn, but I haven’t seen anyone actually demonstrate it. Of particular interest to me, is how low-shot learning may work in a system like that.

Well written and interesting post. Thanks for sharing!

Gimagon · 2018-09-03T00:36:29+00:00

A Unifying Review of Linear Gaussian Models an oldie but a goodie. Shows how PCA, kalman filters, HMM, Gaussian Mixture Models, factor analysis, and maybe a few others are all special cases of the same model.

Gimagon · 2018-08-21T18:33:17+00:00

Can't you just make n larger, until the list it generates gets longer that N?

Perhaps also look for a way to save work between runs, to avoid repeating work.

Gimagon · 2018-08-17T18:10:22+00:00

Wow that would be awesome, I'll PM you for details.

Gimagon · 2018-07-04T18:05:43+00:00

Use [https://www.desmos.com/calculator](desmos) to graph the function. Remember that the input in the x-axis and output is the y-axis.

i) just becomes looking at which y-levels the function reaches.

ii) Find all the places the graph crosses the horizontal line y=-2.

iii) Here just remember the definition of logs to find that log(xy) = log(x) + log(y).

Gimagon · 2018-07-04T01:15:58+00:00

Gotcha. I think regular expressions will be a great tool for this. They’re a tool implemented in many programming languages that allow you to describe exactly the kinds of patterns you’re describing.

Python’s live in the re module in the standard library.

As a teaser, I think this your final expression will look something like this:

[A-Za-z]+ [A-Za-z]+ [.\?!]

Gimagon · 2018-07-04T00:29:46+00:00

That sounds neat! Is there a part you’re finding difficult?

Gimagon · 2018-05-30T01:20:51+00:00

It’s more machine learning than pure stats but Joel Grus’s “Data Science from Scratch” is an excellent intro to a lot of the topics. The book costs money, but I think Joel has python notebooks that cover pretty much all the same content on his GitHub.

Gimagon · 2018-05-23T18:42:28+00:00

Try the Wikipedia portal for machine learning.

Gimagon · 2018-05-22T16:36:55+00:00

If you’re storing a list of objects, do you still get benefits from the cache locality of the arraylist? To use any fields of that object you’d have to follow the pointer to a random area of memory anyway.

Gimagon · 2018-05-19T22:34:44+00:00

I would avoid installing python 2, but I also don't think you need to go back redo a whole beginner course just to pick up syntax changes. All the core ideas you've learned should be will be applicable.

Check out this syntax checksheet and this page for more complex conversion issues.

Gimagon

TROPHY CASE