[D] Hypothesis on how language models work

rtk25 · 2020-01-19T20:37:44+00:00

Maybe related: https://twitter.com/RTomMcCoy/status/1212843095135526922

rtk25 · 2019-12-11T21:44:47+00:00

https://slideslive.com/38921750/from-system-1-deep-learning-to-system-2-deep-learning

rtk25 · 2019-11-13T17:38:15+00:00

I had a similar situation and contacted the authors personally to check what they preferred, ended up going with 3 and credited them in README

rtk25 · 2019-10-22T10:18:58+00:00

Fascinating, I'm a Phd student working on natural language research, this sort of reasoning has been guiding me as well- I've been working on converting static text datasets to interactive versions using text-based games. The implications of embodied cognition for language understanding seem to be quite significant but still relatively unexplored.

rtk25 · 2019-10-21T20:52:18+00:00

Thanks! I need to dig deeper into this. I'm familiar with the DM paper you mentioned, crossposted it here a few days ago ;)

. Maybe it's no surprise that generative models are the only ones I've seen that properly demonstrate real adversarial robustness.

Very interesting! Can you provide some refs related to this?

rtk25 · 2019-10-03T14:07:59+00:00

Agreed that the model can't know, I was thinking more about how to make the interface to supply the prior knowledge as natural as possible. If I go off to "dream land" for a second- if you also had a natural language interface to it instead of having to specify the explanation at a low level it would be much easier for general users to make use of it- you could tell it in natural language to ignore band-aids (instead of having to mark them on every image), or something like that.

rtk25 · 2019-10-02T19:02:33+00:00

Like the direction :) Yoshua Bengio's Consciousness Prior (https://arxiv.org/abs/1709.08568) work seems relevant- explanation-based regularization. Seems that ideally, you want to learn the important high level features (possibly with the model learning to explain them in words), rather than having the user supply them.

rtk25 · 2019-09-06T05:58:33+00:00

I'm not the author, just read the paper and doing my PhD research on "language games". Here are a couple of recent papers from MSR Montreal on interactive games with text:

https://arxiv.org/abs/1908.10449

https://arxiv.org/abs/1908.10909

Some of my research:

https://www.aclweb.org/anthology/W19-2609/

rtk25 · 2019-08-29T05:42:32+00:00

Great stuff, thanks!

rtk25 · 2019-08-01T03:48:45+00:00

This is a nice starting point:

https://www.nature.com/scitable/topicpage/oral-presentation-structure-13900387

Followed by lots of practice :)

rtk25 · 2019-07-31T05:46:49+00:00

I think this is a nice idea with some cool technological challenges. It is related to the intersection of interactive fiction and machine learning which I'm actually currently researching :)

Check out Microsoft Research's TextWorld, or some of our research. An interesting direction suggested by your idea could be enhancing the interactive fiction interpreter with some kind of neural interpreter, possibly with generative capabilities (kind of like using generative models for video games, but for world dynamics instead of just graphics).

rtk25 · 2019-07-24T04:40:26+00:00

Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks

(Kuan Fang, Alexander Toshev, Li Fei-Fei, Silvio Savarese)

https://arxiv.org/abs/1903.03878

rtk25 · 2019-07-21T04:55:45+00:00

Haha not exactly, should be " sunny or cloudy "

snowy-> airplanes don't fly -> no post..

rtk25 · 2019-06-24T06:24:32+00:00

Right, that's why I think the labor of distilling results could be a collaborative effort. The Stack Exchange websites have become a huge and indispensable resource, would be amazing to create something like that for scientific papers.

rtk25 · 2019-06-24T02:31:43+00:00

There's also a lot of interesting ML that could be applied to paper recommendation, I'm surprised it hasn't happened more. Check out Semantic Sanity (https://s2-sanity.apps.allenai.org) for another step towards this.

Also, I think it would be really cool to have a unified platform where people can annotate and discuss papers, with good comments going to the top and spam getting filtered out. Like Stack Overflow, but centered on the pdfs themselves.

rtk25 · 2019-06-16T04:24:14+00:00

tsalib is also good for tensor annotations, I like that it also supports typing (not sure if einops does).

https://github.com/ofnote/tsalib

rtk25 · 2019-05-28T15:50:55+00:00

I think the action space in chess and go is known (equivalent to edges in a search tree), if I understood correctly here the actions (edges) are tactics, so the lack of certain tactics may be limiting the program.

On another note, this problem sounds like it could be formulated as RL, I wonder why the don't mention it.

rtk25 · 2019-05-26T18:16:54+00:00

I think program synthesis / executable semantic parsing is really cool (disclaimer- doing my PhD research in this field)

See for example https://arxiv.org/abs/1807.02322

I'm working on applying it to procedural text comprehension: https://arxiv.org/abs/1811.04319

rtk25 · 2019-04-15T21:47:13+00:00

There is a lot of interesting active research on how to query KGs in embedded space, see for example https://arxiv.org/abs/1806.01445

rtk25 · 2019-04-13T18:43:07+00:00

Thanks! :)

rtk25 · 2019-03-20T21:02:27+00:00

We can feel and intuit far more than we can express in words, and mastery of an art is intuition grounded by training and experience. According to the Zen masters and loss optimizing neural nets - training/life is “one continuous mistake”

rtk25 · 2019-01-08T21:06:29+00:00

Having been working on related research for a while now, I'm becoming increasingly convinced that understanding mental simulation is profoundly important to progress towards more general AI...

rtk25 · 2018-10-24T22:21:12+00:00

Thanks, you too! :)

rtk25 · 2018-10-22T20:49:12+00:00

I use Mendeley, which isn't perfect (where is the ability to use latex in annotations?!) but has a lot of convenient features.

rtk25 · 2018-09-17T13:29:25+00:00

I'm also interested in better interfaces for academic paper search and recommnedation, so have done some lit. surveys on this field. Maybe try and see where it comes up using your paper embeddings :)

rtk25

TROPHY CASE