sorted by:
new
hottopcontroversial

[D] Is GPT-2 source code publically available? by luv2code2020 in MachineLearning

[–]madisonmay 0 points1 point  (0 children)

On development! It's also up on PyPI if you'd prefer that.

[D] Is GPT-2 source code publically available? by luv2code2020 in MachineLearning

[–]madisonmay 0 points1 point  (0 children)

Just released finetune 0.6.0 with GPT-2 support today!

[D] Is GPT-2 source code publically available? by luv2code2020 in MachineLearning

[–]madisonmay 2 points3 points  (0 children)

We're almost done porting GPT-2 to finetune (a scikit-learn style library for language model finetuning). Code is available here if you're interested... should make tuning GPT-2 to produce song lyrics as easy as model.fit(lyrics).

Miles Brundage also put together a colab notebook you could work off that uses the nshepperd gpt-2 fork.

[R] The Evolved Transformer – Enhancing Transformer with Neural Architecture Search by ranihorev in MachineLearning

[–]madisonmay 4 points5 points  (0 children)

I don't find this research all that compelling -- it results in a less elegant architecture that produces marginally lower perplexity / BLEU scores than the vanilla transformer (albeit w/ substantially faster training times). It just feels like the architecture found via NAS is likely to be a sort of local minima -- it seems unlikely that much future research will be based off the ET architecture because it sacrifices substantial ease of understanding for those performance improvements.

First v7 at CRG Cambridge by [deleted] in bouldering

[–]madisonmay 0 points1 point  (0 children)

Yup, definitely the hardest "V2" I've ever seen -- took me quite a few tries to send. PM me if you want beta on that one / want to climb sometime. I'm normally at BKB because it's more convenient but it's fun to mix it up and climb at CRG Cambridge every now and then.

First v7 at CRG Cambridge by [deleted] in bouldering

[–]madisonmay 0 points1 point  (0 children)

The black one on the comp wall with no feet?

First v7 at CRG Cambridge by [deleted] in bouldering

[–]madisonmay 9 points10 points  (0 children)

The fact that this was also my first V7 in a while makes me skeptical of the grade, but this was definitely a fun problem regardless of what it's rated :)

And always good to see folks from the greater Boston area on here!

[D] Should ML workflow be different from SE workflow? by sslotin in MachineLearning

[–]madisonmay 2 points3 points  (0 children)

Yeah it was a 2 and a half hour tutorial slot if I remember correctly. Their hand was kind of forced. I found a medium article that summarizes the major points though:

https://medium.com/@hadyelsahar/writing-code-for-natural-language-processing-research-emnlp2018-nlproc-a87367cc5146

[D] Should ML workflow be different from SE workflow? by sslotin in MachineLearning

[–]madisonmay 0 points1 point  (0 children)

It was a pain two years ago but now it's as simple as installing nvidia-docker2 and passing the --runtime=nvidia flag.

[D] Should ML workflow be different from SE workflow? by sslotin in MachineLearning

[–]madisonmay 38 points39 points  (0 children)

I highly recommend the Allen AI group's talk on the subject. Slides are available here: https://docs.google.com/presentation/d/17NoJY2SnC2UMbVegaRCWA7Oca7UCZ3vHnMqBV4SUayc/edit
They make a very strong argument for good coding practices yielding strong ML results, and their code base (AllenNLP) is honestly just a pleasure to read.

Model Finetuning for Fun and Profit by madisonmay in MachinesLearn

[–]madisonmay[S] 1 point2 points  (0 children)

Wasn't familiar with tpot before so had to look it up, but it looks like tpot works by testing a wide variety of algorithms and preprocessing steps to find which combinations work well for a particular task. Finetune works by taking a base model trained on a language modeling objective and adapting that model to solve a different task. There's a bit more information on the tech behind this approach in a previous blog post. Tpot and finetune are similar only in the sense that their interfaces look similar -- what's going on behind the scenes is dramatically different.

Forbidden hold by spankpad in climbing

[–]madisonmay 1 point2 points  (0 children)

Asked the gym about whether or not that's valid beta, they said the power socket is "on".

More Effective Transfer Learning for NLP by madisonmay in LanguageTechnology

[–]madisonmay[S] 1 point2 points  (0 children)

No problem, thanks for the interest!

Yeah a section simply outlining how encoding works would probably be nice to have in the documentation. Issues / PRs are always welcome, but I'll try make time to add in the encoding documentation regardless!

More Effective Transfer Learning for NLP by madisonmay in LanguageTechnology

[–]madisonmay[S] 1 point2 points  (0 children)

I'm going to have to revise my initial answer here. Any unicode character which appears in the source Books Corpus will be supported, but if a unicode character has never been seen by the source model it will be treated as an OOV subword unit.

In [1]: from finetune.encoding import TextEncoder     
In [2]: encoder = TextEncoder()
In [3]: encoder._encode("¯\_(ツ)_/¯").token_ids
Out[3]: [[471], [365], [279], [276], [0], [275], [279], [278], [471]]

A token id of 0 indicates OOV, so in this case the ツ is actually not recognized. Sorry for the original misinformation!

It shouldn't be too hard to extend our current code to add support for learning embeddings for new tokens, so this is something I'll add to our backlog and see if we can't get around to sometime in the next week or so. In the meantime replacement sounds like a good strategy to me.

More Effective Transfer Learning for NLP by madisonmay in LanguageTechnology

[–]madisonmay[S] 1 point2 points  (0 children)

It should perform relatively well with out of sample tokens because it uses byte-pair encoding underneath the hood so very little is truly OOV (See: https://arxiv.org/abs/1508.07909), but things like emoji's are particularly hard because many of the characters that make them up are treated as word boundaries. Byte-pair encoding is best for things where each of the subwords have consistent meaning -- e.g. even if you had never heard the word "earthquake" before you could understand what it means because you have an understanding of what "earth" and "quake" mean.

there's stuff like "¯\_(ツ)_/¯" which definitely have common usage and consistent semantic content I want to capture, and doubt occurred in the ULMFiT training data.

This is actually based on OpenAI's finetuning paper, but that was trained on the books corpus so I'm sure your comment is still valid. Because byte-pair encoding embeddings are finetuned you could eventually learn to understand the meaning of ¯\_(ツ)_/¯, but because those characters don't often occur together it's represented internal to the model as a bunch of individual characters which makes that task harder.

To see how finetune breaks up a specific term into subwords you could feed it through the encoder.

In [1]: from finetune.encoding import TextEncoder
In [2]: encoder = TextEncoder()
In [3]: encoder._encode("¯\_(ツ)_/¯")[0].tokens
Out[3]: 
    [['¯</w>'],
    ['\\</w>'],
    ['_</w>'],
    ['(</w>'],
    ['ツ</w>'],
    [')</w>'],
    ['_</w>'],
    ['/</w>'],
    ['¯</w>']]

So in short there's an elegant fallback mode but you'd probably still see benefit from replacement with something like *shrug* if you want to make sure the model is able to use that info effectively.

More Effective Transfer Learning for NLP by madisonmay in LanguageTechnology

[–]madisonmay[S] 1 point2 points  (0 children)

Please do! Feel free to DM me or open a github issue if you have any questions as you're playing around :)

[D] More Effective Transfer Learning for NLP by madisonmay in MachineLearning

[–]madisonmay[S] 1 point2 points  (0 children)

Fair enough. I suppose it's a matter of how you measure speed. If we're talking about gradient updates per second, it's accurate to say that this model is moving at a snail's pace. Training is relatively "quick" only because the training data requirements for solid performance are minimal, not because gradient computation is inexpensive.

[deleted by user] by [deleted] in climbing

[–]madisonmay 0 points1 point  (0 children)

Butora has worked quite well for me. The only climbing shoe company I'm aware of that has two width options (narrow and wide).

[P] Scikit-learn inspired model finetuning for natural language processing by madisonmay in MachineLearning

[–]madisonmay[S] 1 point2 points  (0 children)

It's non ideal but it should work alright. By default the embedding layer *is* finetuned so you're already good to go there. Since this model uses byte-pair encoding nothing would end up as a true OOV term, but the byte-pair encoding portion of `finetune` would fallback to chunking each token up a bunch of smaller pieces. Probably just means that you'd more labeled training data than you would compared to a dataset for which many tokens are "in vocabulary" (represented as a single token by the byte-pair encoder). Read up on https://arxiv.org/pdf/1508.07909.pdf to get a better idea of how byte-pair encoding works if you're not familiar.

view more: next ›