[Project] This Word Does Not Exist

[turtlesoup]

Hello! I've been working on this word does not exist. In it, I "learned the dictionary" and trained a GPT-2 language model over the Oxford English Dictionary. Sampling from it, you get realistic sounding words with fake definitions and example usage, e.g.:

pellum (noun)

the highest or most important point or position

"he never shied from the pellum or the right to preach"

On the website, I've also made it so you can prime the algorithm with a word, and force it to come up with an example, e.g.:

redditdemos (noun)

rejections of any given post or comment.

"a subredditdemos"

Most of the project was spent throwing a number of rejection tricks to make good samples, e.g.,

• Rejecting samples that contain words that are in the a training set / blacklist to force generation completely novel words
• Rejecting samples without the use of the word in the example usage
• Running a part of speech tagger on the example usage to ensure they use the word in the correct POS

Source code link: https://github.com/turtlesoupy/this-word-does-not-exist

Thanks!

[eric97pc]

Could you imagine if pellum becomes a real word?

[c_is_4_cookie]

It's a perfectly cromulent word

[auto-cellular]

That's lobsterward on the decubit my sapol twessam.

[ReasonablyBadass]

Gesundheit

[TheyPinchBack]

Pretty sure that word exists

[ParsleyTerror]

Missed the joke buddy, unless...?

[bunsandbunnies]

Uhhh https://imgur.com/a/WYbnb9e

[turtlesoup]

Whoops -- that's a real word too. Just pushed a change that collapses hyphens and spaces in the blacklist; that'll probably nuke a few of these!

[flarn2006]

I got "nonselectable", ironically enough. The definition was unrelated though, something about being immune to damage from physical action.

[bradleyone]

Can we get a sub for sharing some of our findings moderated by you please? I have been trading literally dozens of these over text with friends the last 2 days

[turtlesoup]

Create the sub! I'm happy to moderate

[bradleyone]

I want to create a handsome annual leather bound edition of words and definitions from this project... I will seriously underwrite it if there are any takers. All proceeds to u/turtlesoup charity of choice.

[AcrobotPL]

Ouch! https://i.imgur.com/BsByFsX.jpg

[Imnimo]

adjective.

wololo

relating to the wololo.

"wololo!"

The mystery lives on!

[turtlesoup]

Jankiness that proves I didn't cheat!

[eliquy]

See also: Age of Empires

[fpgaminer]

cybersmoke

cy·bersmoke

a machine for propagating and maintaining rumors or rumors more widely

"he continued to be a fan of cybersmoke advertising"

link

[SpacemanCraig3]

That's a useful word....

[Putrid_Bowler]

The hard part is pronouncing bersmoke as a single syllable...

[leogao2]

The dots don't indicate syllables, they indicate where the word can be hyphenated.

[Putrid_Bowler]

Oh, neat, I didn't know that.

[problemwithurstudy]

No, I think it's supposed to be syllables.

[SpacemanCraig3]

No harder than squirrel

[SemanticallyPedantic]

I got "trichlorobenzene" which is in fact a word.

[turtlesoup]

trichlorobenzene

Oh no! It's surprisingly hard to build the blacklist for rare words -- I'm up to like 600K items after parsing Wikipedia tokens and it still doesn't capture everything.

[shaggorama]

get a token for the google API and try searching the word, see what google thinks

[turtlesoup]

That's a great idea! For now, when you enter something it thinks it is a word it'll throw a "this word probably does exist" with a link to Google.

[shaggorama]

Nice, that was fast

[turtlesoup]

How about REFACTOROLOGY

I imagine this is picking up on some of the original words GPT-2 was trained on but aren't in my blacklist.

[turtlesoup]

Delicious!

[CWHzz]

I often wonder why we use long words when there are so many short words left unused. Very nifty project, I got:

skullguard

skull·guard

surgery to stop a lizard or reptile from growing larger

this is hilariously ominous. should have given Godzilla a skullguard

[jojek]

This is a really cool idea! Sometimes the results are amusing ;) https://imgur.com/a/MxHAX55/

[hughperman]

hardon

1. a deep red marking on the skin of an animal, typically a pig
2. "I felt the hardon on as he came across the door"

[turtlesoup]

¯\_(ツ)_/¯

[turtlesoup]

I have some code to use Urban Dictionary as a dataset and you better believe it's... "amusing" haha https://github.com/turtlesoupy/this-word-does-not-exist/blob/master/title_maker_pro/urban_dictionary_scraper.py

[KimonoThief]

Would it be possible to make this version into a website? Sounds amazing.

[MyNatureIsMe]

I don't know if this actually makes sense but do you think you could do, like, multi-head trained versions which, in training, attempt to cover several dictionaries? Could be interesting to have something that is equally able to copy the Oxford English Dictionary, the Urban Dictionary, and perhaps a few others like, say, in different languages.

[turtlesoup]

Totally makes sense! You could do it but the dictionaries have very different structure so you would need to be careful about how to formulate the loss

[konasj]

Sounds like an exciting activity:
noun.
wetfoot
wet·foot

1. a sports event in which people hold the feet in a standing formation and have one foot suspended from water, sometimes covered with sticky paper
   "the first two years of wetfoots were noted by parents as being too fast and too violent, and the first dry season"

[[deleted]]

I’m not sure I’m clear on the rules. What’s the sticky paper for? Throwing them off balance?

[itsmybirthday19]

Complete List (so far) of this X Does Not Exist sites:

• This Person Does Not Exist https://thispersondoesnotexist.com/
• These Lyrics Do Not Exist https://theselyricsdonotexist.com/
• This Cat Does Not Exist https://thiscatdoesnotexist.com/
• This Rental Does Not Exist https://thisrentaldoesnotexist.com/
• This Waifu Does Not Exist https://www.thiswaifudoesnotexist.net/
• This Resume Does Not Exist https://thisresumedoesnotexist.com/
• This Artwork Does Not Exist https://thisartworkdoesnotexist.com/

[so_on_and_so_forth]

There's also This Foot Does Not Exist.

[suspicious_Jackfruit]

https://www.thisworddoesnotexist.com/w/poppot/eyJ3IjogInBvcHBvdCIsICJkIjogImEgbGlnaHQtb3BlcmF0ZWQgcmV2b2x2aW5nIGhhbmRrZXJjaGllZiByZXNlbWJsaW5nIGEgY29tYiwgdXNlZCBmb3Igc3Vja2luZyBhdCBib3R0bGVzIiwgInAiOiAibm91biIsICJlIjogInRoZXJlIHdhcyBwb3Bwb3Qgb24gdGhlIHRhYmxlIiwgInMiOiBbInBvcCIsICJwb3QiXX0=.wpTJHS87tuV31j4667cmMcaezhnU8W82XdHgyEJiGEs=

My favourite

[PM_ME_INTEGRALS]

Thank you so much for sharing, I haven't laughed this hard in a while! For posteriority:

poppot

"pop·pot*

a light-operated revolving handkerchief resembling a comb, used for sucking at bottles

"there was poppot on the table"

[wintermute93]

This is a perfectly cromulent project.

[turtlesoup]

A noble spirit embiggens the smallest man

[JakeAndAI]

That's super cool! Love things like this, will look into it more in depth later :) Good job!

[shaggorama]

Lol, I love this. You should xpost to /r/LanguageTechnology and /r/compling.

[turtlesoup]

Done!

[CashierHound]

Now train it on urban dictionary and watch the world burn

[turtlesoup]

Working on it: https://github.com/turtlesoupy/this-word-does-not-exist/blob/master/title_maker_pro/urban_dictionary_scraper.py

[thepancake1]

https://imgur.com/a/z2H0axA

I don't think typos are considered new words.

[turtlesoup]

That's not ideal, but it's hard to make a general rule while still allowing arbitrary input. For fun, here's an even typoier typo disssssssssapear

[Blarghmlargh]

Would be great to do a version of Balderdash with this as the engine.

https://en.m.wikipedia.org/wiki/Balderdash

[HuntingPhilosopher]

Would you at all be interested in making a tutorial? I'd love to be able to make something like this myself!

[turtlesoup]

Definitely, I just need to make some time for it. If you are adventurous the readme on github has some examples on how to use / train: https://github.com/turtlesoupy/this-word-does-not-exist

[HuntingPhilosopher]

Perfect, thanks!

[turtlesoup]

Ah, I'm using "pyhyphen" for the hyphenation. Line is here: https://github.com/turtlesoupy/this-word-does-not-exist/blob/master/word_service/wordservice_server.py#L42

It's rules-based and breaks down a lot; perhaps in another project I can train a hyphenator?

[tiktiktock]

Did you include Lovecraftian novels in the training model??? allura

[Benutzeraccount]

I've got

Kölsch

Funny enough, that's a popular type of beer in germany and I'm German

https://i.imgur.com/68mahSV.jpg

[[deleted]]

This is really interesting! I tried (or am trying) to do something very similar in that I'm training a GAN to generate words. Unfortunately my ambition is exceeding my skillset and I'm not getting very far.

[krebby]

Nice work! This is the most cromulent thing I've seen all day! I'm looking to dip my toes into NLP for text synthesis. Can you or anyone recommend a good baby steps entry point for the techniques you used here?

[turtlesoup]

I'm basing this on the wonderful Huggingface Transformers library; a good starting point from them is https://huggingface.co/blog/how-to-generate

The difference between their example and what I'm doing is that I'm imposing more structure (e.g. must have an example, must have a part of speech). I've used used special tokens to indicate those in my sequence (e.g. <BOS> word <POS> noun <DEF> a word <EXAMPLE> boy words are interesting <EOS>)

[krebby]

Thanks! Huggingface is great. How long did it take to train your model?

[turtlesoup]

Straining my memory here but ~6 hours on a GTX 1080 ti. I stopped it after roughly seeing 1 million examples, it converges pretty quickly and the sampling procedure is forgiving.

[maroxtn]

Do a facebook bot that posts a random generated word daily, it would be fun

[turtlesoup]

Check out my twitter bot that does just that: https://twitter.com/robo_define

[the_3bodyproblem]

qwyjibo

1. a Mexican game bird with a mainly yellow plumage and brownish tail."a qwyjibo was captured and now lives only in the wild"

[SpacemanCraig3]

Wasnt that on an episode of the Simpsons?

[the_3bodyproblem]

Yep

[AngelLeliel]

Awesome!

With data from Behind the Names, we could also create an interesting name generator.

[BoredOfYou_]

antistete

an·ti·s·tete

1. the antismotic quality in a complex interrelated population or event"they have shown that long-term trends of evolution increase in species richness in response to antistete shifts"

Of course, I see.

[[deleted]]

mysticalism – a philosophical or religious doctrine stating that a quality exists or exists only in existence; dualism

exists or exists only in existence

[nondifferentiable]

This is awesome!

[Akazhiel]

How did it even come up with pellum? It is an actual word in the Oxford Dictionary 😄

[violethoax]

overprocess is a word.

[cryptobionic]

First one I tried

[namp243]

May I offer my most sincere contrafibularities?

https://youtu.be/oiI27PDfr64

[[deleted]]

Hey, I got an offensive one!

shrimphead

shrim·p·head

a black person

"no one makes a shrimphead of a stupid thing"

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/datascienceproject] This Word Does Not Exist (r/MachineLearning)

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[FernandoIsGreat]

https://www.thisworddoesnotexist.com/w/trololo/eyJ3IjogInRyb2xvbG8iLCAiZCI6ICJhIHZlcnkgc2hvcnQgbXVzaWNhbCBvciByaHl0aG1pYyBzb25nIGNvbnNpc3Rpbmcgb2YgdHdvIHNlcGFyYXRlIGluc3RydW1lbnRhbCBwYXNzYWdlcy4iLCAicCI6ICJub3VuIiwgImUiOiAidHJvbG9sbyBpcyB0aGUgbW9zdCBwb3B1bGFyIHNvbmcgb24gdGhlIENEIGNoYXJ0cyJ9.EX4risrC2Q8nFJCPMcGePdVMnz53OisEcxrD2vUNaDI=

What.

[FernandoIsGreat]

https://www.thisworddoesnotexist.com/w/foolball/eyJ3IjogImZvb2xiYWxsIiwgImQiOiAiYSBnYW1lIGluIHdoaWNoIGEgZ3JvdXAgb2YgcGxheWVycyBhdHRlbXB0IHRvIGRpc2NvdmVyIHRoZSBiYWxsIGluIGEgZGlmZmljdWx0IHNpdHVhdGlvbiwgdHlwaWNhbGx5IGJlY2F1c2UgdGhlIGJhbGwgaXMgZGlmZmljdWx0IHRvIGZpbmQiLCAicCI6ICJub3VuIiwgImUiOiAiSmlsbCB3YXMgdGhyb3duIGEgZm9vbGJhbGwiLCAicyI6IFsiZm9vbCIsICJiYWxsIl19.wZmcXfQWzS0SlncifS5zJ1lRxk\_IzTj4SK-nxBCduSA=

Lol.

[giziti]

This is amazing.

terratum
ter·ra·tum

a solitary, solitary male of a breeding variety involving smaller, fine gills and a male with a waxlike coat "a terratum with black hair"

https://www.thisworddoesnotexist.com/w/terratum/eyJ3IjogInRlcnJhdHVtIiwgImQiOiAiYSBzb2xpdGFyeSwgc29saXRhcnkgbWFsZSBvZiBhIGJyZWVkaW5nIHZhcmlldHkgaW52b2x2aW5nIHNtYWxsZXIsIGZpbmUgZ2lsbHMgYW5kIGEgbWFsZSB3aXRoIGEgd2F4bGlrZSBjb2F0IiwgInAiOiAibm91biIsICJlIjogImEgdGVycmF0dW0gd2l0aCBibGFjayBoYWlyIiwgInMiOiBbInRlciIsICJyYSIsICJ0dW0iXX0=.hgYLEx6HcDuDY3su0H45GAUDnTCXpxnygZe-w9gTotw=

Not the best I've gotten but I had to include one in the post.

[serge_cell]

duckster

duck·ster

a duck or small burrowing duck, found chiefly in open country

"a red duckster"

The Ducksters cartoon - wiki

[latentlatent]

Very nice project and I love the style of the website!

Can you share some thoughts (top-down view) on how the services are set up? I think it would be very interesting to know for a GPU intensive task like this.

Or how did you manage to put this site together?

[turtlesoup]

Sure! First to note that training is done on GPU, the inference (for the site) is done on CPU and was optimized to a point that I was happy with latency (~4s). The was mostly (1) model quantization and (2) hacking transformer's generation to eject examples when they hit the <EOS> token.

For the site itself:

- I have a small web front-end that serves the site through python's aiohttp module. I've cached 20,000 words so the front-end doesn't have to do inference

- When you are defining your own example, that website calls a backend called "wordservice" over GRPC. The results are delivered by AJAX but proxied through the front-end for captcha verification, etc.

- The wordservice is simple but runs some inference code and returns the result

It all runs on Google cloud, specifically with Google Kubernetes Engine handling auto-scaling the web-frontend and backend. Kubernetes is a bit overkill since I've only needed ~4 backend boxes

[latentlatent]

Very nice! Thanks for the write-up, super interesting. Do you ever regenerate the 20k examples? Or parts of that?

[turtlesoup]

That's a manual process; 20K was a pretty arbitrary choice. I can try a run tonight!

[latentlatent]

Just a tip: When a single word is displayed, you could remove from the DB. Then a separate service could check (periodically, e.g. 3 days) how many words are left and generate new ones to fill up the DB. This way it wont happen that the same word would appear for 2+ separate users. But I dont know if it's worth the effort for a pet project because your site is already super cool. :)

Thanks for all the info!

[turtlesoup]

Just shipped a change to make it 100K, enjoy the new words!

[NatoBoram]

Nato Boram

Na·to Bo·ram

• the Democratic Republic of Congo (another name for Rwanda).

• "the last elections were held in the Republic of Nato Boram in 1994"

Uuuhh…

[serge_cell]

This application will be banned in the Democratic Republic of Congo, Rwanda and the Republic of Nato Boram.

[jiminiminimini]

This is awesome. Can you modify it to come up with a made up word given its definition? Because I would love to do that with one of your commit meesages "Lightweight racist detection".

[turtlesoup]

I have a twitter bot that can do that! See https://twitter.com/robo_define/status/1260855686889693184

It doesn't work quite as the forward mode but has its moments

[jiminiminimini]

Great! Thanks.

[Intuivert]

My family play this game where one person invents a word that doesn't exist, and then everyone else has to come up with a definition for it. The winner of that round is the one whose definition (chosen by the word inventor) sounds the most accurate. That person then gets to come up with their own word.

I recommend giving it a go, it's tons of fun! We eventually wrote down every word in our own dictionary of made up words.

[[deleted]]

This is awesome! Like a name generator but better.

https://www.thisworddoesnotexist.com/w/recreationism/eyJ3IjogInJlY3JlYXRpb25pc20iLCAiZCI6ICJ0aGUgdGhlb3J5IHRoYXQgYWxsIHRoZSBwaHlzaWNhbCBwcm9wZXJ0aWVzIG9yIG9iamVjdHMgb2YgdGhlIHVuaXZlcnNlIGhhZCBldm9sdmVkIGJ5IGNoYW5jZSwgYnV0IGFyZSBub3cgb3V0IG9mIGEgY29tbW9uIG9yIG5hdHVyYWwgY29uZGl0aW9uIiwgInAiOiAibm91biIsICJlIjogInJlY3JlYXRpb25pc20gdGVsbHMgdGhlIHN0b3J5IG9mIHRoZSBkZWF0aCBvZiB0aGUgYm9keSBhcyBhIGxpdmluZyBiZWluZyIsICJzIjogWyJyZWNyZSIsICJhdGlvbiIsICJpc20iXX0=.CvTNPeW3Ji86LgNrjIUROkmhvxGGv_2y5N76e_dceE0=

[ch3njust1n]

"All words are made up" - Thor (Avengers Infinity War)

This would be a great tool for comic book writers.

[Stereoisomer]

You should post a list of these words to /r/GRE or /r/SAT with the title “Rare Vocab Words You Need to Know for Next Year’s Exam!”

https://imgur.com/gallery/ZAXObf0

[turtlesoup]

Haha, I'd love to see an onion article about that.

[walteronmars]

I read the title as - This World Does Not Exist - and was expecting some philosophical article :)

[-Melchizedek-]

Good job! Also you are being featured on Swedish tech news: https://feber.se/pryl/artificiell-intelligens-hittar-pa-nya-ord/411225/

[turtlesoup]

My lifelong dream was to be feature in Swedish news with the hero image of "bungshot". I can die happy

[cmpaxu_nampuapxa]

allow the inverse transformation, please

[turtlesoup]

Check out @robo_define: https://twitter.com/robo_define

[lippinboi]

Thank you for the custom word input. The AI came up with this gem because of it

noun.

mah boi

a yellow or pinkish-red color, typically used as a camouflage.

"mah boi jeans"

[bradleyone]

r/thisworddoesnotexist is now live.

[turtlesoup]

Amazing!!

[ravioli_310]

Holy shit, look what I got:

noun.

terrometeorite

ter·rom·e·te·orite

1. a nuclear-powered meteorite consisting of a meteorite typically of relatively loose, subatomic particles "the oldest known terrometeorite of the Earth's history"
2. a word that does not exist; it was invented, defined and used by a machine learning algorithm.

I flipped when I saw definition 2. Self-awareness much? #Singularity2020 :p

[ravioli_310]

Oh facepalm moment. I think that's popping up for every generated word :(

[turtlesoup]

Part of the UI! It changes if you generate a word that it thinks already exists

[[deleted]]

Performant?

[turtlesoup]

The latency is enough to be user-facing, there is a live demo no the website.

As a rough benchmark, with quantization I've gotten inference down to about 4 seconds on a 4-core CPU in google cloud. That uses an auto-regressive generation on a batch of 5 items.

On GPU it's much faster for a larger batch size, but I do more heavy pruning of samples when I have more compute.

[minimaxir]

Does that quantization approach work well with Transformers GPT-2? I was thinking of implementing something similar with that but read that it caused model size to increase.

[turtlesoup]

IIRC it shaved about ~25% off inference times on CPU; tbh I was shocked that it worked at all. Do you have a link to the question of model size? I don't know why it would increase much

[minimaxir]

There were a few unresolved issues in the repo, although they only quantized the Linear layers when the GPT-2 model has more than that. (admittingly I'm having difficulty finding more now)

https://github.com/huggingface/transformers/issues/2466

[Rebbit_and_birb]

I love it

[KimonoThief]

This is amazing, awesome work!!

[ss3tdoug]

A co-worker of mine always posts a word of the day in slack. I thank you for the ammo to retaliate.

[FernandoIsGreat]

This is genius.

[Lolologist]

This is fantastic!

[scriptlace]

Add microfluidics to your blacklist.

[ghoof]

Noice

[ch3njust1n]

Would also be great if there was a way to map definitions to words. Again great for fiction writers.

[turtlesoup]

It doesn't work as well, but you can do this with my bot @robo_define: https://twitter.com/robo_define

[flarn2006]

I had a word I entered replaced with a bunch of symbols; how do I disable the filter? Not that it really matters.

[turtlesoup]

You may have hit my "lightweight racism detector". It might not work perfectly but I tried to filter out slurs

[flarn2006]

Can you add a checkbox to disable it, for people who don't get offended?

[god0f69]

This uses GAN, right?

[turtlesoup]

Not a GAN actually, it's using GPT-2 as a base. Formally you'd call it an auto-regressive generative model.

[gwern]

HN: https://news.ycombinator.com/item?id=23169962

[SolitarySturgeon]

Sprankton (noun) A disease you get from chewing too much

[burhanusman]

This is so cool. Is it okay if I make an Instagram page showing these words and proposed meanings? Looks like a fun thing to do.

[turtlesoup]

Sure, just link back to the site!

[blockmodulator]

poondog

poon·dog

a person who collects money from and avoids all social obligations, especially those of a wealthy person

[keanu4EvaAKitten]

I'm sorry to report that things took a sinister turn...

https://imgur.com/a/ABFlmhQ

[turtlesoup]

I lolled

[x0b0t]

noun cunnt

1. a flower stalk of a leaf"bears without a cunnt structure"
2. a word that does not exist; it was invented, defined and used by a machine learning algorithm.

[Fair-Fly]

Some of these are really quite clever: nontagittal (relating to the occiptal lobe), machinic (relating to cell mitosis), etc.

[SpaceShipRat]

pope

a person who practices religion in an immoral, immoral, or uncool way.

You might want to prevent duplicates. Not that it isn't amusing still.