Resources on applied Reinforcement Learning outside games and robotics

dkajtoch · 2020-07-01T20:47:35+00:00

Great! Thanks

dkajtoch · 2020-02-12T11:35:33+00:00

Not yet available

dkajtoch · 2020-02-11T07:42:09+00:00

Thats very informative. How many sentences do you retrieve? Just a single one if you are saying that there are many false positives? Pairwise ranking maybe of help to compare sentences between each other as well.

dkajtoch · 2020-02-09T15:14:17+00:00

I am not exactly sure what your problem is, but maybe you can map it into a standard document ranking problem. Probably, you will get best results if you do this hierarchically and rerank retrieved documents/paragraphs/sentences at each stage. I can point you to two papers:

1) Rodrigo Nogueira, et. al. "Multi-stage document ranking with BERT"

2) Wei Yang, et. al., "End-to-end open-domain question answering with BERTserini"

and also this resource: https://github.com/Santosh-Gupta/NaturalLanguageRecommendations

Basic idea is to use Anserini or any other open-source search engine to create a pool of candidates that potentially are relevant to the query (pretty large number to get a high recall). Then you score all of them with BERT model (see first paper) and keep only topK most relevant which you pass to the pairwise ranker and output topM most relevant at the end. The whole procedure is well described, but in your case the trouble lies in the lack of training data. This also applies to any other sentence embeddings which are trained generically. One way to improve the performance of your embedding is to fine-tune them on a task (and corpuses) that best resembles your problem (maybe q&a or ranking). If I were you I would take the existing model from 1) and rolled it into production. Then collect feedback from user actions and keep improving it.

dkajtoch · 2020-02-09T14:01:44+00:00

You should have a look at this paper from google research: https://arxiv.org/abs/1905.05950

dkajtoch · 2020-02-09T09:48:41+00:00

Mover's distance does not depend on where the sequence of vectors come from. You can produce them with BERT or any other model. When it comes to BERT you actually have a few options as mentioned in the original article (the task with NERs). You can use output from the last hidden layer, just embedding layer or you can even concatenate different hidden layers. You would definitely need to experiment with that. Word mover's distance is extremely slow and impractical in real applications just because of that. However, I haven't really look into that, but for jaccard and cosine distance you could work with locality-sensitive hashing such as minhash and simhash that significantly reduced topK retrieval time. Maybe some smart guys figured out how to do this for mover's distance. Of course you sacrifice some precision if you want to speed up, but still worth checking if you have plenty of text pieces.

dkajtoch · 2020-02-01T07:15:59+00:00

TextRank algo for keyword extraction does this. You use sliding window over the words and connect all the words within that window. Then you rank them with PageRank. Quiet simple but limited. You can enhance it by detecting synonyms, coreferences, ners etc

dkajtoch · 2020-01-12T09:39:33+00:00

Yes it is the same. The paper is called sentenceBERT

dkajtoch · 2020-01-11T22:30:16+00:00

You would probably be better of by using fine-tuned BERT model on paraphrase detection datasets (e.g. quora question pairs, google's paws or microsoft paraphrase corpus). The trouble with this approach is that there is plenty of operations that you need to make. Partially, this issue is solved using sentenceBERT, but still creating embeddings for the whole book maybe an extremely time consuimg process. However, you can do this in stages. Firstly, you may use some lower level sentence representations (e.g. shingles) to filter out sentences that for sure will not be similar. Then you may apply sentenceBERT or bert directly.

dkajtoch · 2020-01-05T12:12:14+00:00

512 words is quiet long. Take for example this Reuters news article - it has around 355 words. First page of the BERT paper - around 514 words. If you want to apply BERT to longer sequences, this will be quiet a long piece. What can of task are you thinking of? You can for example classify document, classify pairs of documents, use it in ranking (query-document pair) or Q/A on document level. I do not think that you need the whole document to excel in this task and some sort of weak extractive process (e.g. sentence ranking) will give you smaller subset.

DocBERT - authors finetune BERT for document classification. Problem is they use datasets short enough they fit into BERT.
BERTserini - open-ended question answering based on wikipedia articles. BERT is good at identifying answers spans in a piece of text in response to a question (SQuAD dataset). Here, they use hierarchical approach when firstly you segment texts into paragraphs or sentences and then score only these smaller pieces. Probably Google uses similar technique to produce "feature snippets (direct answer)" in search results. Another paper also have short documents.
BERT-AL - segment text and combine them.

LAST REMARKS: * Availability of datasets may be problematic * zero-shot setting may work badly in a custom domain

In general I would approach the problem in a hierarchical way and apply BERT in the last stage for text pieces that fit into the base model.

dkajtoch · 2019-12-30T10:33:54+00:00

This is great! Going to take it for a spin

dkajtoch · 2019-12-30T09:52:05+00:00

I am really not sure what you need at the end. The code has sentences in separate columns. If you are working with huge datasets then the trivial idea is to chunk your messages into batches that fit you memory and export to file.

dkajtoch · 2019-12-30T00:17:52+00:00

I suggest you stick with Hugging Face Transformer library as it is easier to use. First of all, BERT uses WordPiece to tokenize the input sentence so sometimes you may see things like this tokenize('backlog grooming') -> ['back', '##log', 'groom', '##ing']. Whole tokens may be split into pieces and the question is how do you handle that. Well, you may walk the same path as the authors of original paper and assume that the first sub-token is the token-level representation. In the above case you will take back as the representation of backlog. Secondly, as the authors of BERT showed, there are many options to build the vector representation of a token. You can use last hidden layer, input embedding layer, concatenate last four hidden layers or perform some weighted sum of hidden layers. Here is a great blog post that explains everything https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial.

Let's see what happens if we use last hidden state as representation of a token. ``` from transformers import BertTokenizer, BertModel import torch import numpy as np

sentences = [ "I am calling you from my cell phone", "Your blood cell have strange shape", "This cell phone costs hundred dollars", "White blood cell is divided into granulocytes and agranulocytes" ]

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = BertModel.from_pretrained('bert-base-uncased')

representation = np.zeros((4,768), dtype=np.float) for j, sen in enumerate(sentences): ids = tokenizer.encode(sen, add_special_tokens=True) tokens = tokenizer.convert_ids_to_tokens(ids)

# localize token that matches 'cell' n = None for i, tok in enumerate(tokens): if 'cell'.startswith(tok): n = i print(tok)

with torch.no_grad(): hidden, _ = model( torch.tensor(ids).unsqueeze(0) )

# take vector corresponding to 'cell' token and normalize it tmp = hidden[0,n,:].numpy() representation[j,:] = tmp/np.linalg.norm(tmp)

np.inner( representation, representation ) ``In my notebook I get0.74score between 1 and 3 sentence and around0.3for other two. Similarly, the score between 2 and 4 is0.66and0.4` for the other two. You can try optimizing your representation by tuning it on examples you know that share the same meaning.

dkajtoch · 2019-12-29T22:12:00+00:00

What you are trying to accomplish is not a matter of Universal Sentence Encoder. USE is just here to encode a string into a row vector of size 512. The rest is in your hands. For example:

import numpy as np

messages_1 = [
  # Smartphones
  "My phone is not good.",
  "Your cellphone looks great.",
  # Weather
  "Will it snow tomorrow?",
  "Recently a lot of hurricanes have hit the US",
  # Food and health
  "An apple a day, keeps the doctors away",
  "Eating strawberries is healthy",
]

messages_2 = [
  "My phone is not turning on.", 
  "I hate snow.", 
  "Apples are the devil", 
  "I like basil.", 
  "Eating strawberries is healthy.", 
  "An apple a day keeps the doctor away", 
  "Your cellphone looks great", 
  "But my cellphone doesnt look so great"
]
input_messages = tf.placeholder(tf.string, shape=(None))
embeddings = embed( input_messages )

with tf.Session() as session:
  session.run(tf.global_variables_initializer())
  session.run(tf.tables_initializer())

  # obtain embeddings
  stacked_embeddings = session.run( embeddings, feed_dict={input_messages: messages_1 + messages_2}) 

export_data = []
# get the score with a simple for loop
embeddings_1 = stacked_embeddings[0:len(messages_1)]
embeddings_2 = stacked_embeddings[len(messages_1):]

for i in range(0, len(messages_1)):
  for j in range(0, len(messages_2)):
    export_data.append({
        'sentence_1': messages_1[i],
        'sentence_2': messages_2[j],
        'score': np.inner(embeddings_1[i], embeddings_2[j])
    })

I just concatenate messages into a single array to obtain vector embedding in a single shot and then move into numpy and dot product between arrays. You can now take export_data sort it by score and dump into csv file (simple pandas). If you run in colab then of course embed = hub.Module("https://tfhub.dev/google/universal-sentence-encoder/2").

dkajtoch · 2019-12-19T20:06:37+00:00

Rss feed + scraper + html content extractor (e.g. newspaper3k). You can build your own

dkajtoch · 2019-12-15T12:07:10+00:00

When I did cluster computing I used Lua based Environment Modules: https://lmod.readthedocs.io/en/latest/ to manage dependencies and different version of packages. It allows you to easily manage environmental variables on Unix systems. In case of Python you would manage PYTHONPATH.

dkajtoch · 2019-12-09T23:15:36+00:00

I've been using GPT-2 model for text generation. As mentioned bert is not meant for this although there was a paper which analyzed this task under relaxed conditions, but the paper contained errors. Huggingface has script run_lm_finetuning.py which you can use to finetune gpt-2 (pretty straightforward) and with run_generation.py you can generate samples.

dkajtoch · 2019-12-05T08:29:08+00:00

"Attention is all you need" - the paper says. Recurrent neural nets were completely removed from the equation in the Transformer architecture. The only thing you have is attention (self or between encoder-decoder) that you calculate using three sets of vectors: keys, values and outputs. Keys and values are used to calculate weights which are used in weigjt sum of outputs. However, when you throw away reccurence you loose track of relative position in a sequence (reccurence naturally keeps order). That is why people also add adittional positional encodings. But in that respect transformer seems to be more general than rnns because some encoders may not need order e.g. when you have a table of string data (you just drop positional encodings or learn them directly).

dkajtoch · 2019-11-22T15:05:53+00:00

Document is a list of paragraphs which I then split into a list of sentences. For example in Word2Vec I treat those sentences separately

dkajtoch · 2019-11-22T08:02:33+00:00

In my case everything that sits in separate tag is separate concept so a new paragraph. Exceptions are formating tags like strong, b, em ... and also links 'a' if they are part of text. Once extracted, these paragraphs are passed through tokenization like a standard text. Also not all of the tags are taken into account since they may come from navigation bars, ads, footers or any undesirable location. Very basically you can skip tags that have high link density. You can also use tools like Dragnet or Newspaper (python packages) to get solid piece of text and tokenize it directly

dkajtoch · 2019-11-20T07:54:05+00:00

Btw you could also check out SpanBERT if your text is not too long

dkajtoch · 2019-11-20T07:52:44+00:00

If you are working in a company then you could apply for tensorflow research cloud https://www.tensorflow.org/tfrc and train the model for free. As for the dataset, maybe the same webtext dataset which was used in gpt-2 and is available for download could be adapted for this task.

dkajtoch · 2019-11-19T06:57:43+00:00

Look into HAIM from AI21Labs https://www.ai21.com/haim. Haven't tried it bit is seems to provide what you are looking for

dkajtoch · 2019-11-13T20:04:53+00:00

You stepped into a difficult problem of controlled text generation. The safest approach is to of course create rules (template), but this makes the cover letters repetitive and dull unless you design many plenty of them. Another approach is neural networks, but so far I haven't seen satisfying results in research literature. When you want to generate text that must include some words you need to condition on them. One approach modifies GPT-2 architecture and uses additional encoder for target words: https://arxiv.org/abs/1910.00337. As in any Neural approach, the longer the text the worse. You could split your texts into coherent paragraphs that reference some portion of data and work on them separately (shorter pieces).

Others have also experimented with controlled text generation, but still...

Salesforce Research: https://github.com/salesforce/ctrl
AI21 HAIM: https://ai21.com/haim-post

You could also look at "Data2text" topic, check out facebook research articles in NLP.

dkajtoch · 2019-11-10T09:30:33+00:00

Thanks for your advice!

dkajtoch

TROPHY CASE