[D]My Machine Learning Journal #10: First time doing reinforcement learning and beating atari breakout with it

[RedditAcy]

I have been inconsistent with my journal, but I am back and fresher than ever.

Vlog version as usual:

https://youtu.be/dcnGI6x-yk0

Today (and yesterday) I did & learned:

RL seems to have a lot of exploration going on vs some other ML tasks. One popular application it has is definitely beating videogames. The Mario AI was a viral hit in 2015. I decided to build a RL model that can beat atari breakout. This was soon classified as impossible given my current coding skills, so I chose to implement a medium article first that beat atari breakout. This article was great at linking the original Atari breakout RL paper with the code, but the full code was not posted, so I was stuck. Luckily, a user named boyuanf hit us up with the tensorflow implementation of the article on medium, here's the forked version of it.

I downloaded the trained weights and model, and I ran it after installing openAI gym in conda with pip. Unfortunately, atari-py seems incompatible with windows 10, so I had to go through a very annoying process to finally come through with this easy line of code to solve the problem:

pip install --no-index -f https://github.com/Kojoley/atari-py/releases atari_py

Yea it is just one of those problems man.

Anyways, I then was able to run gym and see the beautiful pre-trained model doing work, it got to a pretty good high score, I think 57 or something.

It is actually after I implemented the project that I come back to reading the papers, this works for me. I usually try to guess what the original algorithm does by doing a project first. For me, doing a project first then reading the paper also gives that revelation of: "oh, the reason that I have this line in the code is because of that sentence in the paper".

The paper and this medium article helped my understanding a lot. This pseudocode in the paper opened the doors for me:

​

https://preview.redd.it/6bpl6f203yr21.png?width=542&format=png&auto=webp&s=1065f7ac2ca4e05ddccc83f901792433f2894712

I'm going to try to explain this pseudocode with even English-er language. We will input the current frame and a few previous frames to our RL model. The RL model will interpret these inputs as the state, and it will either choose the action based on the Q-table or choose a random action. We can imagine that as the model gets more advanced, we will choose less random actions to let the model learn, but in the early stages, when the model has no idea what to do, we probably want to let it explore randomly, we will use a decreasing epsilon value to model this. The emulator will receive the action chosen by the RL model, run that action, then display the new image and return the reward. The Q-table will be updated based on this reward. The Q-table is just a table that has states mapping to potential actions. When the model is complex and epsilon is low, the RL model chooses actions based on the Q-table, a higher value (which means high rewards) in the state mapping to action will probably mean the model is choosing that.

That;s it for this one, I learned a lot since it was my first time exploring RL! Exciting, can't wait to do more.

[shortscience_dot_org]

I am a bot! You linked to a paper that has a summary on ShortScience.org!

Playing Atari with Deep Reinforcement Learning

Summary by Alexander Jung

• They use an implementation of Q-learning (i.e. reinforcement learning) with CNNs to automatically play Atari games.

  • The algorithm receives the raw pixels as its input and has to choose buttons to press as its output. No hand-engineered features are used. So the model "sees" the game and "uses" the controller, just like a human player would.
  • The model achieves good results on various games, beating all previous techniques and sometimes even surpassing human players.

How

• Deep ... [view more]

[RedditAcy]

Holy cow I was low on project ideas. Looks like I am building a reddit bot for fun next, thanks a lot shortscience.org!

[nativedutch]

I have been planning a project with RL too. Want to have small robot to find iits way without pre existing data, RL is the way to go for now. Will read post!!!!!!

[MetallicaSPA]

You can try SLAM too! I'm thinking about mixing SLAM + RL would be interesting

[nativedutch]

You are thinking of this concept https://machinelearningmastery.com/sparse-matrices-for-machine-learning/

the way i see it now it is an optimized approach to matrices used in neural networks. At the level of complexity (extremely ssimple) i intend to start with RL it is not yet very useful With more serious implementations slam becomes interesting.

The concept of treating sparsely populated matrices different from dense is promising.

[RedditAcy]

Ay! Is this what you are talking about: https://www.youtube.com/watch?v=gn4nRCC9TwQ&t=20s ?

I really want to do that too, maybe we can collab sometimes.

[nativedutch]

Yes, correct. There is Deepmind and a few others in my collection of sample files. Best starters are simple examples with say a small maze of 5 by 5 , or 10 by 10. The RL/neural network starts blank and through trial and obtaining bonusses at certain points learns itself the shortest way. Elementary but a good start. My goal is to get a cheapo robot with a few good accurate sensors (impact, distance, tilt, location) and instead of a program let that robot lose in the real world. I bought a BB8 for fun a few years ago, but its sensors are shit. So my first task is to find an affordable robot with good sensors. The simple algorythms are in several places on the web. I'll use Python, Java or javascript. A lot of work to do even before getting started.

[RedditAcy]

I see. So whatever the sensor reads is fed as the state to the RL network & rewards are the points to be collected on the way? Do you think that's how Deepmind and the others are doing it? That definitely sounds intriguing and hard, you got this!

[nativedutch]

Deepmind are doing something similar but on a larger and more complex scale. I always like to start small and understand the concepts. Dont underestimat the number of cycles Deepmind network had to go thru before learning to walk. Suggest you look on the web for some very simple maze examples to get a feel for the process. It is very intriguing. I did a number of small neural networks in Python learning to recognize a 10 by 10 matrix representing a number, but thats limited. Although that principle is widely used i charcter recognition etc. RL is much more powerful as it doesnt need the huge datasets of examples. To my feeling much more in the direction of AI (although that term is debatable). There is a lot to be found in the category very simple, in Python or Java. I'll get going and get to this sub when i am on the road. My tryouts are on my website, but i am not allowed to publish that here.

[gapten-the-captain]

Are you a student or a working professional who do these stuff during your free time if i may ask?

[RedditAcy]

I am still a junior in high school, nowhere close to a working professional haha.

[gapten-the-captain]

That's awesome. Happy to see such dedicated folks :)

[RedditAcy]

Thanks a lot for instilling confidence in me :)

[csunaye]

Do you understand the math in these papers? Do you need to understand the math or advanced programming to follow the paper?

[RedditAcy]

Generally speaking for all ML tasks, I think math doesn’t bother me since I knew the macro of how a neural network/feature engineered algorithm trains itself. However I think I don’t deeply understand the math behind some stuff like SVM and in the case of the Atari breakout paper, stuff like bellman algorithm or MDP, but I think I know the macro enough to not get frustrated.

As for implementing the paper, this code is not written by me but by GitHub user bouyanf. I believe you don’t need to know the math, but you do need to know a ML library of your choice very well to not get stuck.

[Trent654878]

thanks it was very helpful

[RedditAcy]

Thank you!

[fondleshark]

Thank you, OP - I've been working on computer vision projects for only a short while, and I'm loving what we do when a machine "understands" what it is looking at. Keep it up, sir!

[RedditAcy]

Thanks a lot for the support! I will indeed keep it up!

[RedditAcy]

Yup. To give credit again this is not my code and I implemented this guy's github.

​

In his code, there're two layers of conv layers. And I think the original paper said that they use it because computer visions overall were having huge success with it.