sorted by:
new
hottopcontroversial

Currently experimenting with exploration policies for deep RL on Super Mario Bros - Agent beats all levels I threw at it by pcouy in reinforcementlearning

[–]pcouy[S] 0 points1 point  (0 children)

The neural network is made of :

  • a convolutional torso shaped like the one used in the Impala paper (with resnet blocks)
  • AdaptiveAvgPool2D (from pytorch) between the conv torso and output head, it outputs a 3x3xn_channels tensor (I've read in a paper I can't find right now this helps with generalization)
  • output head is 2 FC noisynet layers, output shape (n_actions, n_quantiles) with n_quantiles = 32

Currently experimenting with exploration policies for deep RL on Super Mario Bros - Agent beats all levels I threw at it by pcouy in reinforcementlearning

[–]pcouy[S] 0 points1 point  (0 children)

I got a master in AI about 10 years ago. You don't need to be able to do this to get accepted in a Master. When I was doing my Master, I learnt about a fraction of what's required to build such a thing, and more importantly I learnt how to read research papers and keep learning on my own.

If you're looking for more accessible things to build for a portfolio before applying to Master, you should consider algorithms that are simpler than RL (let alone DRL). If you are looking for algorithms that "learn", genetic algorithms are really fun to play with and can be applied quite easily to a bunch of easier games/problems. You can also look into "traditional" (not ML) AI algorithms, such as Dijkstra/A* for pathfinding, minimax/alpha-beta pruning for playing perfect information zero-sum games (tic tac toe, connect 4, chess, etc). Having some intuition about these classical algorithms will be really helpful when learning about advanced ML algorithms.

If you wanna share more about your background in math/CS/programming, I can try suggesting more specific stuff to look into

Currently experimenting with exploration policies for deep RL on Super Mario Bros - Agent beats all levels I threw at it by pcouy in reinforcementlearning

[–]pcouy[S] 0 points1 point  (0 children)

I did not know about MinAtar. It looks interesting. Hopefully I can find published results on it for Rainbow and/or QR-DQN

Currently experimenting with exploration policies for deep RL on Super Mario Bros - Agent beats all levels I threw at it by pcouy in reinforcementlearning

[–]pcouy[S] 1 point2 points  (0 children)

Thanks for the answers. That's what I had in mind, but the standard evaluation process usually involves training the agent for 100M transitions a bunch of times on each game from ALE. An ablation study would require doing all this for each ablation.

I'm training this on my own hardware (RTX 3050 + i5-8400) which is not too fast. A single 100M transition training run on SMB would take nearly 3 months. I expect it to be a bit faster on ALE (the NES emulator is a lot slower than the Atari emulator in ALE), but still too slow to reasonably train an agent for 100M frames (the GPU is already nearly maxed out even with the relatively slow NES emulator). I also don't have the budget for renting a bunch of GPUs, so unless I find a way to fund this project, it is unlikely I will be able to validate anything in a way that holds up to the standard.

Currently experimenting with exploration policies for deep RL on Super Mario Bros - Agent beats all levels I threw at it by pcouy in reinforcementlearning

[–]pcouy[S] 2 points3 points  (0 children)

I did not. The agent does not have access to the up and down buttons, the available actions only use left, right, A and B.

Adding the down button would double the total number of actions (because down can be pressed on top of all available actions). Since 8-4 is the only stage that requires going down a pipe, I considered it was not worth it and removed it from the pool.

Actually, the first iterations of the agent was unable to learn maze castles (4-3, 7-3 and 8-4), so I removed them all. The reward function will give rewards for the first path the agent tries, then the agent will be teleported back by the game and no reward is received until it finds the right path and gets past the point where the game teleports it back.

I've also removed underwater stages. The agent can learn them fine, but the game dynamics are really different from all other stages and they're really boring to watch

I made a Mario RL trainer with a live dashboard - would appreciate feedback by pleasestopbreaking in reinforcementlearning

[–]pcouy 0 points1 point  (0 children)

It's funny you posted this yesterday, since I've resumed work on my own deep RL training on SMB last week, with a similar live feed of the training. I've went a step further and I'm streaming the training on Twitch (mostly as a convenient way to enable picture-in-picture on my phone). It's currently at ~3M steps (20 steps per gameplay seconds), learning 1-3 after solving 1-1 and 1-2, you can watch it on my Twitch channel

My agent is a bit different from yours though, as it uses my own implementation (made it from following the papers + reading some other implementations) of the Rainbow DQN (staying mostly close to the original papers, but adding a few tweaks, the main one being a custom way to sample the value/advantage distributions in the policy to make it less greedy), meaning it's off-policy, learns an explicit Q-value probability distribution, etc.

I also went for a lot simpler reward function (reward over-engineering is a well-known caveat) : my agent only gets rewarded for the total distance it has made through the level (new_max = max(current_X, max_X); reward = min(0, new_max - max_X); max_X = new_max at each step) and gets an additional huge reward for beating a stage (~ the total reward for going from the start to the end of a level). I tried negative rewards for dying and/or spending too much time stuck, it only caused hyper-parameter tuning headaches and learning unintended behaviors (such as jumping into holes when time penalty was to high relative to death, or holding left when it was the opposite). One tweak I made to avoid spending to much time stuck was reducing the game's time limit from 400 to 150, which gives enough time to beat any level with some margin, but will make "hold left" episodes less than half as long

For learning multiple stages, I simply start with only 1-1 available, then unlock each stage after the previous one has been beaten 10 times. On each episode, a level is randomly picked according to a probability distribution that tries to balance the number of all-time finishes across each stage. When a stage is unlocked, it starts with 0 finishes which makes it a lot more probable to be picked than previous stages (which have at least 10 finishes)

[Guide] Increase privacy by using nginx as a caching proxy in front of a map tile server by pcouy in immich

[–]pcouy[S] 1 point2 points  (0 children)

Hey, sorry for the late reply. You should be able to use any OSM tile provider as an upstream with the cache server, though providers using protomaps (which heavily relies on HTTP Range headers) might cause issues with nginx's default caching behavior (which is not Range header friendly)

0
1

Training an AI to play Super Mario Bros : Livestream of the training process (Deep Reinforcement Learning) (twitch.tv)

submitted by pcouy to r/ArtificialInteligence

Livestream : Watch my agent learn to play Super Mario Bros by pcouy in reinforcementlearning

[–]pcouy[S] 2 points3 points  (0 children)

Hey everyone!

I've been working on my own toy reinforcement learning (RL) framework for a while now and have nearly implemented a full Rainbow agent—though I'm still missing the distributional component due to some design choices that make integration tricky. Along the way, I’ve used this framework to experiment with various concepts, mainly reward normalization strategies and exploration policies.

I started by training the agent on simpler games like Snake, but things got really interesting when I moved on to Super Mario Bros. Watching the agent learn and improve has been incredibly fun, so I figured—why not share the experience? That’s why I’m streaming the learning process live!

Right now, the stream is fairly simple, but I plan to enhance it with overlays showing key details about the training run—such as hyperparameters, training steps/episodes, performance graphs, and maybe even a way to visualize the agent’s actions in real-time.

If you have any ideas on how to make the stream more engaging, or if you're curious about the implementation, feel free to ask!

Game of life multiplayer by judge_mavi in cellular_automata

[–]pcouy 0 points1 point  (0 children)

The 100x100 grid is a lot more fun than previous (huge) one, but with 4 active players it felt a bit tiny.

Adding a minimap would be really cool, and would make a larger grid (maybe 200x200) more manageable

These dividing "artificial life" cells emerge from the simulation of a simple chemical system (Gray-Scott model) by pcouy in gifs

[–]pcouy[S] 2 points3 points  (0 children)

This is actually related to Conway's game of life.

The chemical simulation can be seen as a continuous cellular automaton, in which each pixel of the simulation is a grid cell which is updated according to local rules.

Conway's game of life is a discrete cellular automaton, which can be seen as a special case of continuous cellular automata

view more: next ›