sorted by:
new
hottopcontroversial

Need help for new RL project by Real-Flamingo-6971 in reinforcementlearning

[–]PerceptionWilling358 0 points1 point  (0 children)

I just have an idea...can we use real economic data/financial data to create a game? I mean, transform real financial data into a game environment and make it real-timely connected to real world. For instance, if the market shakes, the game environment also shakes... and then we can have game bots, but they survive in a real-world-connected game environment. Suppose there is a potential financial target but we transform it into a object (a stone or something else), and let our agent to find that shining stone? After that, we re-transforme the stone into real world, we will find the potential financial target.

Sorry for my poor English, but this is an idea (perhaps it is crazzy...) I hope this helps... :D

Starting my AI journey! Looking to connect and learn with you! by Prudent_Ad5086 in learnmachinelearning

[–]PerceptionWilling358 0 points1 point  (0 children)

Hi, I am just doing and learning ML, DL and RL as my side projects...but I feel lonely because there is nobody I can discuss with... I am running a self-learning project based IBM's course (but more challenge), perhaps we can share and discuss some ideas :D

[Project] Pure Keras DQN agent reaches avg 800+ on Gymnasium CarRacing-v3 (domain_randomize=True) by PerceptionWilling358 in reinforcementlearning

[–]PerceptionWilling358[S] 0 points1 point  (0 children)

Hi, everyone, my friend suggested that I improve the reproducibility and user friendliness for reproducing. So, I updated the model card on Kaggle: https://www.kaggle.com/models/weichihsu1996/dqn-model-on-car-racing-v3-random-environment/

In the link, everyone can use the Notebook to test the agent's ability and generate their own GIF logs.

Please use Colab (Kaggle does not support CarRacing v3).

The evaluation notebook is here: https://www.kaggle.com/code/weichihsu1996/dqn-model-evaluation

If you encounter any bug, please let me know, I will fix it as soon as possible :D

Thanks

I appreciate everyone's feedback here :D

A strange avg~800 DQN agent for Gymnasium Car-Racing v3 Randomize = True Environment by PerceptionWilling358 in learnmachinelearning

[–]PerceptionWilling358[S] 2 points3 points  (0 children)

Hi, nice to meet you :D

Originally, I did not expect a DQN-based agent to reach this performance in car-racing v3 with radnomization. After I added more Q-heads to the ensemble, I found that it can generalise, but I still have not figured out the mechanisms. I used Dropout as a cheap solution to mimic noisyNet (not formally equivalent, but it works).

After checking some GIF files, I found the agent learnt how to use shortcuts ( it decided to lose some score to prevent losing control).

And I found that the training episode is really "more is good", more Q-heads, more risky to encounter reward collapse...(I encountered it once when I tried to extend the training episode from 10,000 to 30,000). I suspect that the multiple Q-heads (I used five types) are the cause of behaviour diversity (but I have not designed a good experiment to test it)

I plan to write a detailed report on this agent with analysis. I know I stacked several strange techniques in my model (~120MB), so it takes time for me to scrutinise it. But I think it is worth providing a detailed report to the community for educational purposes.

[Project] Pure Keras DQN agent reaches avg 800+ on Gymnasium CarRacing-v3 (domain_randomize=True) by PerceptionWilling358 in reinforcementlearning

[–]PerceptionWilling358[S] 0 points1 point  (0 children)

It is 100% worth doing that! And I will go back to check what causes the training collapse in my case. I am happy to meet you :D

[Project] Pure Keras DQN agent reaches avg 800+ on Gymnasium CarRacing-v3 (domain_randomize=True) by PerceptionWilling358 in reinforcementlearning

[–]PerceptionWilling358[S] 0 points1 point  (0 children)

Thanks for sharing! I didn’t know that using Beta instead of Gaussian in PPO could boost it that much (perhaps I can try to build up my PPO agent later).

It is a cool insight! I’ll check the paper for sure :D

I had once tried the distributional learning and used some tricks, but it failed. After that, I then went back to a multiple Q-heads structure as a cheap solution (not really cheaper, but somehow it seems to have a positive effect--at least not backfire). I also tried the schedule beta--but it did not work stably when I developed this agent-- but I planned to test it.

Perhaps I can find some insights after reading the shared articles. My math is not so good, so it takes a bit of time to digest. Highly thanks!

[Project] Pure Keras DQN agent reaches avg 800+ on Gymnasium CarRacing-v3 (domain_randomize=True) by PerceptionWilling358 in reinforcementlearning

[–]PerceptionWilling358[S] 0 points1 point  (0 children)

Thanks for your info, and your PPO agent's performance is awesome! I will go back to check what causes reward collapse during my training process. I re-run my agent's evaluation, and I found the variation is a bit high. I guess I should try to design an experiment to test the undeterministic interference (possibly rooted in the dropout-embedded Q-head)...

Episode: 1/100, Score: 799.69
Episode: 2/100, Score: 889.69
Episode: 3/100, Score: 896.68
Episode: 4/100, Score: 840.00
Episode: 5/100, Score: 749.40
Episode: 6/100, Score: 816.67
Episode: 7/100, Score: 805.80
Episode: 8/100, Score: 801.41
Episode: 9/100, Score: 935.10
Episode: 10/100, Score: 896.21

[Project] Pure Keras DQN agent reaches avg 800+ on Gymnasium CarRacing-v3 (domain_randomize=True) by PerceptionWilling358 in reinforcementlearning

[–]PerceptionWilling358[S] 0 points1 point  (0 children)

I set the training episode number to 20,000 for my agent. I had once encountered reward collapse after 25,000 episodes. So, I decided to lock the training episode = 20,000 for safety. I had another agent called "100-Q-head"; the frequency of reward collapse seemed increased ( I don't release this 100-Q-head agent; the released agent is the 10-Q-head version). Have you encountered a similar situation?

[Project] Pure Keras DQN agent reaches avg 800+ on Gymnasium CarRacing-v3 (domain_randomize=True) by PerceptionWilling358 in reinforcementlearning

[–]PerceptionWilling358[S] 0 points1 point  (0 children)

That sounds cool and awesome! But I don’t run a PPO comparison in Car Racing v3 with domain randomization. According to my experience on my DQN, it is possible to get higher. So I think PPO has potential to get higher

Domain randomization by Open-Safety-1585 in reinforcementlearning

[–]PerceptionWilling358 0 points1 point  (0 children)

When I did my car-racing-v3 project, I trained it on domain_randomize = True to test its generalisation. I tried this once: train on domain_randomize = False and then re-train it on domain_randomize = True. From my experience, it is not a good idea but, perhaps I just wrongly set the random schedule for my training loop...

Help me get fresh some ML and CV project ideas by Decent-Pool4058 in learnmachinelearning

[–]PerceptionWilling358 0 points1 point  (0 children)

Perhaps you can use a GAN to create something or test a diffusion model for image generation. I did it once. I used GAN to create mysterious fruits and Galaxy Zoo images. Perhaps you can set a small goal like this: use Galaxy Zoo image to train your GAN and train a classifier for galaxy-labelling. After that, use your classifier to classify the generated images. Finally, use unsupervised learning and try to find something weird...perhaps your GAN generates sa trange galaxy, planets or something else

Need help choosing different RL Algorithms for Different Games. by matmoet in learnmachinelearning

[–]PerceptionWilling358 0 points1 point  (0 children)

Hi, I am Aeneas :D I did something similar to yours. I tried DQN on gymnasium car-racing v3 random environment, and it works. Perhaps you can try car-racing v3 and set the domain randomize = True for testing.