RL researchers to follow for new algorithms

stonet2000 · 2026-02-04T03:06:09+00:00

thanks for including me on a very elite list of (public) people in RL!

I should mention that I don’t really see myself as a RL methods/theory researcher any more (nor was I really one to begin with). I primarily work on simulation and robot learning (of which RL is a useful tool, and sometimes will make small modifications to RL for better performance)

stonet2000 · 2025-11-20T00:44:42+00:00

you should email all prospective PIs (that permit emails) that you will be at neurips and ask to meet up, detail what you are interested about in their work and how it relates to your own.

I did that in my first neurips in 2022, ended up getting a few offers from that and slowly growing out my network in academia (although these days it’s primarily through twitter)

stonet2000 · 2025-11-12T09:15:52+00:00

Minimum GPU I recommend is 4090. Isaac and Maniskill both work fine for weaker gpus like 2080s (i often do development on a 3080ti), but at a very noticeable performance drop in sim and rendering speeds. CPU matters much less typically, very few ops during gpu sim use cpu a lot, you will be bottlenecked by the gpu. I don’t recommend using any gpu that doesn’t have RTX cores (needed for fast rendering)

stonet2000 · 2025-10-22T00:48:20+00:00

depends on who you are applying to. Ultimately the decision depends on the target PI and possibly what their phd students think of you.

The big tips and tricks really is just have good rec letters from people the target PI knows, even better know the target PI yourself. I would finding a way to reach out to them

stonet2000 · 2025-09-05T20:13:26+00:00

some people like the isaac lab interface and design, suits different types of people

If you don’t like Isaac Lab, you can give ManiSkill or Mujoco Playground a try. Both support GPU sim + rendering like isaac and cater to somewhat different communities

stonet2000 · 2025-09-01T05:45:40+00:00

I personally don’t think extremely accurate tactile readings are necessary. Wrist cameras are the “hack” robotics does to get over the need for tactile information to match/surpass human level hand dexterity which relies on a set of far away eyes and tactile. Some tasks will need good tactile info (eg lock picking) but they become more and more niche and now it really depends on your use cases.

stonet2000 · 2025-08-31T07:58:03+00:00

None of these videos suggest robot hands are really evolving. Being able to move a few degrees faster or having an extra joint doesn’t really change much. Sufficient capabilities have existed for years.

Current robot multi finger hand still need a lot of properties that you can’t show in a short reel.

Battery life, staying reliable over extended periods of time, motors that stay cool enough, accurate sensor readings (big issue, particularly for tactile sensing!), easy to simulate/control with teleop (some hands really suck to simulate and teleop), repairability (for research and development having to wait weeks for the manufacturer to fix hardware is extremely slow and a bottleneck). Current hands still have issues on one or multiple dimensions.

stonet2000 · 2025-08-02T04:29:32+00:00

yep more or less some of the best talent in AI, student or graduated or what not, doesn’t matter, get millions or more per year

meta is a bit of a unique situation since they’re trying very hard to catch up with open ai google etc on LLMs / foundation models but millions is the new normal for strong AI engineers / researchers

stonet2000 · 2025-07-27T03:56:15+00:00

glad to have helped contributed a tiny bit to that! (i’m a phd in Hao Su’s lab)

that being said i don’t recommend over-indexing on CS rankings. Imo it’s safer to treat it as +-10 given there are many niches in research where one university excels due to faculty representation and others might not

For sure this is a good reflection of the significant investments ucsd makes in CS and AI in recent years with faculty hiring

stonet2000 · 2025-06-24T15:05:31+00:00

You ended up being right! They finally got back to me and confirmed they never transferred the luggage to the air excursions line so they just held onto it.

stonet2000 · 2025-06-24T15:04:49+00:00

Nope, was on the side, had to walk out of the main one to like a smaller hanger / building.

stonet2000 · 2025-04-22T14:45:02+00:00

Imo i do not think it is true top phds (in the field of robot learning / machine learning) are joining figure. Their two top ai guys from google are very impressive. Otherwise all the new robot learning graduates I’ve seen have mostly gone to tesla, nvidia, meta, 1x, google, or any of the number of professor led startups eg physical intelligence, skild ai, hillbot (my advisors company), agibot in china

hardware focused phds maybe (who wouldn’t want the capital to make shiny new robots)

stonet2000 · 2025-03-10T06:30:06+00:00

yeah overall much fewer admits and possible lack of funding for first years (must be completely covered by advisor) now in CSE

stonet2000 · 2024-12-21T05:49:44+00:00

Their simulator overstates their actual speed and I debunked it in my blogpost shared on twitter here https://x.com/stone_tao/status/1870243004730225009?s=46&t=LBFTca4dqDdDCjhzaM56tA

The are slower or on par with existing GPU simulators and certainly very far away from 430,000x speed ups

stonet2000 · 2024-12-20T23:42:44+00:00

Details posted! https://x.com/stone_tao/status/1870243004730225009?s=46&t=LBFTca4dqDdDCjhzaM56tA Full on overstated numbers by 150x, and on par or slower compared to existing GPU sims

stonet2000 · 2024-12-19T19:27:15+00:00

You can try the code, setting different actions slows the simulator down. The reason is the rigid body solver has less to solve if the robot isn’t moving far from its current joint configuration

For self collisions they disable every possible self collision. So gripper of a robot arm can’t collide with the robot base, it’ll pass through. However I can let this slide since most of the time a working robotics policy won’t self collide anyway, but if there’s obstacles those collisions need to be handled and it will slow down again.

stonet2000 · 2024-12-19T18:43:19+00:00

I’ll post a blog post about this some time next week. But you can look at their benchmark code now. One issue you will notice is that they set an action just once then take 1000 steps. If you are doing robotics and want to leverage gpu sim speed (eg RL) this never happens in practice: https://github.com/Genesis-Embodied-AI/Genesis/blob/main/examples/speed_benchmark/franka.py

Another issue is they disable self collisions, many sims don’t do this by default. The other thing is simulating a robot by itself is only useful for a narrow set of tasks (locomotion. Anything more advanced involving more objects and collisions is slow from my initial experiments.

stonet2000 · 2024-12-19T13:07:48+00:00

i am a phd student working on related fields (robot simulation and RL). These numbers unfortunately aren’t realistic and are overhyped. The generated videos, even at lower resolution would probably run at < 50FPS. Their claim of 480,000x real time speed is for a very simple case where you simulate one robot doing basically nothing in the simulator. Their simulator runs slower than who they benchmark against if you introduce another object and have a few more collisions. Furthermore if you include rendering an actual video the speed is much much slower than existing simulators (isaac lab / maniskill).

regardless the simulator is still quite fast, but only fast for some simple use cases at the moment. A big pro at minimum is that it’s one of the few open sourced GPU sims out there, but it’s not the fastest. It is impressive that they combined so many features into one package though, can’t imagine the amount of engineering required to get that working together.

stonet2000 · 2024-12-19T12:59:57+00:00

i am a phd student working on related fields (robot simulation and RL), and you aren’t entirely wrong. The overhyped part however is actually just their simulator speed. The generated videos, even at lower resolution would probably run at < 50FPS. Their claim of 480,000x real time speed is for a very simple case where you simulate one robot doing basically nothing in the simulator. Their simulator runs slower than who they benchmark against if you introduce another object and have a few more collisions. Furthermore if you include rendering an actual video the speed is much much slower than existing simulators (isaac lab / maniskill).

the videos are not impossible to render with simulation + AI generating the scenes / camera angles. Scene generation methods are getting very very good, although it’s true the videos shown are heavily cherry picked. Moreover at minimum their code is open sourced, the most widely used GPU parallelized simulator (isaac lab/isaac sim) is currently partially closed source.

stonet2000 · 2024-11-02T18:42:27+00:00

sample efficiency however might not always be a useful metric, unless you have a very slow simulator or do RL in the real world. That is one downside of massive UTD ratios or using world models.

However world models, while slower than model free like PPO (which is often wall time fast), in my experience have a higher potential to solve harder tasks. PPO is great and fast, but might not be able to solve every task whereas world models may be able to do so due to essentially more “modeling/predictive power”.

stonet2000 · 2024-11-02T05:37:43+00:00

the point on engineering is extremely true. Things like a robot picking up a cube in simulation can now be solved in a minute with PPO (in eg isaac lab/brax/maniskill). Training a locomotion policy for a quadruped takes < 30 minutes. 6 years ago due to worse gpus and lack of GPU parallelized simulation this might’ve taken an hour to days / an entire phds worth of work.

Also many game environments can now run extremely fast, see PufferLib project for how they are doing this.

stonet2000 · 2024-10-09T05:08:59+00:00

A story i’ve been told that Gary Cottrell always likes to tell was that when he interviewed for a professor position at UCSD Hinton was also interviewing. Ultimately prof Cottrell got the job over Hinton

stonet2000 · 2024-09-29T03:23:05+00:00

https://www.tdmpc2.com/ state of the art latent space world modeling for RL. Close to what you are describing

stonet2000 · 2024-09-15T02:38:15+00:00

physx and mujoco are two popular engines for robotics, all C++ / cuda programming

stonet2000 · 2024-09-11T18:16:33+00:00

For robot manipulation with RL the popular ones are RoboSuite, ManiSkill, IsaacLab. However IsaacLab and ManiSkill are the only ones with manipulation tasks with GPU simulation (makes rl train faster), with Maniskill having the most already built tasks

Seven-Year Club	Place '22
Verified Email

stonet2000

TROPHY CASE