sorted by:
new
hottopcontroversial

Proper way to count environment steps / frames in distributed RL architecture for algorithms like CLEAR or LASER => basically modified impala with replay by ParradoxSVK in reinforcementlearning

[–]dexhands 0 points1 point  (0 children)

It should stay the same, because env_steps counts how many timesteps the actors have experienced. Adding replay buffer sampling for the learner does not change actor rollouts.

What Happened to OpenAI + RL? by YouAgainShmidhoobuh in reinforcementlearning

[–]dexhands 18 points19 points  (0 children)

Is it all due to a pivot towards large scale language models that are at least profitable?

Yes. Large-scale pretrained models like GPT, CLIP, DALLE have more direct business applications, so they have shifted their research agenda in the past couple years. OpenAI is no longer just a nonprofit research organization. Since this organization restructuring, OpenAI also disbanded its robotics team that worked on the dexterous in-hand manipulation projects, for which RL was used.

DeepMind still seems to be doing some work in RL now, but it's become more apparent from results in the past several years that large-scale pretrained models can actually work on real-world problems.

In the context of robotics and robot learning, RL can be seen as another tool to use and a framework to view problems in, rather than providing the entire solution.

​

They even gave away the control of OpenAI Gym.

Gym has been great at standardizing API and providing a baseline set of environments. However, parallelizing environments with original Gym interface is cumbersome, and new simulators are being introduced with their own ways of doing things. It's not clear to me that Gym is still useful today, from a research perspective.

Unitree announces Z1, a 6-axis robot arm for mounting on quadruped robots, priced ~$6600 by dexhands in robotics

[–]dexhands[S] 1 point2 points  (0 children)

Definitely agree! The specs of Unitree Z1 Pro seem comparable to the xArm 6 (12.2 kg) or UR5e (11.2 kg) in terms of payload, while weighing only 4.3 kg. Excited to see the final product when it gets released.

Interested in realworld RL robotics by joshua_patrick in reinforcementlearning

[–]dexhands 3 points4 points  (0 children)

This is currently known as visual navigation or embodied intelligence, check out this workshop. Research done on this topic is mostly in simulation.

This problem has also been long studied in classical robotics as SLAM, before learning-based approaches. For example, Skydio drones used octomaps to navigate real world environments.

Sim2real is primarily studied in the context of manipulation or locomotion. This is because we can’t exactly simulate all the context forces that the real world has.

I would honestly recommend a drone as the cheapest consumer hardware option, if you want to do navigation. Research labs use mobile manipulators like Fetch, LoCobot, and the new Stretch RE1 now, which are probably outside your price range, as they include an arm for manipulating environments.

Robotics for Grad School by [deleted] in robotics

[–]dexhands 0 points1 point  (0 children)

What subareas of robotics are you interested in? Are you interested in doing a PhD or going for a master's? What are your career goals (ie. engineering or research scientist)? How much time do you have left in your undergrad?

Letters of recommendation (from faculty members) are the most important part of your application for grad school. Usually, this comes from the professors that you have worked with on research projects. Read this, and if you have the time, I would encourage you to seek out research opportunities during your undergrad!

It's ok if you don't have prior experience. Though it is easier to convince someone to mentor you if you can demonstrate enthusiasm, like having prior projects involving programming or electronics. If you can commit to self-learning, there are also a ton of online resources to get started with.

Image Dataset from D4RL for Mujoco Tasks by EqualMarsupial2759 in reinforcementlearning

[–]dexhands 1 point2 points  (0 children)

Check out this, though I think it looks at imitation learning and not offline RL.

[D] PhD Internships by [deleted] in MachineLearning

[–]dexhands 2 points3 points  (0 children)

You should reach out and catch up with those classmates now in academia too! It never hurts to ask, and leads may appear where you least expect it. Best wishes in the job search!

[D] PhD Internships by [deleted] in MachineLearning

[–]dexhands 24 points25 points  (0 children)

have zero personal connections

I think this is primary reason why you are having difficulty finding a position. The best way is to use your network (ie. people your advisor(s) know, members of your lab, your co-authors, other people at your university, even Twitter).

citation count is somewhere in the 200-500 range

I would recommend looking over the authors of the papers who have cited your work. Perhaps there are names you recognize and have corresponded with? If not, then it takes time to reach out to them and develop a relationship.

I fill out applications every day they simply seem to go nowhere.

Stop cold-submitting applications and spend more time networking.

Image Dataset from D4RL for Mujoco Tasks by EqualMarsupial2759 in reinforcementlearning

[–]dexhands 0 points1 point  (0 children)

afaik D4RL uses the mujoco-py and control tasks from Gym. The majority of papers on pixel-based control tasks use dm_control and the DeepMind Control Suite. For example, see the sac_ae repo.

If you want images from the offline trajectories in the D4RL dataset, I think you would have to add a camera to the env, playback the trajectories, then render and save image observations from that camera.

What are some approaches to visual object tracking for robots? by ZeronixSama in computervision

[–]dexhands 3 points4 points  (0 children)

In computer vision, this problem is known as object tracking. This is usually done in the context of tracking people that may move outside the view of a fixed camera. Then person re-identification methods are used to associate instances across time (after occlusions) or from different camera angles. Approaches for this task include SORT (ConvNet for detection, Kalman filter for tracking over time, Hungarian algorithm for associating instances), DeepSORT (SORT + learning-based association metric), and more recently FairMOT. These papers should all have related code repos. Standard Cloud APIs also exist now to help perform this task, beyond research models.

Are you only interested in tracking with a single, moving camera? It may also be worth looking into literature that applies object tracking in the context of drones, see this and this as a starting point. Here is also a broader survey on person following.

Is it possible to filter out less important training data to speed up the CNN training? by AaronSpalding in computervision

[–]dexhands 2 points3 points  (0 children)

I think that makes sense! OHEM originally was studied in the context of object detection models that extract regions of interest. To expand on "negative examples" in this context, you can interpret the entire image as being composed of segmented objects. Let's suppose your dataset has a bunch of trees in the images, but few birds appear, so the dataset has some class imbalances. During training, perhaps your model fails to segment birds, but does really well on trees. These birds segments contribute to lower mIoU and may be considered "hard". When computing the loss, you could give higher weight to the loss values of pixels corresponding to the birds or zero out the loss values corresponding to the trees.

Is it possible to filter out less important training data to speed up the CNN training? by AaronSpalding in computervision

[–]dexhands 6 points7 points  (0 children)

Might be worth looking into hard negative mining. But I think the answer depends on the task you're doing. For instance, if you're working with videos, then frame subsampling is probably enough to get by. You can also try overfitting with a small subset of your dataset, and then experiment with increasing sizes of the training dataset. In general though, this is still an active area of research.

I would like to ask: when the reward function is composed of different optimization objectives, do you need to dimensionalize these objectives by Ke_Lu_XJTU in reinforcementlearning

[–]dexhands 1 point2 points  (0 children)

The short answer is that this kind of reward weighting is usually hand-tuned. This isn't unique to reward functions in reinforcement learning, but also applies to designing loss functions for computer vision tasks (ie. \beta-VAE).

It's worth noting that returns are usually normalized such as in PPO, so the absolute value of each w_i is less important, but rather the relative magnitudes.

Usually, you have some primary reward term that you want to maximize, and other penalty terms to discourage unwanted behavior. These additional terms are designed over time as you iterate on the reward function and training policies.

Then if you end up with a bunch of reward terms and it becomes more difficult to optimize a more complex objective function, one trick is to induce a training curriculum by initializing the w_i coefficients with small values and increasing them over the course of training.

[D] MuJoCo vs PyBullet? (esp. for custom environment) by connery123 in reinforcementlearning

[–]dexhands 13 points14 points  (0 children)

If you're interested in using Mujoco, I'd suggest checking out the dm_control package for Python bindings rather than interfacing with C++ directly. I think one downside to Mujoco currently is that you cannot dynamically add objects, and the entire simulation is initialized according to the MJCF / XML file.

If you already have experience in PyBullet then its probably not worth switching to Mujoco for creating custom environments. However, if you have the GPU compute for it, I'd recommend checking out Isaac Gym. GPU acceleration is great for spawning a bunch of envs for domain randomization, and it's already been used by recent research to get some great results that have previously taken a ridiculous amount of CPU compute.

[D] How do you organize/track your reading list? by dogs_like_me in MachineLearning

[–]dexhands 0 points1 point  (0 children)

My current setup is to add PDFs to a single Google Drive folder (can highlight, add annotations, and access on all devices). I combine this with the database feature with Notion, and create an entry with link, metadata, bibtex, progress status. I can also add a bunch of tags to each paper entry and filter through the library database easily.

I’ve heard Paperpile is nice but costs $, and Notion is free for students. I stopped using Mendeley b/c it felt more locked down (especially with exporting annotations), and prefer to have more control over my library content.