Reinforcement Learning in the HighwayEnv

Links to the code: social attention | behavioral cloning & implicit Q learning | state space models.

I started these experiments because many robotics observations are naturally sets rather than fixed feature vectors. A detector, for example, returns a variable collection of object boxes for which the ordering is arbitrary. This motivates permutation-invariant encoders such as Deep Sets and Social Attention. I implemented both in HighwayEnv; in my behavioral-cloning experiments, the social-attention encoder produced the better driving policy, although the comparison was small and not exhaustively tuned.

From online RL to imitation learning

My initial DQN and SAC experiments were unstable. The continuous-action SAC agent behaved erratically, while the discrete-action experiments exhibited high variance and occasional catastrophic forgetting. A latent state-space model with cross-entropy-method planning was also unsuccessful; the repository records an average planning return of 11.99 and correctly treats it as a negative result rather than a competitive method.

I therefore shifted to behavioral cloning and Implicit Q-Learning. I collected demonstrations manually and found that the learned policy reproduced recognizable habits in the data, such as staying in the rightmost lane when possible and accelerating until blocked. In a 50-episode evaluation, the revised, more safety-aware demonstration set yielded a mean behavioral-cloning return of 25.69. The IQL implementation reached 27.10, but I do not regard that number as conclusive because the original reward distribution was heavily concentrated near one and required filtering.

Lessons from the dataset

The most useful outcome was understanding the data problems. Rewards were almost always close to one, with crashes forming a very small fraction of the transitions. The demonstration data were also imbalanced because the ego vehicle selected IDLE most of the time. I used focal loss for the latter issue and recollected demonstrations after discovering that the first input-recording hook stored incorrect actions. These details mattered more than changing the RL algorithm.

Potential-based shaping for robot navigation

I also used the same implementation of IQL, combined with reward shaping, to explore the potential for learning-based navigation using an OpenCV-based simulator (for the RoboMaster University League robotics competition) wrapped as a Gym environment. The observation is based on the localization result from a particle filter (also implemented by the team I lead). While the code isn’t publicly available, you can view how it performs below:

The shaded robot represents the state estimated by a particle filter. I used the potential-based shaping term

with

where $\alpha,\beta>0$, $c,\ell>0$, and $d(s)$ is the minimum distance from the current state to an obstacle. This second experiment reinforced the same conclusion: the structure and quality of the training signal can dominate the choice of learning algorithm.