This “Imitation” Python library provides open-source implementations of imitation and reward learning algorithms in PyTorch

In domains where reward functions are clearly defined, such as games, reinforcement learning (RL) has outperformed human performance. Unfortunately, it is difficult or impossible for many real-world tasks to design the reward function procedurally. Instead, they should immediately absorb a reward feature or policy from user feedback. Moreover, even when a reward function can be formulated, as in the case of an agent winning a game, the resulting objective may need to be more sparse for RL to be solved efficiently. Therefore, imitation learning is frequently used to initialize the policy in state-of-the-art results for RL.

In this article, they provide imitation, a library that offers excellent, reliable, and modular implementations of seven reward and imitation learning algorithms. Importantly, the interfaces of their algorithms are consistent, making it easier to train and compare different methods. Additionally, contemporary backends like PyTorch and Stable Baselines3 are used to build the imitation. Earlier libraries, on the other hand, frequently supported multiple algorithms, were no longer actively updated, and were built on outdated frameworks. As a benchmark for experiments, imitation has many important applications. According to previous research, small implementation details in imitation learning algorithms can significantly affect performance.

Imitation aims to simplify the process of creating new reward and imitation learning algorithms in addition to providing reliable baselines. If a poor experimental basis is used, it can lead to false positive results. Their techniques have been carefully calibrated and compared to previous solutions to overcome this difficulty. They also perform static type checking and have tests covering 98% of their code. Their implementations are modular, allowing users to flexibly modify the architecture of the reward or policy network, the RL algorithm, and the optimizer without modifying the code.

By subclassing and overriding required methods, algorithms can be extended. Additionally, imitation offers convenient ways to handle routine activities such as collecting deployments, which helps promote the creation of brand new algorithms. The fact that the model is built using state-of-the-art frameworks like PyTorch and Stable Baselines3 is an added bonus. In contrast, many current implementations of imitation and reward learning algorithms were released years ago and have yet to be updated. This is especially valid for reference implementations made available alongside the original releases, such as the GAIL and AIRL codebases.

However, even popular libraries such as Stable Baselines2 are no longer under active development. They compare alternative libraries on a variety of metrics in the table above. Although it is not possible to include all implementations of imitation and reward learning algorithms, this table includes all widely used imitation learning libraries to the best of their knowledge. They find that imitation equals or surpasses alternatives in all parameters. APRel scores high but focuses on preference comparison algorithms learning from low-dimensional features. This is complementary to the model, which provides a wider range of algorithms and emphasizes scalability at the cost of greater implementation complexity. PyTorch implementations are available on GitHub.

Check Paper and GithubGenericName. All credit for this research goes to the researchers on this project. Also don’t forget to register. our Reddit page and discord channelwhere we share the latest AI research news, cool AI projects, and more.