Cooperation of multiple reinforcement learning algorithms

Reinforcement learning methods are essential components in the development of autonomous robotic systems. They must enable these systems to learn, through trial and error and without additional intervention by their designers, the actions that must be performed, and those that must be avoided, in order to achieve their mission.

Two main classes of algorithms have historically been developed in the literature: those based on the use of an internal model of the world, and in particular of transitions between states, and those without an internal model. The former consumes a lot of computational resources (i.e. the computations required to deduce the action that seems to lead to the best consequences as predicted by the internal model), but enables us to react to changes in the environment in just a few trials, by reusing the knowledge previously learned about the structure of the environment thanks to the internal model; the latter is extremely inexpensive (no model, so no estimation of the consequences of the action), but at the cost of slow convergence of learning and very poor adaptability to change (i.e.g. hundreds of trials are required to update the values associated with actions following a change in the environment). It might therefore seem logical to seek to benefit from the complementarities of these two approaches by combining them. However, the cooperation of multiple reinforcement learning systems has, until now, been little explored in the machine learning literature.

The good properties of such an approach were initially developed in the context of the study of animal behavior. Indeed, the cohabitation of multiple learning systems, and the existence of distinct neural substrates, have been clearly demonstrated in neuroscience. Several computational models have been proposed to account for the way animals coordinate their multiple learning systems. These models are a source of inspiration for the design of robotic systems. This import has been mainly focused on navigation, but need not be limited to it. Finally, the limits of these methods, whose scientific objective is the simulation of animal behavior and not operational efficiency, are perfectly surmountable in the engineering context, by getting rid of biological constraints.