Adding a New Agent

Coach’s modularity makes adding an agent a simple and clean task.We suggest using the followingJupyter notebook tutorialto ramp up on this process. In general, it involves the following steps:

  • Implement your algorithm in a new file. The agent can inherit base classes such as ValueOptimizationAgent orActorCriticAgent, or the more generic Agent base class.

Note

ValueOptimizationAgent, PolicyOptimizationAgent and Agent are abstract classes.learn_from_batch() should be overriden with the desired behavior for the algorithm being implemented.If deciding to inherit from Agent, also choose_action() should be overriden.

  1. def learn_from_batch(self, batch) -> Tuple[float, List, List]:
  2.     """
  3.     Given a batch of transitions, calculates their target values and updates the network.
  4.     :param batch: A list of transitions
  5.     :return: The total loss of the training, the loss per head and the unclipped gradients
  6.     """
  7.  
  8. def choose_action(self, curr_state):
  9.     """
  10.     choose an action to act with in the current episode being played. Different behavior might be exhibited when training
  11.      or testing.
  12.  
  13.     :param curr_state: the current state to act upon.
  14.     :return: chosen action, some action value describing the action (q-value, probability, etc)
  15.     """

  • Implement your agent’s specific network head, if needed, at the implementation for the framework of your choice.For example architectures/neon_components/heads.py. The head will inherit the generic base class Head.A new output type should be added to configurations.py, and a mapping between the new head and output type shouldbe defined in the get_output_head() function at architectures/neon_components/general_network.py

  • Define a new parameters class that inherits AgentParameters.The parameters class defines all the hyperparameters for the agent, and is initialized with 4 main components:

    • algorithm: A class inheriting AlgorithmParameters which defines any algorithm specific parameters

    • exploration: A class inheriting ExplorationParameters which defines the exploration policy parameters.There are several common exploration policies built-in which you can use, and are defined underthe exploration sub directory. You can also define your own custom exploration policy.

    • memory: A class inheriting MemoryParameters which defined the memory parameters.There are several common memory types built-in which you can use, and are defined under the memoriessub directory. You can also define your own custom memory.

    • networks: A dictionary defining all the networks that will be used by the agent. The keys of the dictionarydefine the network name and will be used to access each network through the agent class.The dictionary values are a class inheriting NetworkParameters, which define the network structureand parameters.

Additionally, set the path property to return the path to your agent class in the following format:

<path to python module>:<name of agent class>

For example,

  1. class RainbowAgentParameters(AgentParameters):
  2. def __init__(self):
  3.     super().__init__(algorithm=RainbowAlgorithmParameters(),
  4.                      exploration=RainbowExplorationParameters(),
  5.                      memory=RainbowMemoryParameters(),
  6.                      networks={"main": RainbowNetworkParameters()})
  7.  
  8. @property
  9. def path(self):
  10.     return 'rainbow.rainbow_agent:RainbowAgent'
  • (Optional) Define a preset using the new agent type with a given environment, and the hyper-parameters that shouldbe used for training on that environment.