burlap.behavior.singleagent.learning.actorcritic

Class ActorCritic

java.lang.Object

burlap.behavior.singleagent.MDPSolver

burlap.behavior.singleagent.learning.actorcritic.ActorCritic

All Implemented Interfaces:

LearningAgent, MDPSolverInterface

public class ActorCritic
extends MDPSolver
implements LearningAgent

This is a general class structure for implementing Actor-critic learning. The kind of actor critic learning performed can be modified by swapping out different Actor and Critic objects. The general structure of the learning algorithm is for the Actor class to be queried for an action given the current state of the world. That action is taken and a resulting state is observed. The Critic is then asked to critique this behavior which is returned in a CritiqueResult object and then passed along to the Actor so that the actor may update is behavior accordingly.

In addition to learning, this algorithm can also be used for planning using the planFromState(State) method. If you plan to use it for planning, you should call the initializeForPlanning(int) method before calling the planFromState(State).

Author:

James MacGlashan

Field Summary

Fields

Modifier and Type	Field and Description
protected Actor	actor The actor component to use.
protected Critic	critic The critic component to use
protected java.util.LinkedList<Episode>	episodeHistory The saved and most recent learning episodes this agent has performed.
protected int	maxEpisodeSize The maximum number of steps of an episode before the agent will manually terminate it.This is defaulted to Integer.MAX_VALUE.
protected int	numEpisodesForPlanning The number of simulated learning episodes to use when the planFromState(State) method is called.
protected int	numEpisodesToStore The number of most recent learning episodes to store.

Fields inherited from class burlap.behavior.singleagent.MDPSolver

actionTypes, debugCode, domain, gamma, hashingFactory, model, usingOptionModel

Constructor Summary

Constructors

Constructor and Description
ActorCritic(SADomain domain, double gamma, Actor actor, Critic critic) Initializes the learning algorithm.
ActorCritic(SADomain domain, double gamma, Actor actor, Critic critic, int maxEpisodeSize) Initializes the learning algorithm.

Method Summary

Modifier and Type	Method and Description
void	addActionType(ActionType a) Adds an additional action the solver that is not included in the domain definition.
java.util.List<Episode>	getAllStoredLearningEpisodes()
Episode	getLastLearningEpisode()
Policy	getPolicy() Returns the policy/actor of this learning algorithm.
void	initializeForPlanning(int numEpisodesForPlanning) Sets the number of simulated episodes to use for planning when the planFromState(State) method is called.
void	planFromState(State initialState)
void	resetSolver() This method resets all solver results so that a solver can be restarted fresh as if had never solved the MDP.
Episode	runLearningEpisode(Environment env)
Episode	runLearningEpisode(Environment env, int maxSteps)
void	setNumEpisodesToStore(int numEps)

Methods inherited from class burlap.behavior.singleagent.MDPSolver

applicableActions, getActionTypes, getDebugCode, getDomain, getGamma, getHashingFactory, getModel, setActionTypes, setDebugCode, setDomain, setGamma, setHashingFactory, setModel, solverInit, stateHash, toggleDebugPrinting

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

actor

protected Actor actor

The actor component to use.

critic

protected Critic critic

The critic component to use

maxEpisodeSize

protected int maxEpisodeSize

The maximum number of steps of an episode before the agent will manually terminate it.This is defaulted to Integer.MAX_VALUE.

numEpisodesForPlanning

protected int numEpisodesForPlanning

The number of simulated learning episodes to use when the planFromState(State) method is called.

episodeHistory

protected java.util.LinkedList<Episode> episodeHistory

The saved and most recent learning episodes this agent has performed.

numEpisodesToStore

protected int numEpisodesToStore

The number of most recent learning episodes to store.

Constructor Detail

ActorCritic

public ActorCritic(SADomain domain,
                   double gamma,
                   Actor actor,
                   Critic critic)

Initializes the learning algorithm.

Parameters:

domain - the domain in which to learn

gamma - the discount factor

actor - the actor component to use to select actions

critic - the critic component to use to critique

ActorCritic

public ActorCritic(SADomain domain,
                   double gamma,
                   Actor actor,
                   Critic critic,
                   int maxEpisodeSize)

Initializes the learning algorithm.

Parameters:

domain - the domain in which to learn

gamma - the discount factor

actor - the actor component to use to select actions

critic - the critic component to use to critique

maxEpisodeSize - the maximum number of steps the agent will take in a learning episode before the agent gives up.

Method Detail

initializeForPlanning

public void initializeForPlanning(int numEpisodesForPlanning)

Sets the number of simulated episodes to use for planning when the planFromState(State) method is called. If the RewardFunction and TerminalFunction are not set, the planFromState(State) method will throw a runtime exception.

Parameters:

numEpisodesForPlanning - the number of simulated episodes to run for planning.

addActionType

public void addActionType(ActionType a)

Description copied from interface: MDPSolverInterface

Adds an additional action the solver that is not included in the domain definition. For instance, an Option should be added using this method.

Specified by:

addActionType in interface MDPSolverInterface

Overrides:

addActionType in class MDPSolver

Parameters:

a - the action to add to the solver

runLearningEpisode

public Episode runLearningEpisode(Environment env)

Specified by:

runLearningEpisode in interface LearningAgent

runLearningEpisode

public Episode runLearningEpisode(Environment env,
                                  int maxSteps)

Specified by:

runLearningEpisode in interface LearningAgent

getLastLearningEpisode

public Episode getLastLearningEpisode()

setNumEpisodesToStore

public void setNumEpisodesToStore(int numEps)

getAllStoredLearningEpisodes

public java.util.List<Episode> getAllStoredLearningEpisodes()

planFromState

public void planFromState(State initialState)

resetSolver

public void resetSolver()

Description copied from interface: MDPSolverInterface

This method resets all solver results so that a solver can be restarted fresh as if had never solved the MDP.

Specified by:

resetSolver in interface MDPSolverInterface

Specified by:

resetSolver in class MDPSolver

getPolicy

public Policy getPolicy()

Returns the policy/actor of this learning algorithm. Note that all Actor objects are also Policy objects.

Returns:

the policy/actor of this learning algorithm.