burlap.behavior.singleagent.learning.tdmethods

Class QLearning

java.lang.Object

burlap.behavior.singleagent.MDPSolver

burlap.behavior.singleagent.learning.tdmethods.QLearning

All Implemented Interfaces:

LearningAgent, MDPSolverInterface, Planner, QFunction, QProvider, ValueFunction

Direct Known Subclasses:

SarsaLam

public class QLearning
extends MDPSolver
implements QProvider, LearningAgent, Planner

Tabular Q-learning algorithm [1]. This implementation will work correctly with Options [2]. The implementation can either be used for learning or planning, the latter of which is performed by running many learning episodes in succession in a SimulatedEnvironment. If you are going to use this algorithm for planning, call the initializeForPlanning(int) method before calling planFromState(State). The number of episodes used for planning can be determined by a threshold maximum number of episodes, or by a maximum change in the Q-function threshold.

By default, this agent will use an epsilon-greedy policy with epsilon=0.1. You can change the learning policy to anything with the setLearningPolicy(burlap.behavior.policy.Policy) policy.

If you want to use a custom learning rate decay schedule rather than a constant learning rate, use the setLearningRateFunction(burlap.behavior.learningrate.LearningRate).

1. Watkins, Christopher JCH, and Peter Dayan. "Q-learning." Machine learning 8.3-4 (1992): 279-292.

2. Sutton, Richard S., Doina Precup, and Satinder Singh. "Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning." Artificial intelligence 112.1 (1999): 181-211.

Author:

James MacGlashan

Nested Class Summary

Nested classes/interfaces inherited from interface burlap.behavior.valuefunction.QProvider

QProvider.Helper

Field Summary

Fields

Modifier and Type	Field and Description
protected int	eStepCounter A counter for counting the number of steps in an episode that have been taken thus far
protected Policy	learningPolicy The learning policy to use.
protected LearningRate	learningRate The learning rate function used.
protected int	maxEpisodeSize The maximum number of steps that will be taken in an episode before the agent terminates a learning episode
protected double	maxQChangeForPlanningTermination The maximum allowable change in the Q-function during an episode before the planning method terminates.
protected double	maxQChangeInLastEpisode The maximum Q-value change that occurred in the last learning episode.
protected int	numEpisodesForPlanning The maximum number of episodes to use for planning
protected java.util.Map<HashableState,QLearningStateNode>	qFunction The tabular mapping from states to Q-values
protected QFunction	qInitFunction The object that defines how Q-values are initialized.
protected boolean	shouldDecomposeOptions Whether options should be decomposed into actions in the returned Episode objects.
protected int	totalNumberOfSteps The total number of learning steps performed by this agent.

Fields inherited from class burlap.behavior.singleagent.MDPSolver

actionTypes, debugCode, domain, gamma, hashingFactory, model, usingOptionModel

Constructor Summary

Constructors

Constructor and Description
QLearning(SADomain domain, double gamma, HashableStateFactory hashingFactory, double qInit, double learningRate) Initializes Q-learning with 0.1 epsilon greedy policy, the same Q-value initialization everywhere, and places no limit on the number of steps the agent can take in an episode.
QLearning(SADomain domain, double gamma, HashableStateFactory hashingFactory, double qInit, double learningRate, int maxEpisodeSize) Initializes Q-learning with 0.1 epsilon greedy policy, the same Q-value initialization everywhere.
QLearning(SADomain domain, double gamma, HashableStateFactory hashingFactory, double qInit, double learningRate, Policy learningPolicy, int maxEpisodeSize) Initializes the same Q-value initialization everywhere.
QLearning(SADomain domain, double gamma, HashableStateFactory hashingFactory, QFunction qInit, double learningRate, Policy learningPolicy, int maxEpisodeSize) Initializes the algorithm.

Method Summary

Modifier and Type	Method and Description
int	getLastNumSteps() Returns the number of steps taken in the last episode;
protected double	getMaxQ(HashableState s) Returns the maximum Q-value in the hashed stated.
protected QValue	getQ(HashableState s, Action a) Returns the Q-value for a given hashed state and action.
protected java.util.List<QValue>	getQs(HashableState s) Returns the possible Q-values for a given hashed stated.
protected QLearningStateNode	getStateNode(HashableState s) Returns the QLearningStateNode object stored for the given hashed state.
void	initializeForPlanning(int numEpisodesForPlanning) Sets the RewardFunction, TerminalFunction, and the number of simulated episodes to use for planning when the planFromState(State) method is called.
void	loadQTable(java.lang.String path) Loads the q-function table located on disk at the specified path.
GreedyQPolicy	planFromState(State initialState) Plans from the input state and then returns a GreedyQPolicy that greedily selects the action with the highest Q-value and breaks ties uniformly randomly.
protected void	QLInit(SADomain domain, double gamma, HashableStateFactory hashingFactory, QFunction qInitFunction, double learningRate, Policy learningPolicy, int maxEpisodeSize) Initializes the algorithm.
double	qValue(State s, Action a) Returns the QValue for the given state-action pair.
java.util.List<QValue>	qValues(State s) Returns a List of QValue objects for ever permissible action for the given input state.
void	resetSolver() This method resets all solver results so that a solver can be restarted fresh as if had never solved the MDP.
Episode	runLearningEpisode(Environment env)
Episode	runLearningEpisode(Environment env, int maxSteps)
void	setLearningPolicy(Policy p) Sets which policy this agent should use for learning.
void	setLearningRateFunction(LearningRate lr) Sets the learning rate function to use
void	setMaximumEpisodesForPlanning(int n) Sets the maximum number of episodes that will be performed when the planFromState(State) method is called.
void	setMaxQChangeForPlanningTerminaiton(double m) Sets a max change in the Q-function threshold that will cause the planFromState(State) to stop planning when it is achieved.
void	setQInitFunction(QFunction qInit) Sets how to initialize Q-values for previously unexperienced state-action pairs.
void	toggleShouldDecomposeOption(boolean toggle) Sets whether the primitive actions taken during an options will be included as steps in returned EpisodeAnalysis objects.
double	value(State s) Returns the value function evaluation of the given state.
void	writeQTable(java.lang.String path) Writes the q-function table stored in this object to the specified file path.

Methods inherited from class burlap.behavior.singleagent.MDPSolver

addActionType, applicableActions, getActionTypes, getDebugCode, getDomain, getGamma, getHashingFactory, getModel, setActionTypes, setDebugCode, setDomain, setGamma, setHashingFactory, setModel, solverInit, stateHash, toggleDebugPrinting

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface burlap.behavior.singleagent.MDPSolverInterface

addActionType, getActionTypes, getDebugCode, getDomain, getGamma, getHashingFactory, getModel, setActionTypes, setDebugCode, setDomain, setGamma, setHashingFactory, setModel, solverInit, toggleDebugPrinting

Field Detail

qFunction

protected java.util.Map<HashableState,QLearningStateNode> qFunction

The tabular mapping from states to Q-values

qInitFunction

protected QFunction qInitFunction

The object that defines how Q-values are initialized.

learningRate

protected LearningRate learningRate

The learning rate function used.

learningPolicy

protected Policy learningPolicy

The learning policy to use. Typically these will be policies that link back to this object so that they change as the Q-value estimate change.

maxEpisodeSize

protected int maxEpisodeSize

The maximum number of steps that will be taken in an episode before the agent terminates a learning episode

eStepCounter

protected int eStepCounter

A counter for counting the number of steps in an episode that have been taken thus far

numEpisodesForPlanning

protected int numEpisodesForPlanning

The maximum number of episodes to use for planning

maxQChangeForPlanningTermination

protected double maxQChangeForPlanningTermination

The maximum allowable change in the Q-function during an episode before the planning method terminates.

maxQChangeInLastEpisode

protected double maxQChangeInLastEpisode

The maximum Q-value change that occurred in the last learning episode.

shouldDecomposeOptions

protected boolean shouldDecomposeOptions

Whether options should be decomposed into actions in the returned Episode objects.

totalNumberOfSteps

protected int totalNumberOfSteps

The total number of learning steps performed by this agent.

Constructor Detail

QLearning

public QLearning(SADomain domain,
                 double gamma,
                 HashableStateFactory hashingFactory,
                 double qInit,
                 double learningRate)

Initializes Q-learning with 0.1 epsilon greedy policy, the same Q-value initialization everywhere, and places no limit on the number of steps the agent can take in an episode. By default the agent will only save the last learning episode and a call to the planFromState(State) method will cause the valueFunction to use only one episode for planning; this should probably be changed to a much larger value if you plan on using this algorithm as a planning algorithm.

Parameters:

domain - the domain in which to learn

gamma - the discount factor

hashingFactory - the state hashing factory to use for Q-lookups

qInit - the initial Q-value to user everywhere

learningRate - the learning rate

QLearning

public QLearning(SADomain domain,
                 double gamma,
                 HashableStateFactory hashingFactory,
                 double qInit,
                 double learningRate,
                 int maxEpisodeSize)

Initializes Q-learning with 0.1 epsilon greedy policy, the same Q-value initialization everywhere. By default the agent will only save the last learning episode and a call to the planFromState(State) method will cause the valueFunction to use only one episode for planning; this should probably be changed to a much larger value if you plan on using this algorithm as a planning algorithm.

Parameters:

domain - the domain in which to learn

gamma - the discount factor

hashingFactory - the state hashing factory to use for Q-lookups

qInit - the initial Q-value to user everywhere

learningRate - the learning rate

maxEpisodeSize - the maximum number of steps the agent will take in a learning episode for the agent stops trying.

QLearning

public QLearning(SADomain domain,
                 double gamma,
                 HashableStateFactory hashingFactory,
                 double qInit,
                 double learningRate,
                 Policy learningPolicy,
                 int maxEpisodeSize)

Initializes the same Q-value initialization everywhere. Note that if the provided policy is derived from the Q-value of this learning agent (as it should be), you may need to set the policy to point to this object after call this constructor; the constructor will not do this automatically in case it was by design to use the policy that was learned in some other domain. By default the agent will only save the last learning episode and a call to the planFromState(State) method will cause the valueFunction to use only one episode for planning; this should probably be changed to a much larger value if you plan on using this algorithm as a planning algorithm.

Parameters:

domain - the domain in which to learn

gamma - the discount factor

hashingFactory - the state hashing factory to use for Q-lookups

qInit - the initial Q-value to user everywhere

learningRate - the learning rate

learningPolicy - the learning policy to follow during a learning episode.

maxEpisodeSize - the maximum number of steps the agent will take in a learning episode for the agent stops trying.

QLearning

public QLearning(SADomain domain,
                 double gamma,
                 HashableStateFactory hashingFactory,
                 QFunction qInit,
                 double learningRate,
                 Policy learningPolicy,
                 int maxEpisodeSize)

Initializes the algorithm. Note that if the provided policy is derived from the Q-value of this learning agent (as it should be), you may need to set the policy to point to this object after call this constructor; the constructor will not do this automatically in case it was by design to use the policy that was learned in some other domain. By default the agent will only save the last learning episode and a call to the planFromState(State) method will cause the valueFunction to use only one episode for planning; this should probably be changed to a much larger value if you plan on using this algorithm as a planning algorithm.

Parameters:

domain - the domain in which to learn

gamma - the discount factor

hashingFactory - the state hashing factory to use for Q-lookups

qInit - a QFunction object that can be used to initialize the Q-values.

learningRate - the learning rate

learningPolicy - the learning policy to follow during a learning episode.

maxEpisodeSize - the maximum number of steps the agent will take in a learning episode for the agent stops trying.

Method Detail

QLInit

protected void QLInit(SADomain domain,
                      double gamma,
                      HashableStateFactory hashingFactory,
                      QFunction qInitFunction,
                      double learningRate,
                      Policy learningPolicy,
                      int maxEpisodeSize)

Initializes the algorithm. By default the agent will only save the last learning episode and a call to the planFromState(State) method will cause the valueFunction to use only one episode for planning; this should probably be changed to a much larger value if you plan on using this algorithm as a planning algorithm.

Parameters:

domain - the domain in which to learn

gamma - the discount factor

hashingFactory - the state hashing factory to use for Q-lookups

qInitFunction - a QFunction object that can be used to initialize the Q-values.

learningRate - the learning rate

learningPolicy - the learning policy to follow during a learning episode.

maxEpisodeSize - the maximum number of steps the agent will take in a learning episode for the agent stops trying.

initializeForPlanning

public void initializeForPlanning(int numEpisodesForPlanning)

Sets the RewardFunction, TerminalFunction, and the number of simulated episodes to use for planning when the planFromState(State) method is called.

Parameters:

numEpisodesForPlanning - the number of simulated episodes to run for planning.

setLearningRateFunction

public void setLearningRateFunction(LearningRate lr)

Sets the learning rate function to use

Parameters:

lr - the learning rate function to use

setQInitFunction

public void setQInitFunction(QFunction qInit)

Sets how to initialize Q-values for previously unexperienced state-action pairs.

Parameters:

qInit - a QFunction object that can be used to initialize the Q-values.

setLearningPolicy

public void setLearningPolicy(Policy p)

Sets which policy this agent should use for learning.

Parameters:

p - the policy to use for learning.

setMaximumEpisodesForPlanning

public void setMaximumEpisodesForPlanning(int n)

Sets the maximum number of episodes that will be performed when the planFromState(State) method is called.

Parameters:

n - the maximum number of episodes that will be performed when the planFromState(State) method is called.

setMaxQChangeForPlanningTerminaiton

public void setMaxQChangeForPlanningTerminaiton(double m)

Sets a max change in the Q-function threshold that will cause the planFromState(State) to stop planning when it is achieved.

Parameters:

m - the maximum allowable change in the Q-function before planning stops

getLastNumSteps

public int getLastNumSteps()

Returns the number of steps taken in the last episode;

Returns:

the number of steps taken in the last episode;

toggleShouldDecomposeOption

public void toggleShouldDecomposeOption(boolean toggle)

Sets whether the primitive actions taken during an options will be included as steps in returned EpisodeAnalysis objects. The default value is true. If this is set to false, then EpisodeAnalysis objects returned from a learning episode will record options as a single "action" and the steps taken by the option will be hidden.

Parameters:

toggle - whether to decompose options into the primitive actions taken by them or not.

qValues

public java.util.List<QValue> qValues(State s)

Description copied from interface: QProvider

Returns a List of QValue objects for ever permissible action for the given input state.

Specified by:

qValues in interface QProvider

Parameters:

s - the state for which Q-values are to be returned.

Returns:

a List of QValue objects for ever permissible action for the given input state.

qValue

public double qValue(State s,
                     Action a)

Description copied from interface: QFunction

Returns the QValue for the given state-action pair.

Specified by:

qValue in interface QFunction

Parameters:

s - the input state

a - the input action

Returns:

the QValue for the given state-action pair.

getQs

protected java.util.List<QValue> getQs(HashableState s)

Returns the possible Q-values for a given hashed stated.

Parameters:

s - the hashed state for which to get the Q-values.

Returns:

the possible Q-values for a given hashed stated.

getQ

protected QValue getQ(HashableState s,
                      Action a)

Returns the Q-value for a given hashed state and action.

Parameters:

s - the hashed state

a - the action

Returns:

the Q-value for a given hashed state and action; null is returned if there is not Q-value currently stored.

value

public double value(State s)

Description copied from interface: ValueFunction

Returns the value function evaluation of the given state. If the value is not stored, then the default value specified by the ValueFunctionInitialization object of this class is returned.

Specified by:

value in interface ValueFunction

Parameters:

s - the state to evaluate.

Returns:

the value function evaluation of the given state.

getStateNode

protected QLearningStateNode getStateNode(HashableState s)

Returns the QLearningStateNode object stored for the given hashed state. If no QLearningStateNode object. is stored, then it is created and has its Q-value initialize using this objects QFunction data member.

Parameters:

s - the hashed state for which to get the QLearningStateNode object

Returns:

the QLearningStateNode object stored for the given hashed state. If no QLearningStateNode object.

getMaxQ

protected double getMaxQ(HashableState s)

Returns the maximum Q-value in the hashed stated.

Parameters:

s - the state for which to get he maximum Q-value;

Returns:

the maximum Q-value in the hashed stated.

planFromState

public GreedyQPolicy planFromState(State initialState)

Plans from the input state and then returns a GreedyQPolicy that greedily selects the action with the highest Q-value and breaks ties uniformly randomly.

Specified by:

planFromState in interface Planner

Parameters:

initialState - the initial state of the planning problem

Returns:

a GreedyQPolicy.

runLearningEpisode

public Episode runLearningEpisode(Environment env)

Specified by:

runLearningEpisode in interface LearningAgent

runLearningEpisode

public Episode runLearningEpisode(Environment env,
                                  int maxSteps)

Specified by:

runLearningEpisode in interface LearningAgent

resetSolver

public void resetSolver()

Description copied from interface: MDPSolverInterface

This method resets all solver results so that a solver can be restarted fresh as if had never solved the MDP.

Specified by:

resetSolver in interface MDPSolverInterface

Specified by:

resetSolver in class MDPSolver

writeQTable

public void writeQTable(java.lang.String path)

Writes the q-function table stored in this object to the specified file path. Uses a standard YAML approach, which means the HashableState and underlying Domain states must have JavaBean like properties; i.e., have a default constructor and getters and setters (or public data members) for all relevant fields.

Parameters:

path - the path to write the value function

loadQTable

public void loadQTable(java.lang.String path)

Loads the q-function table located on disk at the specified path. Expects the file to be a Yaml representation of a Java Map from HashableState to QLearningStateNode.

Parameters:

path - the path to the save value function table