public class QLearning extends MDPSolver implements QFunction, LearningAgent, Planner
SimulatedEnvironment
.
If you are going to use this algorithm for planning, call the initializeForPlanning(burlap.oomdp.singleagent.RewardFunction, burlap.oomdp.core.TerminalFunction, int)
method before calling planFromState(burlap.oomdp.core.states.State)
.
The number of episodes used for planning can be determined
by a threshold maximum number of episodes, or by a maximum change in the Q-function threshold.
setLearningPolicy(burlap.behavior.policy.Policy)
policy.
setLearningRateFunction(burlap.behavior.learningrate.LearningRate)
.
QFunction.QFunctionHelper
Modifier and Type | Field and Description |
---|---|
protected java.util.LinkedList<EpisodeAnalysis> |
episodeHistory
the saved previous learning episodes
|
protected int |
eStepCounter
A counter for counting the number of steps in an episode that have been taken thus far
|
protected Policy |
learningPolicy
The learning policy to use.
|
protected LearningRate |
learningRate
The learning rate function used.
|
protected int |
maxEpisodeSize
The maximum number of steps that will be taken in an episode before the agent terminates a learning episode
|
protected double |
maxQChangeForPlanningTermination
The maximum allowable change in the Q-function during an episode before the planning method terminates.
|
protected double |
maxQChangeInLastEpisode
The maximum Q-value change that occurred in the last learning episode.
|
protected int |
numEpisodesForPlanning
The maximum number of episodes to use for planning
|
protected int |
numEpisodesToStore
The number of the most recent learning episodes to store.
|
protected java.util.Map<HashableState,QLearningStateNode> |
qIndex
The tabular mapping from states to Q-values
|
protected ValueFunctionInitialization |
qInitFunction
The object that defines how Q-values are initialized.
|
protected boolean |
shouldAnnotateOptions
Whether decomposed options should have their primitive actions annotated with the options name in the returned
EpisodeAnalysis objects. |
protected boolean |
shouldDecomposeOptions
Whether options should be decomposed into actions in the returned
EpisodeAnalysis objects. |
protected int |
totalNumberOfSteps
The total number of learning steps performed by this agent.
|
actions, debugCode, domain, gamma, hashingFactory, mapToStateIndex, rf, tf
Constructor and Description |
---|
QLearning(Domain domain,
double gamma,
HashableStateFactory hashingFactory,
double qInit,
double learningRate)
Initializes Q-learning with 0.1 epsilon greedy policy, the same Q-value initialization everywhere, and places no limit on the number of steps the
agent can take in an episode.
|
QLearning(Domain domain,
double gamma,
HashableStateFactory hashingFactory,
double qInit,
double learningRate,
int maxEpisodeSize)
Initializes Q-learning with 0.1 epsilon greedy policy, the same Q-value initialization everywhere.
|
QLearning(Domain domain,
double gamma,
HashableStateFactory hashingFactory,
double qInit,
double learningRate,
Policy learningPolicy,
int maxEpisodeSize)
Initializes the same Q-value initialization everywhere.
|
QLearning(Domain domain,
double gamma,
HashableStateFactory hashingFactory,
ValueFunctionInitialization qInit,
double learningRate,
Policy learningPolicy,
int maxEpisodeSize)
Initializes the algorithm.
|
Modifier and Type | Method and Description |
---|---|
java.util.List<EpisodeAnalysis> |
getAllStoredLearningEpisodes() |
EpisodeAnalysis |
getLastLearningEpisode() |
int |
getLastNumSteps()
Returns the number of steps taken in the last episode;
|
protected double |
getMaxQ(HashableState s)
Returns the maximum Q-value in the hashed stated.
|
protected QValue |
getQ(HashableState s,
GroundedAction a)
Returns the Q-value for a given hashed state and action.
|
QValue |
getQ(State s,
AbstractGroundedAction a)
Returns the
QValue for the given state-action pair. |
protected java.util.List<QValue> |
getQs(HashableState s)
Returns the possible Q-values for a given hashed stated.
|
java.util.List<QValue> |
getQs(State s)
Returns a
List of QValue objects for ever permissible action for the given input state. |
protected QLearningStateNode |
getStateNode(HashableState s)
Returns the
QLearningStateNode object stored for the given hashed state. |
void |
initializeForPlanning(RewardFunction rf,
TerminalFunction tf,
int numEpisodesForPlanning)
Sets the
RewardFunction , TerminalFunction ,
and the number of simulated episodes to use for planning when
the planFromState(burlap.oomdp.core.states.State) method is called. |
GreedyQPolicy |
planFromState(State initialState)
Plans from the input state and then returns a
GreedyQPolicy that greedily
selects the action with the highest Q-value and breaks ties uniformly randomly. |
protected void |
QLInit(Domain domain,
double gamma,
HashableStateFactory hashingFactory,
ValueFunctionInitialization qInitFunction,
double learningRate,
Policy learningPolicy,
int maxEpisodeSize)
Initializes the algorithm.
|
void |
resetSolver()
This method resets all solver results so that a solver can be restarted fresh
as if had never solved the MDP.
|
EpisodeAnalysis |
runLearningEpisode(Environment env) |
EpisodeAnalysis |
runLearningEpisode(Environment env,
int maxSteps) |
void |
setLearningPolicy(Policy p)
Sets which policy this agent should use for learning.
|
void |
setLearningRateFunction(LearningRate lr)
Sets the learning rate function to use
|
void |
setMaximumEpisodesForPlanning(int n)
Sets the maximum number of episodes that will be performed when the
planFromState(State) method is called. |
void |
setMaxQChangeForPlanningTerminaiton(double m)
Sets a max change in the Q-function threshold that will cause the
planFromState(State) to stop planning
when it is achieved. |
void |
setNumEpisodesToStore(int numEps) |
void |
setQInitFunction(ValueFunctionInitialization qInit)
Sets how to initialize Q-values for previously unexperienced state-action pairs.
|
void |
toggleShouldAnnotateOptionDecomposition(boolean toggle)
Sets whether options that are decomposed into primitives will have the option that produced them and listed.
|
void |
toggleShouldDecomposeOption(boolean toggle)
Sets whether the primitive actions taken during an options will be included as steps in produced EpisodeAnalysis objects.
|
double |
value(State s)
Returns the value function evaluation of the given state.
|
addNonDomainReferencedAction, getActions, getAllGroundedActions, getDebugCode, getDomain, getGamma, getHashingFactory, getRf, getRF, getTf, getTF, setActions, setDebugCode, setDomain, setGamma, setHashingFactory, setRf, setTf, solverInit, stateHash, toggleDebugPrinting, translateAction
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
addNonDomainReferencedAction, getActions, getDebugCode, getDomain, getGamma, getHashingFactory, getRf, getRF, getTf, getTF, setActions, setDebugCode, setDomain, setGamma, setHashingFactory, setRf, setTf, solverInit, toggleDebugPrinting
protected java.util.Map<HashableState,QLearningStateNode> qIndex
protected ValueFunctionInitialization qInitFunction
protected LearningRate learningRate
protected Policy learningPolicy
protected int maxEpisodeSize
protected int eStepCounter
protected int numEpisodesForPlanning
protected double maxQChangeForPlanningTermination
protected double maxQChangeInLastEpisode
protected java.util.LinkedList<EpisodeAnalysis> episodeHistory
protected int numEpisodesToStore
protected boolean shouldDecomposeOptions
EpisodeAnalysis
objects.protected boolean shouldAnnotateOptions
EpisodeAnalysis
objects.protected int totalNumberOfSteps
public QLearning(Domain domain, double gamma, HashableStateFactory hashingFactory, double qInit, double learningRate)
planFromState(State)
method
will cause the valueFunction to use only one episode for planning; this should probably be changed to a much larger value if you plan on using this
algorithm as a planning algorithm.domain
- the domain in which to learngamma
- the discount factorhashingFactory
- the state hashing factory to use for Q-lookupsqInit
- the initial Q-value to user everywherelearningRate
- the learning ratepublic QLearning(Domain domain, double gamma, HashableStateFactory hashingFactory, double qInit, double learningRate, int maxEpisodeSize)
planFromState(State)
method
will cause the valueFunction to use only one episode for planning; this should probably be changed to a much larger value if you plan on using this
algorithm as a planning algorithm.domain
- the domain in which to learngamma
- the discount factorhashingFactory
- the state hashing factory to use for Q-lookupsqInit
- the initial Q-value to user everywherelearningRate
- the learning ratemaxEpisodeSize
- the maximum number of steps the agent will take in a learning episode for the agent stops trying.public QLearning(Domain domain, double gamma, HashableStateFactory hashingFactory, double qInit, double learningRate, Policy learningPolicy, int maxEpisodeSize)
planFromState(State)
method
will cause the valueFunction to use only one episode for planning; this should probably be changed to a much larger value if you plan on using this
algorithm as a planning algorithm.domain
- the domain in which to learngamma
- the discount factorhashingFactory
- the state hashing factory to use for Q-lookupsqInit
- the initial Q-value to user everywherelearningRate
- the learning ratelearningPolicy
- the learning policy to follow during a learning episode.maxEpisodeSize
- the maximum number of steps the agent will take in a learning episode for the agent stops trying.public QLearning(Domain domain, double gamma, HashableStateFactory hashingFactory, ValueFunctionInitialization qInit, double learningRate, Policy learningPolicy, int maxEpisodeSize)
planFromState(State)
method
will cause the valueFunction to use only one episode for planning; this should probably be changed to a much larger value if you plan on using this
algorithm as a planning algorithm.domain
- the domain in which to learngamma
- the discount factorhashingFactory
- the state hashing factory to use for Q-lookupsqInit
- a ValueFunctionInitialization
object that can be used to initialize the Q-values.learningRate
- the learning ratelearningPolicy
- the learning policy to follow during a learning episode.maxEpisodeSize
- the maximum number of steps the agent will take in a learning episode for the agent stops trying.protected void QLInit(Domain domain, double gamma, HashableStateFactory hashingFactory, ValueFunctionInitialization qInitFunction, double learningRate, Policy learningPolicy, int maxEpisodeSize)
planFromState(State)
method
will cause the valueFunction to use only one episode for planning; this should probably be changed to a much larger value if you plan on using this
algorithm as a planning algorithm.domain
- the domain in which to learngamma
- the discount factorhashingFactory
- the state hashing factory to use for Q-lookupsqInitFunction
- a ValueFunctionInitialization
object that can be used to initialize the Q-values.learningRate
- the learning ratelearningPolicy
- the learning policy to follow during a learning episode.maxEpisodeSize
- the maximum number of steps the agent will take in a learning episode for the agent stops trying.public void initializeForPlanning(RewardFunction rf, TerminalFunction tf, int numEpisodesForPlanning)
RewardFunction
, TerminalFunction
,
and the number of simulated episodes to use for planning when
the planFromState(burlap.oomdp.core.states.State)
method is called. If the
RewardFunction
and TerminalFunction
are not set, the planFromState(burlap.oomdp.core.states.State)
method will throw a runtime exception.rf
- the reward function to use for planningtf
- the terminal function to use for planningnumEpisodesForPlanning
- the number of simulated episodes to run for planning.public void setLearningRateFunction(LearningRate lr)
lr
- the learning rate function to usepublic void setQInitFunction(ValueFunctionInitialization qInit)
qInit
- a ValueFunctionInitialization
object that can be used to initialize the Q-values.public void setLearningPolicy(Policy p)
p
- the policy to use for learning.public void setMaximumEpisodesForPlanning(int n)
planFromState(State)
method is called.n
- the maximum number of episodes that will be performed when the planFromState(State)
method is called.public void setMaxQChangeForPlanningTerminaiton(double m)
planFromState(State)
to stop planning
when it is achieved.m
- the maximum allowable change in the Q-function before planning stopspublic int getLastNumSteps()
public void toggleShouldDecomposeOption(boolean toggle)
toggle
- whether to decompose options into the primitive actions taken by them or not.public void toggleShouldAnnotateOptionDecomposition(boolean toggle)
toggle
- whether to annotate the primitive actions of options with the calling option's name.public java.util.List<QValue> getQs(State s)
QFunction
List
of QValue
objects for ever permissible action for the given input state.public QValue getQ(State s, AbstractGroundedAction a)
QFunction
QValue
for the given state-action pair.protected java.util.List<QValue> getQs(HashableState s)
s
- the hashed state for which to get the Q-values.protected QValue getQ(HashableState s, GroundedAction a)
s
- the hashed statea
- the actionpublic double value(State s)
ValueFunction
value
in interface ValueFunction
s
- the state to evaluate.protected QLearningStateNode getStateNode(HashableState s)
QLearningStateNode
object stored for the given hashed state. If no QLearningStateNode
object.
is stored, then it is created and has its Q-value initialize using this objects ValueFunctionInitialization
data member.s
- the hashed state for which to get the QLearningStateNode
objectQLearningStateNode
object stored for the given hashed state. If no QLearningStateNode
object.protected double getMaxQ(HashableState s)
s
- the state for which to get he maximum Q-value;public GreedyQPolicy planFromState(State initialState)
GreedyQPolicy
that greedily
selects the action with the highest Q-value and breaks ties uniformly randomly.planFromState
in interface Planner
initialState
- the initial state of the planning problemGreedyQPolicy
.public EpisodeAnalysis runLearningEpisode(Environment env)
runLearningEpisode
in interface LearningAgent
public EpisodeAnalysis runLearningEpisode(Environment env, int maxSteps)
runLearningEpisode
in interface LearningAgent
public EpisodeAnalysis getLastLearningEpisode()
public void setNumEpisodesToStore(int numEps)
public java.util.List<EpisodeAnalysis> getAllStoredLearningEpisodes()
public void resetSolver()
MDPSolverInterface
resetSolver
in interface MDPSolverInterface
resetSolver
in class MDPSolver