burlap.behavior.singleagent

Class MDPSolver

java.lang.Object

burlap.behavior.singleagent.MDPSolver

All Implemented Interfaces:

MDPSolverInterface

Direct Known Subclasses:

ActorCritic, ARTDP, BeliefSparseSampling, DeterministicPlanner, DifferentiableSparseSampling, DynamicProgramming, FittedVI, GradientDescentSarsaLam, LSPI, PotentialShapedRMax, QLearning, QLTutorial, QMDP, SparseSampling, UCT, VITutorial

public abstract class MDPSolver
extends java.lang.Object
implements MDPSolverInterface

The abstract super class to use for various MDP solving algorithms, including both planning and learning algorithms. It implements the MDPSolverInterface and provides the common data members and method implementations that most all algorithms will need to use and provides methods for manipulating them that are common.

Author:

James MacGlashan

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.List<Action>	actions The list of actions this solver can use.
protected int	debugCode The debug code use for calls to DPrint
protected Domain	domain The domain to solve
protected double	gamma The MDP discount factor
protected HashableStateFactory	hashingFactory The hashing factory to use for hashing states in tabular solvers
protected java.util.Map<HashableState,HashableState>	mapToStateIndex A mapping to internal stored hashed states (HashableState) that are stored.
protected RewardFunction	rf The task reward function
protected TerminalFunction	tf The terminal function for identifying terminal states

Constructor Summary

Constructors

Constructor and Description
MDPSolver()

Method Summary

Methods

Modifier and Type	Method and Description
void	addNonDomainReferencedAction(Action a) Adds an additional action the solver that is not included in the domain definition.
java.util.List<Action>	getActions() Returns a copy of all actions this solver uses for reasoning; including added actions that are not part of the domain specification (e.g., Options).
protected java.util.List<GroundedAction>	getAllGroundedActions(State s) Returns all grounded actions in the provided state for all the actions that this valueFunction can use.
int	getDebugCode() Returns the debug code used by this solver for calls to DPrint
Domain	getDomain() Returns the Domain this solver solves.
double	getGamma() Returns gamma, the discount factor used by this solver
HashableStateFactory	getHashingFactory() Returns the HashableStateFactory this solver uses.
RewardFunction	getRf() Returns the RewardFunction this solver uses.
RewardFunction	getRF() Returns the RewardFunction this solver uses.
TerminalFunction	getTf() Returns the TerminalFunction this solver uses.
TerminalFunction	getTF() Returns the TerminalFunction this solver uses.
abstract void	resetSolver() This method resets all solver results so that a solver can be restarted fresh as if had never solved the MDP.
void	setActions(java.util.List<Action> actions) Sets the action set the solver should use.
void	setDebugCode(int code) Sets the debug code to be used by calls to DPrint
void	setDomain(Domain domain) Sets the domain of this solver.
void	setGamma(double gamma) Sets gamma, the discount factor used by this solver
void	setHashingFactory(HashableStateFactory hashingFactory) Sets the HashableStateFactory used to hash states for tabular solvers.
void	setRf(RewardFunction rf) Sets the reward function used by this solver
void	setTf(TerminalFunction tf) Sets the terminal state function used by this solver
void	solverInit(Domain domain, RewardFunction rf, TerminalFunction tf, double gamma, HashableStateFactory hashingFactory) Initializes the solver with the common elements.
HashableState	stateHash(State s) A shorthand method for hashing a state.
void	toggleDebugPrinting(boolean toggle) Toggles whether the solver's calls to DPrint should be printed.
protected GroundedAction	translateAction(GroundedAction a, java.util.Map<java.lang.String,java.lang.String> matching) Takes a source GroundedAction and a matching between object instances of two different states and returns a GroundedAction with parameters using the matched parameters if the GroundedAction is an instance of AbstractObjectParameterizedGroundedAction.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

domain

protected Domain domain

The domain to solve

hashingFactory

protected HashableStateFactory hashingFactory

The hashing factory to use for hashing states in tabular solvers

rf

protected RewardFunction rf

The task reward function

tf

protected TerminalFunction tf

The terminal function for identifying terminal states

gamma

protected double gamma

The MDP discount factor

actions

protected java.util.List<Action> actions

The list of actions this solver can use. May include non-domain specified actions like Options.

mapToStateIndex

protected java.util.Map<HashableState,HashableState> mapToStateIndex

A mapping to internal stored hashed states (HashableState) that are stored. Useful since two identical states may have different object instance name identifiers that can affect the parameters in GroundedActions.

debugCode

protected int debugCode

The debug code use for calls to DPrint

Constructor Detail

MDPSolver

public MDPSolver()

Method Detail

resetSolver

public abstract void resetSolver()

Description copied from interface: MDPSolverInterface

This method resets all solver results so that a solver can be restarted fresh as if had never solved the MDP.

Specified by:

resetSolver in interface MDPSolverInterface

solverInit

public void solverInit(Domain domain,
              RewardFunction rf,
              TerminalFunction tf,
              double gamma,
              HashableStateFactory hashingFactory)

Description copied from interface: MDPSolverInterface

Initializes the solver with the common elements.

Specified by:

solverInit in interface MDPSolverInterface

Parameters:

domain - the domain to be solved.

rf - the reward function

tf - the terminal state function

gamma - the MDP discount factor

hashingFactory - the hashing factory used to store states (may be set to null if the solver is not tabular)

addNonDomainReferencedAction

public void addNonDomainReferencedAction(Action a)

Description copied from interface: MDPSolverInterface

Adds an additional action the solver that is not included in the domain definition. For instance, an Option should be added using this method.

Specified by:

addNonDomainReferencedAction in interface MDPSolverInterface

Parameters:

a - the action to add to the solver

setActions

public void setActions(java.util.List<Action> actions)

Description copied from interface: MDPSolverInterface

Sets the action set the solver should use.

Specified by:

setActions in interface MDPSolverInterface

Parameters:

actions - the actions the solver should use.

getActions

public java.util.List<Action> getActions()

Description copied from interface: MDPSolverInterface

Returns a copy of all actions this solver uses for reasoning; including added actions that are not part of the domain specification (e.g., Options). Modifying the returned list will not modify the action list this solver uses.

Specified by:

getActions in interface MDPSolverInterface

Returns:

a List of all actions this solver uses.

getTF

public TerminalFunction getTF()

Description copied from interface: MDPSolverInterface

Returns the TerminalFunction this solver uses.

Specified by:

getTF in interface MDPSolverInterface

Returns:

the TerminalFunction this solver uses.

getRF

public RewardFunction getRF()

Description copied from interface: MDPSolverInterface

Returns the RewardFunction this solver uses.

Specified by:

getRF in interface MDPSolverInterface

Returns:

the RewardFunction this solver uses.

setHashingFactory

public void setHashingFactory(HashableStateFactory hashingFactory)

Description copied from interface: MDPSolverInterface

Sets the HashableStateFactory used to hash states for tabular solvers.

Specified by:

setHashingFactory in interface MDPSolverInterface

Parameters:

hashingFactory - the HashableStateFactory used to hash states for tabular solvers.

getHashingFactory

public HashableStateFactory getHashingFactory()

Description copied from interface: MDPSolverInterface

Returns the HashableStateFactory this solver uses.

Specified by:

getHashingFactory in interface MDPSolverInterface

Returns:

the HashableStateFactory this solver uses.

setRf

public void setRf(RewardFunction rf)

Description copied from interface: MDPSolverInterface

Sets the reward function used by this solver

Specified by:

setRf in interface MDPSolverInterface

Parameters:

rf - the reward function to be used by this solver

setTf

public void setTf(TerminalFunction tf)

Description copied from interface: MDPSolverInterface

Sets the terminal state function used by this solver

Specified by:

setTf in interface MDPSolverInterface

Parameters:

tf - the terminal function to be used by this solver

getGamma

public double getGamma()

Description copied from interface: MDPSolverInterface

Returns gamma, the discount factor used by this solver

Specified by:

getGamma in interface MDPSolverInterface

Returns:

gamma, the discount factor used by this solver

setGamma

public void setGamma(double gamma)

Description copied from interface: MDPSolverInterface

Sets gamma, the discount factor used by this solver

Specified by:

setGamma in interface MDPSolverInterface

Parameters:

gamma - the discount factor used by this solver

setDebugCode

public void setDebugCode(int code)

Description copied from interface: MDPSolverInterface

Sets the debug code to be used by calls to DPrint

Specified by:

setDebugCode in interface MDPSolverInterface

Parameters:

code - the code to be used by DPrint

getDebugCode

public int getDebugCode()

Description copied from interface: MDPSolverInterface

Returns the debug code used by this solver for calls to DPrint

Specified by:

getDebugCode in interface MDPSolverInterface

Returns:

the debug code used by this solver for calls to DPrint

toggleDebugPrinting

public void toggleDebugPrinting(boolean toggle)

Description copied from interface: MDPSolverInterface

Toggles whether the solver's calls to DPrint should be printed.

Specified by:

toggleDebugPrinting in interface MDPSolverInterface

Parameters:

toggle - whether to print the calls to DPrint

setDomain

public void setDomain(Domain domain)

Description copied from interface: MDPSolverInterface

Sets the domain of this solver. NOTE: this will also reset the actions this solver uses to the actions of the provided domain. If you have previously added non-domain referenced actions through the MDPSolverInterface.addNonDomainReferencedAction(burlap.oomdp.singleagent.Action) method, you will have to do so again.

Specified by:

setDomain in interface MDPSolverInterface

Parameters:

domain - the domain this solver should use.

getDomain

public Domain getDomain()

Description copied from interface: MDPSolverInterface

Returns the Domain this solver solves.

Specified by:

getDomain in interface MDPSolverInterface

Returns:

the Domain this solver solves.

getRf

public RewardFunction getRf()

Description copied from interface: MDPSolverInterface

Returns the RewardFunction this solver uses.

Specified by:

getRf in interface MDPSolverInterface

Returns:

the RewardFunction this solver uses.

getTf

public TerminalFunction getTf()

Description copied from interface: MDPSolverInterface

Returns the TerminalFunction this solver uses.

Specified by:

getTf in interface MDPSolverInterface

Returns:

the TerminalFunction this solver uses.

translateAction

protected GroundedAction translateAction(GroundedAction a,
                             java.util.Map<java.lang.String,java.lang.String> matching)

Takes a source GroundedAction and a matching between object instances of two different states and returns a GroundedAction with parameters using the matched parameters if the GroundedAction is an instance of AbstractObjectParameterizedGroundedAction. This method is useful a stored state and action pair in the valueFunction data structure has different object name identifiers than a query state that is otherwise identical. The matching is from the state in which the source action is applied to some target state that is not provided to this method.

Parameters:

a - the source action that needs to be translated

matching - a map from object instance names to other object instance names.

Returns:

and new GroundedAction with object parametrization that follow from the matching

stateHash

public HashableState stateHash(State s)

A shorthand method for hashing a state.

Parameters:

s - the state to hash

Returns:

a StateHashTuple produce from this planners StateHashFactory.

getAllGroundedActions

protected java.util.List<GroundedAction> getAllGroundedActions(State s)

Returns all grounded actions in the provided state for all the actions that this valueFunction can use.

Parameters:

s - the source state for which to get all GroundedActions.

Returns:

all GroundedActions.