burlap.behavior.singleagent.learning.modellearning

Class Model

java.lang.Object

burlap.behavior.singleagent.learning.modellearning.Model

Direct Known Subclasses:

TabularModel

public abstract class Model
extends java.lang.Object

This abstract class is used by model learning algorithms to learn the transition dynamics, reward function, and terminal function, of the world through experience. The interface is used to return the modeled reward function, terminal function, and transition dynamics (or a sample from the transition dynamics) which can be used with any planning algorithm to generate a policy for how to act in the real world. It also specifies transitionIsModeled(State, GroundedAction) method, which can be used to determine when the model is confident in its predictions from a given input state and action. This is useful for RMax algorithms which treat "unknown" states-action (or under sampled state-actions) specially.

Author:

James MacGlashan

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.Random	rand Random number generator

Constructor Summary

Constructors

Constructor and Description
Model()

Method Summary

Methods

Modifier and Type	Method and Description
abstract RewardFunction	getModelRF() Returns the learned reward function in this model
abstract TerminalFunction	getModelTF() Returns the learned terminal function in this model
abstract java.util.List<TransitionProbability>	getTransitionProbabilities(State s, GroundedAction ga) Returns this model's transition probabilities for the given source state and action
abstract java.util.List<AbstractGroundedAction>	getUnmodeledActionsForState(State s) Returns a list specifying the actions for which the transition dynamics are not yet "known."
abstract void	resetModel() Resets the model data so that learning can begin anew.
State	sampleModel(State s, GroundedAction ga) A method to sample this model's transition dynamics for the given state and action.
abstract State	sampleModelHelper(State s, GroundedAction ga) A helper method to sample this model's transition dynamics for the given state and action.
protected State	sampleTransitionFromTransitionProbabilities(State s, GroundedAction ga) Will return a sampled outcome state by calling the getTransitionProbabilities(State, GroundedAction) method and randomly drawing a state according to its distribution
abstract boolean	stateTransitionsAreModeled(State s) Indicates whether this model "knows" the transition dynamics from the given input state for all applicable actions.
abstract boolean	transitionIsModeled(State s, GroundedAction ga) Indicates whether this model "knows" how the transition dynamics from the given input state and action work.
abstract void	updateModel(State s, GroundedAction ga, State sprime, double r, boolean sprimeIsTerminal) Causes this model to be updated with a new interaction with the world.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

rand

protected java.util.Random rand

Random number generator

Constructor Detail

Model

public Model()

Method Detail

getModelRF

public abstract RewardFunction getModelRF()

Returns the learned reward function in this model

Returns:

the learned reward function in this model

getModelTF

public abstract TerminalFunction getModelTF()

Returns the learned terminal function in this model

Returns:

the learned terminal function in this model

transitionIsModeled

public abstract boolean transitionIsModeled(State s,
                          GroundedAction ga)

Indicates whether this model "knows" how the transition dynamics from the given input state and action work.

Parameters:

s - the state that is checked

ga - the action to take in state s

Returns:

true if the transition dynamics from the input state and action are "known;" false otherwise.

stateTransitionsAreModeled

public abstract boolean stateTransitionsAreModeled(State s)

Indicates whether this model "knows" the transition dynamics from the given input state for all applicable actions.

Parameters:

s - the state that is checked.

Returns:

true if the transition dynamics for all actions are "known;" false otherwise.

getUnmodeledActionsForState

public abstract java.util.List<AbstractGroundedAction> getUnmodeledActionsForState(State s)

Returns a list specifying the actions for which the transition dynamics are not yet "known."

Parameters:

s - the state for which the unmodeled actions should be returned.

Returns:

a List of AbstractGroundedAction objects

sampleModel

public final State sampleModel(State s,
                GroundedAction ga)

A method to sample this model's transition dynamics for the given state and action.

Parameters:

s - The source state

ga - the action taken in the source state

Returns:

a new State object instance, which is a sampled result of taking the given action in the given state.

sampleModelHelper

public abstract State sampleModelHelper(State s,
                      GroundedAction ga)

A helper method to sample this model's transition dynamics for the given state and action.

Parameters:

s - a copied source state which can be directly modified.

ga - the action taken in the source state

Returns:

a State which is a sampled result of taking the given action in the given state. This may be the same object as State s.

getTransitionProbabilities

public abstract java.util.List<TransitionProbability> getTransitionProbabilities(State s,
                                                               GroundedAction ga)

Returns this model's transition probabilities for the given source state and action

Parameters:

s - the source state

ga - an action taken in the source state

Returns:

the list of the possible transition probabilities; should sum to 1 and states will probability zero do *not* have to be included.

updateModel

public abstract void updateModel(State s,
               GroundedAction ga,
               State sprime,
               double r,
               boolean sprimeIsTerminal)

Causes this model to be updated with a new interaction with the world.

Parameters:

s - a source state

ga - an action that was taken in the source state

sprime - the state to which the agent transitioned for taking action

r - a reward that was received for taking the given action in the given source state and transitioning to the given next state

sprimeIsTerminal - whether the next state is a terminal state or not

resetModel

public abstract void resetModel()

Resets the model data so that learning can begin anew.

sampleTransitionFromTransitionProbabilities

protected State sampleTransitionFromTransitionProbabilities(State s,
                                                GroundedAction ga)

Will return a sampled outcome state by calling the getTransitionProbabilities(State, GroundedAction) method and randomly drawing a state according to its distribution

Parameters:

s - a source state

ga - the action taken in the source state

Returns:

a sampled outcome state