burlap.behavior.singleagent.vfa.common

Class LinearFVVFA

java.lang.Object

burlap.behavior.singleagent.vfa.common.LinearFVVFA

All Implemented Interfaces:

DifferentiableStateActionValue, DifferentiableStateValue, ParametricFunction, ParametricFunction.ParametricStateActionFunction, ParametricFunction.ParametricStateFunction

public class LinearFVVFA
extends java.lang.Object
implements DifferentiableStateValue, DifferentiableStateActionValue

This class can be used to perform linear value function approximation, either for a states or state-actions (Q-values). It takes as input a StateToFeatureVectorGenerator which defines the state features on which linear function approximation is performed. In the case of Q-value function approximation, the state features are replicated for each action with all other action's associated state features set to zero, thereby allowing for unique predictions for each action.

This class can be used for either state-value functions or state-action-value functions, but only one of them. Which one is used is determined implicitly by whether the first function input is set with the evaluate(burlap.oomdp.core.states.State) method or the evaluate(burlap.oomdp.core.states.State, burlap.oomdp.core.AbstractGroundedAction) method.

Author:

James MacGlashan.

Nested Class Summary

Nested classes/interfaces inherited from interface burlap.behavior.singleagent.vfa.ParametricFunction

ParametricFunction.ParametricStateActionFunction, ParametricFunction.ParametricStateFunction

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.Map<AbstractGroundedAction,java.lang.Integer>	actionOffset A feature index offset for each action when using Q-value function approximation.
protected int	currentActionOffset
protected FunctionGradient	currentGradient
protected double[]	currentStateFeatures
protected double	currentValue
protected double	defaultWeight A default weight value for the functions weights.
protected StateToFeatureVectorGenerator	fvGen The state feature vector generator used for linear value function approximation.
protected State	lastState
protected double[]	stateActionWeights The function weights when performing Q-value function approximation.
protected double[]	stateWeights The function weights when performing state value function approximation.

Constructor Summary

Constructors

Constructor and Description
LinearFVVFA(StateToFeatureVectorGenerator fvGen, double defaultWeightValue) Initializes.

Method Summary

Methods

Modifier and Type	Method and Description
LinearFVVFA	copy() Returns a copy of this ParametricFunction.
double	evaluate(State s) Sets the input of this function to the given State and returns the value of it.
double	evaluate(State s, AbstractGroundedAction a) Sets the input of this function to the given State and AbstractGroundedAction and returns the value of it.
protected void	expandStateActionWeights(int num) Expands the state-action function weight vector by a fixed sized and initializes their value to the default weight value set for this object.
java.util.Map<AbstractGroundedAction,java.lang.Integer>	getActionOffset() Returns the Map of feature index offsets into the full feature vector for each action
int	getActionOffset(AbstractGroundedAction a)
double	getDefaultWeight()
StateToFeatureVectorGenerator	getFvGen()
double	getParameter(int i) Returns the value of the ith parameter value
FunctionGradient	gradient(State s) Returns the gradient of this function
FunctionGradient	gradient(State s, AbstractGroundedAction a) Returns the gradient of this function.
void	initializeStateActionWeightVector(int size, double v) Resets the state-action function weight array to a new array of the given sized and default value.
void	initializeStateWeightVector(int size, double v) Resets the state function weight array to a new array of the given sized and default value.
int	numParameters() Returns the number of parameters defining this function.
void	resetParameters() Resets the parameters of this function to default values.
void	setActionOffset(AbstractGroundedAction a, int offset) Sets the Map of feature index offset into the full feature vector for the given action
void	setActionOffset(java.util.Map<AbstractGroundedAction,java.lang.Integer> actionOffset) Sets the Map of feature index offsets into the full feature vector for each action
void	setParameter(int i, double p) Sets the value of the ith parameter to given value

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fvGen

protected StateToFeatureVectorGenerator fvGen

The state feature vector generator used for linear value function approximation.

actionOffset

protected java.util.Map<AbstractGroundedAction,java.lang.Integer> actionOffset

A feature index offset for each action when using Q-value function approximation.

stateWeights

protected double[] stateWeights

The function weights when performing state value function approximation.

stateActionWeights

protected double[] stateActionWeights

The function weights when performing Q-value function approximation.

defaultWeight

protected double defaultWeight

A default weight value for the functions weights.

currentStateFeatures

protected double[] currentStateFeatures

currentActionOffset

protected int currentActionOffset

currentValue

protected double currentValue

currentGradient

protected FunctionGradient currentGradient

lastState

protected State lastState

Constructor Detail

LinearFVVFA

public LinearFVVFA(StateToFeatureVectorGenerator fvGen,
           double defaultWeightValue)

Initializes. This object will be set to perform either state value function approximation or state-action function approximation once a call to either evaluate(burlap.oomdp.core.states.State) or evaluate(burlap.oomdp.core.states.State, burlap.oomdp.core.AbstractGroundedAction) is made. If the former method is called first, then this object will be tasked with state value function approximation. If the latter method is called first, then this object will be tasked with state-action value function approximation.

Parameters:

fvGen - The state feature vector generator that produces the features used for either linear state value function approximation or state-action-value function approximation.

defaultWeightValue - The default weight value of all function weights.

Method Detail

evaluate

public double evaluate(State s,
              AbstractGroundedAction a)

Description copied from interface: ParametricFunction.ParametricStateActionFunction

Sets the input of this function to the given State and AbstractGroundedAction and returns the value of it.

Specified by:

evaluate in interface ParametricFunction.ParametricStateActionFunction

Parameters:

s - the input State

a - the input action

Returns:

the value of this function evaluated on the State and AbstractGroundedAction

evaluate

public double evaluate(State s)

Description copied from interface: ParametricFunction.ParametricStateFunction

Sets the input of this function to the given State and returns the value of it.

Specified by:

evaluate in interface ParametricFunction.ParametricStateFunction

Parameters:

s - the State to input to the function

Returns:

the value of this function evaluated on the input State

gradient

public FunctionGradient gradient(State s)

Description copied from interface: DifferentiableStateValue

Returns the gradient of this function

Specified by:

gradient in interface DifferentiableStateValue

Parameters:

s - the input state

Returns:

the gradient

gradient

public FunctionGradient gradient(State s,
                        AbstractGroundedAction a)

Description copied from interface: DifferentiableStateActionValue

Returns the gradient of this function.

Specified by:

gradient in interface DifferentiableStateActionValue

Parameters:

s - the input State

a - the input AbstractGroundedAction

Returns:

the FunctionGradient of this function at the input

numParameters

public int numParameters()

Description copied from interface: ParametricFunction

Returns the number of parameters defining this function. Note that some implementations my have a dynamic number of parameters that grows or shrinks over time. Consult the documentation for the specific implementation for more information.

Specified by:

numParameters in interface ParametricFunction

Returns:

the number of parameters defining this function.

getParameter

public double getParameter(int i)

Description copied from interface: ParametricFunction

Returns the value of the ith parameter value

Specified by:

getParameter in interface ParametricFunction

Parameters:

i - the parameter index

Returns:

the double value of the ith parameter

setParameter

public void setParameter(int i,
                double p)

Description copied from interface: ParametricFunction

Sets the value of the ith parameter to given value

Specified by:

setParameter in interface ParametricFunction

Parameters:

i - the index of the parameter to set

p - the parameter value to which it should be set

resetParameters

public void resetParameters()

Description copied from interface: ParametricFunction

Resets the parameters of this function to default values.

Specified by:

resetParameters in interface ParametricFunction

getActionOffset

public int getActionOffset(AbstractGroundedAction a)

expandStateActionWeights

protected void expandStateActionWeights(int num)

Expands the state-action function weight vector by a fixed sized and initializes their value to the default weight value set for this object.

Parameters:

num - the number of function weights to add to the state-action function weight vector

getFvGen

public StateToFeatureVectorGenerator getFvGen()

getDefaultWeight

public double getDefaultWeight()

initializeStateWeightVector

public void initializeStateWeightVector(int size,
                               double v)

Resets the state function weight array to a new array of the given sized and default value.

Parameters:

size - the dimensionality of the weights

v - the default value to which the weights will be set

initializeStateActionWeightVector

public void initializeStateActionWeightVector(int size,
                                     double v)

Resets the state-action function weight array to a new array of the given sized and default value.

Parameters:

size - the dimensionality of the weights

v - the default value to which the weights will be set

getActionOffset

public java.util.Map<AbstractGroundedAction,java.lang.Integer> getActionOffset()

Returns the Map of feature index offsets into the full feature vector for each action

Returns:

the Map of feature index offsets into the full feature vector for each action

setActionOffset

public void setActionOffset(java.util.Map<AbstractGroundedAction,java.lang.Integer> actionOffset)

Sets the Map of feature index offsets into the full feature vector for each action

Parameters:

actionOffset - the Map of feature index offsets into the full feature vector for each action

setActionOffset

public void setActionOffset(AbstractGroundedAction a,
                   int offset)

Sets the Map of feature index offset into the full feature vector for the given action

Parameters:

a - the action whose feature vector index is to be set

offset - the feature index offset for the action

copy

public LinearFVVFA copy()

Description copied from interface: ParametricFunction

Returns a copy of this ParametricFunction.

Specified by:

copy in interface ParametricFunction

Returns:

a copy of this ParametricFunction.