burlap.behavior.singleagent.learnbydemo.mlirl.commonrfs

Class LinearStateDifferentiableRF

java.lang.Object

burlap.behavior.singleagent.learnbydemo.mlirl.support.DifferentiableRF

burlap.behavior.singleagent.learnbydemo.mlirl.commonrfs.LinearStateDifferentiableRF

All Implemented Interfaces:

RewardFunction

public class LinearStateDifferentiableRF
extends DifferentiableRF

A class for defining a linear state DifferentiableRF. The features of the reward function are produced by a StateToFeatureVectorGenerator. By default, the reward function is defined as: R(s, a, s') = w * f(s'), where w is the weight vector (the parameters) of this object, * is the dot product operator, and f(s') is the feature vector for state s'. Alternatively, the reward function may be defined R(s, a, s') = w * f(s), (that is, using the feature vector for the previous state) by using the LinearStateDifferentiableRF(burlap.behavior.singleagent.vfa.StateToFeatureVectorGenerator, int, boolean) constructor or the setFeaturesAreForNextState(boolean)} method and setting the featuresAreForNextState boolean to false.

Author:

James MacGlashan.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	featuresAreForNextState Whether features are based on the next state or previous state.
protected StateToFeatureVectorGenerator	fvGen The state feature vector generator.

Fields inherited from class burlap.behavior.singleagent.learnbydemo.mlirl.support.DifferentiableRF

dim, parameters

Constructor Summary

Constructors

Constructor and Description
LinearStateDifferentiableRF(StateToFeatureVectorGenerator fvGen, int dim) Initializes.
LinearStateDifferentiableRF(StateToFeatureVectorGenerator fvGen, int dim, boolean featuresAreForNextState) Initializes.

Method Summary

Methods

Modifier and Type	Method and Description
protected DifferentiableRF	copyHelper() A helper method for making a copy of this reward function.
double[]	getGradient(State s, GroundedAction ga, State sp) Returns the gradient of the reward function for the given state transition.
double	reward(State s, GroundedAction a, State sprime) Returns the reward received when action a is executed in state s and the agent transitions to state sprime.
void	setFeaturesAreForNextState(boolean featuresAreForNextState) Sets whether features for the reward function are generated from the next state or previous state.

Methods inherited from class burlap.behavior.singleagent.learnbydemo.mlirl.support.DifferentiableRF

copy, getParameterDimension, getParameters, randomizeParameters, setParameter, setParameters, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

featuresAreForNextState

protected boolean featuresAreForNextState

Whether features are based on the next state or previous state. Default is for the next state (true).

fvGen

protected StateToFeatureVectorGenerator fvGen

The state feature vector generator.

Constructor Detail

LinearStateDifferentiableRF

public LinearStateDifferentiableRF(StateToFeatureVectorGenerator fvGen,
                           int dim)

Initializes. The reward function will use the features for the next state.

Parameters:

fvGen - the state feature vector generator

dim - the dimensionality of the state features that will be produced

LinearStateDifferentiableRF

public LinearStateDifferentiableRF(StateToFeatureVectorGenerator fvGen,
                           int dim,
                           boolean featuresAreForNextState)

Initializes.

Parameters:

fvGen - the state feature vector generator

dim - the dimensionality of the state features that will be produced

featuresAreForNextState - If true, then the features will be generated from the next state in the (s, a, s') transition. If false, then the previous state.

Method Detail

setFeaturesAreForNextState

public void setFeaturesAreForNextState(boolean featuresAreForNextState)

Sets whether features for the reward function are generated from the next state or previous state.

Parameters:

featuresAreForNextState - If true, then the features will be generated from the next state in the (s, a, s') transition. If false, then the previous state.

copyHelper

protected DifferentiableRF copyHelper()

Description copied from class: DifferentiableRF

A helper method for making a copy of this reward function. THe parameters and dimensionality do not have to be copied, because they will be copied in the public DifferentiableRF.copy() method.

Specified by:

copyHelper in class DifferentiableRF

Returns:

a copy of this reward function.

getGradient

public double[] getGradient(State s,
                   GroundedAction ga,
                   State sp)

Description copied from class: DifferentiableRF

Returns the gradient of the reward function for the given state transition.

Specified by:

getGradient in class DifferentiableRF

Parameters:

s - the source state

ga - the action taken in the source state

sp - the resulting state from the action

Returns:

the gradient of the reward function for the given transition.

reward

public double reward(State s,
            GroundedAction a,
            State sprime)

Description copied from interface: RewardFunction

Returns the reward received when action a is executed in state s and the agent transitions to state sprime.

Parameters:

s - the state in which the action was executed

a - the action executed

sprime - the state to which the agent transitioned

Returns:

the reward received when action a is executed in state s and the agent transitions to state sprime.