burlap.behavior.singleagent.vfa.fourier

## Class FourierBasis

• java.lang.Object
• burlap.behavior.singleagent.vfa.fourier.FourierBasis
• All Implemented Interfaces:
FeatureDatabase

```public class FourierBasis
extends java.lang.Object
implements FeatureDatabase```
An implementation of Fourier Basis functions [1]. This class expects a normalized state variable/feature vector of input states, if it is not normalized, behavior is not well defined. Therefore consider using the `ConcatenatedObjectFeatureVectorGenerator` generator with the normalization flag set to convert the OO-MDP `State` objects into the necessary input vector. The higher order the basis functions, the higher the VFA resolution is. Typically, order n will produce (n+1)^d state basis functions (and a copy for each action), where d is the number of state variables. Since this grows quickly, a way to manage the complexity is to simplify the number of coefficient vectors. That is, each basis function is a function of the dot product of the input state variable vector and a coefficient vector {0...n}^d and normally all possible coefficient vectors (and their corresponding basis functions) for order n are produced. However, this class can be told to limit the permitted coefficient vectors to those that have no more than k non-zero entries in the coefficient vector. When k = 1, all features are treated as indepdent resulting in n*d basis functions.

When using a learning algorithm like `GradientDescentSarsaLam` with Fourier basis functions, it is typically a good idea to use the `FourierBasisLearningRateWrapper`, which scales the normal learning rate by the inverse of the norm of a basis function's coefficient vector.

1. G.D. Konidaris, S. Osentoski and P.S. Thomas. Value Function Approximation in Reinforcement Learning using the Fourier Basis. In Proceedings of the Twenty-Fifth Conference on Artificial Intelligence, pages 380-385, August 2011.

Author:
James MacGlashan
• ### Field Summary

Fields
Modifier and Type Field and Description
`protected java.util.Map<java.lang.String,java.lang.Integer>` `actionFeatureMultiplier`
A map for returning a multiplier to the number of state features for each action.
`protected java.util.List<short[]>` `coefficientVectors`
The coefficient vectors used
`protected StateToFeatureVectorGenerator` `featureVectorGenerator`
The OO-MDP `State` to feature vector/variable generator.
`protected int` `maxNonZeroCoefficents`
The maximum number of non-zero coefficient entries permitted in a coefficient vector
`protected int` `nextActionMultiplier`
The next action Fourier basis function size multiplier to use for the next newly seen action.
`protected int` `numStateVariables`
The number of state varibles on which the produced basis functions operate
`protected int` `order`
The order of the Fourier basis functions.
• ### Constructor Summary

Constructors
Constructor and Description
```FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator, int order)```
Initializes.
```FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator, int order, int maxNonZeroCoefficents)```
Initializes.
• ### Method Summary

Methods
Modifier and Type Method and Description
`double` ```basisValue(double[] input, int basisFunction)```
Returns the basis function value for the given state input for the given basis function index.
`double` `coefficientNorm(int i)`
Returns the norm of the coefficient vector for the given basis function index.
`void` `freezeDatabaseState(boolean toggle)`
Enable or disable the database from generating new features.
`protected void` `generateCoefficientVectors()`
Generates all coefficient vectors given the number of state variables and the maximum number of non-zero coefficient element entries.
`protected void` ```generateCoefficientVectorsHelper(int index, short[] vector, int numNonZeroEntries)```
Recursive cofficient generator helper method.
`ValueFunctionApproximation` `generateVFA(double defaultWeightValue)`
Creates and returns a linear VFA object over this Fourier basis feature database.
`java.util.List<ActionFeaturesQuery>` ```getActionFeaturesSets(State s, java.util.List<GroundedAction> actions)```
Returns non-zero action features for all of provided actions in state s.
`protected int` `getActionMultiplier(GroundedAction ga)`
This method returns the action multiplier for the specified grounded action.
`short[]` `getCoefficientVector(int i)`
Returns the coefficient vector for the given basis function index.
`java.util.List<StateFeature>` `getStateFeatures(State s)`
Returns non-zero state features for a given state.
`int` `numberOfFeatures()`
Returns the number of features this database tracks.
`void` `setCoefficientVectors(java.util.List<short[]> coefficientVectors)`
Forces the set of coefficient vectors (and thereby Fourier basis functions) used.
• ### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`
• ### Field Detail

• #### numStateVariables

`protected int numStateVariables`
The number of state varibles on which the produced basis functions operate
• #### featureVectorGenerator

`protected StateToFeatureVectorGenerator featureVectorGenerator`
The OO-MDP `State` to feature vector/variable generator. Should produced normalized values.
• #### coefficientVectors

`protected java.util.List<short[]> coefficientVectors`
The coefficient vectors used
• #### maxNonZeroCoefficents

`protected int maxNonZeroCoefficents`
The maximum number of non-zero coefficient entries permitted in a coefficient vector
• #### order

`protected int order`
The order of the Fourier basis functions.
• #### actionFeatureMultiplier

`protected java.util.Map<java.lang.String,java.lang.Integer> actionFeatureMultiplier`
A map for returning a multiplier to the number of state features for each action. Effectively this ensures a unique feature ID for each Fourier basis function for each action.
• #### nextActionMultiplier

`protected int nextActionMultiplier`
The next action Fourier basis function size multiplier to use for the next newly seen action.
• ### Constructor Detail

• #### FourierBasis

```public FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator,
int order)```
Initializes. The coefficient vectors used by this Fourier Basis function will be generated lazily when the first features for an input state/state-action pair are queried. The maximum number of non-zero coefficient entries in a coefficient vector will be set to the maixmum (the state variable dimensionality).
Parameters:
`featureVectorGenerator` - the state feature vector generator that turns OO-MDP `State` objects into double arrays.
`order` - the Fourier basis order
• #### FourierBasis

```public FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator,
int order,
int maxNonZeroCoefficents)```
Initializes. The coefficient vectors used by this Fourier Basis function will be generated lazily when the first features for an input state/state-action pair are queried. Setting maxNonZeroCoefficents to one results in treating each state variable as indepdent and thereby producing order*d basis functions (for each action) where d is the number of state variables. Larger values of maxNonZeroCoefficents will result in more variable dependency conbinations.
Parameters:
`featureVectorGenerator` - the state feature vector generator that turns OO-MDP `State` objects into double arrays
`order` - the fourier basis order
`maxNonZeroCoefficents` - the maximum number of entries in coeffient vectors that can have non-zero values.
• ### Method Detail

• #### setCoefficientVectors

`public void setCoefficientVectors(java.util.List<short[]> coefficientVectors)`
Forces the set of coefficient vectors (and thereby Fourier basis functions) used. Use this method only if you want to fine tune the basis functions used.
Parameters:
`coefficientVectors` - the coefficient vectors used to produce the Fourier basis functions.
• #### basisValue

```public double basisValue(double[] input,
int basisFunction)```
Returns the basis function value for the given state input for the given basis function index. The basisFunction index may be greater than the number of coefficient vectors because it may refer to an state-action feature's basis function (and there is a copy of each basis function for each action). The basis function used will be the one associated with the coefficient vector at index basisFunction % m, where m is the number of this object's coefficient vectors.
Parameters:
`input` - the state variables
`basisFunction` - the basis function index
Returns:
the value of the basis function for the given input state variables
• #### getStateFeatures

`public java.util.List<StateFeature> getStateFeatures(State s)`
Description copied from interface: `FeatureDatabase`
Returns non-zero state features for a given state. This method should be implemented if it is to be used with algorithms that compute V-values from state features or other values that are independent of the actions.
Specified by:
`getStateFeatures` in interface `FeatureDatabase`
Parameters:
`s` - the state for which features should be returned
Returns:
the features for state s
• #### getActionFeaturesSets

```public java.util.List<ActionFeaturesQuery> getActionFeaturesSets(State s,
java.util.List<GroundedAction> actions)```
Description copied from interface: `FeatureDatabase`
Returns non-zero action features for all of provided actions in state s. This method should be implemented if it is to be used with algorithms that compute Q-values from action features or other values are the dependent on the actions. Note that features for different actions should return different feature ids.
Specified by:
`getActionFeaturesSets` in interface `FeatureDatabase`
Parameters:
`s` - the state for which features should be returned
`actions` - the action set for which the action features should be returned
Returns:
returns the set set of action features for each action in actions
• #### freezeDatabaseState

`public void freezeDatabaseState(boolean toggle)`
Description copied from interface: `FeatureDatabase`
Enable or disable the database from generating new features. This method only needs to do anything if it creates features in an online fashion. That is, if the set of possible state features is defined at construction, then this method does not need to do anything. If features are generated as needed, such as with an instance-based feature set, then when the database is told to be frozen no new instances should be created.
Specified by:
`freezeDatabaseState` in interface `FeatureDatabase`
• #### numberOfFeatures

`public int numberOfFeatures()`
Description copied from interface: `FeatureDatabase`
Returns the number of features this database tracks.
Specified by:
`numberOfFeatures` in interface `FeatureDatabase`
Returns:
the number of features this database tracks.
• #### getCoefficientVector

`public short[] getCoefficientVector(int i)`
Returns the coefficient vector for the given basis function index. The basisFunction index may be greater than the number of coefficient vectors because it may refer to an state-action feature's basis function (and there is a copy of each basis function for each action). The coefficient vector returned is at the index basisFunction % m, where m is the number of this object's coefficient vectors.
Parameters:
`i` - the basis function index
Returns:
the coefficient vector for the given basis function
• #### coefficientNorm

`public double coefficientNorm(int i)`
Returns the norm of the coefficient vector for the given basis function index. The basisFunction index may be greater than the number of coefficient vectors because it may refer to an state-action feature's basis function (and there is a copy of each basis function for each action). The coefficient vector returned is at the index basisFunction % m, where m is the number of this object's coefficient vectors.
Parameters:
`i` - the basis function index
Returns:
the norm of the coefficient vector for the given basis function
• #### generateVFA

`public ValueFunctionApproximation generateVFA(double defaultWeightValue)`
Creates and returns a linear VFA object over this Fourier basis feature database.
Parameters:
`defaultWeightValue` - the default feature weight value to use for all features
Returns:
a linear VFA object over this Fourier basis feature database.
• #### generateCoefficientVectors

`protected void generateCoefficientVectors()`
Generates all coefficient vectors given the number of state variables and the maximum number of non-zero coefficient element entries.
• #### generateCoefficientVectorsHelper

```protected void generateCoefficientVectorsHelper(int index,
short[] vector,
int numNonZeroEntries)```
Recursive cofficient generator helper method. Once a permitted coefficient vector is fully generated, it is copied and added to this object's list of coefficient vectors.
Parameters:
`index` - the index into the coefficient vector that needs to have its values filled in.
`vector` - the coefficient vector generated thus far
`numNonZeroEntries` - the number of non-zero coefficient vector entires currently in the vector.
• #### getActionMultiplier

`protected int getActionMultiplier(GroundedAction ga)`
This method returns the action multiplier for the specified grounded action. If the action is not stored, a new action multiplier will created, stored, and returned. If the action is parameterized a runtime exception is thrown.
Parameters:
`ga` - the grounded action for which the multiplier will be returned
Returns:
the action multiplier to be applied to a state feature id.