burlap.behavior.functionapproximation.dense.fourier

Class FourierBasis

java.lang.Object

burlap.behavior.functionapproximation.dense.fourier.FourierBasis

All Implemented Interfaces:

DenseStateFeatures

public class FourierBasis
extends java.lang.Object
implements DenseStateFeatures

An implementation of Fourier Basis functions [1]. This class expects a normalized state variable/feature vector of input states, if it is not normalized, behavior is not well defined. Therefore, you may want to consider using the NormalizedVariableFeatures. The higher order the basis functions, the higher the VFA resolution is. Typically, order n will produce (n+1)^d state basis functions (and a copy for each action), where d is the number of state variables. Since this grows quickly, a way to manage the complexity is to simplify the number of coefficient vectors. That is, each basis function is a function of the dot product of the input state variable vector and a coefficient vector {0...n}^d and normally all possible coefficient vectors (and their corresponding basis functions) for order n are produced. However, this class can be told to limit the permitted coefficient vectors to those that have no more than k non-zero entries in the coefficient vector. When k = 1, all features are treated as indepdent resulting in n*d basis functions.

When using a learning algorithm like GradientDescentSarsaLam with Fourier basis functions, it is typically a good idea to use the FourierBasisLearningRateWrapper, which scales the normal learning rate by the inverse of the norm of a basis function's coefficient vector.

1. G.D. Konidaris, S. Osentoski and P.S. Thomas. Value Function Approximation in Reinforcement Learning using the Fourier Basis. In Proceedings of the Twenty-Fifth Conference on Artificial Intelligence, pages 380-385, August 2011.

Author:

James MacGlashan

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.Map<Action,java.lang.Integer>	actionFeatureMultiplier A map for returning a multiplier to the number of state features for each action.
protected java.util.List<short[]>	coefficientVectors The coefficient vectors used
protected DenseStateFeatures	inputFeatures The input features over which fourier basis functions will be generated
protected int	maxNonZeroCoefficients The maximum number of non-zero coefficient entries permitted in a coefficient vector
protected int	nextActionMultiplier The next action Fourier basis function size multiplier to use for the next newly seen action.
protected int	numStateVariables The number of state variables on which the produced basis functions operate
protected int	order The order of the Fourier basis functions.

Constructor Summary

Constructors

Constructor and Description
FourierBasis(DenseStateFeatures inputFeatures, int order) Initializes.
FourierBasis(DenseStateFeatures inputFeatures, int order, int maxNonZeroCoefficients) Initializes.

Method Summary

Modifier and Type	Method and Description
double	basisValue(double[] input, int basisFunction) Returns the basis function value for the given state input for the given basis function index.
double	coefficientNorm(int i) Returns the norm of the coefficient vector for the given basis function index.
FourierBasis	copy() Returns a copy of this DenseStateFeatures
double[]	features(State s) Returns a feature vector represented as a double array for a given input state.
protected void	generateCoefficientVectors() Generates all coefficient vectors given the number of state variables and the maximum number of non-zero coefficient element entries.
protected void	generateCoefficientVectorsHelper(int index, short[] vector, int numNonZeroEntries) Recursive cofficient generator helper method.
DenseLinearVFA	generateVFA(double defaultWeightValue) Creates and returns a linear VFA object over this Fourier basis feature database.
short[]	getCoefficientVector(int i) Returns the coefficient vector for the given basis function index.
void	setCoefficientVectors(java.util.List<short[]> coefficientVectors) Forces the set of coefficient vectors (and thereby Fourier basis functions) used.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

numStateVariables

protected int numStateVariables

The number of state variables on which the produced basis functions operate

inputFeatures

protected DenseStateFeatures inputFeatures

The input features over which fourier basis functions will be generated

coefficientVectors

protected java.util.List<short[]> coefficientVectors

The coefficient vectors used

maxNonZeroCoefficients

protected int maxNonZeroCoefficients

The maximum number of non-zero coefficient entries permitted in a coefficient vector

order

protected int order

The order of the Fourier basis functions.

actionFeatureMultiplier

protected java.util.Map<Action,java.lang.Integer> actionFeatureMultiplier

A map for returning a multiplier to the number of state features for each action. Effectively this ensures a unique feature ID for each Fourier basis function for each action.

nextActionMultiplier

protected int nextActionMultiplier

The next action Fourier basis function size multiplier to use for the next newly seen action.

Constructor Detail

FourierBasis

public FourierBasis(DenseStateFeatures inputFeatures,
                    int order)

Initializes. The coefficient vectors used by this Fourier Basis function will be generated lazily when the first features for an input state/state-action pair are queried. The maximum number of non-zero coefficient entries in a coefficient vector will be set to the maixmum (the state variable dimensionality).

Parameters:

inputFeatures - the state feature vector generator that turns OO-MDP State objects into double arrays.

order - the Fourier basis order

FourierBasis

public FourierBasis(DenseStateFeatures inputFeatures,
                    int order,
                    int maxNonZeroCoefficients)

Initializes. The coefficient vectors used by this Fourier Basis function will be generated lazily when the first features for an input state are queried. Setting maxNonZeroCoefficents to one results in treating each state variable as independent and thereby producing order*d basis functions (for each action) where d is the number of state variables. Larger values of maxNonZeroCoefficents will result in more variable dependency combinations.

Parameters:

inputFeatures - The input features over which fourier basis functions will be generated

order - the fourier basis order

maxNonZeroCoefficients - the maximum number of entries in coefficient vectors that can have non-zero values.

Method Detail

setCoefficientVectors

public void setCoefficientVectors(java.util.List<short[]> coefficientVectors)

Forces the set of coefficient vectors (and thereby Fourier basis functions) used. Use this method only if you want to fine tune the basis functions used.

Parameters:

coefficientVectors - the coefficient vectors used to produce the Fourier basis functions.

basisValue

public double basisValue(double[] input,
                         int basisFunction)

Returns the basis function value for the given state input for the given basis function index. The basisFunction index may be greater than the number of coefficient vectors because it may refer to an state-action feature's basis function (and there is a copy of each basis function for each action). The basis function used will be the one associated with the coefficient vector at index basisFunction % m, where m is the number of this object's coefficient vectors.

Parameters:

input - the state variables

basisFunction - the basis function index

Returns:

the value of the basis function for the given input state variables

features

public double[] features(State s)

Description copied from interface: DenseStateFeatures

Returns a feature vector represented as a double array for a given input state.

Specified by:

features in interface DenseStateFeatures

Parameters:

s - the input state to turn into a feature vector.

Returns:

the feature vector represented as a double array.

getCoefficientVector

public short[] getCoefficientVector(int i)

Returns the coefficient vector for the given basis function index. The basisFunction index may be greater than the number of coefficient vectors because it may refer to an state-action feature's basis function (and there is a copy of each basis function for each action). The coefficient vector returned is at the index basisFunction % m, where m is the number of this object's coefficient vectors.

Parameters:

i - the basis function index

Returns:

the coefficient vector for the given basis function

coefficientNorm

public double coefficientNorm(int i)

Returns the norm of the coefficient vector for the given basis function index. The basisFunction index may be greater than the number of coefficient vectors because it may refer to an state-action feature's basis function (and there is a copy of each basis function for each action). The coefficient vector returned is at the index basisFunction % m, where m is the number of this object's coefficient vectors.

Parameters:

i - the basis function index

Returns:

the norm of the coefficient vector for the given basis function

generateVFA

public DenseLinearVFA generateVFA(double defaultWeightValue)

Creates and returns a linear VFA object over this Fourier basis feature database.

Parameters:

defaultWeightValue - the default feature weight value to use for all features

Returns:

a linear VFA object over this Fourier basis feature database.

generateCoefficientVectors

protected void generateCoefficientVectors()

Generates all coefficient vectors given the number of state variables and the maximum number of non-zero coefficient element entries.

generateCoefficientVectorsHelper

protected void generateCoefficientVectorsHelper(int index,
                                                short[] vector,
                                                int numNonZeroEntries)

Recursive cofficient generator helper method. Once a permitted coefficient vector is fully generated, it is copied and added to this object's list of coefficient vectors.

Parameters:

index - the index into the coefficient vector that needs to have its values filled in.

vector - the coefficient vector generated thus far

numNonZeroEntries - the number of non-zero coefficient vector entires currently in the vector.

copy

public FourierBasis copy()

Description copied from interface: DenseStateFeatures

Returns a copy of this DenseStateFeatures

Specified by:

copy in interface DenseStateFeatures

Returns:

a copy of this DenseStateFeatures