public class FourierBasis extends java.lang.Object implements DenseStateFeatures
NormalizedVariableFeatures
.
The higher order the basis functions, the higher the VFA resolution is. Typically, order n will produce (n+1)^d state basis functions (and a copy for each action), where d is the number of state variables. Since this grows quickly,
a way to manage the complexity is to simplify the number of coefficient vectors. That is, each basis function is a function of the dot product of the input state variable vector and a coefficient vector {0...n}^d
and normally all possible coefficient vectors (and their corresponding basis functions) for
order n are produced. However, this class can be told to limit the permitted coefficient vectors to those that have no more than k nonzero entries in the coefficient vector. When k = 1, all features are treated as indepdent resulting
in n*d basis functions.
When using a learning algorithm like GradientDescentSarsaLam
with Fourier basis functions, it is typically a good idea to use the FourierBasisLearningRateWrapper
, which scales the normal learning rate by the inverse of the norm
of a basis function's coefficient vector.
1. G.D. Konidaris, S. Osentoski and P.S. Thomas. Value Function Approximation in Reinforcement Learning using the Fourier Basis. In Proceedings of the TwentyFifth Conference on Artificial Intelligence, pages 380385, August 2011.
Modifier and Type  Field and Description 

protected java.util.Map<Action,java.lang.Integer> 
actionFeatureMultiplier
A map for returning a multiplier to the number of state features for each action.

protected java.util.List<short[]> 
coefficientVectors
The coefficient vectors used

protected DenseStateFeatures 
inputFeatures
The input features over which fourier basis functions will be generated

protected int 
maxNonZeroCoefficients
The maximum number of nonzero coefficient entries permitted in a coefficient vector

protected int 
nextActionMultiplier
The next action Fourier basis function size multiplier to use for the next newly seen action.

protected int 
numStateVariables
The number of state variables on which the produced basis functions operate

protected int 
order
The order of the Fourier basis functions.

Constructor and Description 

FourierBasis(DenseStateFeatures inputFeatures,
int order)
Initializes.

FourierBasis(DenseStateFeatures inputFeatures,
int order,
int maxNonZeroCoefficients)
Initializes.

Modifier and Type  Method and Description 

double 
basisValue(double[] input,
int basisFunction)
Returns the basis function value for the given state input for the given basis function index.

double 
coefficientNorm(int i)
Returns the norm of the coefficient vector for the given basis function index.

FourierBasis 
copy()
Returns a copy of this
DenseStateFeatures 
double[] 
features(State s)
Returns a feature vector represented as a double array for a given input state.

protected void 
generateCoefficientVectors()
Generates all coefficient vectors given the number of state variables and the maximum number of nonzero coefficient element entries.

protected void 
generateCoefficientVectorsHelper(int index,
short[] vector,
int numNonZeroEntries)
Recursive cofficient generator helper method.

DenseLinearVFA 
generateVFA(double defaultWeightValue)
Creates and returns a linear VFA object over this Fourier basis feature database.

short[] 
getCoefficientVector(int i)
Returns the coefficient vector for the given basis function index.

void 
setCoefficientVectors(java.util.List<short[]> coefficientVectors)
Forces the set of coefficient vectors (and thereby Fourier basis functions) used.

protected int numStateVariables
protected DenseStateFeatures inputFeatures
protected java.util.List<short[]> coefficientVectors
protected int maxNonZeroCoefficients
protected int order
protected java.util.Map<Action,java.lang.Integer> actionFeatureMultiplier
protected int nextActionMultiplier
public FourierBasis(DenseStateFeatures inputFeatures, int order)
inputFeatures
 the state feature vector generator that turns OOMDP State
objects into double arrays.order
 the Fourier basis orderpublic FourierBasis(DenseStateFeatures inputFeatures, int order, int maxNonZeroCoefficients)
inputFeatures
 The input features over which fourier basis functions will be generatedorder
 the fourier basis ordermaxNonZeroCoefficients
 the maximum number of entries in coefficient vectors that can have nonzero values.public void setCoefficientVectors(java.util.List<short[]> coefficientVectors)
coefficientVectors
 the coefficient vectors used to produce the Fourier basis functions.public double basisValue(double[] input, int basisFunction)
input
 the state variablesbasisFunction
 the basis function indexpublic double[] features(State s)
DenseStateFeatures
features
in interface DenseStateFeatures
s
 the input state to turn into a feature vector.public short[] getCoefficientVector(int i)
i
 the basis function indexpublic double coefficientNorm(int i)
i
 the basis function indexpublic DenseLinearVFA generateVFA(double defaultWeightValue)
defaultWeightValue
 the default feature weight value to use for all featuresprotected void generateCoefficientVectors()
protected void generateCoefficientVectorsHelper(int index, short[] vector, int numNonZeroEntries)
index
 the index into the coefficient vector that needs to have its values filled in.vector
 the coefficient vector generated thus farnumNonZeroEntries
 the number of nonzero coefficient vector entires currently in the vector.public FourierBasis copy()
DenseStateFeatures
DenseStateFeatures
copy
in interface DenseStateFeatures
DenseStateFeatures