public class FourierBasis extends java.lang.Object implements FeatureDatabase
ConcatenatedObjectFeatureVectorGenerator
generator with the normalization flag set to convert the OOMDP State
objects into the necessary input vector.
The higher order the basis functions, the higher the VFA resolution is. Typically, order n will produce (n+1)^d state basis functions (and a copy for each action), where d is the number of state variables. Since this grows quickly,
a way to manage the complexity is to simplify the number of coefficient vectors. That is, each basis function is a function of the dot product of the input state variable vector and a coefficient vector {0...n}^d
and normally all possible coefficient vectors (and their corresponding basis functions) for
order n are produced. However, this class can be told to limit the permitted coefficient vectors to those that have no more than k nonzero entries in the coefficient vector. When k = 1, all features are treated as indepdent resulting
in n*d basis functions.
When using a learning algorithm like GradientDescentSarsaLam
with Fourier basis functions, it is typically a good idea to use the FourierBasisLearningRateWrapper
, which scales the normal learning rate by the inverse of the norm
of a basis function's coefficient vector.
1. G.D. Konidaris, S. Osentoski and P.S. Thomas. Value Function Approximation in Reinforcement Learning using the Fourier Basis. In Proceedings of the TwentyFifth Conference on Artificial Intelligence, pages 380385, August 2011.
Modifier and Type  Field and Description 

protected java.util.Map<GroundedAction,java.lang.Integer> 
actionFeatureMultiplier
A map for returning a multiplier to the number of state features for each action.

protected java.util.List<short[]> 
coefficientVectors
The coefficient vectors used

protected StateToFeatureVectorGenerator 
featureVectorGenerator
The OOMDP
State to feature vector/variable generator. 
protected int 
maxNonZeroCoefficents
The maximum number of nonzero coefficient entries permitted in a coefficient vector

protected int 
nextActionMultiplier
The next action Fourier basis function size multiplier to use for the next newly seen action.

protected int 
numStateVariables
The number of state varibles on which the produced basis functions operate

protected int 
order
The order of the Fourier basis functions.

Constructor and Description 

FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator,
int order)
Initializes.

FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator,
int order,
int maxNonZeroCoefficents)
Initializes.

Modifier and Type  Method and Description 

double 
basisValue(double[] input,
int basisFunction)
Returns the basis function value for the given state input for the given basis function index.

double 
coefficientNorm(int i)
Returns the norm of the coefficient vector for the given basis function index.

FourierBasis 
copy()
Returns a deep copy of this feature database.

void 
freezeDatabaseState(boolean toggle)
Enable or disable the database from generating new features.

protected void 
generateCoefficientVectors()
Generates all coefficient vectors given the number of state variables and the maximum number of nonzero coefficient element entries.

protected void 
generateCoefficientVectorsHelper(int index,
short[] vector,
int numNonZeroEntries)
Recursive cofficient generator helper method.

LinearVFA 
generateVFA(double defaultWeightValue)
Creates and returns a linear VFA object over this Fourier basis feature database.

java.util.List<ActionFeaturesQuery> 
getActionFeaturesSets(State s,
java.util.List<GroundedAction> actions)
Returns nonzero action features for all of provided actions in state s.

protected int 
getActionMultiplier(GroundedAction ga)
This method returns the action multiplier for the specified grounded action.

short[] 
getCoefficientVector(int i)
Returns the coefficient vector for the given basis function index.

java.util.List<StateFeature> 
getStateFeatures(State s)
Returns nonzero state features for a given state.

int 
numberOfFeatures()
Returns the number of features this database tracks.

void 
setCoefficientVectors(java.util.List<short[]> coefficientVectors)
Forces the set of coefficient vectors (and thereby Fourier basis functions) used.

protected int numStateVariables
protected StateToFeatureVectorGenerator featureVectorGenerator
State
to feature vector/variable generator. Should produced normalized values.protected java.util.List<short[]> coefficientVectors
protected int maxNonZeroCoefficents
protected int order
protected java.util.Map<GroundedAction,java.lang.Integer> actionFeatureMultiplier
protected int nextActionMultiplier
public FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator, int order)
featureVectorGenerator
 the state feature vector generator that turns OOMDP State
objects into double arrays.order
 the Fourier basis orderpublic FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator, int order, int maxNonZeroCoefficents)
featureVectorGenerator
 the state feature vector generator that turns OOMDP State
objects into double arraysorder
 the fourier basis ordermaxNonZeroCoefficents
 the maximum number of entries in coeffient vectors that can have nonzero values.public void setCoefficientVectors(java.util.List<short[]> coefficientVectors)
coefficientVectors
 the coefficient vectors used to produce the Fourier basis functions.public double basisValue(double[] input, int basisFunction)
input
 the state variablesbasisFunction
 the basis function indexpublic java.util.List<StateFeature> getStateFeatures(State s)
FeatureDatabase
getStateFeatures
in interface FeatureDatabase
s
 the state for which features should be returnedpublic java.util.List<ActionFeaturesQuery> getActionFeaturesSets(State s, java.util.List<GroundedAction> actions)
FeatureDatabase
getActionFeaturesSets
in interface FeatureDatabase
s
 the state for which features should be returnedactions
 the action set for which the action features should be returnedpublic void freezeDatabaseState(boolean toggle)
FeatureDatabase
freezeDatabaseState
in interface FeatureDatabase
public int numberOfFeatures()
FeatureDatabase
numberOfFeatures
in interface FeatureDatabase
public short[] getCoefficientVector(int i)
i
 the basis function indexpublic double coefficientNorm(int i)
i
 the basis function indexpublic LinearVFA generateVFA(double defaultWeightValue)
defaultWeightValue
 the default feature weight value to use for all featuresprotected void generateCoefficientVectors()
protected void generateCoefficientVectorsHelper(int index, short[] vector, int numNonZeroEntries)
index
 the index into the coefficient vector that needs to have its values filled in.vector
 the coefficient vector generated thus farnumNonZeroEntries
 the number of nonzero coefficient vector entires currently in the vector.protected int getActionMultiplier(GroundedAction ga)
ga
 the grounded action for which the multiplier will be returnedpublic FourierBasis copy()
FeatureDatabase
copy
in interface FeatureDatabase