public class FourierBasis extends java.lang.Object implements FeatureDatabase
ConcatenatedObjectFeatureVectorGenerator
generator with the normalization flag set to convert the OO-MDP State
objects into the necessary input vector.
The higher order the basis functions, the higher the VFA resolution is. Typically, order n will produce (n+1)^d state basis functions (and a copy for each action), where d is the number of state variables. Since this grows quickly,
a way to manage the complexity is to simplify the number of coefficient vectors. That is, each basis function is a function of the dot product of the input state variable vector and a coefficient vector {0...n}^d
and normally all possible coefficient vectors (and their corresponding basis functions) for
order n are produced. However, this class can be told to limit the permitted coefficient vectors to those that have no more than k non-zero entries in the coefficient vector. When k = 1, all features are treated as indepdent resulting
in n*d basis functions.
When using a learning algorithm like GradientDescentSarsaLam
with Fourier basis functions, it is typically a good idea to use the FourierBasisLearningRateWrapper
, which scales the normal learning rate by the inverse of the norm
of a basis function's coefficient vector.
1. G.D. Konidaris, S. Osentoski and P.S. Thomas. Value Function Approximation in Reinforcement Learning using the Fourier Basis. In Proceedings of the Twenty-Fifth Conference on Artificial Intelligence, pages 380-385, August 2011.Modifier and Type | Field and Description |
---|---|
protected java.util.Map<java.lang.String,java.lang.Integer> |
actionFeatureMultiplier
A map for returning a multiplier to the number of state features for each action.
|
protected java.util.List<short[]> |
coefficientVectors
The coefficient vectors used
|
protected StateToFeatureVectorGenerator |
featureVectorGenerator
The OO-MDP
State to feature vector/variable generator. |
protected int |
maxNonZeroCoefficents
The maximum number of non-zero coefficient entries permitted in a coefficient vector
|
protected int |
nextActionMultiplier
The next action Fourier basis function size multiplier to use for the next newly seen action.
|
protected int |
numStateVariables
The number of state varibles on which the produced basis functions operate
|
protected int |
order
The order of the Fourier basis functions.
|
Constructor and Description |
---|
FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator,
int order)
Initializes.
|
FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator,
int order,
int maxNonZeroCoefficents)
Initializes.
|
Modifier and Type | Method and Description |
---|---|
double |
basisValue(double[] input,
int basisFunction)
Returns the basis function value for the given state input for the given basis function index.
|
double |
coefficientNorm(int i)
Returns the norm of the coefficient vector for the given basis function index.
|
void |
freezeDatabaseState(boolean toggle)
Enable or disable the database from generating new features.
|
protected void |
generateCoefficientVectors()
Generates all coefficient vectors given the number of state variables and the maximum number of non-zero coefficient element entries.
|
protected void |
generateCoefficientVectorsHelper(int index,
short[] vector,
int numNonZeroEntries)
Recursive cofficient generator helper method.
|
ValueFunctionApproximation |
generateVFA(double defaultWeightValue)
Creates and returns a linear VFA object over this Fourier basis feature database.
|
java.util.List<ActionFeaturesQuery> |
getActionFeaturesSets(State s,
java.util.List<GroundedAction> actions)
Returns non-zero action features for all of provided actions in state s.
|
protected int |
getActionMultiplier(GroundedAction ga)
This method returns the action multiplier for the specified grounded action.
|
short[] |
getCoefficientVector(int i)
Returns the coefficient vector for the given basis function index.
|
java.util.List<StateFeature> |
getStateFeatures(State s)
Returns non-zero state features for a given state.
|
int |
numberOfFeatures()
Returns the number of features this database tracks.
|
void |
setCoefficientVectors(java.util.List<short[]> coefficientVectors)
Forces the set of coefficient vectors (and thereby Fourier basis functions) used.
|
protected int numStateVariables
protected StateToFeatureVectorGenerator featureVectorGenerator
State
to feature vector/variable generator. Should produced normalized values.protected java.util.List<short[]> coefficientVectors
protected int maxNonZeroCoefficents
protected int order
protected java.util.Map<java.lang.String,java.lang.Integer> actionFeatureMultiplier
protected int nextActionMultiplier
public FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator, int order)
featureVectorGenerator
- the state feature vector generator that turns OO-MDP State
objects into double arrays.order
- the Fourier basis orderpublic FourierBasis(StateToFeatureVectorGenerator featureVectorGenerator, int order, int maxNonZeroCoefficents)
featureVectorGenerator
- the state feature vector generator that turns OO-MDP State
objects into double arraysorder
- the fourier basis ordermaxNonZeroCoefficents
- the maximum number of entries in coeffient vectors that can have non-zero values.public void setCoefficientVectors(java.util.List<short[]> coefficientVectors)
coefficientVectors
- the coefficient vectors used to produce the Fourier basis functions.public double basisValue(double[] input, int basisFunction)
input
- the state variablesbasisFunction
- the basis function indexpublic java.util.List<StateFeature> getStateFeatures(State s)
FeatureDatabase
getStateFeatures
in interface FeatureDatabase
s
- the state for which features should be returnedpublic java.util.List<ActionFeaturesQuery> getActionFeaturesSets(State s, java.util.List<GroundedAction> actions)
FeatureDatabase
getActionFeaturesSets
in interface FeatureDatabase
s
- the state for which features should be returnedactions
- the action set for which the action features should be returnedpublic void freezeDatabaseState(boolean toggle)
FeatureDatabase
freezeDatabaseState
in interface FeatureDatabase
public int numberOfFeatures()
FeatureDatabase
numberOfFeatures
in interface FeatureDatabase
public short[] getCoefficientVector(int i)
i
- the basis function indexpublic double coefficientNorm(int i)
i
- the basis function indexpublic ValueFunctionApproximation generateVFA(double defaultWeightValue)
defaultWeightValue
- the default feature weight value to use for all featuresprotected void generateCoefficientVectors()
protected void generateCoefficientVectorsHelper(int index, short[] vector, int numNonZeroEntries)
index
- the index into the coefficient vector that needs to have its values filled in.vector
- the coefficient vector generated thus farnumNonZeroEntries
- the number of non-zero coefficient vector entires currently in the vector.protected int getActionMultiplier(GroundedAction ga)
ga
- the grounded action for which the multiplier will be returned