ExponentialDecayLR

java.lang.Object
- burlap.behavior.learningrate.ExponentialDecayLR

All Implemented Interfaces:

LearningRate
```
public class ExponentialDecayLR
extends java.lang.Object
implements LearningRate
```
This class provides a learning rate that decays exponentially with time according to r^t, where r is in [0,1] and t is the time step, from an initial learning rate. A minimum learning rate value can be specified so that the learning rate is never rounded to zero. By default, the learning rate may decrease to Double.MIN_NORMAL, which is the smallest fraction a double value can hold. This class may be specified to use a universal learning rate that is shared regardless of state and action, or it can be set to have a different learning rate for each state (or state feature) that is decayed independently of other states, or it may also be specified to have a learning rate that is independently decayed for each state-action pair (or state feature-action pair). However, the state-action decay will ignore any parameterizations of actions.

Author:

James MacGlashan

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`protected class`	`ExponentialDecayLR.MutableDouble` A class for storing a mutable double value object
`protected class`	`ExponentialDecayLR.StateWiseLearningRate` A class for storing a learning rate for a state, or a learning rate for each action for a given state

Field Summary

Fields
Modifier and Type	Field and Description
`protected double`	`decayRate` The exponential base by which the learning rate is decayed
`protected java.util.Map<java.lang.Integer,ExponentialDecayLR.StateWiseLearningRate>`	`featureWiseMap` The state feature dependent or state feature-action dependent learning rates
`protected HashableStateFactory`	`hashingFactory` How to hash and perform equality checks of states
`protected double`	`initialLearningRate` The initial learning rate value
`protected int`	`lastPollTime` The last agent time at which they polled the learning rate
`protected double`	`minimumLR` The minimum learning rate
`protected java.util.Map<HashableState,ExponentialDecayLR.StateWiseLearningRate>`	`stateWiseMap` The state dependent or state-action dependent learning rates
`protected double`	`universalLR` The state independent learning rate
`protected boolean`	`useStateActionWise` Whether the learning rate is dependent on state-actions
`protected boolean`	`useStateWise` Whether the learning rate is dependent on the state

Constructor Summary

Constructors
Constructor and Description
`ExponentialDecayLR(double initialLearningRate, double decayRate)` Initializes with an initial learning rate and decay rate for a state independent learning rate.
`ExponentialDecayLR(double initialLearningRate, double decayRate, double minimumLearningRate)` Initializes with an initial learning rate and decay rate for a state independent learning rate that will decay to a value no smaller than minimumLearningRate
`ExponentialDecayLR(double initialLearningRate, double decayRate, double minimumLearningRate, HashableStateFactory hashingFactory, boolean useSeparateLRPerStateAction)` Initializes with an initial learning rate and decay rate for a state or state-action (or state feature-action) dependent learning rate that will decay to a value no smaller than minimumLearningRate If this learning rate function is to be used for state state features, rather than states, then the hashing factory can be null;
`ExponentialDecayLR(double initialLearningRate, double decayRate, HashableStateFactory hashingFactory, boolean useSeparateLRPerStateAction)` Initializes with an initial learning rate and decay rate for a state or state-action (or state feature-action) dependent learning rate.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected ExponentialDecayLR.StateWiseLearningRate`	`getFeatureWiseLearningRate(int feature)` Returns the learning rate data structure for the given state feature.
`protected ExponentialDecayLR.StateWiseLearningRate`	`getStateWiseLearningRate(State s)` Returns the learning rate data structure for the given state.
`protected double`	`nextLRVal(double cur)` Returns the value of an input current learning rate after it has been decayed by one time step.
`double`	`peekAtLearningRate(int featureId)` A method for looking at the current learning rate for a state (-action) feature without having it altered.
`double`	`peekAtLearningRate(State s, Action ga)` A method for looking at the current learning rate for a state-action pair without having it altered.
`double`	`pollLearningRate(int agentTime, int featureId)` A method for returning the learning rate for a given state (-action) feature and then decaying the learning rate as defined by this class.
`double`	`pollLearningRate(int agentTime, State s, Action ga)` A method for returning the learning rate for a given state action pair and then decaying the learning rate as defined by this class.
`void`	`resetDecay()` Causes any learnign rate decay to reset to where it started.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - initialLearningRate
```
protected double initialLearningRate
```
    The initial learning rate value
  - decayRate
```
protected double decayRate
```
    The exponential base by which the learning rate is decayed
  - minimumLR
```
protected double minimumLR
```
    The minimum learning rate
  - universalLR
```
protected double universalLR
```
    The state independent learning rate
  - stateWiseMap
```
protected java.util.Map<HashableState,ExponentialDecayLR.StateWiseLearningRate> stateWiseMap
```
    The state dependent or state-action dependent learning rates
  - featureWiseMap
```
protected java.util.Map<java.lang.Integer,ExponentialDecayLR.StateWiseLearningRate> featureWiseMap
```
    The state feature dependent or state feature-action dependent learning rates
  - useStateWise
```
protected boolean useStateWise
```
    Whether the learning rate is dependent on the state
  - useStateActionWise
```
protected boolean useStateActionWise
```
    Whether the learning rate is dependent on state-actions
  - hashingFactory
```
protected HashableStateFactory hashingFactory
```
    How to hash and perform equality checks of states
  - lastPollTime
```
protected int lastPollTime
```
    The last agent time at which they polled the learning rate
- Constructor Detail
  - ExponentialDecayLR
```
public ExponentialDecayLR(double initialLearningRate,
                          double decayRate)
```
    Initializes with an initial learning rate and decay rate for a state independent learning rate. Minimum learning rate that can be returned will be Double.MIN_NORMAL
    
    Parameters:
    
    initialLearningRate - the initial learning rate
    
    decayRate - the exponential base by which the learning rate is decayed
  - ExponentialDecayLR
```
public ExponentialDecayLR(double initialLearningRate,
                          double decayRate,
                          double minimumLearningRate)
```
    Initializes with an initial learning rate and decay rate for a state independent learning rate that will decay to a value no smaller than minimumLearningRate
    
    Parameters:
    
    initialLearningRate - the initial learning rate
    
    decayRate - the exponential base by which the learning rate is decayed
    
    minimumLearningRate - the smallest value to which the learning rate will decay
  - ExponentialDecayLR
```
public ExponentialDecayLR(double initialLearningRate,
                          double decayRate,
                          HashableStateFactory hashingFactory,
                          boolean useSeparateLRPerStateAction)
```
    Initializes with an initial learning rate and decay rate for a state or state-action (or state feature-action) dependent learning rate. Minimum learning rate that can be returned will be Double.MIN_NORMAL. If this learning rate function is to be used for state state features, rather than states, then the hashing factory can be null;
    
    Parameters:
    
    initialLearningRate - the initial learning rate for each state or state-action
    
    decayRate - the exponential base by which the learning rate is decayed
    
    hashingFactory - how to hash and compare states
    
    useSeparateLRPerStateAction - whether to have an independent learning rate for each state-action pair, rather than just each state
  - ExponentialDecayLR
```
public ExponentialDecayLR(double initialLearningRate,
                          double decayRate,
                          double minimumLearningRate,
                          HashableStateFactory hashingFactory,
                          boolean useSeparateLRPerStateAction)
```
    Initializes with an initial learning rate and decay rate for a state or state-action (or state feature-action) dependent learning rate that will decay to a value no smaller than minimumLearningRate If this learning rate function is to be used for state state features, rather than states, then the hashing factory can be null;
    
    Parameters:
    
    initialLearningRate - the initial learning rate for each state or state-action
    
    decayRate - the exponential base by which the learning rate is decayed
    
    minimumLearningRate - the smallest value to which the learning rate will decay
    
    hashingFactory - how to hash and compare states
    
    useSeparateLRPerStateAction - whether to have an independent learning rate for each state-action pair, rather than just each state
- Method Detail
  - peekAtLearningRate
```
public double peekAtLearningRate(State s,
                                 Action ga)
```
    Description copied from interface: LearningRate
    
    A method for looking at the current learning rate for a state-action pair without having it altered.
    
    Specified by:
    
    peekAtLearningRate in interface LearningRate
    
    Parameters:
    
    s - the state for which the learning rate should be returned
    
    ga - the action from which the learning rate should be returned
    
    Returns:
    
    the current learning rate for the given state-action pair
  - pollLearningRate
```
public double pollLearningRate(int agentTime,
                               State s,
                               Action ga)
```
    Description copied from interface: LearningRate
    
    A method for returning the learning rate for a given state action pair and then decaying the learning rate as defined by this class.
    
    Specified by:
    
    pollLearningRate in interface LearningRate
    
    Parameters:
    
    agentTime - the time index of the agent when polling.
    
    s - the state for which the learning rate should be returned
    
    ga - the action from which the learning rate should be returned
    
    Returns:
    
    the current learning rate for the given state-action pair
  - peekAtLearningRate
```
public double peekAtLearningRate(int featureId)
```
    Description copied from interface: LearningRate
    
    A method for looking at the current learning rate for a state (-action) feature without having it altered.
    
    Specified by:
    
    peekAtLearningRate in interface LearningRate
    
    Parameters:
    
    featureId - the state feature for which the learning rate should be returned
    
    Returns:
    
    the current learning rate for the given state feature-action pair
  - pollLearningRate
```
public double pollLearningRate(int agentTime,
                               int featureId)
```
    Description copied from interface: LearningRate
    
    A method for returning the learning rate for a given state (-action) feature and then decaying the learning rate as defined by this class.
    
    Specified by:
    
    pollLearningRate in interface LearningRate
    
    Parameters:
    
    agentTime - the time index of the agent when polling.
    
    featureId - the state feature for which the learning rate should be returned
    
    Returns:
    
    the current learning rate for the given state feature-action pair
  - resetDecay
```
public void resetDecay()
```
    Description copied from interface: LearningRate
    
    Causes any learnign rate decay to reset to where it started.
    
    Specified by:
    
    resetDecay in interface LearningRate
  - getStateWiseLearningRate
```
protected ExponentialDecayLR.StateWiseLearningRate getStateWiseLearningRate(State s)
```
    Returns the learning rate data structure for the given state. An entry will be created if it does not already exist.
    
    Parameters:
    
    s - the state to get a learning rate for
    
    Returns:
    
    the learning rate data structure for the given state
  - getFeatureWiseLearningRate
```
protected ExponentialDecayLR.StateWiseLearningRate getFeatureWiseLearningRate(int feature)
```
    Returns the learning rate data structure for the given state feature. An entry will be created if it does not already exist.
    
    Parameters:
    
    feature - the state feature id to get a learning rate for
    
    Returns:
    
    the learning rate data structure for the given state feature
  - nextLRVal
```
protected double nextLRVal(double cur)
```
    Returns the value of an input current learning rate after it has been decayed by one time step.
    
    Parameters:
    
    cur - the currently learning rate to be decayed by one time step.
    
    Returns:
    
    the value of an input current learning rate after it has been decayed by one time step.

Class ExponentialDecayLR

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

initialLearningRate

decayRate

minimumLR

universalLR

stateWiseMap

featureWiseMap

useStateWise

useStateActionWise

hashingFactory

lastPollTime

Constructor Detail

ExponentialDecayLR

ExponentialDecayLR

ExponentialDecayLR

ExponentialDecayLR

Method Detail

peekAtLearningRate

pollLearningRate

peekAtLearningRate

pollLearningRate

resetDecay

getStateWiseLearningRate

getFeatureWiseLearningRate

nextLRVal