Episode

java.lang.Object
- burlap.behavior.singleagent.Episode

```
public class Episode
extends java.lang.Object
```
This class is used to keep track of all events that occur in an episode. This class should be created by either calling the constructor with the initial state of the episode, or by calling the default constructor and then calling the initializeInState(State) method to set the initial state of the episode, before recording any transitions. It is then advised that transitions are recorded with the transition(Action, State, double) method, which takes as input the next state to which the agent transitioned, the action applied in the last recorded state, and the reward received fro the transition.
When querying about the state, action, and reward sequences, use the state(int), action(int), and reward(int) methods. These methods take as input the time step of the element you want. Note that t = 0 refers to the initial state step so calling getState(0) and getAction(0) will return the initial state and the action taken in the initial state, respectively. However, rewards are always received in the next time step from the state and action that produced them. Therefore, getReward(0) is undefined. Instead, the first reward received will be at time step 1: getReward(1).

Author:

James MacGlashan

Field Summary

Fields
Modifier and Type	Field and Description
`java.util.List<Action>`	`actionSequence` The sequence of actions taken
`java.util.List<java.lang.Double>`	`rewardSequence` The sequence of rewards received.
`java.util.List<State>`	`stateSequence` The sequence of states observed

Constructor Summary

Constructors
Constructor and Description
`Episode()` Creates a new EpisodeAnalysis object.
`Episode(State initialState)` Initializes a new EpisodeAnalysis object with the initial state in which the episode started.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`Action`	`action(int t)` Returns the action taken in the state at time step t.
`java.lang.String`	`actionString()` Returns a string representing the actions taken in this episode.
`java.lang.String`	`actionString(java.lang.String delimiter)` Returns a string representing the actions taken in this episode.
`void`	`addAction(Action ga)` Adds a GroundedAction to the action sequence.
`void`	`addReward(double r)` Adds a reward to the reward sequence.
`void`	`addState(State s)` Adds a state to the state sequence.
`void`	`appendAndMergeEpisodeAnalysis(Episode e)` This method will append execution results in e to this object's results.
`Episode`	`copy()` Returns a copy of this `Episode`.
`double`	`discountedReturn(double discountFactor)` Will return the discounted return received from the first state in the episode to the last state in the episode.
`void`	`initializeInState(State initialState)` Initializes this object with the initial state in which the episode started.
`static void`	`main(java.lang.String[] args)`
`int`	`maxTimeStep()` Returns the maximum time step index in this episode which is the `numTimeSteps()`-1.
`int`	`numActions()` Returns the number of actions, which is 1 less than the number of states.
`int`	`numTimeSteps()` Returns the number of time steps in this episode, which is equivalent to the number of states.
`static Episode`	`parseEpisode(java.lang.String episodeString)`
`static Episode`	`read(java.lang.String path)` Reads an episode that was written to a file and turns into an EpisodeAnalysis object.
`static java.util.List<Episode>`	`readEpisodes(java.lang.String directoryPath)` Takes a path to a directory containing .episode files and reads them all into a `List` of `Episode` objects.
`double`	`reward(int t)` Returns the reward received at timestep t.
`java.lang.String`	`serialize()`
`State`	`state(int t)` Returns the state observed at time step t.
`void`	`transition(Action usingAction, State nextState, double r)` Records a transition event where the agent applied the usingAction action in the last state in this object's state sequence, transitioned to state nextState, and received reward r,.
`void`	`transition(EnvironmentOutcome eo)` Records a transition event from the `EnvironmentOutcome`.
`void`	`write(java.lang.String path)` Writes this episode to a file.
`static void`	`writeEpisodes(java.util.List<Episode> episodes, java.lang.String directoryPath, java.lang.String baseFileName)` Takes a `List` of `Episode` objects and writes them to a directory.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - stateSequence
```
public java.util.List<State> stateSequence
```
    The sequence of states observed
  - actionSequence
```
public java.util.List<Action> actionSequence
```
    The sequence of actions taken
  - rewardSequence
```
public java.util.List<java.lang.Double> rewardSequence
```
    The sequence of rewards received. Note the reward stored at index i is the reward received at time step i+1.
- Constructor Detail
  - Episode
```
public Episode()
```
    Creates a new EpisodeAnalysis object. Before recording transitions, the initializeInState(State) method should be called to set the initial state of the episode.
  - Episode
```
public Episode(State initialState)
```
    Initializes a new EpisodeAnalysis object with the initial state in which the episode started.
    
    Parameters:
    
    initialState - the initial state of the episode
- Method Detail
  - initializeInState
```
public void initializeInState(State initialState)
```
    Initializes this object with the initial state in which the episode started.
    
    Parameters:
    
    initialState - the initial state of the episode
  - addState
```
public void addState(State s)
```
    Adds a state to the state sequence. In general, it is recommended that initializeInState(State) method along with subsequent calls to the transition(Action, State, double) method is used instead, but this method can be used to manually add a state.
    
    Parameters:
    
    s - the state to add
  - addAction
```
public void addAction(Action ga)
```
    Adds a GroundedAction to the action sequence. In general, it is recommended that initializeInState(State) method along with subsequent calls to the transition(Action, State, double) method is used instead, but this method can be used to manually add a GroundedAction.
    
    Parameters:
    
    ga - the GroundedAction to add
  - addReward
```
public void addReward(double r)
```
    Adds a reward to the reward sequence. In general, it is recommended that initializeInState(State) method along with subsequent calls to the transition(Action, State, double) method is used instead, but this method can be used to manually add a reward.
    
    Parameters:
    
    r - the reward to add
  - transition
```
public void transition(Action usingAction,
                       State nextState,
                       double r)
```
    Records a transition event where the agent applied the usingAction action in the last state in this object's state sequence, transitioned to state nextState, and received reward r,.
    
    Parameters:
    
    usingAction - the action the agent used that caused the transition
    
    nextState - the next state to which the agent transitioned
    
    r - the reward the agent received for this transition.
  - transition
```
public void transition(EnvironmentOutcome eo)
```
    Records a transition event from the EnvironmentOutcome. Assumes that the last state recorded in this Episode is the same as the previous state (EnvironmentOutcome.o in the EnvironmentOutcome
    
    Parameters:
    
    eo - an EnvironmentOutcome specifying a new transition for this episode.
  - state
```
public State state(int t)
```
    Returns the state observed at time step t. t=0 refers to the initial state.
    
    Parameters:
    
    t - the time step of the episode
    
    Returns:
    
    the state at time step t
  - action
```
public Action action(int t)
```
    Returns the action taken in the state at time step t. t=0 refers to the action taken in the initial state.
    
    Parameters:
    
    t - the time step of the episode
    
    Returns:
    
    the action taken at time step t
  - reward
```
public double reward(int t)
```
    Returns the reward received at timestep t. Note that the fist received reward will be at time step 1, which is the reward received after taking the first action in the initial state.
    
    Parameters:
    
    t - the time step of the episode
    
    Returns:
    
    the ith reward received in this episode
  - numTimeSteps
```
public int numTimeSteps()
```
    Returns the number of time steps in this episode, which is equivalent to the number of states. Note that there will be no action in the last time step.
    
    Returns:
    
    the number of time steps in this episode
  - maxTimeStep
```
public int maxTimeStep()
```
    Returns the maximum time step index in this episode which is the numTimeSteps()-1. Note that there is will be no action in the last time step.
    
    Returns:
    
    the maximum time step index in this episode
  - numActions
```
public int numActions()
```
    Returns the number of actions, which is 1 less than the number of states.
    
    Returns:
    
    the number of actions
  - discountedReturn
```
public double discountedReturn(double discountFactor)
```
    Will return the discounted return received from the first state in the episode to the last state in the episode.
    
    Parameters:
    
    discountFactor - the discount factor to compute the discounted return; should be on [0, 1]
    
    Returns:
    
    the discounted return of the episode
  - appendAndMergeEpisodeAnalysis
```
public void appendAndMergeEpisodeAnalysis(Episode e)
```
    This method will append execution results in e to this object's results. Note that it is assumed that the initial state in e is the last state recorded in this object. This method is useful for appending the results of an option's execution to a episode.
    
    Parameters:
    
    e - the execution results to append to this episode.
  - actionString
```
public java.lang.String actionString()
```
    Returns a string representing the actions taken in this episode. Actions are separated by ';' characters.
    
    Returns:
    
    a string representing the actions taken in this episode
  - actionString
```
public java.lang.String actionString(java.lang.String delimiter)
```
    Returns a string representing the actions taken in this episode. Actions are separated by the provided delimiter string.
    
    Parameters:
    
    delimiter - the delimiter to separate actions in the string.
    
    Returns:
    
    a string representing the actions taken in this episode
  - writeEpisodes
```
public static void writeEpisodes(java.util.List<Episode> episodes,
                                 java.lang.String directoryPath,
                                 java.lang.String baseFileName)
```
    Takes a List of Episode objects and writes them to a directory. The format of the file names will be "baseFileName{index}.episode" where {index} represents the index of the episode in the list. States must be serializable.
    
    Parameters:
    
    episodes - the list of episodes to write to disk
    
    directoryPath - the directory path in which the episodes will be written
    
    baseFileName - the base file name to use for the episode files
  - write
```
public void write(java.lang.String path)
```
    Writes this episode to a file. If the the directory for the specified file path do not exist, then they will be created. If the file extension is not ".episode" will automatically be added. States must be serializable.
    
    Parameters:
    
    path - the path to the file in which to write this episode.
  - readEpisodes
```
public static java.util.List<Episode> readEpisodes(java.lang.String directoryPath)
```
    Takes a path to a directory containing .episode files and reads them all into a List of Episode objects.
    
    Parameters:
    
    directoryPath - the path to the directory containing the episode files
    
    Returns:
    
    a List of Episode objects.
  - read
```
public static Episode read(java.lang.String path)
```
    Reads an episode that was written to a file and turns into an EpisodeAnalysis object.
    
    Parameters:
    
    path - the path to the episode file.
    
    Returns:
    
    an EpisodeAnalysis object.
  - serialize
```
public java.lang.String serialize()
```
  - copy
```
public Episode copy()
```
    Returns a copy of this Episode.
    
    Returns:
    
    a copy of this Episode.
  - parseEpisode
```
public static Episode parseEpisode(java.lang.String episodeString)
```
  - main
```
public static void main(java.lang.String[] args)
```

Class Episode

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

stateSequence

actionSequence

rewardSequence

Constructor Detail

Episode

Episode

Method Detail

initializeInState

addState

addAction

addReward

transition

transition

state

action

reward

numTimeSteps

maxTimeStep

numActions

discountedReturn

appendAndMergeEpisodeAnalysis

actionString

actionString

writeEpisodes

write

readEpisodes

read

serialize

copy

parseEpisode

main