burlap.domain.singleagent.rlglue

Class RLGlueEnvironment

java.lang.Object

burlap.domain.singleagent.rlglue.RLGlueEnvironment

All Implemented Interfaces:

org.rlcommunity.rlglue.codec.EnvironmentInterface

public class RLGlueEnvironment
extends java.lang.Object
implements org.rlcommunity.rlglue.codec.EnvironmentInterface

This class can be used to take a BURLAP domain and task with discrete actions and turn it into an RLGlue environment with which other RLGlue agents can interact. Because RLGLue requires flat vector representations of states, you must provide a DenseStateFeatures to flatten the BURLAP states; it should always return arrays of the same length for all visitable states. Additionally, RLGlue does not support action preconditions, so each action must be available everywhere.

Note that RLGlue does not support observations of terminal states; it only gives the final reward upon entering a terminal state. Therefore, this class will not terminate in a terminal state indicated by the provided TerminalFunction. Instead, it will allow one more transition from the terminal state, which will transition back to itself with reward zero, which is mathematically equivalent to transitioning to terminal state and observing it.

Author:

James MacGlashan

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.Map<java.lang.Integer,Action>	actionMap A mapping from action index identifiers (that RLGlue will use) to BURLAP actions and their parametrization specified as the index of objects in a state.
protected State	curState The current state of the environment
protected double	discount The discount factor of the task
protected SADomain	domain The BURLAP domain
protected boolean	isEpisodic Whether this task is episodic (false will indicate that it is continuing)
protected org.rlcommunity.rlglue.codec.taskspec.ranges.DoubleRange	rewardRange The reward function value range
protected DenseStateFeatures	stateFlattener Used to flatten states into a vector representation
protected StateGenerator	stateGenerator The state generator for generating states for each episode
protected int	terminalVisits Indicates the number of times a terminal state has been visited by the agent within the same episode.
protected boolean	usedConstructorState Whether the state generated from the state generator to gather auxiliary information (like the number of objects of each class) has yet be used as a starting state for an RLGlue episode.
protected org.rlcommunity.rlglue.codec.taskspec.ranges.DoubleRange[]	valueRanges The value ranges for the vector representation of the state

Constructor Summary

Constructors

Constructor and Description
RLGlueEnvironment(SADomain domain, StateGenerator stateGenerator, DenseStateFeatures stateFlattener, org.rlcommunity.rlglue.codec.taskspec.ranges.DoubleRange[] valueRanges, org.rlcommunity.rlglue.codec.taskspec.ranges.DoubleRange rewardRange, boolean isEpisodic, double discount) Constructs with all the BURLAP information necessary for generating an RLGlue Environment.

Method Summary

Modifier and Type	Method and Description
protected org.rlcommunity.rlglue.codec.types.Observation	convertIntoObservation(State s) Takes a OO-MDP state and converts it into an RLGlue observation
void	env_cleanup()
java.lang.String	env_init()
java.lang.String	env_message(java.lang.String arg0)
org.rlcommunity.rlglue.codec.types.Observation	env_start()
org.rlcommunity.rlglue.codec.types.Reward_observation_terminal	env_step(org.rlcommunity.rlglue.codec.types.Action arg0)
void	load() Loads this environment into RLGlue
void	load(java.lang.String hostAddress, java.lang.String port) Loads this environment into RLGLue with the specified host address and port

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

domain

protected SADomain domain

The BURLAP domain

stateGenerator

protected StateGenerator stateGenerator

The state generator for generating states for each episode

stateFlattener

protected DenseStateFeatures stateFlattener

Used to flatten states into a vector representation

valueRanges

protected org.rlcommunity.rlglue.codec.taskspec.ranges.DoubleRange[] valueRanges

The value ranges for the vector representation of the state

terminalVisits

protected int terminalVisits

Indicates the number of times a terminal state has been visited by the agent within the same episode. This variable is used because RLGLue does not support observations into terminal states and so a terminal flag will only be set once the agent has taken one action in the terminal state, which will transition back to itself.

rewardRange

protected org.rlcommunity.rlglue.codec.taskspec.ranges.DoubleRange rewardRange

The reward function value range

isEpisodic

protected boolean isEpisodic

Whether this task is episodic (false will indicate that it is continuing)

discount

protected double discount

The discount factor of the task

curState

protected State curState

The current state of the environment

actionMap

protected java.util.Map<java.lang.Integer,Action> actionMap

A mapping from action index identifiers (that RLGlue will use) to BURLAP actions and their parametrization specified as the index of objects in a state.

usedConstructorState

protected boolean usedConstructorState

Whether the state generated from the state generator to gather auxiliary information (like the number of objects of each class) has yet be used as a starting state for an RLGlue episode. When this value is false, the state generated in the constructor will be passed as the initial state of a new episodes. After that, this value is set to true and the states used for each RLGlue episode are generated fresh from the state generator.

Constructor Detail

RLGlueEnvironment

public RLGlueEnvironment(SADomain domain,
                         StateGenerator stateGenerator,
                         DenseStateFeatures stateFlattener,
                         org.rlcommunity.rlglue.codec.taskspec.ranges.DoubleRange[] valueRanges,
                         org.rlcommunity.rlglue.codec.taskspec.ranges.DoubleRange rewardRange,
                         boolean isEpisodic,
                         double discount)

Constructs with all the BURLAP information necessary for generating an RLGlue Environment.

Parameters:

domain - the BURLAP domain

stateGenerator - a generated for generating states at the start of each episode.

stateFlattener - used to flatten states into a numeric representation

valueRanges - the value ranges of the flattened vector state

rewardRange - the reward function value range

isEpisodic - whether the task is episodic or continuing

discount - the discount factor to use for the task

Method Detail

load

public void load()

Loads this environment into RLGlue

load

public void load(java.lang.String hostAddress,
                 java.lang.String port)

Loads this environment into RLGLue with the specified host address and port

Parameters:

hostAddress - the RLGlue host address

port - the RLGlue port

env_cleanup

public void env_cleanup()

Specified by:

env_cleanup in interface org.rlcommunity.rlglue.codec.EnvironmentInterface

env_init

public java.lang.String env_init()

Specified by:

env_init in interface org.rlcommunity.rlglue.codec.EnvironmentInterface

env_message

public java.lang.String env_message(java.lang.String arg0)

Specified by:

env_message in interface org.rlcommunity.rlglue.codec.EnvironmentInterface

env_start

public org.rlcommunity.rlglue.codec.types.Observation env_start()

Specified by:

env_start in interface org.rlcommunity.rlglue.codec.EnvironmentInterface

env_step

public org.rlcommunity.rlglue.codec.types.Reward_observation_terminal env_step(org.rlcommunity.rlglue.codec.types.Action arg0)

Specified by:

env_step in interface org.rlcommunity.rlglue.codec.EnvironmentInterface

convertIntoObservation

protected org.rlcommunity.rlglue.codec.types.Observation convertIntoObservation(State s)

Takes a OO-MDP state and converts it into an RLGlue observation

Parameters:

s - the OO-MDP state

Returns:

an RLGlue Observation