burlap.behavior.stochasticgames.agents.naiveq.history

Class SGQWActionHistory

java.lang.Object

burlap.mdp.stochasticgames.agent.SGAgentBase

burlap.behavior.stochasticgames.agents.naiveq.SGNaiveQLAgent

burlap.behavior.stochasticgames.agents.naiveq.history.SGQWActionHistory

All Implemented Interfaces:

QFunction, QProvider, ValueFunction, SGAgent

public class SGQWActionHistory
extends SGNaiveQLAgent

A Tabular Q-learning [1] algorithm for stochastic games formalisms that augments states with the actions each agent took in n previous time steps.

1. Watkins, Christopher JCH, and Peter Dayan. "Q-learning." Machine learning 8.3-4 (1992): 279-292.

Author:

James MacGlashan

Nested Class Summary

Nested classes/interfaces inherited from interface burlap.behavior.valuefunction.QProvider

QProvider.Helper

Field Summary

Fields

Modifier and Type	Field and Description
protected HistoryState	curHState
protected int	historySize The size of action history to store.

Fields inherited from class burlap.behavior.stochasticgames.agents.naiveq.SGNaiveQLAgent

agentNum, discount, hashFactory, learningRate, policy, qInit, qMap, stateRepresentations, storedMapAbstraction, totalNumberOfSteps

Fields inherited from class burlap.mdp.stochasticgames.agent.SGAgentBase

agentType, domain, internalRewardFunction, world, worldAgentName

Constructor Summary

Constructors

Constructor and Description
SGQWActionHistory(SGDomain d, double discount, double learningRate, HashableStateFactory hashFactory, int historySize) Initializes the learning algorithm using 0.1 epsilon greedy learning strategy/policy

Method Summary

Modifier and Type	Method and Description
Action	action(State s) This method is called by the world when it needs the agent to choose an action
void	gameStarting(World w, int agentNum) This method is called by the world when a new game is starting.
void	observeOutcome(State s, JointAction jointAction, double[] jointReward, State sprime, boolean isTerminal) This method is called by the world when every agent in the world has taken their action.
SGQWActionHistory	setAgentDetails(java.lang.String agentName, SGAgentType type)

Methods inherited from class burlap.behavior.stochasticgames.agents.naiveq.SGNaiveQLAgent

gameTerminated, getMaxQValue, qValue, qValues, setLearningRate, setQValueInitializer, setStoredMapAbstraction, setStrategy, stateHash, storedQ, value

Methods inherited from class burlap.mdp.stochasticgames.agent.SGAgentBase

agentName, agentType, getInternalRewardFunction, init, init, setInternalRewardFunction

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

historySize

protected int historySize

The size of action history to store.

curHState

protected HistoryState curHState

Constructor Detail

SGQWActionHistory

public SGQWActionHistory(SGDomain d,
                         double discount,
                         double learningRate,
                         HashableStateFactory hashFactory,
                         int historySize)

Initializes the learning algorithm using 0.1 epsilon greedy learning strategy/policy

Parameters:

d - the domain in which the agent will act

discount - the discount factor

learningRate - the learning rate

hashFactory - the state hashing factory to use

historySize - the number of previous steps to remember and with which to augment the state space

Method Detail

setAgentDetails

public SGQWActionHistory setAgentDetails(java.lang.String agentName,
                                         SGAgentType type)

Overrides:

setAgentDetails in class SGNaiveQLAgent

gameStarting

public void gameStarting(World w,
                         int agentNum)

Description copied from interface: SGAgent

This method is called by the world when a new game is starting.

Specified by:

gameStarting in interface SGAgent

Overrides:

gameStarting in class SGNaiveQLAgent

Parameters:

w - the world in which the game is starting

agentNum - the agent number of the agent in the world

observeOutcome

public void observeOutcome(State s,
                           JointAction jointAction,
                           double[] jointReward,
                           State sprime,
                           boolean isTerminal)

Description copied from interface: SGAgent

This method is called by the world when every agent in the world has taken their action. It conveys the result of the joint action.

Specified by:

observeOutcome in interface SGAgent

Overrides:

observeOutcome in class SGNaiveQLAgent

Parameters:

s - the state in which the last action of each agent was taken

jointAction - the joint action of all agents in the world

jointReward - the joint reward of all agents in the world

sprime - the next state to which the agent transitioned

isTerminal - whether the new state is a terminal state

action

public Action action(State s)

Description copied from interface: SGAgent

This method is called by the world when it needs the agent to choose an action

Specified by:

action in interface SGAgent

Overrides:

action in class SGNaiveQLAgent

Parameters:

s - the current state of the world

Returns:

the action this agent wishes to take