burlap.behavior.stochasticgames.madynamicprogramming

Class MADynamicProgramming

java.lang.Object

burlap.behavior.stochasticgames.madynamicprogramming.MADynamicProgramming

All Implemented Interfaces:

MultiAgentQSourceProvider

Direct Known Subclasses:

MAValueIteration

public abstract class MADynamicProgramming
extends java.lang.Object
implements MultiAgentQSourceProvider

An abstract value function based planning algorithm base for sequential stochastic games that require the computation of Q-values for each agent for each joint action. Value function updates are performed using a Bellman-like backup operator; however, planning for different solution concepts is achieved by providing different backup operators via the SGBackupOperator object.

Note that the agent definitions can only be changed up until planning begins. Once planning has begun, the agent definitions must remain fixed for consistency of planning results. If the the client tries to change the agent definitions after planning has already begun, then a runtime exception will be thrown.

Since value function planning algorithms compute multi-agent Q-values, this object implements the MultiAgentQSourceProvider interface.

Author:

James MacGlashan

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	MADynamicProgramming.BackupBasedQSource A QSourceForSingleAgent implementation which stores a value function for an agent and produces Joint action Q-values by marginalizing over the transition dynamics the reward and discounted next state value.
class	MADynamicProgramming.JointActionTransitions A class for holding all of the transition dynamic information for a given joint action in a given state.

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.List<SGAgentType>	agentDefinitions The agent types
protected SGBackupOperator	backupOperator The backup operating defining the solution concept to use.
protected double	discount The discount factor in [0, 1]
protected SGDomain	domain The domain in which planning is to be performed
protected HashableStateFactory	hashingFactory The state hashing factory used to query the value function for individual states
protected JointModel	jointModel The joint action model to use in planning.
protected JointRewardFunction	jointRewardFunction The joint reward function
protected boolean	planningStarted Whether planning has begun or not.
protected AgentQSourceMap.HashMapAgentQSourceMap	qSources The Hash map backed multi-agent Q-source in which to store Q-values.
protected TerminalFunction	terminalFunction The state terminal function.
protected ValueFunction	vInit The Q-value initialization function to use.

Constructor Summary

Constructors

Constructor and Description
MADynamicProgramming()

Method Summary

Modifier and Type	Method and Description
double	backupAllValueFunctions(State s) Backups the state value function for all agent's value functions in state s.
AgentQSourceMap	getQSources() Returns an object that can provide Q-value sources for each agent.
boolean	hasStartedPlanning() Indicates whether planning has begun or not.
void	initMAVF(SGDomain domain, java.util.List<SGAgentType> agentDefinitions, JointRewardFunction jointRewardFunction, TerminalFunction terminalFunction, double discount, HashableStateFactory hashingFactory, ValueFunction vInit, SGBackupOperator backupOperator) Initializes all the main datstructres of the value function valueFunction
abstract void	planFromState(State s) Calling this method causes planning to be performed from State s.
void	setAgentDefinitions(java.util.List<SGAgentType> agentDefinitions) Sets/changes the agent definitions to use in planning.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

domain

protected SGDomain domain

The domain in which planning is to be performed

agentDefinitions

protected java.util.List<SGAgentType> agentDefinitions

The agent types

jointModel

protected JointModel jointModel

The joint action model to use in planning.

jointRewardFunction

protected JointRewardFunction jointRewardFunction

The joint reward function

terminalFunction

protected TerminalFunction terminalFunction

The state terminal function.

discount

protected double discount

The discount factor in [0, 1]

hashingFactory

protected HashableStateFactory hashingFactory

The state hashing factory used to query the value function for individual states

vInit

protected ValueFunction vInit

The Q-value initialization function to use.

backupOperator

protected SGBackupOperator backupOperator

The backup operating defining the solution concept to use.

qSources

protected AgentQSourceMap.HashMapAgentQSourceMap qSources

The Hash map backed multi-agent Q-source in which to store Q-values.

planningStarted

protected boolean planningStarted

Whether planning has begun or not.

Constructor Detail

MADynamicProgramming

public MADynamicProgramming()

Method Detail

initMAVF

public void initMAVF(SGDomain domain,
                     java.util.List<SGAgentType> agentDefinitions,
                     JointRewardFunction jointRewardFunction,
                     TerminalFunction terminalFunction,
                     double discount,
                     HashableStateFactory hashingFactory,
                     ValueFunction vInit,
                     SGBackupOperator backupOperator)

Initializes all the main datstructres of the value function valueFunction

Parameters:

domain - the domain in which to perform planning

agentDefinitions - the definitions of the agents involved in the planning problem.

jointRewardFunction - the joint reward function

terminalFunction - the terminal state function

discount - the discount factor

hashingFactory - the state hashing factorying to use to lookup Q-values for individual states

vInit - the value function initialization function to use

backupOperator - the solution concept backup operator to use.

hasStartedPlanning

public boolean hasStartedPlanning()

Indicates whether planning has begun or not. Once planning has begun, the agent defintions cannot be changed or a runtime exception will be thrown.

Returns:

true is planning has started; false if it has not.

setAgentDefinitions

public void setAgentDefinitions(java.util.List<SGAgentType> agentDefinitions)

Sets/changes the agent definitions to use in planning. This can only be change up until planning begins, after which a runtime exception will be thrown. To check if the planning has already begun, use the hasStartedPlanning() method.

Parameters:

agentDefinitions - the definitions of agents involve in the planning problem.

planFromState

public abstract void planFromState(State s)

Calling this method causes planning to be performed from State s.

Parameters:

s - the state from which planning is to be performed.

getQSources

public AgentQSourceMap getQSources()

Description copied from interface: MultiAgentQSourceProvider

Returns an object that can provide Q-value sources for each agent.

Specified by:

getQSources in interface MultiAgentQSourceProvider

Returns:

a AgentQSourceMap object.

backupAllValueFunctions

public double backupAllValueFunctions(State s)

Backups the state value function for all agent's value functions in state s.

Parameters:

s - the state in which the value functions should be backed up.

Returns:

the maximum value-function change from this backup.