burlap.behavior.stochasticgames.madynamicprogramming.policies

Class EGreedyMaxWellfare

java.lang.Object

burlap.behavior.stochasticgames.JointPolicy

burlap.behavior.stochasticgames.madynamicprogramming.MAQSourcePolicy

burlap.behavior.stochasticgames.madynamicprogramming.policies.EGreedyMaxWellfare

All Implemented Interfaces:

EnumerablePolicy, Policy

public class EGreedyMaxWellfare
extends MAQSourcePolicy
implements EnumerablePolicy

An epsilon greedy joint policy, in which the joint aciton with the highest aggregate Q-values for each agent is returned a 1-epsilon fraction of the time and a random joint action an epsilon fraction of the time. Ties are broken deterministically (the first joint aciton with the maximum value is selected), but can be set to default to break ties randomly. The former is useful to maintain consistency between agents selecting their action indepdently from each other. This policy is typically used for CoCo-Q agents.

Author:

James MacGlashan

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	breakTiesRandomly Whether ties should be broken randomly or not.
protected double	epsilon The epsilon parameter specifying how often random joint actions are returned
protected MultiAgentQSourceProvider	qSourceProvider The multi-agent q-source provider
protected java.util.Random	rand A random object used for sampling

Fields inherited from class burlap.behavior.stochasticgames.JointPolicy

agentsInJointPolicy, agentsSynchronizedSoFar, lastSyncedState, lastSynchronizedJointAction

Constructor Summary

Constructors

Constructor and Description
EGreedyMaxWellfare(double epsilon) Initializes for a given epsilon value.
EGreedyMaxWellfare(double epsilon, boolean breakTiesRandomly) Initializes for a given epsilon value and whether to break ties randomly.
EGreedyMaxWellfare(MultiAgentQLearning actingAgent, double epsilon) Initializes for a multi-agent Q-learning object and epsilon value.
EGreedyMaxWellfare(MultiAgentQLearning actingAgent, double epsilon, boolean breakTiesRandomly) Initializes for a multi-agent Q-learning object and epsilon value.

Method Summary

Modifier and Type	Method and Description
Action	action(State s) This method will return an action sampled by the policy for the given state.
double	actionProb(State s, Action a) Returns the probability/probability density that the given action will be taken in the given state.
JointPolicy	copy() Creates a copy of this joint policy and returns it.
boolean	definedFor(State s) Specifies whether this policy is defined for the input state.
java.util.List<ActionProb>	policyDistribution(State s) This method will return action probability distribution defined by the policy.
void	setBreakTiesRandomly(boolean breakTiesRandomly) Whether to break ties randomly or deterministically.
void	setQSourceProvider(MultiAgentQSourceProvider provider) Sets the MultiAgentQSourceProvider that will be used to define this object's joint policy.
void	setTargetAgent(int agentNum) Sets the target privileged agent from which this joint policy is defined.

Methods inherited from class burlap.behavior.stochasticgames.JointPolicy

getAgentsInJointPolicy, getAgentSynchronizedActionSelection, getAllJointActions, setAgentsInJointPolicy, setAgentsInJointPolicyFromWorld, setAgentTypesInJointPolicy

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

qSourceProvider

protected MultiAgentQSourceProvider qSourceProvider

The multi-agent q-source provider

epsilon

protected double epsilon

The epsilon parameter specifying how often random joint actions are returned

rand

protected java.util.Random rand

A random object used for sampling

breakTiesRandomly

protected boolean breakTiesRandomly

Whether ties should be broken randomly or not.

Constructor Detail

EGreedyMaxWellfare

public EGreedyMaxWellfare(double epsilon)

Initializes for a given epsilon value. The set of agents for which joint actions are returned and multi-agent q-source provider will need to be set manually with the methods JointPolicy.setAgentsInJointPolicy(List), and setQSourceProvider(MultiAgentQSourceProvider) before the policy can be queried. Note that the MultiAgentQLearning and MultiAgentDPPlanningAgent agents may do this themselves. Consult the documentation to check.

Parameters:

epsilon - the fraction of the time [0, 1] that the agent selections random actions.

EGreedyMaxWellfare

public EGreedyMaxWellfare(double epsilon,
                          boolean breakTiesRandomly)

Initializes for a given epsilon value and whether to break ties randomly. The set of agents for which joint actions are returned and multi-agent q-source provider will need to be set manually with the methods JointPolicy.setAgentsInJointPolicy(List) , and setQSourceProvider(MultiAgentQSourceProvider) before the policy can be queried. Note that the MultiAgentQLearning and MultiAgentDPPlanningAgent agents may do this themselves. Consult the documentation to check.

Parameters:

epsilon - the fraction of the time [0, 1] that the agent selections random actions.

breakTiesRandomly - whether ties should be broken randomly (true) or not (false)

EGreedyMaxWellfare

public EGreedyMaxWellfare(MultiAgentQLearning actingAgent,
                          double epsilon)

Initializes for a multi-agent Q-learning object and epsilon value. The set of agents for which joint actions are to be returned must be subsequently defined with the JointPolicy.setAgentsInJointPolicy(List). Note that the MultiAgentQLearning and MultiAgentDPPlanningAgent agents may do this themselves. Consult the documentation to check.

Parameters:

actingAgent - the agent who will use this policy.

epsilon - the fraction of the time [0, 1] that the agent selections random actions.

EGreedyMaxWellfare

public EGreedyMaxWellfare(MultiAgentQLearning actingAgent,
                          double epsilon,
                          boolean breakTiesRandomly)

Initializes for a multi-agent Q-learning object and epsilon value. The set of agents for which joint actions are to be returned must be subsequently defined with the JointPolicy.setAgentsInJointPolicy(List). Note that the MultiAgentQLearning and MultiAgentDPPlanningAgent agents may do this themselves. Consult the documentation to check.

Parameters:

actingAgent - the agent who will use this policy.

epsilon - the fraction of the time [0, 1] that the agent selections random actions.

breakTiesRandomly - whether ties should be broken randomly (true) or not (false)

Method Detail

setBreakTiesRandomly

public void setBreakTiesRandomly(boolean breakTiesRandomly)

Whether to break ties randomly or deterministically. The former is useful for exploration during learning. The latter is useful to synchronize action selection for agents that must select an action indepdently from the same joint policy.

Parameters:

breakTiesRandomly - true if ties will be broken randomly; false if ties will be broken detemrinistically.

setQSourceProvider

public void setQSourceProvider(MultiAgentQSourceProvider provider)

Description copied from class: MAQSourcePolicy

Sets the MultiAgentQSourceProvider that will be used to define this object's joint policy.

Specified by:

setQSourceProvider in class MAQSourcePolicy

Parameters:

provider - the MultiAgentQSourceProvider that will be used to define this object's joint policy.

action

public Action action(State s)

Description copied from interface: Policy

This method will return an action sampled by the policy for the given state. If the defined policy is stochastic, then multiple calls to this method for the same state may return different actions. The sampling should be with respect to defined action distribution that is returned by getActionDistributionForState

Specified by:

action in interface Policy

Parameters:

s - the state for which an action should be returned

Returns:

a sample action from the action distribution; null if the policy is undefined for s

actionProb

public double actionProb(State s,
                         Action a)

Description copied from interface: Policy

Returns the probability/probability density that the given action will be taken in the given state.

Specified by:

actionProb in interface Policy

Parameters:

s - the state of interest

a - the action that may be taken in the state

Returns:

the probability/probability density

policyDistribution

public java.util.List<ActionProb> policyDistribution(State s)

Description copied from interface: EnumerablePolicy

This method will return action probability distribution defined by the policy. The action distribution is represented by a list of ActionProb objects, each which specifies a grounded action and a probability of that grounded action being taken. The returned list does not have to include actions with probability 0.

Specified by:

policyDistribution in interface EnumerablePolicy

Parameters:

s - the state for which an action distribution should be returned

Returns:

a list of possible actions taken by the policy and their probability.

definedFor

public boolean definedFor(State s)

Description copied from interface: Policy

Specifies whether this policy is defined for the input state.

Specified by:

definedFor in interface Policy

Parameters:

s - the input state to test for whether this policy is defined

Returns:

true if this policy is defined for State s, false otherwise.

setTargetAgent

public void setTargetAgent(int agentNum)

Description copied from class: JointPolicy

Sets the target privileged agent from which this joint policy is defined.

Specified by:

setTargetAgent in class JointPolicy

Parameters:

agentNum - the target agent.

copy

public JointPolicy copy()

Description copied from class: JointPolicy

Creates a copy of this joint policy and returns it. This is useful when generating different agents using the same kind of policy, but have different target agents evaluating it.

Specified by:

copy in class JointPolicy

Returns:

a copy of this joint policy.