d3rlpy.ope.DiscreteFQE¶

class d3rlpy.ope.DiscreteFQE(*, algo=None, learning_rate=0.0001, optim_factory=<d3rlpy.models.optimizers.AdamFactory object>, encoder_factory='default', q_func_factory='mean', batch_size=100, n_frames=1, n_steps=1, gamma=0.99, n_critics=1, bootstrap=False, share_encoder=False, target_update_interval=100, use_gpu=False, scaler=None, action_scaler=None, impl=None, **kwargs)[source]¶

Fitted Q Evaluation for discrete action-space.

FQE is an off-policy evaluation method that approximates a Q function \(Q_\theta (s, a)\) with the trained policy \(\pi_\phi(s)\).

\[L(\theta) = \mathbb{E}_{s_t, a_t, r_{t+1} s_{t+1} \sim D} [(Q_\theta(s_t, a_t) - r_{t+1} - \gamma Q_{\theta'}(s_{t+1}, \pi_\phi(s_{t+1})))^2]\]

The trained Q function in FQE will estimate evaluation metrics more accurately than learned Q function during training.

References

Le et al., Batch Policy Learning under Constraints.

Parameters

algo (d3rlpy.algos.base.AlgoBase) – algorithm to evaluate.
learning_rate (float) – learning rate.
optim_factory (d3rlpy.models.optimizers.OptimizerFactory or str) – optimizer factory.
encoder_factory (d3rlpy.models.encoders.EncoderFactory or str) – encoder factory.
q_func_factory (d3rlpy.models.q_functions.QFunctionFactory or str) – Q function factory.
batch_size (int) – mini-batch size.
n_frames (int) – the number of frames to stack for image observation.
n_steps (int) – N-step TD calculation.
gamma (float) – discount factor.
n_critics (int) – the number of Q functions for ensemble.
bootstrap (bool) – flag to bootstrap Q functions.
share_encoder (bool) – flag to share encoder network.
target_update_interval (int) – interval to update the target network.
use_gpu (bool, int or d3rlpy.gpu.Device) – flag to use GPU, device ID or device.
scaler (d3rlpy.preprocessing.Scaler or str) – preprocessor. The available options are [‘pixel’, ‘min_max’, ‘standard’]
augmentation (d3rlpy.augmentation.AugmentationPipeline or list(str)) – augmentation pipeline.
impl (d3rlpy.metrics.ope.torch.FQEImpl) – algorithm implementation.

Methods

build_with_dataset(dataset)¶

Instantiate implementation object with MDPDataset object.

Parameters: dataset (d3rlpy.dataset.MDPDataset) – dataset.
Return type: None

build_with_env(env)¶

Instantiate implementation object with OpenAI Gym object.

Parameters: env (gym.core.Env) – gym-like environment.
Return type: None

create_impl(observation_shape, action_size)[source]¶

Instantiate implementation objects with the dataset shapes.

This method will be used internally when fit method is called.

Parameters

observation_shape (Sequence[int]) – observation shape.
action_size (int) – dimension of action-space.

Return type

None

fit(episodes, n_epochs=1000, save_metrics=True, experiment_name=None, with_timestamp=True, logdir='d3rlpy_logs', verbose=True, show_progress=True, tensorboard=True, eval_episodes=None, save_interval=1, scorers=None, shuffle=True)¶

Trains with the given dataset.

algo.fit(episodes)

Parameters

episodes (List[d3rlpy.dataset.Episode]) – list of episodes to train.
n_epochs (int) – the number of epochs to train.
save_metrics (bool) – flag to record metrics in files. If False, the log directory is not created and the model parameters are not saved during training.
experiment_name (Optional[str]) – experiment name for logging. If not passed, the directory name will be {class name}_{timestamp}.
with_timestamp (bool) – flag to add timestamp string to the last of directory name.
logdir (str) – root directory name to save logs.
verbose (bool) – flag to show logged information on stdout.
show_progress (bool) – flag to show progress bar for iterations.
tensorboard (bool) – flag to save logged information in tensorboard (additional to the csv data)
eval_episodes (Optional[List[d3rlpy.dataset.Episode]]) – list of episodes to test.
save_interval (int) – interval to save parameters.
scorers (Optional[Dict[str, Callable[[Any, List[d3rlpy.dataset.Episode]], float]]]) – list of scorer functions used with eval_episodes.
shuffle (bool) – flag to shuffle transitions on each epoch.

Return type

None

fit_batch_online(env, buffer=None, explorer=None, n_epochs=1000, n_steps_per_epoch=1000, n_updates_per_epoch=1000, eval_interval=10, eval_env=None, eval_epsilon=0.0, save_metrics=True, save_interval=1, experiment_name=None, with_timestamp=True, logdir='d3rlpy_logs', verbose=True, show_progress=True, tensorboard=True, timelimit_aware=True)¶

Start training loop of batch online deep reinforcement learning.

Parameters

env (d3rlpy.envs.batch.BatchEnv) – gym-like environment.
buffer (Optional[d3rlpy.online.buffers.BatchBuffer]) – replay buffer.
explorer (Optional[d3rlpy.online.explorers.Explorer]) – action explorer.
n_epochs (int) – the number of epochs to train.
n_steps_per_epoch (int) – the number of steps per epoch.
update_interval – the number of steps per update.
n_updates_per_epoch (int) – the number of updates per epoch.
eval_interval (int) – the number of epochs before evaluation.
eval_env (Optional[gym.core.Env]) – gym-like environment. If None, evaluation is skipped.
eval_epsilon (float) – \(\epsilon\)-greedy factor during evaluation.
save_metrics (bool) – flag to record metrics. If False, the log directory is not created and the model parameters are not saved.
save_interval (int) – the number of epochs before saving models.
experiment_name (Optional[str]) – experiment name for logging. If not passed, the directory name will be {class name}_online_{timestamp}.
with_timestamp (bool) – flag to add timestamp string to the last of directory name.
logdir (str) – root directory name to save logs.
verbose (bool) – flag to show logged information on stdout.
show_progress (bool) – flag to show progress bar for iterations.
tensorboard (bool) – flag to save logged information in tensorboard (additional to the csv data)
timelimit_aware (bool) – flag to turn terminal flag False when TimeLimit.truncated flag is True, which is designed to incorporate with gym.wrappers.TimeLimit.

Return type

None

fit_online(env, buffer=None, explorer=None, n_steps=1000000, n_steps_per_epoch=10000, update_interval=1, update_start_step=0, eval_env=None, eval_epsilon=0.0, save_metrics=True, save_interval=1, experiment_name=None, with_timestamp=True, logdir='d3rlpy_logs', verbose=True, show_progress=True, tensorboard=True, timelimit_aware=True)¶

Start training loop of online deep reinforcement learning.

Parameters

env (gym.core.Env) – gym-like environment.
buffer (Optional[d3rlpy.online.buffers.Buffer]) – replay buffer.
explorer (Optional[d3rlpy.online.explorers.Explorer]) – action explorer.
n_steps (int) – the number of total steps to train.
n_steps_per_epoch (int) – the number of steps per epoch.
update_interval (int) – the number of steps per update.
update_start_step (int) – the steps before starting updates.
eval_env (Optional[gym.core.Env]) – gym-like environment. If None, evaluation is skipped.
eval_epsilon (float) – \(\epsilon\)-greedy factor during evaluation.
save_metrics (bool) – flag to record metrics. If False, the log directory is not created and the model parameters are not saved.
save_interval (int) – the number of epochs before saving models.
experiment_name (Optional[str]) – experiment name for logging. If not passed, the directory name will be {class name}_online_{timestamp}.
with_timestamp (bool) – flag to add timestamp string to the last of directory name.
logdir (str) – root directory name to save logs.
verbose (bool) – flag to show logged information on stdout.
show_progress (bool) – flag to show progress bar for iterations.
tensorboard (bool) – flag to save logged information in tensorboard (additional to the csv data)
timelimit_aware (bool) – flag to turn terminal flag False when TimeLimit.truncated flag is True, which is designed to incorporate with gym.wrappers.TimeLimit.

Return type

None

classmethod from_json(fname, use_gpu=False)¶

Returns algorithm configured with json file.

The Json file should be the one saved during fitting.

from d3rlpy.algos import Algo

# create algorithm with saved configuration
algo = Algo.from_json('d3rlpy_logs/<path-to-json>/params.json')

# ready to load
algo.load_model('d3rlpy_logs/<path-to-model>/model_100.pt')

# ready to predict
algo.predict(...)

Parameters

fname (str) – file path to params.json.
use_gpu (Optional[Union[bool, int, d3rlpy.gpu.Device]]) – flag to use GPU, device ID or device.

Returns

algorithm.

Return type

d3rlpy.base.LearnableBase

get_loss_labels()¶

Return type: List[str]

get_params(deep=True)¶

Returns the all attributes.

This method returns the all attributes including ones in subclasses. Some of scikit-learn utilities will use this method.

params = algo.get_params(deep=True)

# the returned values can be used to instantiate the new object.
algo2 = AlgoBase(**params)

Parameters: deep (bool) – flag to deeply copy objects such as impl.
Returns: attribute values in dictionary.
Return type: Dict[str, Any]

load_model(fname)¶

Load neural network parameters.

algo.load_model('model.pt')

Parameters: fname (str) – source file path.
Return type: None

predict(x)¶

Returns greedy actions.

# 100 observations with shape of (10,)
x = np.random.random((100, 10))

actions = algo.predict(x)
# actions.shape == (100, action size) for continuous control
# actions.shape == (100,) for discrete control

Parameters: x (Union[numpy.ndarray, List[Any]]) – observations
Returns: greedy actions
Return type: numpy.ndarray

predict_value(x, action, with_std=False)¶

Returns predicted action-values.

# 100 observations with shape of (10,)
x = np.random.random((100, 10))

# for continuous control
# 100 actions with shape of (2,)
actions = np.random.random((100, 2))

# for discrete control
# 100 actions in integer values
actions = np.random.randint(2, size=100)

values = algo.predict_value(x, actions)
# values.shape == (100,)

values, stds = algo.predict_value(x, actions, with_std=True)
# stds.shape  == (100,)

Parameters

x (Union[numpy.ndarray, List[Any]]) – observations
action (Union[numpy.ndarray, List[Any]]) – actions
with_std (bool) – flag to return standard deviation of ensemble estimation. This deviation reflects uncertainty for the given observations. This uncertainty will be more accurate if you enable bootstrap flag and increase n_critics value.

Returns

predicted action-values

Return type

Union[numpy.ndarray, Tuple[numpy.ndarray, numpy.ndarray]]

sample_action(x)¶

Returns sampled actions.

The sampled actions are identical to the output of predict method if the policy is deterministic.

Parameters: x (Union[numpy.ndarray, List[Any]]) – observations.
Returns: sampled actions.
Return type: numpy.ndarray

save_model(fname)¶

Saves neural network parameters.

algo.save_model('model.pt')

Parameters: fname (str) – destination file path.
Return type: None

save_params(logger)¶

Saves configurations as params.json.

Parameters: logger (d3rlpy.logger.D3RLPyLogger) – logger object.
Return type: None

save_policy(fname, as_onnx=False)¶

Save the greedy-policy computational graph as TorchScript or ONNX.

# save as TorchScript
algo.save_policy('policy.pt')

# save as ONNX
algo.save_policy('policy.onnx', as_onnx=True)

The artifacts saved with this method will work without d3rlpy. This method is especially useful to deploy the learned policy to production environments or embedding systems.