Getting Started

This tutorial is also available on Google Colaboratory

Install

First of all, let’s install d3rlpy on your machine:

$ pip install d3rlpy

See more information at Installation.

Note

If core dump error occurs in this tutorial, please try Install from source.

Note

d3rlpy supports Python 3.7+. Make sure which version you use.

Note

If you use GPU, please setup CUDA first.

Prepare Dataset

You can make your own dataset without any efforts. In this tutorial, let’s use integrated datasets to start. If you want to make a new dataset, see Replay Buffer.

d3rlpy provides suites of datasets for testing algorithms and research. See more documents at Datasets.

from d3rlpy.datasets import get_cartpole # CartPole-v1 dataset
from d3rlpy.datasets import get_pendulum # Pendulum-v1 dataset
from d3rlpy.datasets import get_atari    # Atari 2600 task datasets
from d3rlpy.datasets import get_d4rl     # D4RL datasets

Here, we use the CartPole dataset to instantly check training results.

dataset, env = get_cartpole()

Setup Algorithm

There are many algorithms avaiable in d3rlpy. Since CartPole is the simple task, let’s start from DQN, which is the Q-learnig algorithm proposed as the first deep reinforcement learning algorithm.

from d3rlpy.algos import DQNConfig

# if you don't use GPU, set device=None instead.
dqn = DQNConfig().create(device="cuda:0")

# initialize neural networks with the given observation shape and action size.
# this is not necessary when you directly call fit or fit_online method.
dqn.build_with_dataset(dataset)

See more algorithms and configurations at Algorithms.

Setup Metrics

Collecting evaluation metrics is important to train algorithms properly. d3rlpy provides Evaluator classes to compute evaluation metrics.

from d3rlpy.metrics import TDErrorEvaluator

# calculate metrics with training dataset
td_error_evaluator = TDErrorEvaluator(episodes=dataset.episodes)

Since evaluating algorithms without access to environment is still difficult, the algorithm can be directly evaluated with EnvironmentEvaluator if the environment is available to interact.

from d3rlpy.metrics import EnvironmentEvaluator

# set environment in scorer function
env_evaluator = EnvironmentEvaluator(env)

# evaluate algorithm on the environment
rewards = env_evaluator(dqn, dataset=None)

See more metrics and configurations at Metrics.

Start Training

Now, you have everything to start offline training.

dqn.fit(
    dataset,
    n_steps=10000,
    evaluators={
        'td_error': td_error_evaluator,
        'environment': env_evaluator,
    },
)

See more about logging at Logging.

Once the training is done, your algorithm is ready to make decisions.

observation, _ = env.reset()

# return actions based on the greedy-policy
action = dqn.predict(np.expand_dims(observation, axis=0))

# estimate action-values
value = dqn.predict_value(np.expand_dims(observation, axis=0), action)

Save and Load

d3rlpy provides several ways to save trained models.

import d3rlpy

# save full parameters and configurations in a single file.
dqn.save('dqn.d3')
# load full parameters and build algorithm
dqn2 = d3rlpy.load_learnable("dqn.d3")

# save full parameters only
dqn.save_model('dqn.pt')
# load full parameters with manual setup
dqn3 = DQN()
dqn3.build_with_dataset(dataset)
dqn3.load_model('dqn.pt')

# save the greedy-policy as TorchScript
dqn.save_policy('policy.pt')
# save the greedy-policy as ONNX
dqn.save_policy('policy.onnx')

See more information at After Training Policies (Save and Load).