d3rlpy.dataset.MixedReplayBuffer¶

class d3rlpy.dataset.MixedReplayBuffer(primary_replay_buffer, secondary_replay_buffer, secondary_mix_ratio)[source]¶

A class combining two replay buffer instances.

This replay buffer implementation combines two replay buffers (e.g. offline buffer and online buffer). The primary replay buffer is exposed to methods such as append. Mini-batches are sampled from each replay buffer based on secondary_mix_ratio.

import d3rlpy

# offline dataset
dataset, env = d3rlpy.datasets.get_cartpole()

# online replay buffer
online_buffer = d3rlpy.dataset.create_fifo_replay_buffer(
    limit=100000,
    env=env,
)

# combine two replay buffers
replay_buffer = d3rlpy.dataset.MixedReplayBuffer(
    primary_replay_buffer=online_buffer,
    secondary_replay_buffer=dataset,
    secondary_mix_ratio=0.5,
)

Parameters:

primary_replay_buffer (d3rlpy.dataset.ReplayBufferBase) – Primary replay buffer.
secondary_replay_buffer (d3rlpy.dataset.ReplayBufferBase) – Secondary replay buffer.
secondary_mix_ratio (float) – Ratio to sample mini-batches from the secondary replay buffer.

Methods

append(observation, action, reward)[source]¶

Appends observation, action and reward to buffer.

Parameters:

observation (Union[ndarray[Any, dtype[Any]], Sequence[ndarray[Any, dtype[Any]]]]) – Observation.
action (Union[int, ndarray[Any, dtype[Any]]]) – Action.
reward (Union[float, ndarray[Any, dtype[Any]]]) – Reward.

Return type:

None

append_episode(episode)[source]¶

Appends episode to buffer.

Parameters:: episode (EpisodeBase) – Episode.
Return type:: None

clip_episode(terminated)[source]¶

Clips current episode.

Parameters:: terminated (bool) – Flag to represent environmental termination. This flag should be False if the episode is terminated by timeout.
Return type:: None

dump(f)[source]¶

Dumps buffer data.

with open('dataset.h5', 'w+b') as f:
    replay_buffer.dump(f)

Parameters:: f (BinaryIO) – IO object to write to.
Return type:: None

classmethod from_episode_generator(episode_generator, buffer, transition_picker=None, trajectory_slicer=None, writer_preprocessor=None)[source]¶

Builds ReplayBuffer from episode generator.

Parameters:

episode_generator (EpisodeGeneratorProtocol) – Episode generator implementation.
buffer (BufferProtocol) – Buffer implementation.
transition_picker (Optional[TransitionPickerProtocol]) – Transition picker implementation for Q-learning-based algorithms.
trajectory_slicer (Optional[TrajectorySlicerProtocol]) – Trajectory slicer implementation for Transformer-based algorithms.
writer_preprocessor (Optional[WriterPreprocessProtocol]) – Writer preprocessor implementation.

Returns:

Replay buffer.

Return type:

ReplayBuffer

classmethod load(f, buffer, episode_cls=<class 'd3rlpy.dataset.components.Episode'>, transition_picker=None, trajectory_slicer=None, writer_preprocessor=None)[source]¶

Builds ReplayBuffer from dumped data.

This method reconstructs replay buffer dumped by dump method.

with open('dataset.h5', 'rb') as f:
    replay_buffer = ReplayBuffer.load(f, buffer)

Parameters:

f (BinaryIO) – IO object to read from.
buffer (BufferProtocol) – Buffer implementation.
episode_cls (Type[EpisodeBase]) – Eisode class used to reconstruct data.
transition_picker (Optional[TransitionPickerProtocol]) – Transition picker implementation for Q-learning-based algorithms.
trajectory_slicer (Optional[TrajectorySlicerProtocol]) – Trajectory slicer implementation for Transformer-based algorithms.
writer_preprocessor (Optional[WriterPreprocessProtocol]) – Writer preprocessor implementation.

Returns:

Replay buffer.

Return type:

ReplayBuffer

sample_trajectory(length)[source]¶

Samples a partial trajectory.

Parameters:: length (int) – Length of partial trajectory.
Returns:: Partial trajectory.
Return type:: PartialTrajectory

sample_trajectory_batch(batch_size, length)[source]¶

Samples a mini-batch of partial trajectories.

Parameters:

batch_size (int) – Mini-batch size.
length (int) – Length of partial trajectories.

Returns:

Mini-batch.

Return type:

TrajectoryMiniBatch

sample_transition()[source]¶

Samples a transition.

Returns:: Transition.
Return type:: Transition

sample_transition_batch(batch_size)[source]¶

Samples a mini-batch of transitions.

Parameters:: batch_size (int) – Mini-batch size.
Returns:: Mini-batch.
Return type:: TransitionMiniBatch

size()[source]¶

Returns number of episodes.

Returns:: Number of episodes.
Return type:: int

Attributes

buffer¶

dataset_info¶

episodes¶

primary_replay_buffer¶

secondary_replay_buffer¶

trajectory_slicer¶

transition_count¶

transition_picker¶