Skip to content

Commit

Permalink
Merge pull request #2 from hill-a/master
Browse files Browse the repository at this point in the history 
update master
  • Loading branch information
@XMaster96
XMaster96 authored Jul 20, 2019
2 parents 004e3fb + dc31d83 commit b9e0fc0
Show file tree
Hide file tree
Showing 65 changed files with 2,461 additions and 2,425 deletions.
17 changes: 16 additions & 1 deletion .github/ISSUE_TEMPLATE/issue-template.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,31 @@ about: How to create an issue for this repository

---

**Important Note: We do not do technical support, nor consulting** and don't answer personal questions per email.

If you have any questions, feel free to create an issue with the tag [question].
If you wish to suggest an enhancement or feature request, add the tag [feature request].
If you are submitting a bug report, please fill in the following details.
If you are submitting a bug report, please fill in the following details.

**Describe the bug**
A clear and concise description of what the bug is.

**Code example**
Please try to provide a minimal example to reproduce the bug. Error messages and stack traces are also helpful.

Please use the [markdown code blocks](https://help.github.com/en/articles/creating-and-highlighting-code-blocks)
for both code and stack traces.

```python
from stable_baselines import ...

```

```bash
Traceback (most recent call last): File ...

```

**System Info**
Describe the characteristic of your environment:
* Describe how the library was installed (pip, docker, source, ...)
Expand Down
48 changes: 27 additions & 21 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ You can read a detailed presentation of Stable Baselines in the [Medium article]

These algorithms will make it easier for the research community and industry to replicate, refine, and identify new ideas, and will create good baselines to build projects on top of. We expect these tools will be used as a base around which new ideas can be added, and as a tool for comparing a new approach against existing ones. We also hope that the simplicity of these tools will allow beginners to experiment with a more advanced toolset, without being buried in implementation details.

**Note: despite its simplicity of use, Stable Baselines (SB) assumes you have some knowledge about Reinforcement Learning (RL).** You should not utilize this library without some practice. To that extent, we provide good resources in the [documentation](https://stable-baselines.readthedocs.io/en/master/guide/rl.html) to get started with RL.

## Main differences with OpenAI Baselines

This toolset is a fork of OpenAI Baselines, with a major structural refactoring, and code cleanups:
Expand All @@ -28,26 +30,25 @@ This toolset is a fork of OpenAI Baselines, with a major structural refactoring,
| Common interface | :heavy_check_mark: | :heavy_minus_sign: <sup>(3)</sup> |
| Tensorboard support | :heavy_check_mark: | :heavy_minus_sign: <sup>(4)</sup> |
| Ipython / Notebook friendly | :heavy_check_mark: | :x: |
| PEP8 code style | :heavy_check_mark: | :heavy_minus_sign: <sup>(5)</sup> |
| PEP8 code style | :heavy_check_mark: | :heavy_check_mark: <sup>(5)</sup> |
| Custom callback | :heavy_check_mark: | :heavy_minus_sign: <sup>(6)</sup> |

<sup><sup>(1): Forked from previous version of OpenAI baselines, however missing refactoring for HER.</sup></sup><br>
<sup><sup>(1): Forked from previous version of OpenAI baselines, with now SAC in addition</sup></sup><br>
<sup><sup>(2): Currently not available for DDPG, and only from the run script. </sup></sup><br>
<sup><sup>(3): Only via the run script.</sup></sup><br>
<sup><sup>(4): Rudimentary logging of training information (no loss nor graph). </sup></sup><br>
<sup><sup>(5): WIP on OpenAI's side (you can do it OpenAI! :cat:)</sup></sup><br>
<sup><sup>(5): EDIT: you did it OpenAI! :cat:</sup></sup><br>
<sup><sup>(6): Passing a callback function is only available for DQN</sup></sup><br>

## Documentation

Documentation is available online: [https://stable-baselines.readthedocs.io/](https://stable-baselines.readthedocs.io/)

## RL Baselines Zoo: A Collection of 70+ Trained RL Agents
## RL Baselines Zoo: A Collection of 100+ Trained RL Agents

[RL Baselines Zoo](https://github.com/araffin/rl-baselines-zoo). is a collection of pre-trained Reinforcement Learning agents using
Stable-Baselines.
[RL Baselines Zoo](https://github.com/araffin/rl-baselines-zoo). is a collection of pre-trained Reinforcement Learning agents using Stable-Baselines.

It also provides basic scripts for training, evaluating agents and recording videos.
It also provides basic scripts for training, evaluating agents, tuning hyperparameters and recording videos.

Goals of this repository:

Expand All @@ -72,7 +73,7 @@ sudo apt-get update && sudo apt-get install cmake libopenmpi-dev python3-dev zli
```

#### Mac OS X
Installation of system packages on Mac requires [Homebrew](https://brew.sh). With Homebrew installed, run the follwing:
Installation of system packages on Mac requires [Homebrew](https://brew.sh). With Homebrew installed, run the following:
```bash
brew install cmake openmpi
```
Expand Down Expand Up @@ -115,6 +116,8 @@ for i in range(1000):
action, _states = model.predict(obs)
obs, rewards, dones, info = env.step(action)
env.render()

env.close()
```

Or just train a model with a one liner if [the environment is registered in Gym](https://github.com/openai/gym/wiki/Environments) and if [the policy is registered](https://stable-baselines.readthedocs.io/en/master/guide/custom_policy.html):
Expand Down Expand Up @@ -143,25 +146,25 @@ All the following examples can be executed online using Google colab notebooks:

| **Name** | **Refactored**<sup>(1)</sup> | **Recurrent** | ```Box``` | ```Discrete``` | ```MultiDiscrete``` | ```MultiBinary``` | **Multi Processing** |
| ------------------- | ---------------------------- | ------------------ | ------------------ | ------------------ | ------------------- | ------------------ | --------------------------------- |
| A2C | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| A2C | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| ACER | :heavy_check_mark: | :heavy_check_mark: | :x: <sup>(5)</sup> | :heavy_check_mark: | :x: | :x: | :heavy_check_mark: |
| ACKTR | :heavy_check_mark: | :heavy_check_mark: | :x: <sup>(5)</sup> | :heavy_check_mark: | :x: | :x: | :heavy_check_mark: |
| DDPG | :heavy_check_mark: | :x: | :heavy_check_mark: | :x: | :x: | :x: | :x: |
| DDPG | :heavy_check_mark: | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: <sup>(4)</sup>|
| DQN | :heavy_check_mark: | :x: | :x: | :heavy_check_mark: | :x: | :x: | :x: |
| GAIL <sup>(2)</sup> | :heavy_check_mark: | :x: | :heavy_check_mark: |:heavy_check_mark:| :x: | :x: | :heavy_check_mark: <sup>(4)</sup> |
| HER <sup>(3)</sup> | :x: <sup>(5)</sup> | :x: | :heavy_check_mark: | :x: | :x: | :x: | :x: |
| HER <sup>(3)</sup> | :heavy_check_mark: | :x: | :heavy_check_mark: | :heavy_check_mark: | :x: | :heavy_check_mark:| :x: |
| PPO1 | :heavy_check_mark: | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: <sup>(4)</sup> |
| PPO2 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| SAC | :heavy_check_mark: | :x: | :heavy_check_mark: | :x: | :x: | :x: | :x: |
| TRPO | :heavy_check_mark: | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: <sup>(4)</sup> |

<sup><sup>(1): Whether or not the algorithm has be refactored to fit the ```BaseRLModel``` class.</sup></sup><br>
<sup><sup>(2): Only implemented for TRPO.</sup></sup><br>
<sup><sup>(3): Only implemented for DDPG.</sup></sup><br>
<sup><sup>(3): Re-implemented from scratch</sup></sup><br>
<sup><sup>(4): Multi Processing with [MPI](https://mpi4py.readthedocs.io/en/stable/).</sup></sup><br>
<sup><sup>(5): TODO, in project scope.</sup></sup>

NOTE: Soft Actor-Critic (SAC) was not part of the original baselines.
NOTE: Soft Actor-Critic (SAC) was not part of the original baselines and HER was reimplemented from scratch.

Actions ```gym.spaces```:
* ```Box```: A N-dimensional box that containes every point in the action space.
Expand Down Expand Up @@ -190,20 +193,23 @@ please tell us when if you want your project to appear on this page ;)
To cite this repository in publications:

```
@misc{stable-baselines,
author = {Hill, Ashley and Raffin, Antonin and Ernestus, Maximilian and Gleave, Adam and Traore, Rene and Dhariwal, Prafulla and Hesse, Christopher and Klimov, Oleg and Nichol, Alex and Plappert, Matthias and Radford, Alec and Schulman, John and Sidor, Szymon and Wu, Yuhuai},
title = {Stable Baselines},
year = {2018},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/hill-a/stable-baselines}},
}
@misc{stable-baselines,
author = {Hill, Ashley and Raffin, Antonin and Ernestus, Maximilian and Gleave, Adam and Traore, Rene and Dhariwal, Prafulla and Hesse, Christopher and Klimov, Oleg and Nichol, Alex and Plappert, Matthias and Radford, Alec and Schulman, John and Sidor, Szymon and Wu, Yuhuai},
title = {Stable Baselines},
year = {2018},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/hill-a/stable-baselines}},
}
```

## Maintainers

Stable-Baselines is currently maintained by [Ashley Hill](https://github.com/hill-a) (aka @hill-a), [Antonin Raffin](https://araffin.github.io/) (aka [@araffin](https://github.com/araffin)), [Maximilian Ernestus](https://github.com/erniejunior) (aka @erniejunior) and [Adam Gleave](https://github.com/adamgleave) (@AdamGleave).

**Important Note: We do not do technical support, nor consulting** and don't answer personal questions per email.


## How To Contribute

To any interested in making the baselines better, there is still some documentation that needs to be done.
Expand Down
25 changes: 11 additions & 14 deletions docs/guide/algos.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,34 +11,31 @@ along with some useful characteristics: support for recurrent policies, discrete
.. A2C ✔️
.. ===== ======================== ========= ======= ============ ================= =============== ================
.. There is an issue with Read The Docs for building the table when the "HER" row is present:
.. Apparently a problem of spacing
.. HER [#f3]_ ❌ [#f5]_ ❌ ✔️ ❌ ❌
============ ======================== ========= =========== ============ ================
Name Refactored [#f1]_ Recurrent ``Box`` ``Discrete`` Multi Processing
============ ======================== ========= =========== ============ ================
A2C ✔️ ✔️ ✔️ ✔️ ✔️
ACER ✔️ ✔️ ❌ [#f5]_ ✔️ ✔️
ACKTR ✔️ ✔️ ❌ [#f5]_ ✔️ ✔️
DDPG ✔️ ❌ ✔️ ❌ ❌
ACER ✔️ ✔️ ❌ [#f4]_ ✔️ ✔️
ACKTR ✔️ ✔️ ❌ [#f4]_ ✔️ ✔️
DDPG ✔️ ❌ ✔️ ❌ ✔️ [#f3]_
DQN ✔️ ❌ ❌ ✔️ ❌
GAIL [#f2]_ ✔️ ✔️ ✔️ ✔️ ✔️ [#f4]_
PPO1 ✔️ ❌ ✔️ ✔️ ✔️ [#f4]_
HER ✔️ ❌ ✔️ ✔️ ❌
GAIL [#f2]_ ✔️ ✔️ ✔️ ✔️ ✔️ [#f3]_
PPO1 ✔️ ❌ ✔️ ✔️ ✔️ [#f3]_
PPO2 ✔️ ✔️ ✔️ ✔️ ✔️
SAC ✔️ ❌ ✔️ ❌ ❌
TRPO ✔️ ❌ ✔️ ✔️ ✔️ [#f4]_
TRPO ✔️ ❌ ✔️ ✔️ ✔️ [#f3]_
============ ======================== ========= =========== ============ ================

.. [#f1] Whether or not the algorithm has be refactored to fit the ``BaseRLModel`` class.
.. [#f2] Only implemented for TRPO.
.. [#f3] Only implemented for DDPG.
.. [#f4] Multi Processing with `MPI`_.
.. [#f5] TODO, in project scope.
.. [#f3] Multi Processing with `MPI`_.
.. [#f4] TODO, in project scope.
.. note::
Non-array spaces such as `Dict` or `Tuple` are not currently supported by any algorithm.
Non-array spaces such as `Dict` or `Tuple` are not currently supported by any algorithm,
except HER for dict when working with gym.GoalEnv

Actions ``gym.spaces``:

Expand Down
Loading

0 comments on commit b9e0fc0

Please sign in to comment.