文章目录
Unit 6: How to change the learning algorithm: Cube
| SUMMARY |
|---|
Estimated time to completion: 2 h
In this unit, we are going to see how to set up the environment in order to be able to use the OpenAI Baselines deepq algorithm. For that, we are going to use a virtual environment. We are going to use the RoboCube environment during the Unit.
| END OF SUMMARY |
|---|
Now, we are going to only change the learning script from using Qlearn to use DeepQ. We won’t change anything else. Here you will also see the power of environments. This allows you to change the learning algorith easily.
So, you already have a version of the CubeEnvironment to use with deepq:
| my_start_deepq.py |
|---|
#!/usr/bin/env python
import os
import gym
from baselines import deepq
import rospy
import rospkg
#from openai_gazebo.cartpole_stay_up import stay_up
import my_one_disk_walk
def main():
rospy.init_node('movingcube_onedisk_walk_gym', anonymous=True, log_level=rospy.WARN)
# Set the path where learned model will be saved
rospack = rospkg.RosPack()
pkg_path = rospack.get_path('my_moving_cube_pkg')
models_dir_path = os.path.join(pkg_path, "models_saved")
if not os.path.exists(models_dir_path):
os.makedirs(models_dir_path)
out_model_file_path = os.path.join(models_dir_path, "movingcube_model.pkl")
max_timesteps = rospy.get_param("/moving_cube/max_timesteps")
buffer_size = rospy.get_param("/moving_cube/buffer_size")
# We convert to float becase if we are using Ye-X notation, it sometimes treats it like a string.
lr = float(rospy.get_param("/moving_cube/lr"))
exploration_fraction = rospy.get_param("/moving_cube/exploration_fraction")
exploration_final_eps = rospy.get_param("/moving_cube/exploration_final_eps")
print_freq = rospy.get_param("/moving_cube/print_freq")
reward_task_learned = rospy.get_param("/moving_cube/reward_task_learned")
def callback(lcl, _glb):
# stop training if reward exceeds 199
aux1 = lcl['t'] > 100
aux2 = sum(lcl['episode_rewards'][-101:-1]) / 100
is_solved = aux1 and aux2 >= reward_task_learned
rospy.logdebug("aux1="+str(aux1))
rospy.logdebug("aux2="+str(aux2))
rospy.logdebug("reward_task_learned="+str(reward_task_learned))
rospy.logdebug("IS SOLVED?="+str(is_solved))
return is_solved
env = gym.make("MyMovingCubeOneDiskWalkEnv-v0")
act = deepq.learn(
env,
network='mlp',
lr=lr,
total_timesteps=max_timesteps,
buffer_size=buffer_size,
exploration_fraction=exploration_fraction,
exploration_final_eps=exploration_final_eps,
print_freq=print_freq, # how many apisodes until you print total rewards and info
param_noise=False,
callback=callback
)
env.close()
rospy.logwarn("Saving model to movingcube_model.pkl")
act.save(out_model_file_path)
if __name__ == '__main__':
main()
| END my_start_deepq.py |
|---|
As you can see, we are loading all the parameters from the param server, therefore, we will have to create a new config file to be loaded when launching this deepq script. Here you have it:
| my_one_disk_walk_openai_params_deepQ.yaml |
|---|
moving_cube: #namespace
# DeepQ Parameters
max_timesteps: 1000 # Maximum time steps of all the steps done throughout all the episodes
buffer_size: 500 # size of the replay buffer
lr: 1e-3 # learning rate for adam optimizer
exploration_fraction: 0.1 # fraction of entire training period over which the exploration rate is annealed
exploration_final_eps: 0.02 # final value of random action probability
print_freq: 1 # how often (Ex: 1 means every episode, 2 every two episode we print ) to print out training progress set to None to disable printing
reward_task_learned: 10000
# Learning General Parameters
n_actions: 5 # We have 3 actions
n_observations: 6 # We have 6 different observations
speed_step: 1.0 # Time to wait in the reset phases
init_roll_vel: 0.0 # Initial speed of the Roll Disk
roll_speed_fixed_value: 100.0 # Speed at which it will move forwards or backwards
roll_speed_increment_value: 10.0 # Increment that could be done in each step
max_distance: 2.0 # Maximum distance allowed for the RobotCube
max_pitch_angle: 0.2 # Maximum Angle radians in Pitch that we allow before terminating episode
max_yaw_angle: 0.1 # Maximum yaw angle deviation, after that it starts getting negative rewards
max_y_linear_speed: 100 # Free fall get to around 30, so we triple it
init_cube_pose:
x: 0.0
y: 0.0
z: 0.0
end_episode_points: 1000 # Points given when ending an episode
move_distance_reward_weight: 1000.0 # Multiplier for the moved distance reward, Ex: inc_d = 0.1 --> 100points
y_linear_speed_reward_weight: 1000.0 # Multiplier for moving fast in the y Axis
y_axis_angle_reward_weight: 1000.0 # Multiplier of angle of yaw, to keep it straight
| END my_one_disk_walk_openai_params_deepQ.yaml |
|---|
Just comment on the deepq related parameters; the other ones are exactly the same as before:
# DeepQ Parameters
max_timesteps: 1000 # Maximum time steps of all the steps done throughout all the episodes
buffer_size: 500 # size of the replay buffer
lr: 1e-3 # learning rate for adam optimizer
exploration_fraction: 0.1 # fraction of entire training period over which the exploration rate is annealed
exploration_final_eps: 0.02 # final value of random action probability
print_freq: 1 # how often (Ex: 1 means every episode, 2 every two episode we print ) to print out training progress set to None to disable printing
reward_task_learned: 10000
max_timesteps: This parameter allows you to set the maximum TOTAL steps throughout the entire learning process. It doesn’t matter if you do 1000 steps in 1000 episodes, or in one episode.
print_freq: We define how many episodes until we print the average reward and other learning info.
reward_task_learned: When the sum of all the rewards in that episode reach this value, it will consider the robot to have learned the task.
But this model has many more options, so please feel free to experiment. Here you have all the parameter options from the deepQ class:
Parameters
-------
env: gym.Env
environment to train on
network: string or a function
neural network to use as a q function approximator. If string, has to be one of the names of registered models in baselines.common.models
(mlp, cnn, conv_only). If a function, should take an observation tensor and return a latent variable tensor, which
will be mapped to the Q function heads (see build_q_func in baselines.deepq.models for details on that)
seed: int or None
prng seed. The runs with the same seed "should" give the same results. If None, no seeding is used.
lr: float
learning rate for adam optimizer
total_timesteps: int
number of env steps to optimizer for
buffer_size: int
size of the replay buffer
exploration_fraction: float
fraction of entire training period over which the exploration rate is annealed
exploration_final_eps: float
final value of random action probability
train_freq: int
update the model every `train_freq` steps.
set to None to disable printing
batch_size: int
size of a batched sampled from replay buffer for training
print_freq: int
how often to print out training progress
set to None to disable printing
checkpoint_freq: int
how often to save the model. This is so that the best version is restored
at the end of the training. If you do not wish to restore the best version at
the end of the training set this variable to None.
learning_starts: int
how many steps of the model to collect transitions for before learning starts
gamma: float
discount factor
target_network_update_freq: int
update the target network every `target_network_update_freq` steps.
prioritized_replay: True
if True prioritized replay buffer will be used.
prioritized_replay_alpha: float
alpha parameter for prioritized replay buffer
prioritized_replay_beta0: float
initial value of beta for prioritized replay buffer
prioritized_replay_beta_iters: int
number of iterations over which beta will be annealed from initial value
to 1.0. If set to None equals to total_timesteps.
prioritized_replay_eps: float
epsilon to add to the TD errors when updating priorities.
param_noise: bool
whether or not to use parameter space noise (https://arxiv.org/abs/1706.01905)
callback: (locals, globals) -> None
function called at every steps with state of the algorithm.
If callback returns true training stops.
load_path: str
path to load the model from. (default: None)
And now we have to follow the same steps you took for the CartPole to setup the Baselines, if you haven’t done it yet.
1. Set Up the Baselines for DeepQ
Sourcing the Python3 venv
The first thing you have to do is to source the Python 3 venv. This is due to the fact that ROS doesn’t directly support Python 3. This means that if you launch programs for ROS with libraries for Python 3, it will most likely give an error.
Execute in WebShell #1
source ~/.catkin_ws_python3/openai_venv/bin/activate
Using ROS TF for Python 3
Because we are using these ROS libraries in our code, we will need to source an special workspace in our Virtual Environment. For this course, this workspace is already provided to you, and it’s called .catkin_ws_python3. So, you will just need to source this workspace.
Execute in WebShell #1
source ~/.catkin_ws_python3/devel/setup.bash
To test that you are using Python 3 and that everythng works, you can execute the following commands:
Execute in WebShell #1
source ~/.catkin_ws_python3/openai_venv/bin/activate
source ~/.catkin_ws_python3/devel/setup.bash
Execute in WebShell #1
To test that the tf2 works with Python 3:
(openai_venv) user:~/catkin_ws$ python
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tf
>>> tf.__file__
'/home/user/.catkin_ws_python3/devel/lib/python3/dist-packages/tf/__init__.py'
>>> import tf2_py
>>> tf2_py.__file__
'/home/user/.catkin_ws_python3/devel/lib/python3/dist-packages/tf2_py/__init__.py'
>>> quit()
It should give you an output like this one, using Python 3.
How to get inside the Python3 VENV every time you open a new shell
You will have to do this very often, so here are the commands:
source ~/.catkin_ws_python3/openai_venv/bin/activate
source ~/.catkin_ws_python3/devel/setup.bash
2. Start the Learning script using DeepQ for the MyMovingCubeOneDiskWalkEnv
We just have to create a new launch file that launches the deepq Python script, instead of the Qlearn learning script:
| my_start_training_deepq_version.launch |
|---|
<launch>
<rosparam command="load" file="$(find my_moving_cube_pkg)/config/my_one_disk_walk_openai_params_deepQ.yaml" />
<!-- Launch the training system -->
<node pkg="my_moving_cube_pkg" name="movingcube_gym" type="my_start_deepq.py" output="screen"/>
</launch>
As you can see, we didn’t change much.
| END my_start_training_deepq_version.launch |
|---|
To start the learning, get inside the venv and launch the deepq launch.
source ~/.catkin_ws_python3/openai_venv/bin/activate
source ~/.catkin_ws_python3/devel/setup.bash
cd /home/user/catkin_ws
rm -rf build devel
catkin_make
source devel/setup.bash
roslaunch my_moving_cube_pkg my_start_training_deepq_version.launch
You should get something like this:
When the total steps done reaches the maximum, you have established or reached the reward for the task to be considered learned. Now, the system will automatically create a pickle file for you, with the model of the learned task inside.
| EXTRA exercise U3.1 |
|---|
Change the parameter so that:
- It spends more time learning
- It stops learning when it reaches a lower reward average
- It prints the learning information less frequently.
| END EXTRA exercise U3.1 |
|---|
And that’s it! You now know how to reuse Task Environments for different learning algorithms.
