Create simulink integrated environment model
If the reference model has appropriate operation input ports, observation output ports and scalar reward output ports, a Simulink model containing the reference model and the RL Agent block can be automatically created. This function returns the environment object, operation specification and observation specification of the model.
createIntegratedEnv
Use the reference model as an environment to create Simulink models for reinforcement learning.
grammar
env = createIntegratedEnv(refModel,newModel)
[env,agentBlock,obsInfo,actInfo] = createIntegratedEnv(___)
[___] = createIntegratedEnv(___,Name,Value)
description
env = createIntegratedEnv(refModel, newModel) creates a Simulink® model with the name specified by newModel, and returns a reinforcement learning environment object env for this model. The new model contains an RL Agent block, and the reference model refModel is used as a reinforcement learning environment to train the agent specified by this block.
[env, agentBlock, obsInfo, actInfo] = createIntegratedEnv(___) returns the block path of the RL Agent block in the new model and the observation and action data specifications of the reference model obsInfo and actInfo.
[___] = createIntegratedEnv(___, Name, Value) uses one or more Name, Value to specify the port, observation value and operation information to create a model and environment interface.
Instance
Create Environment from a Simulink Model
This example shows how to use createIntegratedEnv to create environment objects from the Simulink model of the system that the agent needs to interact with. Such a system is usually called a factory, an open-loop system or a reference system, and the entire (integrated) system including an agent is usually called a closed-loop system.
Open the model.
open_system('rlFlyingRobotEnv');
Initialize state variables and sampling time.
% initial model state variables
theta0 = 0;
x0 = -15;
y0 = 0;
% sample time
Ts = 0.4;
Create a Simulink model IntegratedEnv, which contains a flying robot model connected to the intelligent body in a closed loop. This function also returns the reinforcement learning environment object env, which is used for training.
env=createIntegratedEnv('rlFlyingRobotEnv','IntegratedEnv')
This function can also return the block path of the RL agent block in the new integrated model, as well as the observation and operation data specifications of the reference model.
[~,agentBlk,observationInfo,actionInfo]=createIntegratedEnv('rlFlyingRobotEnv','IntegratedEnv')
This function is useful if you need to modify the description, restriction or name in observationInfo and actionInfo, and then use the function rlSimulinkEnv to create an environment from the integrated model IntegratedEnv.
Create an Integrated Environment with Specified Port Names
This example shows how to use the "name and value" pair to call the function createIntegratedEnv to create an integrated (closed loop) Simulink environment and corresponding environment objects.
The first parameter of the createIntegratedEnv function is the name of the reference Simulink model, which contains the system that interacts with the agent. This type of system is usually called a factory or open loop system. For this example, the reference system is a model of a water tank.
Open the open-loop water tank model
open_system('rlWatertankOpenloop.slx');
When the above situation occurs, click the direction of the arrow to open the example model.
Set the sampling time of the discrete integrator block used to generate observations so that the simulation can be run.
Ts=1;
Since the input port is called u (not action), and the first and third output ports are called y and stop (instead of observation and isdone), use "name and value" when calling the function createIntegratedEnv Assign the correct name to it.
env=createIntegratedEnv('rlWatertankOpenloop','IntegratedWatertank','ActionPortName','u','ObservationPortName','y','IsDonePortName','stop')
This will create a new model IntegratedWatertank, which contains a reference model connected to the agent block in a closed loop. This function also returns the reinforcement learning environment object env, which is used for training.
Input parameters
refModel — Reference model name
string | character vector
Reference model name, specified as a string or character vector. This is the Simulink model, which is used to implement the system that interacts with the agent. Such a system is usually called a factory, an open-loop system or a reference system, and the entire (integrated) system including an agent is usually called a closed-loop system. The new Simulink model uses this reference model as a dynamic model of the reinforcement learning environment.
newModel — New model name
string | character vector
New model name, specified as a string or character vector. createIntegratedEnv uses this name to create a Simulink model, but does not save the model.
Name-Value Pair Arguments
Specify an optional comma-separated name, value-parameter pair. Name is the name of the parameter, and Value is the corresponding value. The name must appear in quotation marks. You can specify multiple name and value pair parameters in any order, such as Name1, Value1, …, NameN, ValueN.
Example : "IsDonePortName", "stopSim" sets the stopSim port of the reference model as the source of the isdone signal.
‘ObservationPortName’ — Reference model observation output port name
“observation” (default) | string | character vector
ObservationPortName" and a string or character vector. When the name of the observation output port of the reference model is not "observation", please specify observationPortName.
‘ActionPortName’ — Reference model action input port name
“action” (default) | string | character vector
ActionPortName" and string or character vector. When the name of the action input port of the reference model is not "action", please specify ActionPortName.
‘IsDonePortName’ — Reference model done flag output port name
“isdone” (default) | string | character vector
Reference model completion flag output port name, specified as a comma-separated pair consisting of "IsDonePortName" and a string or character vector. When the name of the completion flag output port of the reference model is not "isdone", please specify IsDonePortName.
‘ObservationBusElementNames’ — Names of observation bus leaf elements
string array
The name of the leaf element of the observation bus for which to create a canonical, specified as a string array. To create an observation specification for a subset of elements in a Simulink bus object, specify BusElementNames. If BusElementNames is not specified, a data specification will be created for each leaf element in the bus.
ObservationBusElementNames is applicable only when the observation output port is a bus signal.
Example:'ObservationBusElementNames', ["sin""cos"] creates specifications for the observation bus elements named "sin" and "cos".
'ObservationDiscreteElements' — Finite values for observation specifications
cell array of name-value pairs
, specified as a comma-separated pair consisting of "ObservationDiscreteElements" and a cell array of name-value pairs . Each name/value pair consists of an element name and an array of discrete values.
If the observation output port of the reference model is:
-
The bus signal specifies the name of one of the bus leaf elements specified by ObservationBusElementNames .
-
For non-bus signals, specify the name of the observation port, such as ObservationPortName .
The specified discrete value must be convertible to the data type of the specified observation signal.
If you do not specify a discrete value for an observation specification element, the element is continuous.
Example :'ObservationDiscretElements', {'observation', [-1 0 1]} specifies the discrete value of the non-bus observation signal observed with the default port name.
Example :'ObservationDiscretElements',{'gear',[-1 0 1 2],'direction',[1 2 3 4]} specify the discrete values of the "gear" and "direction" leaf elements of the bus action signal.
Output parameters
env — Reinforcement learning environment
SimulinkEnvWithAgent object
The interface of the reinforcement learning environment is returned as a SimulinkEnvWithAgent object.
agentBlock — Block path to the agent block
character vector
The block path of the agent block in the new model is returned as a character vector. To train an agent in a new Simulink model, you must create an agent and specify the agent name in the RL agent block indicated by agentBlock.
For more information on creating agents, see Reinforcement Learning Agents .
obsInfo — Observation data specifications
rlNumericSpec object | rlFiniteSetSpec object | array of data specification objects
Observation data specification, returned as one of the following:
-
A single continuous observation specification rlNumericSpec object
-
RlFiniteSetSpec object for a single discrete observation specification
-
Data specification object array of multiple specifications
actInfo — Action data specifications
rlNumericSpec object | rlFiniteSetSpec object | array of data specification objects
Operation data specification, returned as one of the following:
-
RlNumericSpec object for a single continuous action specification
-
RlFiniteSetSpec object for a single discrete operation specification
-
Data specification object array for multiple action specifications