Machine Learning in Modeling and Control

NI-AML: Advanced Machine Learning, 30. 3. 2023

Zdeněk Buk, Miroslav Čepek, Rodrigo Da Silva Alves, Vojtěch Rybář and Petr Šimánek

Outline

Motivation

Control Task

Model / Simulation

Controller Evolution

Controller Transfer

Examples:

Cart Pole Balancing (Gym, RL)

Simple Car Simulator (Neuroevolution)

Prerequisites

Neural Networks Basics

Evolutionary Algorithms (at least SGA)

Motivation

Learning to Park (Audi Electronic Ventures & NNAISENSE, NIPS 2016)

https://www.youtube.com/watch?v=C01qVU5E1OI

Bionic Softhand (Festo & NNAISENSE, Hannover Messe 2019)

https://www.youtube.com/watch?v=kavHcgITpcI

Problem Overview

Control

Control is a process that involves two components: an environment and a controller. At any given instant, the environment is in one of a potentially infinite number of states

(Subscript[s, t])∈S⊂(ℝ^n),

and the dynamics of the environment are governed by some arbitrary, unknown function f

(Subscript[s, t+1])=f((Subscript[s, t]),(Subscript[a, t])),

of the state (Subscript[s, t]) and the action

(Subscript[a, t])∈A⊂(ℝ^m)

The action is generated by the controller according to:

(Subscript[a, t])=π((Subscript[o, t])), (Subscript[o, t])=h((Subscript[s, t])),

where π is the control function of policy, and (Subscript[o, t]) is the controller’s observation of the state.

Controller’s Behavior

Ω represents set of all possible trajectories that can occur as a consequence of all possible sequences of actions (Subscript[a, t]) starting in (Subscript[s, 0])

The subset of Ω that is generated by a particular controller will be referred to as the controller’s behavior.

The objective of control is to find a controller whose behavior solves a particular task.

Controller Adaptation

Reinforcement Learning

Reinforcement learning (RL) refers to a class of algorithms for solving problems in which a sequence of decisions must be made to maximize a reward or reinforcement received from the environment.

At each decision point, the learning agent in state s∈S, selects an action a∈A that transitions the environment to the next state s′ and imparts a reinforcement signal, r, to the agent.

The learning process is difficult because, unlike in supervised learning tasks, the desired response in each state is not known in advance.

Essential to RL methods is the value function V which maps each problem state to its utility or value with respect to the task being learned.

Neuroevolution

Neuroevolution presents a fundamentally different approach to reinforcement learning tasks.

Neuroevolution leverages the strengths of two biologically inspired areas of artificial intelligence:

Artificial Neural Networks and

Genetic Algorithms.

The basic idea of NE is to search the space of neural network policies directly by using a genetic algorithm.

Learn Model

Evolve Using Model

Evolve Using Model / Transfer

Sensor noise

(Subscript[s, t+1])=Φ((Subscript[s, t]),π((Subscript[s, t])+v)), where v∈(ℝ^(|s|)) is a uniformly distributed random vector

Trajectory noise

(Subscript[s, t+1])=Φ((Subscript[s, t]),π((Subscript[s, t])))+w, where w∈(ℝ^(|s|)) is a uniformly distributed random vector

Transfer Controller

Example: Cart Pole

https://www.gymlibrary.dev/environments/classic_control/cart_pole/

Action Space: Discrete {Left, Right}

Observation Shape: {4}

Observation Space

Cart Position: {-4.8, 4.81}

Cart Velocity: {-∞, ∞}

Pole Angle: {-0.418 rad (-24°), 0.418 rad (24°)}

Pole Angular Velocity: {-∞, ∞}

Interface:

DeviceOpen, DeviceClose

DeviceRead

Device Execute

env=DeviceOpen["SimulatedCartPole"]

DeviceRead[env]

DeviceExecute[env,"Render"]

action=DeviceExecute[env,"RandomAction"]

DeviceExecute[env,"Step",action]

DeviceExecute[env,"Render"]

DeviceClose[env]

Example: Learn Model

Training Data

AbsoluteTiming[actions=RandomChoice[{Left,Right},200000];]

env=DeviceOpen["SimulatedCartPole"]
AbsoluteTiming[
observationsAll=Map[{
DeviceRead[env]["ObservedState"],
#,
DeviceExecute[env,"Step",#]
}&,actions];
]
DeviceClose[env]

(*observationsAll=Import["model-2M-observations.wxf"];*)

(* Filter out the states at the end of episodes *)
observations={#[[1]],#[[2]],#[[3]]["ObservedState"]}&/@DeleteCases[observationsAll,{{_,_,_,_},_Symbol,<|___,"Ended"->True,___|>}];
Length[observations]

observations[[1]]

Riffle[#,"⇒"]&/@RandomSample[observations,3]//TableForm[#,TableHeadings->{None,{"(Subscript[o, t])","","(Subscript[a, t])","","(Subscript[o, t + 1])"}}]&

With[{r=0.1},Graph[DirectedEdge[Round[#[[1]],r],Round[#[[3]],r],#[[2]]]&/@ observations[[;;200]],EdgeLabels->"EdgeTag"]]

observations[[1;;10,1,{1,3}]]

Graphics[{PointSize[Tiny],Point[observations[[1;;200000,1,{1,3}]]]},Frame->True,FrameLabel->{"Cart Position","Pole Angle"}]
Graphics[{PointSize[Tiny],Point[observations[[1;;200000,1,{2,4}]]]},Frame->True,FrameLabel->{"Cart Velocity","Pole Angular Velocity"}]

trainingData=<|"State"->#[[1]],"Action"->#[[2]],"Output"->#[[3]]|>&/@observations;
Length[trainingData]
RandomSample[trainingData,3]

(*Export["model-2M-observations.wxf",observationsAll]
Export["model-2M-observations_filtered.wxf",observations]
Export["model-2M-training_data.wxf",trainingData]*)

Model Definition & Training

model=NetGraph[<|
"net"->NetChain[{CatenateLayer[],LinearLayer[16],Tanh,4}]
|>,
{NetPort["State"]->"net",NetPort["Action"]->"net"},
"State"->{4},
"Action"->NetEncoder[{"Class",{Left,Right},"IndicatorVector"}]
]

res=NetTrain[model,trainingData,All,ValidationSet->Scaled[0.1],MaxTrainingRounds->100,TargetDevice->"CPU",BatchSize->512]

(*res=Import["model-2M-training_results.wxf"]*)

(*Export["model-2M-training_results.wxf",res]
Export["model-2M.wlnet",res["TrainedNet"]]*)

net=res["TrainedNet"]

Simulator Wrapper

ClearAll[randomState]
randomState[]:=RandomReal[{-0.05,0.05},4]

Clear[endQ]
endQ[state_List]:=Or[
		(* Pole Angle *) Abs[state[[3]]]>0.2,
		(* Cart Position *)Abs[state[[1]]]>2.4
	] (* Ignoring the episode length truncation *)
ClearAll[episodeEndCheck]
episodeEndCheck[environment_Association]:=Append[environment,<|"Ended"->#,"ObservedState"->If[#,randomState[],environment["ObservedState"]]|>&@endQ[environment["ObservedState"]]]
SetAttributes[episodeEndCheck,HoldFirst]

ClearAll[simulatedEnvironment]
simulatedEnvironment[model_NetChain|model_NetGraph]:=<|"Model"->model,"Data"-><|"ObservedState"->randomState[]|>|>

ClearAll[gymEnvironment]
gymEnvironment[env_DeviceObject]:=<|"Model"->env,"Data"->DeviceExecute[env,"Reset"]|>

ClearAll[reset,step]
SetAttributes[{reset,step},HoldFirst]
(* Gym Wrapper*)
reset[environment_]:=AppendTo[environment,<|"Data"->DeviceExecute[Evaluate[environment]["Model"],"Reset"]|>]/;MatchQ[Evaluate[environment]["Model"],_DeviceObject]
step[environment_,action_]:=AppendTo[environment,<|"Data"->DeviceExecute[Evaluate[environment]["Model"],"Step",action]|>]/;MatchQ[Evaluate[environment]["Model"],_DeviceObject]
(* NN-model Wrapper*)
reset[environment_]:=AppendTo[environment,<|"Data"-><|"ObservedState"->randomState[]|>|>]/;MatchQ[Evaluate[environment]["Model"],_NetChain|_NetGraph]
step[environment_,action_]:=AppendTo[environment,<|
"Data"->episodeEndCheck@<|
"ObservedState"->environment["Model"][<|"State"->Evaluate[environment]["Data","ObservedState"],"Action"->action|>],
"Reward"->1
|>
|>]/;MatchQ[Evaluate[environment]["Model"],_NetChain|_NetGraph]

ClearAll[render]
render[data_List,action_:None]:=Graphics[
		{
			Black,Thickness[Medium],
			Line[{{-4.8,0},{4.8,0}}],
			Green,Opacity[0.75],EdgeForm[Black],
			Rectangle[{#[[1]]-0.4,-0.25},{#[[1]]+0.4,0.25}],
			Opacity[1],Blue,Thickness[0.015],
			Line[{{#[[1]],0},{#[[1]]+2Cos[π/2-#[[3]]],2Sin[π/2-#[[3]]]}}],
			PointSize[0.025],Red,Point[{#[[1]],0}],
			Text[action/.{Left->"←",Right->"→",None->""},{#[[1]],-0.4}]
		},
		PlotRange->{{-5,5},{-0.5,2.5}}
	]&@data
render[environment_Association,action_:None]:=render[environment["Data","ObservedState"],action]

envModel=simulatedEnvironment[net]

envGym=gymEnvironment[DeviceOpen["SimulatedCartPole"]]

reset[envModel]

step[envModel,Left]

Example: Reinforcement Learning

policyNet=NetInitialize@NetChain[{32,Tanh,8,Tanh,2,SoftmaxLayer[]},"Input"->4,"Output"->NetDecoder[{"Class",{Left,Right}}]]

reinforceLoss=NetGraph[{CrossEntropyLossLayer["Index"],ThreadingLayer[#1*#2&]},{NetPort["DiscountedReward"]->2,{NetPort["Input"],NetPort["Action"]}->1->2},"Action"->NetEncoder[{"Class",{Left,Right}}]]

discountedSum[rewards_List,gamma_]:=Dot[rewards,gamma^Range[0,Length[rewards]-1]];
applyDiscount[rewards_List,gamma_]:=Table[discountedSum[Drop[rewards,i-1],gamma],{i,Length@rewards}]

ClearAll[CollectEpisodes]
SetAttributes[CollectEpisodes,HoldFirst]
CollectEpisodes[env_,policyNet_,totalNumberSteps_Integer,maxStepsPerEp_Integer,discountFactor_]:=Module[{action,actions,epActions,observedState,observedStates,epObservedStates,reward,rewards,discountedRewards,epDiscountedRewards,ended,currentSteps},epActions={};epObservedStates={};epDiscountedRewards={};
currentSteps=0;
(*episode loop*)
While[currentSteps<totalNumberSteps,
observedStates={};rewards={};actions={};
(*start each new episode with a fresh environment*)observedState=reset[env]["Data"]["ObservedState"];
(*step loop*)
Do[
AppendTo[observedStates,observedState];
action=policyNet[observedState];
{observedState,reward,ended}=Lookup[step[env,action]["Data"],{"ObservedState","Reward","Ended"}];
AppendTo[rewards,reward];
AppendTo[actions,action];
If[ended,Break[]];
,
Min[maxStepsPerEp,totalNumberSteps-currentSteps]
];
discountedRewards=applyDiscount[rewards,discountFactor];
epActions=Join[epActions,actions];
epObservedStates=Join[epObservedStates,observedStates];
epDiscountedRewards=Join[epDiscountedRewards,discountedRewards];
currentSteps+=Length@actions;
];
<|"Action"->epActions,"ObservedState"->epObservedStates,"DiscountedReward"->epDiscountedRewards|>
];

ClearAll[reinforceSampler]
SetAttributes[reinforceSampler,HoldFirst]
reinforceSampler[env_,net_,batchSize_]:=Block[{data},data=CollectEpisodes[env,net[#,"RandomSample"]&,batchSize,100,0.99];
data["Input"]=data["ObservedState"];
data];

reinforceSampler[envModel,policyNet,5]

results=NetTrain[policyNet,reinforceSampler[envModel,#Net,#BatchSize]&,All,LossFunction->reinforceLoss,Method->"ADAM",BatchSize->256,MaxTrainingRounds->1000,TargetDevice->"CPU",TrainingProgressMeasurements-><|"Measurement"->NetPort["DiscountedReward"],"Aggregation"->"Mean","Key"->"Reward"|>]

(*results=Import["controller-model-100-2M-training_results.wxf"]*)

trained=results["TrainedNet"];

(*Export["controller-v1-model-256b-1000r-2M.wlnet",trained]
Export["controller-v1-model-256b-1000r-2M-training_results.wxf",results]*)

reset[envGym];
ListAnimate[Table[(step[envGym,a=trained[envGym["Data"]["ObservedState"]]];
render[envGym,a]),100]]

reset[envModel];
ListAnimate[Table[(step[envModel,a=trained[envModel["Data"]["ObservedState"]]];
render[envModel,a]),100]]

Example: Neuroevolution

Controller

Fully Recurrent Neural Network

𝕎=Table[Subscript[w,i,j],{i,3},{j,6}];
MatrixForm[𝕎]

𝕎.{(Subscript[y, 1]),(Subscript[y, 2]),(Subscript[y, 3]),1,(Subscript[x, 1]),(Subscript[x, 2])}

Tanh/@(𝕎.{(Subscript[y, 1]),(Subscript[y, 2]),(Subscript[y, 3]),1,(Subscript[x, 1]),(Subscript[x, 2])})

Controller Implementation

ClearAll[evaluateCtrl];
evaluateCtrl[ctrl_,ctrlState_,inputs_]:=Module[{y},
	y=Tanh/@(ctrl.Join[ctrlState,{1},inputs]);
	{If[y[[1]]<0,Left,Right],y} (* {action, newState} *)
];

evaluateCtrl[randomController[],initCtrlState,{0.1,0.2,0.3,0.4}]

Simulation

Parameters & Initialization

numberOfNeurons=4;
numberOfInputs=4; (* Observation Space *)

initCtrlState=ConstantArray[0,numberOfNeurons];

randomController[]:=RandomReal[{-0.05,0.05},{numberOfNeurons,numberOfNeurons+numberOfInputs+1}];

Simulation Step

envGym[["Data","ObservedState"]]

ClearAll[simulationStep]
SetAttributes[simulationStep,HoldFirst]
simulationStep[env_,ctrl_,ctrlState_]:=Module[
{action,newCtrlState,observation,newObservation},
observation=env[["Data","ObservedState"]];
{action,newCtrlState}=evaluateCtrl[ctrl,ctrlState,observation];
newObservation=step[env,action];
Sow[{observation,action}];
{newCtrlState,(newObservation["Data","Ended"]/.{Missing[___]->False})}
]

ctrl=randomController[];

Reap[simulationStep[envGym,ctrl,initCtrlState]]

reset[envGym]

r=NestWhileList[simulationStep[envGym,ctrl,#[[1]]]&,{initCtrlState,False},!#[[2]]&,1,10]

(* For the simplicity we use a global variable "environment" *)
environment=envModel;

ClearAll[fitness];
fitness[ctrl_?(MatrixQ[#,NumericQ]&),maxSteps_:300]:=Module[{run},
reset[environment];
run=Rest@NestWhileList[simulationStep[environment,ctrl,#[[1]]]&,{initCtrlState,False},!#[[2]]&,1,maxSteps];
Length[run]
]

fitness[randomController[]]

Differential Evolution

𝕨=Table[ToExpression@StringTemplate["w$`1`$`2`"][m,n],{m,numberOfNeurons},{n,numberOfNeurons+numberOfInputs+1}]

Length[Flatten[𝕨]]

environment=envGym;
fitStep=0;
evoLog={};
(res=NMaximize[
fitness[𝕨,400],
Flatten[𝕨],
Method->{
"DifferentialEvolution",
"ScalingFactor"->0.6,
"CrossProbability"->0.5,
"PostProcess"->False,
"SearchPoints"->20,
"RandomSeed"->1
},
MaxIterations->100,
StepMonitor:>(fitStep++)
(*EvaluationMonitor:>(AppendTo[evoLog,{fitness[𝕨],𝕨}])*)
])//AbsoluteTiming
fitStep

(*ListLinePlot[Transpose[N[{Min[#],Mean[#],Max[#]}]&/@Partition[evoLog[[All,1]],20]],PlotTheme->"Detailed",PlotLegends->{"Min","Mean","Max"},Filling->1->{3},FrameLabel->{"Interation","Fitness"},PlotRange->All]*)

resultCtrl=𝕨/.res[[2]];
MatrixPlot[resultCtrl,PlotTheme->"Detailed"]

environment=envGym;
log=Reap[fitness[resultCtrl,800]][[2,1]];
Length[log]

ListAnimate[render[#[[1]],#[[2]]]&/@log]

References

Gomez, Faustino J. “Robust Non-Linear Control through Neuroevolution.” PhD Thesis, Department of Computer Sciences, The University of Texas at Austin, 2003. http://nn.cs.utexas.edu/?gomez:phd03.

Koutník, Jan, Jürgen Schmidhuber, and Faustino Gomez. “Online Evolution of Deep Convolutional Network for Vision-Based Reinforcement Learning.” In From Animals to Animats 13, edited by Angel P. del Pobil, Eris Chinellato, Ester Martinez-Martin, John Hallam, Enric Cervera, and Antonio Morales, 260–69. Cham: Springer International Publishing, 2014. https://doi.org/10.1007/978-3-319-08864-8_25.

Sutton, Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. Cambridge, MA, USA: A Bradford Book, 2018.

Eiben, A. E., and James E. Smith. Introduction to Evolutionary Computing. 2nd ed. Springer Publishing Company, Incorporated, 2015.

Ahmad, Mohamad Faiz, Nor Ashidi Mat Isa, Wei Hong Lim, and Koon Meng Ang. “Differential Evolution: A Recent Review Based on State-of-the-Art Works.” Alexandria Engineering Journal 61, no. 5 (2022): 3831–72. https://doi.org/10.1016/j.aej.2021.09.013.

Price, Kenneth, Rainer M. Storn, and Jouni A. Lampinen. Differential Evolution: A Practical Approach to Global Optimization (Natural Computing Series). Berlin, Heidelberg: Springer-Verlag, 2005.

Buk, Zdeněk, Jan Koutník, and Miroslav Šnorek. “NEAT in HyperNEAT Substituted with Genetic Programming.” In Adaptive and Natural Computing Algorithms, edited by Mikko Kolehmainen, Pekka Toivanen, and Bartlomiej Beliczynski, 243–52. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009. https://doi.org/10.1007/978-3-642-04921-7_25.

Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. “Playing Atari with Deep Reinforcement Learning,” 2013. https://arxiv.org/abs/1312.5602.