Reinforcement‑learning level control for a non‑minimum‑phase propane refrigeration loop.
A deep‑RL controller that holds the refrigeration economizer flash‑drum level steady through the inverse‑response dynamics that defeat fixed‑gain PID, stabilizing the loop and cutting the manual intervention it used to demand. Operators engage it bumplessly to drive the valve directly while the DCS PID block tracks underneath, closed‑loop on the existing Foxboro DCS over OPC‑UA.
The economizer (a flash drum) sits in the propane refrigeration circuit of a cryogenic gas‑processing plant: it separates two‑phase propane, returns vapor to compressor suction, and feeds liquid refrigerant downstream. The level loop manipulates the liquid‑outlet valve to hold vessel level, but the loop is non‑minimum phase, opening the valve first drops level, then delayed flashing raises it. That inverse response, coupled to upstream propane pressure and reboiler load, makes a single PID gain either too aggressive or too sluggish, so operators kept stepping in by hand.
Ai‑OPs trained a Koios PPO deep‑reinforcement‑learning controller (an AiMV policy) in a first‑principles refrigeration simulator to learn the loop's delayed, nonlinear response. Across 13 held‑out validation episodes it held level to a 2.03% mean tracking error (median 2.04%), with 10 of 13 episodes below 2.5%, while a 40× move penalty kept valve action calm. Results are offline simulator validation; plant shadow‑mode trials and operator review precede closed‑loop use.
Ai-OPs
·Refrigeration RL Level Control01
Background & objective
The objective was to stabilize a difficult inverse‑response level loop and reduce the manual intervention it demanded, without adding instrumentation and without giving up the operator's safety net.
The refrigeration economizer, a flash drum, takes a two‑phase propane stream, sends vapor back to compressor suction or interstage, and withdraws liquid refrigerant from the bottom for downstream loads. The level controller manipulates that liquid‑outlet valve to keep vessel level at setpoint. Hold level too low and you starve downstream cooling; too high and you risk carryover to the compressor.
Why the loop is hard: non‑minimum phase
- Open the valve, level drops first: pulling liquid out of the drum lowers level, the expected direction.
- Then the pressure shifts: the same move alters pressure and enthalpy in the vessel, driving more flashing and vapor generation.
- Level rises back up: the delayed vaporization increases inflow and condensation, so level climbs after the initial drop, an inverse response.
- One gain can't win: tuned for the fast drop the PID over‑reacts; tuned for the slow rebound it lags, so operators take the loop to manual during pressure and reboiler‑load swings.
A fixed‑gain PID has one response curve. This loop has two, an immediate drop and a delayed rebound on different timescales. A deep‑RL policy can learn that delayed, nonlinear behavior and act on a trend instead of a single sample.
FIG. 1: ECONOMIZER LEVEL LOOP. A two‑phase stream flashes in the drum; vapor leaves to the compressor and liquid leaves through the level‑control valve. Because outlet flow changes pressure and flashing inside the drum, the level response is coupled and inverting, not a simple integrator.
Ai-OPs
·Refrigeration RL Level Control02
Closed-loop control architecture
A single PPO policy reads eight process tags, inferences once every 10‑second scan on the prior ~5 minutes of runtime, and writes one smoothed valve position, so it acts on the loop's trend rather than on a noisy instantaneous level.
FIG. 2: CONTROL & CONNECTIVITY ARCHITECTURE. One PPO policy drives the level valve and respects a valid‑range guardrail that hands control back to PID outside trained conditions. It reaches the field over OPC‑UA: the Foxboro DCS, an ArchestrA Server gateway, and the on‑prem KoiosLT edge.
What the agent watches
Eight tags, drum level and setpoint, upstream propane pressure, drum pressure, the current valve position, plus inlet‑gas and reboiler‑side valve context. Koios manages historian data, quality and range tolerances; the policy then inferences each 10‑second scan on a 30‑sample window.
What it controls
A single output, the liquid‑outlet valve position, written as a positional command clipped to 0–100%. The policy acts only while every input stays inside its trained valid range; outside that envelope it defers to the existing PID controller, so a poorly‑characterized state never reaches the valve through the model.
Ai-OPs
·Refrigeration RL Level Control2.3
What the controller balances
The PPO policy balances two competing goals, hold level at setpoint and move the valve as little as possible, the exact trade‑off operators had to referee by hand. This configuration is deliberately biased toward calmer valve action.
FIG. 3: CONTROL OBJECTIVES. Level error and valve movement pull in opposite directions; this configuration is biased toward smoother action, a balance to be weighed against response speed during operator review.
03
Bumpless engagement & integration
Operator engages, PID tracks
The operator bumplessly engages the controller from the standard faceplate. In AIOP mode Koios writes the valve output directly, while the DCS PID block tracks that output in the background. Because PID is always following the live position, handing control back, on operator command or a guardrail trip, is bumpless in both directions.
Trained offline, no new hardware
The PPO policy was pretrained to imitate historical controller behavior, then improved with reinforcement learning in a first‑principles refrigeration simulator (up to 8M steps), and exported to ONNX. It reads existing tags and writes the existing loop, so it reached the field with no new instrumentation.
Operators enable or disable the agent per loop, and PID sits underneath at all times, tracking the output. The model never owns the valve outright, full transparency, with PID fallback available on demand or automatically outside the trained envelope.
Ai-OPs
·Refrigeration RL Level Control04
Offline validation results
Across 13 held‑out historical windows the trained policy held vessel level near setpoint with smooth valve action, including through injected surge disturbances designed to test unmeasured inventory changes.
FIG. 4: PER‑EPISODE TRACKING ERROR. Mean PV–SP absolute error per validation episode, from Table 2 of the validation report. Ten of thirteen episodes hold below 2.5% level; the average across all episodes is 2.03%.
FIG. 5: SURGE RECOVERY. A surge disturbance hits mid‑episode; the AiMV valve (dashed) drives hard then settles, returning level to setpoint without sustained offset.
FIG. 6: SUSTAINED DISTURBANCE. Original PID traces vs. the simulated AiMV response; the policy holds level through a late‑episode upset while keeping valve motion comparatively smooth.
These are offline simulator results, evidence of credible closed‑loop behavior, not plant performance. Plant shadow‑mode trials, bump testing against real process response, field instrument checks, and operator review remain prerequisites to closed‑loop deployment.
Ai-OPs
·Refrigeration RL Level Control05
Where this applies
The same pattern, a reinforcement‑learning policy that learns a delayed, nonlinear loop and runs closed‑loop on the existing control system with PID always underneath, generalizes across gas processing, cryogenics and refrigeration. Applications available today on Koios:
About Koios
Koios is Ai‑OPs' self‑hosted machine‑learning inferencing platform for deploying scalable AI models on industrial control systems. It is hardware‑agnostic, runs fully on‑premises and air‑gapped, and writes optimized control back to PLCs, SCADA and DCS in real time. Your models and data stay inside your plant; you own every model Koios trains.
It installs as a single Docker image from a Docker Verified Publisher, or as an offline ISO / virtual‑appliance image, and is licensed as an annual per‑instance subscription.
| Protocols | OPC‑UA · OPC‑DA · EtherNet/IP · Modbus TCP & more |
| Deployment | On‑prem · air‑gapped |
| Install | Single Docker image / ISO |
| Min. hardware | 4‑core 64‑bit · 8 GB RAM · 128 GB SSD · 1 GbE |
| Per license | Unlimited devices, tags & models · historian & diagnostics · support |