Service Health & Resource Alarms

This is the platform layer. When a specific device, tag, model, or scan group looks wrong, start with the layer that owns it. But when many things fail at once, or the whole server slows down, the cause is usually one layer beneath them all: a background service has stopped, or the host is out of CPU, memory, or disk. This page covers how to read the System Alerts banners, diagnose unhealthy services and their heartbeats, respond to resource alarms, and interpret scan-group execution errors.

Whole app is blank or won't load?

If nothing renders at all (blank page, login won't submit, GraphQL errors on every request), this is an outage, not a health alarm. Go to Server Won't Start.

For the shared status colors, the three diagnostic fields, and the auto-clear vs. stuck checklist, see Reading Status, Quality & Errors. This page does not repeat those.

Start with the System Alerts banners

Koios continuously checks four platform signals and surfaces them as alert banners. Read the banners first — they tell you which layer to investigate and link straight to the tool that owns it.

Banner	Severity	What it means	Where it links
Unhealthy Services	Warning, or Error if every service is down	One or more background services are not sending healthy heartbeats	Services
Performance Alarm	Warning	A CPU, Memory, Disk, or Network alarm threshold is currently exceeded	System Health
Failed Devices / Tags / Models / Scan Groups	Error	One or more entities of that type are in a failed state	The matching list page
License (No License / Invalid / Grace Period)	Warning or Error	Licensed functionality is gated by license state	License

Two of these also pin a persistent bar at the bottom of the app until resolved: the license grace period warning and the unhealthy services banner, which names the services that aren't running.

Route by the banner, not the symptom

A single failed service can cascade into dozens of failed tags and models. If you see an Unhealthy Services banner and failed-entity banners together, fix the service first — the entity failures often clear on their own once the service recovers.

The failed-entity and license banners are the entry points to other layers. Follow them to Bad, Missing, or Frozen Tag Values, A Model or Binding Isn't Running, or Licensing Problems. The rest of this page covers services and resource alarms.

Unhealthy or stopped services

Open System > Service Status to see every service. Services are split into two tables:

Application Services — Data Collector, Predict Engine, Expression Evaluator, Parameter Mapping, Performance Monitoring, and Component Engine. These do the platform's real work, and they report heartbeats.
Infrastructure — Celery Worker, the configuration database, the time-series database, the in-memory cache, the web proxy, and the metrics agent. These support the application services.

Each row shows status, CPU, memory, PID, and uptime.

How heartbeats decide health

Application services report a heartbeat on a regular cadence. The status badge combines that heartbeat with the process state:

Status	Color	Meaning	What to do
Running	Teal	Healthy and reporting normally	Nothing
Stopped	Gray	Not running	Start it from the detail drawer
Failed	Red	Crashed or exited with an error	Read the error, then Start or Restart
Starting	Yellow	In the process of starting	Wait for the transition to finish
Stopping	Yellow	In the process of stopping	Wait for the transition to finish
Unresponsive	Orange	Process is alive but not sending healthy heartbeats	Restart it; if it keeps going unresponsive, check the logs and resource alarms
Standalone	Orange	Heartbeats are healthy but the process is not managed by the service manager	Usually a dev/manual-launch artifact; restart to bring it back under management

Unresponsive is the key heartbeat state: the process didn't crash, so it isn't Failed, but it stopped checking in. A service that pins the host CPU or runs out of memory will often go Unresponsive rather than Failed. Hover any badge for a plain-language explanation.

Recover a service

Click a service row to open its detail drawer. The Overview tab shows the description, live metrics (CPU, memory, thread count, PID, uptime), Last Heartbeat (when it last reported in), and any reported Error Info message and detail. Action buttons appear in the footer based on the current state:

Current state	Available actions
Stopped / Failed	Start
Running / Unresponsive	Restart, Stop
Starting / Stopping	None — wait for the transition

Start and Restart run immediately; Stop asks for confirmation. The list refreshes automatically afterward.

Some services have no action buttons

The configuration database, the in-memory cache, and the web proxy cannot be started, stopped, or restarted from the UI — no action buttons appear for them. If one of those is down, the app itself usually can't run: treat it as an outage and see Server Won't Start or Manually Starting Koios.

When a service keeps going Unresponsive or Failed

Open the drawer's Diagnostics tab (application services only). It shows the service's live workload: Pressure (thread-pool utilization as a rolling average), In-Flight (tasks executing now), and Queued (tasks waiting). A badge turns yellow at 70% pressure and red at 90%. Sustained high pressure means the service is overloaded, not broken — reduce its workload or scan rate rather than just restarting it.
For the Predict Engine, the Diagnostics tab lists Top Contributors — the AI models putting the most load on the service. Click a model to open it and check whether one model is starving the rest.
Stream the service's log for the actual traceback. See Logs, and set the log level to Debug there if the default output isn't detailed enough.
Check whether a resource alarm is active (next section). A service that keeps dying is often a symptom of the host running out of memory or disk.

Resource alarms: CPU, memory, disk, network

A Performance Alarm banner, or a red dot on a tab in System > Health, means a hardware threshold has been crossed. The Health page has four tabs, each with a live chart, an alarm limit, and display options. The alarm state is re-evaluated every 10 seconds; a red dot on the tab means that metric's alarm is currently active.

Alarm	Threshold setting	Fires when	First things to check
CPU	CPU Alarm Limit (%)	Overall CPU usage exceeds the limit	Which service is consuming it — the chart's per-service Service Processes breakdown shows the culprit. A hot Predict Engine or Data Collector usually means scan rates are too aggressive for the hardware.
Memory	RAM Alarm Limit (%)	Memory usage exceeds the limit	Per-service memory in the chart legend. Sustained high memory is the most common cause of a service going Unresponsive or Failed. Switch the axis to bytes with Show in Bytes for absolute numbers.
Disk	Storage Alarm Limit (%)	Disk usage exceeds the limit	The live display shows free space, daily change rate, and a projected time-to-full. If it's filling fast, history is the usual cause — tighten retention and compression.
Network	Network Alarm Limit (MB/s)	Throughput exceeds the limit, or (if Alarm on Packet Drops is on) packet drops are detected	The interface selector — confirm you're watching the right interface. Packet drops point at a physical link or driver problem, not Koios.

Each tab's Show Setpoint option draws the threshold as a dashed red line so you can see how close you are. Thresholds save automatically the moment you change them — there's no Save button.

Set thresholds to your normal operating range

A CPU alarm at 90% catches sustained overload without false positives from brief spikes. A storage alarm at 85% gives you time to act before the disk fills. Tune every limit to what "normal" looks like on your deployment.

Disk alarm won't clear

Disk pressure rarely resolves itself. The largest consumer is almost always historical storage. Lower retention windows and enable/tune compression (see the retention settings, below), then confirm the projected time-to-full estimate on the Disk tab is trending back up. Backups also accumulate — old pre-migration and manual dumps in the media volume can be pruned.

A resource alarm caused a service failure

If a resource alarm was active at the same time a service went Unresponsive or Failed, treat the alarm as the root cause. Relieve the resource pressure (reduce scan rates, tighten retention, add capacity), then restart the affected service. Restarting without relieving the pressure just repeats the cycle.

Device housekeeping failures (codes 14, 15, 17)

A running device can report a platform-side failure that isn't about the connection or a tag read — it's the Data Collector failing to complete background work for that device. These surface as device error codes and usually clear on the next cycle once the underlying service or resource pressure eases.

Code	Meaning	What to do
14 — Failed to Update Heartbeat	The device's availability heartbeat could not be written	Check the Data Collector service and the in-memory cache; typically transient under high load.
15 — Failed to Cleanup	Resource cleanup after a device operation failed	Usually transient; if it persists, restart the Data Collector and check for an active resource alarm.
17 — Failed to Write Executions	Scan/execution records could not be written to the database	Check the configuration database health and free disk space, and relieve any active resource alarm.

These are symptoms of service or host pressure, not a device fault. If one recurs, work the services and resource-alarm sections above — the fix is almost always there.

Scan-group errors: Overscan and On-Demand Read/Write

A scan group runs its members on a shared schedule. When the scan group itself is Failed or shows a warning, the error code identifies what went wrong at the group level (distinct from any per-tag or per-binding errors underneath it).

Code	Meaning	What to do
0 — None	Running normally	Nothing
1 — Overscan	Execution exceeded the configured scan rate — the group could not finish one cycle before the next was due	Slow the scan rate, or reduce the number of members. Overscan surfaces as a running-with-warning (amber) state, not a hard failure: the group still works, it just can't keep up. Persistent overscan also drives CPU.
2 — On-Demand Read Failed	The group's shared on-demand read failed	Trace it to the underlying device or tag — this is a device-layer failure surfacing at the group. See Troubleshoot a Connection.
3 — On-Demand Write Failed	The group's shared on-demand write failed	Confirm the target tag is writable and the device accepts the write. See Bad, Missing, or Frozen Tag Values.
99 — Generic Exception	An unexpected exception during scan-group execution	Read the group's error detail and the service logs for the traceback.

Overscan is a capacity signal

Overscan means the work assigned to a scan group is more than the host can complete in the allotted time. Increasing the scan rate (running less often) or splitting members across groups is the fix — not restarting. Left unaddressed, a chronically overscanning group keeps CPU high and can starve other services. On-demand read/write behavior is configured per device and per scan group; see On-Demand Scanning and On-Demand Inference.

The On-Demand Read/Write failures here are the group-level view. When a model binding reports an on-demand read failure (binding code 20), diagnose it from the model side in A Model or Binding Isn't Running.

What's Next

Reading Status, Quality & Errors: the shared status legend, the three diagnostic fields, and the master error-code lookup
Services: the full Service Status page — heartbeats, diagnostics, and start/stop/restart
System Health: the CPU, memory, disk, and network charts and every alarm setting
Logs: stream a service's log and raise its log level to Debug
Data Retention: tighten history and compression to relieve a disk alarm
Server Won't Start: when the whole platform is down, not just one service
Collecting Diagnostics for Support: what to gather before opening a ticket

Data Is Stale, Frozen, or Has Gaps Licensing Problems