--- title: "Service Health & Resource Alarms" description: "Diagnose unhealthy services, heartbeats, and CPU, memory, disk, and network alarms" source_url: https://ai-ops.com/docs/troubleshoot/services-and-alarms --- # Service Health & Resource Alarms This is the platform layer. When a specific device, tag, model, or scan group looks wrong, start with the layer that owns it. But when *many* things fail at once, or the whole server slows down, the cause is usually one layer beneath them all: a background service has stopped, or the host is out of CPU, memory, or disk. This page covers how to read the System Alerts banners, diagnose unhealthy services and their heartbeats, respond to resource alarms, and interpret scan-group execution errors. > [!NOTE] Whole app is blank or won > If nothing renders at all (blank page, login won't submit, GraphQL errors on every request), this is an outage, not a health alarm. Go to [Server Won't Start](https://ai-ops.com/docs/troubleshoot/server-startup.md). For the shared status colors, the three diagnostic fields, and the auto-clear vs. stuck checklist, see [Reading Status, Quality & Errors](https://ai-ops.com/docs/troubleshoot/reading-status-and-errors.md). This page does not repeat those. --- ## Start with the System Alerts banners Koios continuously checks four platform signals and surfaces them as alert banners. Read the banners first — they tell you which layer to investigate and link straight to the tool that owns it. | Banner | Severity | What it means | Where it links | |--------|----------|---------------|----------------| | **Unhealthy Services** | Warning, or Error if *every* service is down | One or more background services are not sending healthy heartbeats | [Services](https://ai-ops.com/docs/system/services.md) | | **Performance Alarm** | Warning | A CPU, Memory, Disk, or Network alarm threshold is currently exceeded | [System Health](https://ai-ops.com/docs/system/performance.md) | | **Failed Devices / Tags / Models / Scan Groups** | Error | One or more entities of that type are in a failed state | The matching list page | | **License** (No License / Invalid / Grace Period) | Warning or Error | Licensed functionality is gated by license state | [License](https://ai-ops.com/docs/system/license.md) | Two of these also pin a persistent bar at the bottom of the app until resolved: the **license grace period** warning and the **unhealthy services** banner, which names the services that aren't running. > [!TIP] Route by the banner, not the symptom > A single failed service can cascade into dozens of failed tags and models. If you see an Unhealthy Services banner *and* failed-entity banners together, fix the service first — the entity failures often clear on their own once the service recovers. The failed-entity and license banners are the entry points to other layers. Follow them to [Bad, Missing, or Frozen Tag Values](https://ai-ops.com/docs/troubleshoot/tag-values.md), [A Model or Binding Isn't Running](https://ai-ops.com/docs/troubleshoot/models.md), or [Licensing Problems](https://ai-ops.com/docs/troubleshoot/licensing.md). The rest of this page covers services and resource alarms. --- ## Unhealthy or stopped services Open **System > Service Status** to see every service. Services are split into two tables: - **Application Services** — Data Collector, Predict Engine, Expression Evaluator, Parameter Mapping, Performance Monitoring, and Component Engine. These do the platform's real work, and they report heartbeats. - **Infrastructure** — Celery Worker, the configuration database, the time-series database, the in-memory cache, the web proxy, and the metrics agent. These support the application services. Each row shows status, CPU, memory, PID, and uptime. ### How heartbeats decide health Application services report a heartbeat on a regular cadence. The status badge combines that heartbeat with the process state: | Status | Color | Meaning | What to do | |--------|-------|---------|------------| | **Running** | Teal | Healthy and reporting normally | Nothing | | **Stopped** | Gray | Not running | Start it from the detail drawer | | **Failed** | Red | Crashed or exited with an error | Read the error, then Start or Restart | | **Starting** | Yellow | In the process of starting | Wait for the transition to finish | | **Stopping** | Yellow | In the process of stopping | Wait for the transition to finish | | **Unresponsive** | Orange | Process is alive but **not sending healthy heartbeats** | Restart it; if it keeps going unresponsive, check the logs and resource alarms | | **Standalone** | Orange | Heartbeats are healthy but the process is **not managed by the service manager** | Usually a dev/manual-launch artifact; restart to bring it back under management | **Unresponsive** is the key heartbeat state: the process didn't crash, so it isn't Failed, but it stopped checking in. A service that pins the host CPU or runs out of memory will often go Unresponsive rather than Failed. Hover any badge for a plain-language explanation. ### Recover a service Click a service row to open its detail drawer. The **Overview** tab shows the description, live metrics (CPU, memory, thread count, PID, uptime), **Last Heartbeat** (when it last reported in), and any reported **Error Info** message and detail. Action buttons appear in the footer based on the current state: | Current state | Available actions | |---------------|-------------------| | Stopped / Failed | Start | | Running / Unresponsive | Restart, Stop | | Starting / Stopping | None — wait for the transition | Start and Restart run immediately; Stop asks for confirmation. The list refreshes automatically afterward. > [!WARNING] Some services have no action buttons > The configuration database, the in-memory cache, and the web proxy cannot be started, stopped, or restarted from the UI — no action buttons appear for them. If one of those is down, the app itself usually can't run: treat it as an outage and see [Server Won't Start](https://ai-ops.com/docs/troubleshoot/server-startup.md) or [Manually Starting Koios](https://ai-ops.com/docs/installation/manually-starting-koios.md). ### When a service keeps going Unresponsive or Failed 1. Open the drawer's **Diagnostics** tab (application services only). It shows the service's live workload: **Pressure** (thread-pool utilization as a rolling average), **In-Flight** (tasks executing now), and **Queued** (tasks waiting). A badge turns yellow at 70% pressure and red at 90%. Sustained high pressure means the service is overloaded, not broken — reduce its workload or scan rate rather than just restarting it. 2. For the Predict Engine, the Diagnostics tab lists **Top Contributors** — the AI models putting the most load on the service. Click a model to open it and check whether one model is starving the rest. 3. Stream the service's log for the actual traceback. See [Logs](https://ai-ops.com/docs/system/logs.md), and set the log level to Debug there if the default output isn't detailed enough. 4. Check whether a resource alarm is active (next section). A service that keeps dying is often a symptom of the host running out of memory or disk. --- ## Resource alarms: CPU, memory, disk, network A **Performance Alarm** banner, or a red dot on a tab in **System > Health**, means a hardware threshold has been crossed. The Health page has four tabs, each with a live chart, an alarm limit, and display options. The alarm state is re-evaluated every 10 seconds; a red dot on the tab means that metric's alarm is currently active. | Alarm | Threshold setting | Fires when | First things to check | |-------|-------------------|------------|-----------------------| | **CPU** | CPU Alarm Limit (%) | Overall CPU usage exceeds the limit | Which service is consuming it — the chart's per-service **Service Processes** breakdown shows the culprit. A hot Predict Engine or Data Collector usually means scan rates are too aggressive for the hardware. | | **Memory** | RAM Alarm Limit (%) | Memory usage exceeds the limit | Per-service memory in the chart legend. Sustained high memory is the most common cause of a service going Unresponsive or Failed. Switch the axis to bytes with **Show in Bytes** for absolute numbers. | | **Disk** | Storage Alarm Limit (%) | Disk usage exceeds the limit | The live display shows free space, daily change rate, and a projected **time-to-full**. If it's filling fast, history is the usual cause — tighten retention and compression. | | **Network** | Network Alarm Limit (MB/s) | Throughput exceeds the limit, or (if **Alarm on Packet Drops** is on) packet drops are detected | The interface selector — confirm you're watching the right interface. Packet drops point at a physical link or driver problem, not Koios. | Each tab's **Show Setpoint** option draws the threshold as a dashed red line so you can see how close you are. Thresholds save automatically the moment you change them — there's no Save button. > [!TIP] Set thresholds to your normal operating range > A CPU alarm at 90% catches sustained overload without false positives from brief spikes. A storage alarm at 85% gives you time to act before the disk fills. Tune every limit to what "normal" looks like on your deployment. ### Disk alarm won't clear Disk pressure rarely resolves itself. The largest consumer is almost always historical storage. Lower retention windows and enable/tune compression (see the retention settings, below), then confirm the projected time-to-full estimate on the Disk tab is trending back up. Backups also accumulate — old pre-migration and manual dumps in the media volume can be pruned. ### A resource alarm caused a service failure If a resource alarm was active at the same time a service went Unresponsive or Failed, treat the alarm as the root cause. Relieve the resource pressure (reduce scan rates, tighten retention, add capacity), then restart the affected service. Restarting without relieving the pressure just repeats the cycle. --- ## Device housekeeping failures (codes 14, 15, 17) A running device can report a platform-side failure that isn't about the connection or a tag read — it's the Data Collector failing to complete background work for that device. These surface as device error codes and usually clear on the next cycle once the underlying service or resource pressure eases. | Code | Meaning | What to do | |------|---------|------------| | **14 — Failed to Update Heartbeat** | The device's availability heartbeat could not be written | Check the Data Collector service and the in-memory cache; typically transient under high load. | | **15 — Failed to Cleanup** | Resource cleanup after a device operation failed | Usually transient; if it persists, restart the Data Collector and check for an active resource alarm. | | **17 — Failed to Write Executions** | Scan/execution records could not be written to the database | Check the configuration database health and free disk space, and relieve any active resource alarm. | These are symptoms of service or host pressure, not a device fault. If one recurs, work the services and resource-alarm sections above — the fix is almost always there. --- ## Scan-group errors: Overscan and On-Demand Read/Write A scan group runs its members on a shared schedule. When the scan group itself is Failed or shows a warning, the error code identifies what went wrong at the group level (distinct from any per-tag or per-binding errors underneath it). | Code | Meaning | What to do | |------|---------|------------| | **0 — None** | Running normally | Nothing | | **1 — Overscan** | Execution exceeded the configured scan rate — the group could not finish one cycle before the next was due | Slow the scan rate, or reduce the number of members. Overscan surfaces as a running-with-warning (amber) state, not a hard failure: the group still works, it just can't keep up. Persistent overscan also drives CPU. | | **2 — On-Demand Read Failed** | The group's shared on-demand read failed | Trace it to the underlying device or tag — this is a device-layer failure surfacing at the group. See [Troubleshoot a Connection](https://ai-ops.com/docs/troubleshoot/connection.md). | | **3 — On-Demand Write Failed** | The group's shared on-demand write failed | Confirm the target tag is writable and the device accepts the write. See [Bad, Missing, or Frozen Tag Values](https://ai-ops.com/docs/troubleshoot/tag-values.md). | | **99 — Generic Exception** | An unexpected exception during scan-group execution | Read the group's error detail and the service logs for the traceback. | > [!WARNING] Overscan is a capacity signal > Overscan means the work assigned to a scan group is more than the host can complete in the allotted time. Increasing the scan rate (running less often) or splitting members across groups is the fix — not restarting. Left unaddressed, a chronically overscanning group keeps CPU high and can starve other services. On-demand read/write behavior is configured per device and per scan group; see [On-Demand Scanning](https://ai-ops.com/docs/devices/on-demand-scanning.md) and [On-Demand Inference](https://ai-ops.com/docs/models/on-demand-inference.md). The On-Demand Read/Write failures here are the group-level view. When a *model binding* reports an on-demand read failure (binding code 20), diagnose it from the model side in [A Model or Binding Isn't Running](https://ai-ops.com/docs/troubleshoot/models.md). --- ## What's Next - [Reading Status, Quality & Errors](https://ai-ops.com/docs/troubleshoot/reading-status-and-errors.md): the shared status legend, the three diagnostic fields, and the master error-code lookup - [Services](https://ai-ops.com/docs/system/services.md): the full Service Status page — heartbeats, diagnostics, and start/stop/restart - [System Health](https://ai-ops.com/docs/system/performance.md): the CPU, memory, disk, and network charts and every alarm setting - [Logs](https://ai-ops.com/docs/system/logs.md): stream a service's log and raise its log level to Debug - [Data Retention](https://ai-ops.com/docs/system/retention.md#tag-range-and-compression): tighten history and compression to relieve a disk alarm - [Server Won't Start](https://ai-ops.com/docs/troubleshoot/server-startup.md): when the whole platform is down, not just one service - [Collecting Diagnostics for Support](https://ai-ops.com/docs/troubleshoot/collecting-diagnostics.md): what to gather before opening a ticket