RTOS Debugging Techniques: Finding and Fixing Real-Time Bugs

Part of the Complete Guide to RTOS for Embedded Systems series.

Debugging RTOS-based firmware is fundamentally different from debugging sequential code. Bugs depend on timing, task priorities, interrupt timing, and resource contention — all of which change when you add breakpoints or printf statements. This article covers practical techniques for finding and fixing the most common RTOS bugs.

Why RTOS Bugs Are Hard to Find

RTOS bugs are notoriously difficult because they are often non-deterministic. A race condition might manifest once in 10,000 executions, only under specific timing conditions. Adding debug output (printf, LED toggles) changes task timing enough to make the bug disappear — a Heisenbug.

Common categories of RTOS bugs:

• Race conditions — two tasks access shared data without proper synchronization. Symptoms: corrupted data, occasional wrong values, intermittent crashes.
• Deadlocks — two tasks each hold a resource the other needs. Symptoms: system freezes, specific tasks stop responding.
• Priority inversion — high-priority task blocked by low-priority task holding a resource. Symptoms: missed deadlines, unexpected latency spikes.
• Stack overflow — task exceeds its stack allocation. Symptoms: random crashes, corrupted variables, hard faults.
• Starvation — low-priority task never gets CPU time. Symptoms: specific tasks appear to hang.
• ISR issues — calling blocking RTOS functions from ISR, or ISR priority misconfiguration. Symptoms: random crashes, data corruption.

The key principle: you cannot effectively debug RTOS issues by adding breakpoints (which stop the entire system) or printf (which changes timing). You need non-intrusive observation tools.

Runtime Statistics and Monitoring

FreeRTOS provides built-in runtime statistics that help identify scheduling and resource issues.

Task runtime stats show CPU utilization per task:

// In FreeRTOSConfig.h
#define configGENERATE_RUN_TIME_STATS 1
#define configUSE_STATS_FORMATTING_FUNCTIONS 1
#define portCONFIGURE_TIMER_FOR_RUN_TIME_STATS() setup_runtime_timer()
#define portGET_RUN_TIME_COUNTER_VALUE() get_runtime_counter()

// In your monitoring task void vStatsTask(void *pvParameters) { char buffer[512]; for (;;) { vTaskGetRunTimeStats(buffer); printf(“Task Abs Time %%Timen”); printf(“%sn”, buffer); vTaskDelay(pdMS_TO_TICKS(10000)); } } “`

This reveals:

• Which tasks consume the most CPU
• Whether the idle task gets any time (if not, your system is overloaded)
• Unexpected CPU usage patterns

Queue and semaphore state — check if queues are consistently full (consumer too slow) or empty (producer too slow):

UBaseType_t waiting = uxQueueMessagesWaiting(xSensorQueue);
UBaseType_t spaces = uxQueueSpacesAvailable(xSensorQueue);
printf("Queue: %u items, %u spacesn", waiting, spaces);

If a queue is consistently full, either increase the queue depth, make the consumer faster, or reduce the production rate.

Using RTOS-Aware Trace Tools

The most powerful debugging approach is using trace tools that record task switches, API calls, and ISR activity non-intrusively.

Segger SystemView — free tool that works with any J-Link debugger. It records task states, ISR execution, API calls, and user events with minimal overhead (uses RTT for data transfer). Displays a timeline view showing exactly when each task ran, what preempted it, and how long each operation took.

Percepio Tracealyzer — commercial tool with advanced visualization. Shows task scheduling as a timeline, identifies priority inversion events, measures response times, and highlights deadline violations. Works with FreeRTOS, Zephyr, ThreadX, and others.

Both tools answer questions that are nearly impossible to answer with printf debugging:

• What was running when the deadline was missed?
• How long was Task A blocked waiting for the mutex?
• Did an ISR preempt the critical section?
• What sequence of events led to the deadlock?

For projects without access to trace tools, implement a lightweight trace buffer:

#define TRACE_SIZE 256
typedef struct { uint32_t timestamp; uint8_t task_id; uint8_t event; uint16_t data; } TraceEntry_t;
static TraceEntry_t trace_buf[TRACE_SIZE];
static volatile uint32_t trace_idx = 0;

void trace_log(uint8_t task_id, uint8_t event, uint16_t data) { uint32_t i = trace_idx % TRACE_SIZE; trace_buf[i].timestamp = DWT->CYCCNT; trace_buf[i].task_id = task_id; trace_buf[i].event = event; trace_buf[i].data = data; trace_idx++; } “`

Inspect trace_buf from the debugger after a crash to reconstruct the sequence of events.

Defensive Programming Patterns

Build detection into your firmware so bugs surface during development, not in the field.

Assertions for impossible conditions:

void vProcessReading(SensorReading_t *reading) {
    configASSERT(reading != NULL);
    configASSERT(reading->channel < MAX_CHANNELS);
    configASSERT(reading->value >= SENSOR_MIN && reading->value <= SENSOR_MAX);
    // ... process
}

FreeRTOS’s configASSERT halts the system and can print file/line information, making it easy to find the source.

Watchdog with task health monitoring:

#define TASK_SENSOR_BIT  (1 << 0)
#define TASK_COMM_BIT    (1 << 1)
#define TASK_CONTROL_BIT (1 << 2)
#define ALL_TASKS_HEALTHY (TASK_SENSOR_BIT | TASK_COMM_BIT | TASK_CONTROL_BIT)

void vWatchdogTask(void *pvParameters) { for (;;) { EventBits_t bits = xEventGroupWaitBits(xHealthGroup, ALL_TASKS_HEALTHY, pdTRUE, pdTRUE, pdMS_TO_TICKS(5000)); if (bits == ALL_TASKS_HEALTHY) { HAL_IWDG_Refresh(&hiwdg); // Feed watchdog } else { // Some task didn’t check in — log which one and let watchdog reset log_error(“Unhealthy tasks: 0x%02X (expected 0x%02X)”, bits, ALL_TASKS_HEALTHY); } } }

// Each task checks in periodically void vSensorTask(void *pvParameters) { for (;;) { read_sensors(); xEventGroupSetBits(xHealthGroup, TASK_SENSOR_BIT); vTaskDelay(pdMS_TO_TICKS(100)); } } “`

This pattern catches deadlocks, infinite loops, stack overflows, and any other condition that prevents a task from running — the exact bugs that are hardest to find with traditional debugging.

Vivek Bhageria

Vivek Bhageria — Lead Firmware R&D Engineer, 12+ years. Ex-Bosch (automotive powertrain), MusicTribe (real-time audio), medical devices. M.Tech BITS Pilani. I write at NerdyElectronics — practical, register-level embedded systems for engineers who want to understand what’s actually happening under the hood.

nerdyelectronics.com/about/