bluetooth central ending up in a state where all connections fail

[avian2]

I'm trying to diagnose an issue with a Zephyr application. I'm using a nrf52832 target. Zephyr 3.7.1. Same issue also reproducible with 3.7.2rc1.

The application uses Bluetooth central to connect a peripheral. The Bluetooth application code is modeled after the samples/bluetooth/central_hr and central_gatt_write examples.

During the lifetime of the application the Bluetooth connection is connected and disconnected many times. The issue is that sometimes Zephyr becomes stuck in a state where the connection to the device will consistently fail as if the peripheral is not present at all. Resetting the CPU via NVIC_SystemReset() restores functionality.

A reliable way to trigger this is to perform connects and disconnects in a loop to the same device with no GATT notifies/writes and after a while Zephyr will end up in the broken state.

I've verified that this is not an issue with the peripheral: Peripheral stays responsive when this happens. I can successfully connect to it from a different device. Power cycling the peripheral also does not help.

The exact symptom is that bt_conn_le_create() will return success, but then the connected() callback is called with conn_err=2. This is the exact same sequence of events that happens normally when trying to connect to a peripheral that is not present. So far my focus has been in trying to determine the difference between a) broken state and peripheral present and b) working state and peripheral absent.

In both cases: old_state is BT_CONN_INITIATING in bt_conn_set_state(). The backtraces [1] and the bt_conn object [2] from the connected() callback are identical.

I don't see any faults or stack overruns in any of the running Zephyr threads ("BT RX", "BT RX pri", "BT RX WQ", "BT LW WQ", "sysworkq", "idle" and "main").

It's also interesting that in the broken state the peripheral is still visible when doing bt_le_scan_start(), it's just the connection that fails.

In the broken state I don't see any CONNECT_IND packets sent to the peripheral in the Bluetooth sniffer.

As far as I can see, this is not a bt_conn_ref/bt_conn_unref counting issue with the application because: 1) the symptom with a dangling connection object in BT_CONN_DISCONNECTED state is different (with a dangling connection bt_conn_le_create() immediately returns -EINVAL when in my case it returns success) 2) logging connections in the broken state before attempting to create a connection using bt_conn_foreach() shows no existing connection objects.

Many thanks for any pointers on how to find what it causing this issue.

[1] Backtrace from the connected() callback when connection fails:

#0  connected (conn=0x200014c0 <acl_conns>, conn_err=2 '\002')
    at /home/.../zephyrproject/.../src/..._ble.c:426
#1  0x0000cc7c in notify_connected (conn=conn@entry=0x200014c0 <acl_conns>)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/conn.c:1719
#2  0x0000d1b0 in bt_conn_set_state (conn=conn@entry=0x200014c0 <acl_conns>, 
    state=state@entry=BT_CONN_DISCONNECTED)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/conn.c:1243
#3  0x0002f8cc in le_conn_complete_cancel (err=<optimized out>)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:1267
#4  0x0000b47a in bt_hci_le_enh_conn_complete (evt=0x2000bd04 <net_buf_data_evt_pool+4>)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:1406
#5  0x0002f97c in enh_conn_complete (evt=<optimized out>)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:1345
#6  0x0002f986 in le_enh_conn_complete (buf=<optimized out>)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:1662
#7  0x0002ef1e in handle_event_common (event=<optimized out>, 
    buf=buf@entry=0x2000c6ec <_net_buf_evt_pool>, 
    handlers=handlers@entry=0x3a378 <meta_events>, num_handlers=num_handlers@entry=10)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:197
#8  0x0002ef3a in handle_event (event=<optimized out>, 
    buf=buf@entry=0x2000c6ec <_net_buf_evt_pool>, 
    handlers=handlers@entry=0x3a378 <meta_events>, num_handlers=num_handlers@entry=10)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:209
#9  0x0000a8c4 in hci_le_meta_event (buf=0x2000c6ec <_net_buf_evt_pool>)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:2825
#10 0x0002ef1e in handle_event_common (event=<optimized out>, 
    buf=buf@entry=0x2000c6ec <_net_buf_evt_pool>, 
    handlers=handlers@entry=0x3a348 <normal_events>, num_handlers=num_handlers@entry=6)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:197
#11 0x0002ef3a in handle_event (event=<optimized out>, 
    buf=buf@entry=0x2000c6ec <_net_buf_evt_pool>, 
    handlers=handlers@entry=0x3a348 <normal_events>, num_handlers=num_handlers@entry=6)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:209
#12 0x0000a86c in hci_event (buf=0x2000c6ec <_net_buf_evt_pool>)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:2949
#13 0x0000a93a in rx_work_handler (work=<optimized out>)
    at /home/.../zephyrproject/zephyr/subsys/bluetooth/host/hci_core.c:4183
#14 0x00024896 in work_queue_main (workq_ptr=0x200013e8 <bt_workq>, p2=<optimized out>, 
    p3=<optimized out>) at /home/.../zephyrproject/zephyr/kernel/work.c:688
#15 0x0002be8e in z_thread_entry (entry=0x24811 <work_queue_main>, p1=<optimized out>, 
    p2=<optimized out>, p3=<optimized out>)
    at /home/.../zephyrproject/zephyr/lib/os/thread_entry.c:48
#16 0xaaaaaaaa in ?? ()

[2] bt_conn passed to the connected() callback when connection fails:

(gdb) print *conn
$7 = {
  handle = 0,
  type = BT_CONN_TYPE_LE,
  role = 0 '\000',
  flags = {0},
  id = 0 '\000',
  sec_level = BT_SECURITY_L1,
  required_sec_level = BT_SECURITY_L1,
  encrypt = 0 '\000',
  err = 2 '\002',
  state = BT_CONN_DISCONNECTED,
  rx_len = 0,
  rx = 0x0 <_vector_table>,
  tx_pending = {
    head = 0x0 <_vector_table>,
    tail = 0x0 <_vector_table>
  },
  tx_complete = {
    head = 0x0 <_vector_table>,
    tail = 0x0 <_vector_table>
  },
  tx_complete_work = {
    node = {
      next = 0x0 <_vector_table>
    },
    handler = 0x3004d <tx_complete_work>,
    queue = 0x0 <_vector_table>,
    flags = 0
  },
  channels = {
    head = 0x0 <_vector_table>,
    tail = 0x0 <_vector_table>
  },
  deferred_work = {
    work = {
      node = {
        next = 0x0 <_vector_table>
      },
      handler = 0xda19 <deferred_work>,
      queue = 0x200028d8 <k_sys_work_q>,
      flags = 256
    },
    timeout = {
      node = {
        {
          head = 0x0 <_vector_table>,
          next = 0x0 <_vector_table>
        },
        {
          tail = 0x0 <_vector_table>,
          prev = 0x0 <_vector_table>
        }
      },
      fn = 0x377a3 <work_timeout>,
      dticks = 0
    },
    queue = 0x200028d8 <k_sys_work_q>
  },
  {
    le = {
      dst = {
        type = 0 '\000',
        a = {
          val = "..."
        }
      },
      init_addr = {
        type = 0 '\000',
        a = {
          val = "\000\000\000\000\000"
        }
      },
      resp_addr = {
        type = 0 '\000',
        a = {
          val = "\000\000\000\000\000"
        }
      },
      interval = 0,
      interval_min = 24,
      interval_max = 40,
      latency = 0,
      timeout = 400,
      pending_latency = 0,
      pending_timeout = 0,
      conn_param_retry_countdown = 0 '\000',
      features = "\000\000\000\000\000\000\000",
      keys = 0x0 <_vector_table>
    }
  },
  tx_data_pull = 0xdc69 <l2cap_data_pull>,
  get_and_clear_cb = 0x2ffeb <acl_get_and_clear_cb>,
  has_data = 0x2ff77 <acl_has_data>,
  l2cap_data_ready = {
    head = 0x0 <_vector_table>,
    tail = 0x0 <_vector_table>
  },
  _conn_ready = {
    next = 0x0 <_vector_table>
  },
  _conn_ready_lock = 0,
  in_ll = 0,
  next_is_frag = false,
  ref = 3
}

[cvinayak]

A reliable way to trigger this is to perform connects and disconnects in a loop to the same device with no GATT notifies/writes and after a while Zephyr will end up in the broken state.

Do you have CONFIG_BT_PRIVACY=y in the peripheral? And, do you always scan and find the Bluetooth device address before you call bt_conn_le_create() ? (Does the peripheral address change after 15 mins?)

Please provide simple samples, build, flash and steps to reproduce for a Zephyr supported nRF5x development boards.

[avian2]

Thanks for your reply!

Do you have CONFIG_BT_PRIVACY=y in the peripheral?

Unfortunately peripheral hardware and firmware is not under my control.

And, do you always scan and find the Bluetooth device address before you call bt_conn_le_create() ? (Does the peripheral address change after 15 mins?)

In the interest of keeping the connection time down scan only happens once, then the application caches the address of the peripheral in flash (so it persists even through the CPU reset).

However, the peripheral address is static and does not change:

When the bug triggers, I can force re-scan and the scan correctly finds the peripheral at the same address as before, even though the central will then not connect to it (with the conn_err=2 error I mention above).

Here is also an excerpt of the application log where I artificially trigger the bug by connecting and disconnecting from the peripheral in a loop. After the bug triggers, I reset the application and the connection to the peripheral succeeds at the same address as before. If the peripheral address changed I would not expect the connection to succeed after the reset using the same address as before.

(many preceding successful connects and disconnects in the same fashion)
[I 2026-01-13 14:24:51.196754 3070056832 server:1027] SPI LOG: connected to XX:XX:XX:88:4C:FD (public)
[I 2026-01-13 14:24:51.250404 3070056832 server:1027] SPI LOG: disconnected from XX:XX:XX:88:4C:FD (public) (reason 22)
[I 2026-01-13 14:24:53.322279 3070056832 server:1027] SPI LOG: connected to XX:XX:XX:88:4C:FD (public)
[I 2026-01-13 14:24:53.530505 3070056832 server:1027] SPI LOG: disconnected from XX:XX:XX:88:4C:FD (public) (reason 22)
(bug triggers here)
[E 2026-01-13 14:24:58.448638 3070056832 server:1027] SPI LOG: failed to connect to XX:XX:XX:88:4C:FD (public) (2)
[E 2026-01-13 14:25:03.412894 3070056832 server:1027] SPI LOG: failed to connect to XX:XX:XX:88:4C:FD (public) (2)
[E 2026-01-13 14:25:08.373541 3070056832 server:1027] SPI LOG: failed to connect to XX:XX:XX:88:4C:FD (public) (2)
[I 2026-01-13 14:25:14.531095 3070056832 server:1027] SPI LOG: reset cause: software
(broken state cleared by calling NVIC_SystemReset)
[I 2026-01-13 14:25:21.376361 3070056832 server:1027] SPI LOG: connected to XX:XX:XX:88:4C:FD (public)
[I 2026-01-13 14:25:21.728039 3070056832 server:1027] SPI LOG: disconnected from XX:XX:XX:88:4C:FD (public) (reason 22)
[I 2026-01-13 14:25:24.234967 3070056832 server:1027] SPI LOG: connected to XX:XX:XX:88:4C:FD (public)
[I 2026-01-13 14:25:24.284402 3070056832 server:1027] SPI LOG: disconnected from XX:XX:XX:88:4C:FD (public) (reason 22)
[I 2026-01-13 14:25:26.255980 3070056832 server:1027] SPI LOG: connected to XX:XX:XX:88:4C:FD (public)

The addresses displayed here are the result of calling bt_addr_le_to_str(bt_conn_get_dst(conn), addr, sizeof(addr)); with the conn pointer passed to the connected() callback.

Please provide simple samples, build, flash and steps to reproduce for a Zephyr supported nRF5x development boards.

At this point I think it's unlikely that I will be able to provide a reduced test case because I don't understand what is the cause of this bug. I have a core dump of the broken and working state and I've been picking it apart but my understanding of Zephyr's Bluetooth code isn't very deep and so far I haven't been able to find and significant difference in the global state of the Bluetooth stack. I'm working on obtaining a debug log from Zephyr's bluetooth code (it's complicated because I don't have UART access on the hardware). Hopefully that will provide more insight.

[cvinayak]

ok. I vaguely remember another user mention similar issue and probably he created a github issue. As this discussion gives no more information to be able to reproduce the issue, and in case you do happen to fix it, feel free to send a pull request.

[avian2]

The only similar issue I found is #50438. My understanding is that issue was related to switching between central and peripheral in a dual role application. My application only uses the central role. I tried compiling my application with both CONFIG_BT_PERIPHERAL=y and n and my issue is reproducible in both cases.

I'm just going to post some more info here for posterity in case it's useful. I think this is beyond my capability to fix.

I'm using CONFIG_BT_LL_SW_SPLIT=y (not using the softdevice controller)

I realized that the artificial way I'm triggering the broken state in my application does an usual thing: I call bt_conn_disconnect() right after I call bt_gatt_subscribe(), without waiting for the subscribe to finish (i.e. not waiting for the params->subscribe callback). This often leads to a disconnect while the subscribe process is still running, leading to random -104 (ECONNRESET) errors in associated functions. For example:

  SPI LOG: bt_att: att_handle_rsp: chan 0x2000be40 err -104 len 0:
  SPI LOG: bt_att: bt_att_req_free: req 0x2000f954
  SPI LOG: bt_gatt: gatt_write_ccc_rsp: err -104
  SPI LOG: bt_gatt: att_err_from_int: -104

These errors don't always lead to the broken state. Sometimes it takes many tens of connects/disconnects and sometimes it breaks in two or three. With 1 connection per second the application typically ends up in a broken state in a minute or so.

However if I correct this so that I disconnect only after the subscribe callback I can't reproduce the issue in this way anymore. So it must be related somehow. But I suspect this only fixes this specific artificial way I'm using of triggering the bug and that there is still an underlying issue somewhere else.

I compared the output given by CONFIG_BT_LOG_LEVEL_DBG=y during many runs, both before and after entering the broken state and I haven't found anything immediately obvious. One thing that stands out is the tx_notify: tx ... cb (nil) user_data (nil) that seems to always happen in the connection immediately before the first one that fails, but I've seen it also a few times that didn't trigger the broken state, so I'm not sure it's relevant:

* SPI LOG: bt_conn: tx_notify: tx 0x20004e24 cb (nil) user_data (nil)
* SPI LOG: bt_conn: tx_free: 0x20004e24
* SPI LOG: bt_conn: tx_notify: raise TX IRQ

Also I'm not trusting this log much since I seem to be often missing messages due to the limited CONFIG_LOG_BUFFER_SIZE that I can fit in available RAM.

Unfortunately I don't have a pair of nrf52832 devkits at hand to test the bt_gatt_subscribe/bt_conn_disconnect method with a modified central-hr example.

[cvinayak]

Thank you for your analysis... I will convert this discussion to an issue so that it is tracked better.