…spin (netdata#22232)

* ebpf.plugin: fix PID accounting shared-memory pool leak and 100% CPU spin

The eBPF plugin's per-PID shared-memory pool (32,768 slots backing
/dev/shm/netdata_shm_integration_ebpf) monotonically fills on any host
with normal process churn, and once the pool is exhausted each module's
apps-read loop spins one CPU core at 100% indefinitely.

ROOT CAUSE

netdata_ebpf_get_shm_pointer_unsafe() is an allocate-AND-set-bit API,
but it is called from two incompatible contexts:

  1. The per-module BPF map iteration (ebpf_read_*_apps_table), which
     is paired with netdata_ebpf_reset_shm_pointer_unsafe() via the
     kill(pid, 0) == ESRCH branch. This is a correct producer/consumer.

  2. The apps and cgroup aggregation helpers (*_sum_pids,
     *_update_*_cgroup, *_resume_apps_data), which iterate live-PID
     lists built from /proc and cgroup snapshots and call the same
     allocating accessor. These paths have NO reset counterpart.

Every second, for every live PID seen in /proc or a tracked cgroup,
every enabled eBPF module unconditionally sets its bit in the shm
slot for that PID. A dead PID's bit is only cleared by the owning
module's BPF-map iteration - but the BPF map's entry was already
deleted in the same cycle (by bpf_map_delete_elem inside
reset_shm_pointer_unsafe), so the next iteration cannot find the PID
and the bit is stuck forever. The slot therefore never reaches
threads == 0 and is never compacted out. Over 15h on a workstation
with ~1,300 live PIDs the pool reaches 100% with ~96% stale entries.

Once the pool is full, get_shm_pointer_unsafe() returns NULL. Every
apps_read loop had a bare `continue;` after the NULL check, which
skipped the `key = next_key;` advance, so bpf_map_get_next_key()
re-returned the same key forever and the thread spun at 100% CPU.

FIXES

  A. Split allocation from lookup. Introduce
     netdata_ebpf_lookup_shm_pointer_unsafe(pid) that returns the
     existing slot or NULL and sets no bit. Replace the 16 aggregation
     call sites with the new API. Keep get_shm_pointer_unsafe() at the
     8 BPF-map iteration call sites where allocation is legitimate and
     paired with reset. Aggregation paths additionally gate on the
     module bit being set so they do not consume zero-initialised
     per-module sub-structs.

  B. Replace `continue;` with `goto end_*_loop;` in all 8 apps-read
     loops so `key = next_key;` always runs, preventing the infinite
     loop even when the pool is legitimately exhausted.

  C. Zero freshly-allocated slots. ebpf_find_or_create_index_pid()
     now memset()s the slot on the create branch, so compacted tails
     and PID reuse cannot leak stale bits or stale per-module
     counters into new occupants.

  D. Use the correct kill() error check. Five modules used
     `if (kill(pid, 0))`, which treats EPERM (cross-UID processes) as
     "process is dead" and erroneously deletes the live process's
     kernel BPF map entry and zeroes its shm data. Switched all of
     them to `if (kill(pid, 0) == -1 && errno == ESRCH)`, matching
     the two modules that already did it correctly. Also applied to
     ebpf_parse_proc_files().

  E. Collapse ebpf_reset_specific_pid_data() to its effective body.
     thread_collecting only ever has NETDATA_EBPF_PIDS_PROC_FILE set,
     so the `idx < PROC_FILE` switch was dead code. The function now
     just forwards to ebpf_del_pid_entry(). The only other reader of
     that bitmask, ebpf_release_pid_data(), had zero callers and was
     removed.

  F. Sweep the shm pool on module exit. New helper
     netdata_ebpf_sweep_shm_for_module_unsafe() clears this module's
     bit across every shm slot and lets the existing compaction path
     release slots that become empty. All eight per-PID modules now
     call it under the shm semaphore in their *_exit path, so runtime
     module restarts no longer leak bits.

VERIFICATION

  - Root cause was captured on a live ebpf.plugin pinned at 64% CPU.
    gdb showed an EBPF_READ_FD thread permanently alternating
    bpf_map_get_next_key/bpf_map_lookup_elem at ebpf_fd.c:770-774,
    and ebpf_stat_values = {total = 32768, current = 32768} while
    ps showed 1,307 live PIDs.

  - The diagnosis was independently confirmed by a parallel review
    (codex, qwen, glm, minimax, opus), all converging on the
    alloc-without-reset asymmetry in the aggregation paths.

  - The modified tree builds cleanly (`cmake --build build --target
    ebpf.plugin`). No new warnings in the changed files; only the
    pre-existing vendored-Judy stringop-overflow warnings remain.

* ebpf.plugin: wipe shm on init, unlink on cleanup, drop unused out-param

Close two residual issues around the per-PID shared-memory integration
pool, spotted post-merge:

1. The POSIX shm object `netdata_shm_integration_ebpf` was neither
   truncated on open nor unlinked on shutdown. `shm_open(..., O_CREAT)`
   on an existing object just reopens it, and `ftruncate` to the
   previous size is a no-op, so the 14 MB mapped region retained the
   previous plugin run's bytes. The slot-allocation path memsets each
   new slot, so this was not a correctness bug — but every slot in
   `[current, total)` kept stale PID numbers and per-module counters
   until system reboot. Now:
   - `netdata_integration_initialize_shm()` memsets the whole mapped
     region immediately after `nd_mmap`, so the next plugin instance
     always starts from a clean slab.
   - `netdata_integration_cleanup_shm()` calls `shm_unlink` so a
     clean shutdown removes the backing object from `/dev/shm`.

2. Drop the unused `bool *created` out-parameter on
   `ebpf_find_or_create_index_pid()`. It was added when the memset
   was factored out of the allocation path; since the memset now
   lives inside the function, no caller needs to know whether the
   slot was freshly allocated. Reported by Copilot on PR netdata#22232.

* ebpf.plugin: address Copilot findings on 22232

Four residual issues spotted by Copilot after the initial round of
changes. All valid.

1. get_shm_pointer_unsafe() no longer bails out when the pool is full.
   The caller path for an already-tracked PID must still reach its
   slot so module bits can be updated or cleared. The inner
   ebpf_find_or_create_index_pid() already returns existing slots
   unconditionally and only rejects *new* allocations when full, so
   the redundant outer guard was both dead weight and actively
   harmful — it made existing PIDs unreachable the moment the pool
   filled.

2. Semaphore lifecycle. netdata_integration_initialize_shm() now
   sem_unlinks NETDATA_EBPF_SHM_INTEGRATION_NAME before sem_open, and
   netdata_integration_cleanup_shm() sem_unlinks on the way out. If
   a previous plugin instance crashed while holding the semaphore,
   sem_open(O_CREAT) on the same name would reuse the existing
   semaphore at its last value (O_CREAT's initial-value argument is
   ignored when the semaphore already exists), and the next run
   would spin on sem_timedwait() timeouts forever. Unlinking
   guarantees the initial value of 1 is honoured.

3. Socket apps iteration: the kernel-map delete for a dead socket
   tuple used to run inside `if (local_pid) { ... }`. When the shm
   pool is full and local_pid is NULL, `deleted` sockets were left
   in the BPF map to be re-iterated every cycle, growing the map
   unbounded. Moved bpf_map_delete_elem(fd, &key) outside the
   local_pid branch; it is conditioned only on `deleted`, which
   captures the socket's own freshness signal and is independent of
   shm pool state. (For SOCKET_IDX the generic
   reset_shm_pointer_unsafe() intentionally skips the delete, since
   the socket map is keyed by a tuple, not by pid — comment added.)

(cherry picked from commit bd38550)

…spin (#22232)

* ebpf.plugin: fix PID accounting shared-memory pool leak and 100% CPU spin

The eBPF plugin's per-PID shared-memory pool (32,768 slots backing
/dev/shm/netdata_shm_integration_ebpf) monotonically fills on any host
with normal process churn, and once the pool is exhausted each module's
apps-read loop spins one CPU core at 100% indefinitely.

ROOT CAUSE

netdata_ebpf_get_shm_pointer_unsafe() is an allocate-AND-set-bit API,
but it is called from two incompatible contexts:

  1. The per-module BPF map iteration (ebpf_read_*_apps_table), which
     is paired with netdata_ebpf_reset_shm_pointer_unsafe() via the
     kill(pid, 0) == ESRCH branch. This is a correct producer/consumer.

  2. The apps and cgroup aggregation helpers (*_sum_pids,
     *_update_*_cgroup, *_resume_apps_data), which iterate live-PID
     lists built from /proc and cgroup snapshots and call the same
     allocating accessor. These paths have NO reset counterpart.

Every second, for every live PID seen in /proc or a tracked cgroup,
every enabled eBPF module unconditionally sets its bit in the shm
slot for that PID. A dead PID's bit is only cleared by the owning
module's BPF-map iteration - but the BPF map's entry was already
deleted in the same cycle (by bpf_map_delete_elem inside
reset_shm_pointer_unsafe), so the next iteration cannot find the PID
and the bit is stuck forever. The slot therefore never reaches
threads == 0 and is never compacted out. Over 15h on a workstation
with ~1,300 live PIDs the pool reaches 100% with ~96% stale entries.

Once the pool is full, get_shm_pointer_unsafe() returns NULL. Every
apps_read loop had a bare `continue;` after the NULL check, which
skipped the `key = next_key;` advance, so bpf_map_get_next_key()
re-returned the same key forever and the thread spun at 100% CPU.

FIXES

  A. Split allocation from lookup. Introduce
     netdata_ebpf_lookup_shm_pointer_unsafe(pid) that returns the
     existing slot or NULL and sets no bit. Replace the 16 aggregation
     call sites with the new API. Keep get_shm_pointer_unsafe() at the
     8 BPF-map iteration call sites where allocation is legitimate and
     paired with reset. Aggregation paths additionally gate on the
     module bit being set so they do not consume zero-initialised
     per-module sub-structs.

  B. Replace `continue;` with `goto end_*_loop;` in all 8 apps-read
     loops so `key = next_key;` always runs, preventing the infinite
     loop even when the pool is legitimately exhausted.

  C. Zero freshly-allocated slots. ebpf_find_or_create_index_pid()
     now memset()s the slot on the create branch, so compacted tails
     and PID reuse cannot leak stale bits or stale per-module
     counters into new occupants.

  D. Use the correct kill() error check. Five modules used
     `if (kill(pid, 0))`, which treats EPERM (cross-UID processes) as
     "process is dead" and erroneously deletes the live process's
     kernel BPF map entry and zeroes its shm data. Switched all of
     them to `if (kill(pid, 0) == -1 && errno == ESRCH)`, matching
     the two modules that already did it correctly. Also applied to
     ebpf_parse_proc_files().

  E. Collapse ebpf_reset_specific_pid_data() to its effective body.
     thread_collecting only ever has NETDATA_EBPF_PIDS_PROC_FILE set,
     so the `idx < PROC_FILE` switch was dead code. The function now
     just forwards to ebpf_del_pid_entry(). The only other reader of
     that bitmask, ebpf_release_pid_data(), had zero callers and was
     removed.

  F. Sweep the shm pool on module exit. New helper
     netdata_ebpf_sweep_shm_for_module_unsafe() clears this module's
     bit across every shm slot and lets the existing compaction path
     release slots that become empty. All eight per-PID modules now
     call it under the shm semaphore in their *_exit path, so runtime
     module restarts no longer leak bits.

VERIFICATION

  - Root cause was captured on a live ebpf.plugin pinned at 64% CPU.
    gdb showed an EBPF_READ_FD thread permanently alternating
    bpf_map_get_next_key/bpf_map_lookup_elem at ebpf_fd.c:770-774,
    and ebpf_stat_values = {total = 32768, current = 32768} while
    ps showed 1,307 live PIDs.

  - The diagnosis was independently confirmed by a parallel review
    (codex, qwen, glm, minimax, opus), all converging on the
    alloc-without-reset asymmetry in the aggregation paths.

  - The modified tree builds cleanly (`cmake --build build --target
    ebpf.plugin`). No new warnings in the changed files; only the
    pre-existing vendored-Judy stringop-overflow warnings remain.

* ebpf.plugin: wipe shm on init, unlink on cleanup, drop unused out-param

Close two residual issues around the per-PID shared-memory integration
pool, spotted post-merge:

1. The POSIX shm object `netdata_shm_integration_ebpf` was neither
   truncated on open nor unlinked on shutdown. `shm_open(..., O_CREAT)`
   on an existing object just reopens it, and `ftruncate` to the
   previous size is a no-op, so the 14 MB mapped region retained the
   previous plugin run's bytes. The slot-allocation path memsets each
   new slot, so this was not a correctness bug — but every slot in
   `[current, total)` kept stale PID numbers and per-module counters
   until system reboot. Now:
   - `netdata_integration_initialize_shm()` memsets the whole mapped
     region immediately after `nd_mmap`, so the next plugin instance
     always starts from a clean slab.
   - `netdata_integration_cleanup_shm()` calls `shm_unlink` so a
     clean shutdown removes the backing object from `/dev/shm`.

2. Drop the unused `bool *created` out-parameter on
   `ebpf_find_or_create_index_pid()`. It was added when the memset
   was factored out of the allocation path; since the memset now
   lives inside the function, no caller needs to know whether the
   slot was freshly allocated. Reported by Copilot on PR #22232.

* ebpf.plugin: address Copilot findings on 22232

Four residual issues spotted by Copilot after the initial round of
changes. All valid.

1. get_shm_pointer_unsafe() no longer bails out when the pool is full.
   The caller path for an already-tracked PID must still reach its
   slot so module bits can be updated or cleared. The inner
   ebpf_find_or_create_index_pid() already returns existing slots
   unconditionally and only rejects *new* allocations when full, so
   the redundant outer guard was both dead weight and actively
   harmful — it made existing PIDs unreachable the moment the pool
   filled.

2. Semaphore lifecycle. netdata_integration_initialize_shm() now
   sem_unlinks NETDATA_EBPF_SHM_INTEGRATION_NAME before sem_open, and
   netdata_integration_cleanup_shm() sem_unlinks on the way out. If
   a previous plugin instance crashed while holding the semaphore,
   sem_open(O_CREAT) on the same name would reuse the existing
   semaphore at its last value (O_CREAT's initial-value argument is
   ignored when the semaphore already exists), and the next run
   would spin on sem_timedwait() timeouts forever. Unlinking
   guarantees the initial value of 1 is honoured.

3. Socket apps iteration: the kernel-map delete for a dead socket
   tuple used to run inside `if (local_pid) { ... }`. When the shm
   pool is full and local_pid is NULL, `deleted` sockets were left
   in the BPF map to be re-iterated every cycle, growing the map
   unbounded. Moved bpf_map_delete_elem(fd, &key) outside the
   local_pid branch; it is conditioned only on `deleted`, which
   captures the socket's own freshness signal and is independent of
   shm pool state. (For SOCKET_IDX the generic
   reset_shm_pointer_unsafe() intentionally skips the delete, since
   the socket map is keyed by a tuple, not by pid — comment added.)

…spin (#22232)

* ebpf.plugin: fix PID accounting shared-memory pool leak and 100% CPU spin

The eBPF plugin's per-PID shared-memory pool (32,768 slots backing
/dev/shm/netdata_shm_integration_ebpf) monotonically fills on any host
with normal process churn, and once the pool is exhausted each module's
apps-read loop spins one CPU core at 100% indefinitely.

ROOT CAUSE

netdata_ebpf_get_shm_pointer_unsafe() is an allocate-AND-set-bit API,
but it is called from two incompatible contexts:

  1. The per-module BPF map iteration (ebpf_read_*_apps_table), which
     is paired with netdata_ebpf_reset_shm_pointer_unsafe() via the
     kill(pid, 0) == ESRCH branch. This is a correct producer/consumer.

  2. The apps and cgroup aggregation helpers (*_sum_pids,
     *_update_*_cgroup, *_resume_apps_data), which iterate live-PID
     lists built from /proc and cgroup snapshots and call the same
     allocating accessor. These paths have NO reset counterpart.

Every second, for every live PID seen in /proc or a tracked cgroup,
every enabled eBPF module unconditionally sets its bit in the shm
slot for that PID. A dead PID's bit is only cleared by the owning
module's BPF-map iteration - but the BPF map's entry was already
deleted in the same cycle (by bpf_map_delete_elem inside
reset_shm_pointer_unsafe), so the next iteration cannot find the PID
and the bit is stuck forever. The slot therefore never reaches
threads == 0 and is never compacted out. Over 15h on a workstation
with ~1,300 live PIDs the pool reaches 100% with ~96% stale entries.

Once the pool is full, get_shm_pointer_unsafe() returns NULL. Every
apps_read loop had a bare `continue;` after the NULL check, which
skipped the `key = next_key;` advance, so bpf_map_get_next_key()
re-returned the same key forever and the thread spun at 100% CPU.

FIXES

  A. Split allocation from lookup. Introduce
     netdata_ebpf_lookup_shm_pointer_unsafe(pid) that returns the
     existing slot or NULL and sets no bit. Replace the 16 aggregation
     call sites with the new API. Keep get_shm_pointer_unsafe() at the
     8 BPF-map iteration call sites where allocation is legitimate and
     paired with reset. Aggregation paths additionally gate on the
     module bit being set so they do not consume zero-initialised
     per-module sub-structs.

  B. Replace `continue;` with `goto end_*_loop;` in all 8 apps-read
     loops so `key = next_key;` always runs, preventing the infinite
     loop even when the pool is legitimately exhausted.

  C. Zero freshly-allocated slots. ebpf_find_or_create_index_pid()
     now memset()s the slot on the create branch, so compacted tails
     and PID reuse cannot leak stale bits or stale per-module
     counters into new occupants.

  D. Use the correct kill() error check. Five modules used
     `if (kill(pid, 0))`, which treats EPERM (cross-UID processes) as
     "process is dead" and erroneously deletes the live process's
     kernel BPF map entry and zeroes its shm data. Switched all of
     them to `if (kill(pid, 0) == -1 && errno == ESRCH)`, matching
     the two modules that already did it correctly. Also applied to
     ebpf_parse_proc_files().

  E. Collapse ebpf_reset_specific_pid_data() to its effective body.
     thread_collecting only ever has NETDATA_EBPF_PIDS_PROC_FILE set,
     so the `idx < PROC_FILE` switch was dead code. The function now
     just forwards to ebpf_del_pid_entry(). The only other reader of
     that bitmask, ebpf_release_pid_data(), had zero callers and was
     removed.

  F. Sweep the shm pool on module exit. New helper
     netdata_ebpf_sweep_shm_for_module_unsafe() clears this module's
     bit across every shm slot and lets the existing compaction path
     release slots that become empty. All eight per-PID modules now
     call it under the shm semaphore in their *_exit path, so runtime
     module restarts no longer leak bits.

VERIFICATION

  - Root cause was captured on a live ebpf.plugin pinned at 64% CPU.
    gdb showed an EBPF_READ_FD thread permanently alternating
    bpf_map_get_next_key/bpf_map_lookup_elem at ebpf_fd.c:770-774,
    and ebpf_stat_values = {total = 32768, current = 32768} while
    ps showed 1,307 live PIDs.

  - The diagnosis was independently confirmed by a parallel review
    (codex, qwen, glm, minimax, opus), all converging on the
    alloc-without-reset asymmetry in the aggregation paths.

  - The modified tree builds cleanly (`cmake --build build --target
    ebpf.plugin`). No new warnings in the changed files; only the
    pre-existing vendored-Judy stringop-overflow warnings remain.

* ebpf.plugin: wipe shm on init, unlink on cleanup, drop unused out-param

Close two residual issues around the per-PID shared-memory integration
pool, spotted post-merge:

1. The POSIX shm object `netdata_shm_integration_ebpf` was neither
   truncated on open nor unlinked on shutdown. `shm_open(..., O_CREAT)`
   on an existing object just reopens it, and `ftruncate` to the
   previous size is a no-op, so the 14 MB mapped region retained the
   previous plugin run's bytes. The slot-allocation path memsets each
   new slot, so this was not a correctness bug — but every slot in
   `[current, total)` kept stale PID numbers and per-module counters
   until system reboot. Now:
   - `netdata_integration_initialize_shm()` memsets the whole mapped
     region immediately after `nd_mmap`, so the next plugin instance
     always starts from a clean slab.
   - `netdata_integration_cleanup_shm()` calls `shm_unlink` so a
     clean shutdown removes the backing object from `/dev/shm`.

2. Drop the unused `bool *created` out-parameter on
   `ebpf_find_or_create_index_pid()`. It was added when the memset
   was factored out of the allocation path; since the memset now
   lives inside the function, no caller needs to know whether the
   slot was freshly allocated. Reported by Copilot on PR #22232.

* ebpf.plugin: address Copilot findings on 22232

Four residual issues spotted by Copilot after the initial round of
changes. All valid.

1. get_shm_pointer_unsafe() no longer bails out when the pool is full.
   The caller path for an already-tracked PID must still reach its
   slot so module bits can be updated or cleared. The inner
   ebpf_find_or_create_index_pid() already returns existing slots
   unconditionally and only rejects *new* allocations when full, so
   the redundant outer guard was both dead weight and actively
   harmful — it made existing PIDs unreachable the moment the pool
   filled.

2. Semaphore lifecycle. netdata_integration_initialize_shm() now
   sem_unlinks NETDATA_EBPF_SHM_INTEGRATION_NAME before sem_open, and
   netdata_integration_cleanup_shm() sem_unlinks on the way out. If
   a previous plugin instance crashed while holding the semaphore,
   sem_open(O_CREAT) on the same name would reuse the existing
   semaphore at its last value (O_CREAT's initial-value argument is
   ignored when the semaphore already exists), and the next run
   would spin on sem_timedwait() timeouts forever. Unlinking
   guarantees the initial value of 1 is honoured.

3. Socket apps iteration: the kernel-map delete for a dead socket
   tuple used to run inside `if (local_pid) { ... }`. When the shm
   pool is full and local_pid is NULL, `deleted` sockets were left
   in the BPF map to be re-iterated every cycle, growing the map
   unbounded. Moved bpf_map_delete_elem(fd, &key) outside the
   local_pid branch; it is conditioned only on `deleted`, which
   captures the socket's own freshness signal and is independent of
   shm pool state. (For SOCKET_IDX the generic
   reset_shm_pointer_unsafe() intentionally skips the delete, since
   the socket map is keyed by a tuple, not by pid — comment added.)

(cherry picked from commit bd38550)