341 lines
14 KiB
Markdown
341 lines
14 KiB
Markdown
# liblcameraDev: libcamera device provider library
|
||
|
||
## Overview
|
||
|
||
`liblcameraDev.so` (`commonLibs/lcameraDev/`) is a Salmanoff common library that
|
||
wraps [libcamera](https://libcamera.org/) and exposes a small, dlopen-able C
|
||
API for acquiring refcounted camera device handles. It is the camera analogue of
|
||
`libxcbXorg.so` and `liblivoxProto1.so`: it owns transport and device lifecycle,
|
||
not stim-buffer semantics.
|
||
|
||
The eventual `lcameraBuff` stim-buff-api (`stimBuffApis/lcameraBuff/`) loads
|
||
this library dynamically (via SMO search paths), resolves cameras from DAP
|
||
`deviceSelector` strings, and shares one capture session per physical camera
|
||
across multiple DAP attachments (e.g. separate H/S/V channel lines).
|
||
|
||
```
|
||
+edev|cam0|colour-hsv-h()|lcameraBuff()|lcameraDev()|lcamera-id:...||
|
||
+edev|cam0|colour-hsv-s()|lcameraBuff()|lcameraDev()|lcamera-id:...||
|
||
+edev|cam0|postrin(...)|negtrin(...)|colour-hsv-v()|lcameraBuff()|lcameraDev()|lcamera-id:...||
|
||
```
|
||
|
||
The **`dev-identifier`** (`cam0` above) is the same on all three lines: one
|
||
logical camera device, three stim-feature attachments (H, S, V). The
|
||
**`deviceSelector`** (provider column) is also typically identical across those
|
||
lines so they resolve to the same physical camera and share one capture session.
|
||
|
||
Layering:
|
||
|
||
```text
|
||
lcameraBuff (stimBuffApi) SMO-facing producers, channel fan-out, intrins
|
||
└─ dlopen ─> lcameraDev (commonLib) libcamera manager, device refcounting
|
||
└─ libcamera0.2 + libcamera-ipa (system packages)
|
||
```
|
||
|
||
Channel splitting, colourspace conversion, threshold masks, and stencils belong
|
||
in a separate shared raster library (`rasterStimulus`, future) and in
|
||
`lcameraBuff`; `lcameraDev` stops at selector resolution, `CameraManager`
|
||
lifecycle, and a refcounted, **acquired** `libcamera::Camera` handle per
|
||
resolved device. Stream negotiation, pixel-format selection, frame buffers,
|
||
and capture timing belong in `lcameraBuff` (and supporting libraries), not here.
|
||
|
||
## Why libcamera (for now)
|
||
|
||
Initial development targets libcamera rather than raw V4L2 so that complex Linux
|
||
camera pipelines (Media Controller graphs, platform ISPs, Raspberry Pi CSI
|
||
modules) are handled by libcamera’s pipeline handlers. Plain USB UVC webcams are
|
||
also supported via libcamera’s `uvcvideo` pipeline on many distributions.
|
||
|
||
This is a **Linux-only** path. It does not provide cross-Unix portability by
|
||
itself; a future `v4l2Linux` provider remains the fallback for generic
|
||
`/dev/videoN` access without libcamera.
|
||
|
||
Runtime packages (Debian/Ubuntu naming):
|
||
|
||
| Package | Purpose |
|
||
|---|---|
|
||
| `libcamera0.2` | Core `libcamera.so`, pipeline handlers, `CameraManager` |
|
||
| `libcamera-ipa` | IPA plugin modules (required on RPi, Rockchip, IPU3, etc.) |
|
||
| `libcamera-dev` | Headers and pkg-config (build only) |
|
||
|
||
`libcamera-v4l2` (`v4l2-compat.so`) is **not** required for native libcamera
|
||
clients; it exists only to wrap legacy V4L2 applications via `LD_PRELOAD`.
|
||
|
||
## DAP roles
|
||
|
||
General line shape (see also `docs/deviceAttachmentPipelineSpec.md`):
|
||
|
||
```text
|
||
sensor-type|dev-identifier|postrin-spec|negtrin-spec|quale-iface-api|stim-buff-api|provider|deviceSelector
|
||
```
|
||
|
||
For camera colour channels:
|
||
|
||
| DAP segment | Value | Role |
|
||
|---|---|---|
|
||
| `dev-identifier` | e.g. `cam0` | User-defined name for this **device instance**; same across H/S/V lines |
|
||
| `quale-iface-api` | `colour-hsv-h()`, `colour-hsv-s()`, `colour-hsv-v()` | Names the stim feature / greyscale channel plane |
|
||
| `stim-buff-api` | `lcameraBuff()` | Salmanoff producer plugin; maps qualeIface → channel |
|
||
| `provider` | `lcameraDev()` | Selects the libcamera-backed device access path |
|
||
| `deviceSelector` | see below | Identifies which physical camera to open (often repeated on each line) |
|
||
|
||
Intrinsic thresholds on the brightness/value channel live in `postrin(...)` /
|
||
`negtrin(...)` segments on the `colour-hsv-v()` line only, following
|
||
`docs/design/intrin-thresholds.md`.
|
||
|
||
Example:
|
||
|
||
```text
|
||
+edev|cam0|colour-hsv-h()|lcameraBuff()|lcameraDev()|lcamera-id:/base/soc/i2c@.../imx219@10||
|
||
+edev|cam0|colour-hsv-s()|lcameraBuff()|lcameraDev()|lcamera-id:/base/soc/i2c@.../imx219@10||
|
||
+edev|cam0|negtrin(interest-thr=40|distraction-thr=70|intolerable-thr=90)|postrin(interest-thr=30|distraction-thr=50|stupefaction-thr=80)|colour-hsv-v()|lcameraBuff()|lcameraDev()|lcamera-id:/base/soc/i2c@.../imx219@10||
|
||
```
|
||
|
||
Compound selector (multiple criteria on one line):
|
||
|
||
```text
|
||
+edev|cam0|colour-hsv-h()|lcameraBuff()|lcameraDev()|lcamera-id:foo bar baz;model:aaaa||
|
||
```
|
||
|
||
All three lines share one underlying `lcameraDev` capture session keyed by the
|
||
resolved libcamera camera ID. `lcameraBuff` must not open the same camera twice.
|
||
|
||
### `lcameraDev()` provider params
|
||
|
||
Reserved for future options. Empty parentheses are valid:
|
||
|
||
```text
|
||
lcameraDev()
|
||
```
|
||
|
||
Possible future params (not implemented in v1):
|
||
|
||
* `ipa-dir=...` — override IPA module search path
|
||
* `log-level=...` — map to `LIBCAMERA_LOG_LEVELS` for this provider instance
|
||
|
||
### `lcameraBuff()` stim-buff-api params
|
||
|
||
Reserved for per-producer options (frame rate caps, pixel format preference).
|
||
Channel selection is driven by the qualeIface name, not by stim-buff-api params.
|
||
|
||
Possible future params:
|
||
|
||
* `fps-hz=30`
|
||
* `width=640|height=480`
|
||
* `pixfmt=NV12` — negotiation hint for `lcameraBuff` stream setup (not
|
||
`lcameraDev`)
|
||
|
||
## Device selector format
|
||
|
||
libcamera’s primary stable identifier is `Camera::id()`: an opaque string,
|
||
unique per camera on the system, stable across reboot when physical topology is
|
||
unchanged. libcamera properties such as `Model` are descriptive but **not**
|
||
guaranteed unique; there is no standard `SerialNumber` property in the core
|
||
property set.
|
||
|
||
### Single criterion
|
||
|
||
`lcameraDev` resolves each criterion using a typed `prefix:value` form. If the
|
||
selector contains no `;` and no `prefix:` delimiter, the entire string is
|
||
treated as a literal `lcamera-id:` value (opaque libcamera ID).
|
||
|
||
| Prefix | Example | Match rule |
|
||
|---|---|---|
|
||
| `lcamera-id:` | `lcamera-id:/base/soc/i2c@1/imx219@10` | Exact `Camera::id()` |
|
||
| *(bare string)* | `/base/soc/i2c@1/imx219@10` | Same as `lcamera-id:` |
|
||
| `index:` | `index:0` | Nth camera from `CameraManager::cameras()` |
|
||
| `model:` | `model:imx219` | Exact match on `properties::Model` |
|
||
| `model-substr:` | `model-substr:Logitech` | Substring match on `properties::Model` |
|
||
| `location:` | `location:external` | `front`, `back`, or `external` |
|
||
|
||
The value after `:` runs to the next `;` or end of string. Values may contain
|
||
spaces (e.g. `lcamera-id:foo bar baz`). Leading/trailing whitespace on each
|
||
value is stripped after parsing.
|
||
|
||
### Compound selectors (semicolon-separated)
|
||
|
||
A `deviceSelector` may contain **multiple criteria** separated by `;`. Every
|
||
criterion must match the **same** enumerated camera (**AND** semantics). This
|
||
lets operators combine a stable libcamera ID with a sanity-check on model name,
|
||
or narrow a fuzzy selector with an extra constraint.
|
||
|
||
```text
|
||
lcamera-id:foo bar baz;model:aaaa
|
||
location:external;model-substr:Logitech
|
||
index:0;model:imx219
|
||
```
|
||
|
||
Parsing rules:
|
||
|
||
1. Split the selector on unescaped `;` into one or more clauses (a single clause
|
||
with no `;` is a compound selector of length 1).
|
||
2. Parse each clause as `prefix:value` (or bare opaque ID if no `:` is present
|
||
in that clause only).
|
||
3. A camera is a candidate only if it satisfies **all** clauses.
|
||
4. **Zero candidates** → attach failure with diagnostics.
|
||
5. **More than one candidate** → attach failure (ambiguous); never pick
|
||
arbitrarily.
|
||
6. Exactly one candidate → resolve session by that camera’s `Camera::id()`.
|
||
|
||
If a value must contain a literal `;`, escape it as `\;` in the selector string
|
||
(same backslash-discard convention as whitespace escaping elsewhere in DAP
|
||
tokens).
|
||
|
||
Resolution algorithm (summary):
|
||
|
||
1. Start `CameraManager` in `lcameraDev_main` (once per process).
|
||
2. Enumerate cameras; build an identity record per camera (`id`, `model`,
|
||
`location`, `systemDevices`).
|
||
3. Parse `deviceSelector` into criterion clauses; apply **all** clauses (AND).
|
||
4. **Zero matches** → attach failure with diagnostic listing known cameras.
|
||
5. **Multiple matches** → attach failure (ambiguous selector); never pick
|
||
arbitrarily.
|
||
6. Return or create a refcounted `LcameraDevice` for the winning `Camera::id()`.
|
||
|
||
Discovery helper: `lcameraDev` should expose a list/print entry point (used by
|
||
`lcameraBuff` verbose mode or a small CLI in `libcamera-tools` style) that prints
|
||
each camera’s `id()`, `model`, and `location` so operators can copy a stable
|
||
`lcamera-id:` into DAP specs after first install.
|
||
|
||
## Shared session and refcounting
|
||
|
||
Unlike X11 windows, a camera has no “sub-objects” inside its feed — the selector
|
||
always designates the whole physical camera. Multiple DAP lines for H, S, and V
|
||
share one **device session** (acquired `libcamera::Camera`), all under the same
|
||
`dev-identifier` (e.g. `cam0`).
|
||
|
||
```text
|
||
dev-identifier cam0 + deviceSelector (resolved) ──> LcameraDeviceSession (refcounted)
|
||
│
|
||
┌─────────────┼─────────────┐
|
||
▼ ▼ ▼
|
||
lcameraBuff lcameraBuff lcameraBuff
|
||
(colour-hsv-h) (colour-hsv-s) (colour-hsv-v + intrins)
|
||
```
|
||
|
||
Rules:
|
||
|
||
* First `getOrCreate` for a resolved camera ID: enumerate, `acquire()` the
|
||
`libcamera::Camera`, create the session entry.
|
||
* Subsequent `getOrCreate` for the same ID: increment refcount; return the same
|
||
session handle.
|
||
* Last `release`: `release()` the libcamera camera and erase the session entry.
|
||
* Different `deviceSelector` strings that resolve to the same libcamera ID share
|
||
one session (even if the selector text differs, e.g. one line uses
|
||
`lcamera-id:...` and another uses a compound selector that resolves to the
|
||
same camera).
|
||
* Attachments with the same `dev-identifier` and equivalent resolved camera are
|
||
the same physical device from SMO’s perspective; channel differences come from
|
||
the qualeIface name on each DAP line.
|
||
|
||
Streaming, frame delivery, and colourspace work are **out of scope** for
|
||
`lcameraDev`; `lcameraBuff` uses the session’s acquired camera handle to set up
|
||
capture on attach.
|
||
|
||
## dlopen API
|
||
|
||
Exported from `liblcameraDev.so` using `extern "C"` symbols (mirroring
|
||
`livoxProto1`). `lcameraBuff` loads the library with `dlopen` + `dlsym` and
|
||
calls through function pointers. Hot-path operations are **`*CReq` coroutine
|
||
invokers** (`sscl::co::ViralNonPostingInvoker`); `main` / `exit` remain
|
||
synchronous.
|
||
|
||
Header: `commonLibs/lcameraDev/lcameraDev.h`.
|
||
|
||
### Lifecycle
|
||
|
||
```c
|
||
typedef void lcameraDev_mainFn(
|
||
const std::shared_ptr<sscl::ComponentThread>& componentThread);
|
||
|
||
typedef void lcameraDev_exitFn(void);
|
||
```
|
||
|
||
`lcameraDev_main` records the Body `ComponentThread`, starts libcamera's
|
||
`CameraManager` (idempotent), and must succeed before any `*CReq` runs.
|
||
|
||
### Device acquisition
|
||
|
||
```cpp
|
||
struct LcameraDevGetOrCreateResult
|
||
{
|
||
std::shared_ptr<CameraSession> deviceSession;
|
||
CameraIdentityRecord resolvedIdentity;
|
||
};
|
||
|
||
typedef sscl::co::ViralNonPostingInvoker<LcameraDevGetOrCreateResult>
|
||
lcameraDev_getOrCreateDeviceCReqFn(const std::string& deviceSelector);
|
||
|
||
typedef sscl::co::ViralNonPostingInvoker<void>
|
||
lcameraDev_releaseDeviceCReqFn(
|
||
const std::shared_ptr<CameraSession>& deviceSession);
|
||
```
|
||
|
||
Failures throw `std::exception`. `lcameraBuff` holds the returned
|
||
`shared_ptr<CameraSession>` and passes it back to `releaseDeviceCReq`. The
|
||
session wraps the acquired `libcamera::Camera`; higher layers configure and
|
||
stream from that handle — `lcameraDev` does not expose frame or stream APIs.
|
||
|
||
### Enumeration (discovery)
|
||
|
||
```cpp
|
||
struct LcameraDevCameraInfo
|
||
{
|
||
std::string id;
|
||
std::string model;
|
||
std::string location;
|
||
};
|
||
|
||
typedef sscl::co::ViralNonPostingInvoker<std::vector<LcameraDevCameraInfo>>
|
||
lcameraDev_enumerateCamerasCReqFn(void);
|
||
```
|
||
|
||
### Manual verification tools
|
||
|
||
When built with `-DENABLE_LIB_lcameraDev=ON`:
|
||
|
||
* `lcameraDev_list_cameras` — runs `enumerateCamerasCReq` on a minimal probe
|
||
`ComponentThread`.
|
||
* `lcameraDev_probe <deviceSelector>` — `getOrCreateDeviceCReq`, then
|
||
`releaseDeviceCReq` (selector and session attach/detach only).
|
||
|
||
## Module layout
|
||
|
||
```text
|
||
commonLibs/lcameraDev/
|
||
CMakeLists.txt
|
||
lcameraDev.h / lcameraDev.cpp Public C API and dlopen exports
|
||
cameraManagerState.h / .cpp CameraManager singleton, session map
|
||
cameraSession.h / .cpp Refcounted acquired-camera session
|
||
cameraIdentity.h / .cpp Discovery identity records
|
||
selectorParse.h / .cpp Compound selector parsing
|
||
selectorResolve.h / .cpp AND-match resolution
|
||
tools/ lcameraDev_list_cameras, lcameraDev_probe
|
||
```
|
||
|
||
Build links against `libcamera` (pkg-config). Does **not** link Salmanoff
|
||
`smocore`; stays a standalone `.so` loadable by `lcameraBuff` and test tools.
|
||
|
||
## Relationship to future work
|
||
|
||
| Component | Responsibility |
|
||
|---|---|
|
||
| `lcameraDev` | libcamera lifecycle, selector resolution, refcounted acquired camera session |
|
||
| `lcameraBuff` | Stream setup, frames, `StimBuffApiDesc`, channel fan-out, intrins |
|
||
| `rasterStimulus` (future) | YUV↔HSV, plane extraction, threshold masks, stencil geometry |
|
||
| `xcbWindow` / `waylandWindow` | Separate capture path; reuse `rasterStimulus` only |
|
||
|
||
If libcamera IDs prove insufficient in practice, selector policies can gain
|
||
`by-id:` / `serial:` fallbacks that scan udev/USB metadata without changing the
|
||
`lcameraBuff` DAP surface — only `lcameraDev` selector resolution changes.
|
||
|
||
## Open questions
|
||
|
||
1. **Hot-unplug** — on camera removal, fail all attached `lcameraBuff` producers
|
||
and drop the session, or attempt re-enumeration by stored `lcamera-id:`?
|
||
2. **IPA packaging** — document per-platform `apt install` requirements in the
|
||
main README when `lcameraBuff` lands (RPi needs `libcamera-ipa`).
|
||
|
||
Stream format, frame timing, and libcamera callback threading are owned by
|
||
`lcameraBuff`, not `lcameraDev`.
|