336 lines
14 KiB
Markdown
336 lines
14 KiB
Markdown
# liblcameraDev: libcamera device provider library
|
||
|
||
## Overview
|
||
|
||
`liblcameraDev.so` (`commonLibs/lcameraDev/`) is a Salmanoff common library that
|
||
wraps [libcamera](https://libcamera.org/) and exposes a small, dlopen-able C
|
||
API for acquiring refcounted camera device handles. It is the camera analogue of
|
||
`libxcbXorg.so` and `liblivoxProto1.so`: it owns transport and device lifecycle,
|
||
not stim-buffer semantics.
|
||
|
||
The eventual `lcameraBuff` stim-buff-api (`stimBuffApis/lcameraBuff/`) loads
|
||
this library dynamically (via SMO search paths), resolves cameras from DAP
|
||
`deviceSelector` strings, and shares one capture session per physical camera
|
||
across multiple DAP attachments (e.g. separate H/S/V channel lines).
|
||
|
||
```
|
||
+edev|cam0|colour-hsv-h()|lcameraBuff()|lcameraDev()|lcamera-id:...||
|
||
+edev|cam0|colour-hsv-s()|lcameraBuff()|lcameraDev()|lcamera-id:...||
|
||
+edev|cam0|postrin(...)|negtrin(...)|colour-hsv-v()|lcameraBuff()|lcameraDev()|lcamera-id:...||
|
||
```
|
||
|
||
The **`dev-identifier`** (`cam0` above) is the same on all three lines: one
|
||
logical camera device, three stim-feature attachments (H, S, V). The
|
||
**`deviceSelector`** (provider column) is also typically identical across those
|
||
lines so they resolve to the same physical camera and share one capture session.
|
||
|
||
Layering:
|
||
|
||
```text
|
||
lcameraBuff (stimBuffApi) SMO-facing producers, channel fan-out, intrins
|
||
└─ dlopen ─> lcameraDev (commonLib) libcamera manager, device refcounting
|
||
└─ libcamera0.2 + libcamera-ipa (system packages)
|
||
```
|
||
|
||
Channel splitting, colourspace conversion, threshold masks, and stencils belong
|
||
in a separate shared raster library (`rasterStimulus`, future) and in
|
||
`lcameraBuff`; `lcameraDev` stops at “here is a stable, acquired libcamera
|
||
`Camera` you can configure and stream from.”
|
||
|
||
## Why libcamera (for now)
|
||
|
||
Initial development targets libcamera rather than raw V4L2 so that complex Linux
|
||
camera pipelines (Media Controller graphs, platform ISPs, Raspberry Pi CSI
|
||
modules) are handled by libcamera’s pipeline handlers. Plain USB UVC webcams are
|
||
also supported via libcamera’s `uvcvideo` pipeline on many distributions.
|
||
|
||
This is a **Linux-only** path. It does not provide cross-Unix portability by
|
||
itself; a future `v4l2Linux` provider remains the fallback for generic
|
||
`/dev/videoN` access without libcamera.
|
||
|
||
Runtime packages (Debian/Ubuntu naming):
|
||
|
||
| Package | Purpose |
|
||
|---|---|
|
||
| `libcamera0.2` | Core `libcamera.so`, pipeline handlers, `CameraManager` |
|
||
| `libcamera-ipa` | IPA plugin modules (required on RPi, Rockchip, IPU3, etc.) |
|
||
| `libcamera-dev` | Headers and pkg-config (build only) |
|
||
|
||
`libcamera-v4l2` (`v4l2-compat.so`) is **not** required for native libcamera
|
||
clients; it exists only to wrap legacy V4L2 applications via `LD_PRELOAD`.
|
||
|
||
## DAP roles
|
||
|
||
General line shape (see also `docs/deviceAttachmentPipelineSpec.md`):
|
||
|
||
```text
|
||
sensor-type|dev-identifier|postrin-spec|negtrin-spec|quale-iface-api|stim-buff-api|provider|deviceSelector
|
||
```
|
||
|
||
For camera colour channels:
|
||
|
||
| DAP segment | Value | Role |
|
||
|---|---|---|
|
||
| `dev-identifier` | e.g. `cam0` | User-defined name for this **device instance**; same across H/S/V lines |
|
||
| `quale-iface-api` | `colour-hsv-h()`, `colour-hsv-s()`, `colour-hsv-v()` | Names the stim feature / greyscale channel plane |
|
||
| `stim-buff-api` | `lcameraBuff()` | Salmanoff producer plugin; maps qualeIface → channel |
|
||
| `provider` | `lcameraDev()` | Selects the libcamera-backed device access path |
|
||
| `deviceSelector` | see below | Identifies which physical camera to open (often repeated on each line) |
|
||
|
||
Intrinsic thresholds on the brightness/value channel live in `postrin(...)` /
|
||
`negtrin(...)` segments on the `colour-hsv-v()` line only, following
|
||
`docs/design/intrin-thresholds.md`.
|
||
|
||
Example:
|
||
|
||
```text
|
||
+edev|cam0|colour-hsv-h()|lcameraBuff()|lcameraDev()|lcamera-id:/base/soc/i2c@.../imx219@10||
|
||
+edev|cam0|colour-hsv-s()|lcameraBuff()|lcameraDev()|lcamera-id:/base/soc/i2c@.../imx219@10||
|
||
+edev|cam0|negtrin(interest-thr=40|distraction-thr=70|intolerable-thr=90)|postrin(interest-thr=30|distraction-thr=50|stupefaction-thr=80)|colour-hsv-v()|lcameraBuff()|lcameraDev()|lcamera-id:/base/soc/i2c@.../imx219@10||
|
||
```
|
||
|
||
Compound selector (multiple criteria on one line):
|
||
|
||
```text
|
||
+edev|cam0|colour-hsv-h()|lcameraBuff()|lcameraDev()|lcamera-id:foo bar baz;model:aaaa||
|
||
```
|
||
|
||
All three lines share one underlying `lcameraDev` capture session keyed by the
|
||
resolved libcamera camera ID. `lcameraBuff` must not open the same camera twice.
|
||
|
||
### `lcameraDev()` provider params
|
||
|
||
Reserved for future options. Empty parentheses are valid:
|
||
|
||
```text
|
||
lcameraDev()
|
||
```
|
||
|
||
Possible future params (not implemented in v1):
|
||
|
||
* `ipa-dir=...` — override IPA module search path
|
||
* `log-level=...` — map to `LIBCAMERA_LOG_LEVELS` for this provider instance
|
||
|
||
### `lcameraBuff()` stim-buff-api params
|
||
|
||
Reserved for per-producer options (frame rate caps, pixel format preference).
|
||
Channel selection is driven by the qualeIface name, not by stim-buff-api params.
|
||
|
||
Possible future params:
|
||
|
||
* `fps-hz=30`
|
||
* `width=640|height=480`
|
||
* `pixfmt=NV12` — negotiation hint passed down to `lcameraDev`
|
||
|
||
## Device selector format
|
||
|
||
libcamera’s primary stable identifier is `Camera::id()`: an opaque string,
|
||
unique per camera on the system, stable across reboot when physical topology is
|
||
unchanged. libcamera properties such as `Model` are descriptive but **not**
|
||
guaranteed unique; there is no standard `SerialNumber` property in the core
|
||
property set.
|
||
|
||
### Single criterion
|
||
|
||
`lcameraDev` resolves each criterion using a typed `prefix:value` form. If the
|
||
selector contains no `;` and no `prefix:` delimiter, the entire string is
|
||
treated as a literal `lcamera-id:` value (opaque libcamera ID).
|
||
|
||
| Prefix | Example | Match rule |
|
||
|---|---|---|
|
||
| `lcamera-id:` | `lcamera-id:/base/soc/i2c@1/imx219@10` | Exact `Camera::id()` |
|
||
| *(bare string)* | `/base/soc/i2c@1/imx219@10` | Same as `lcamera-id:` |
|
||
| `index:` | `index:0` | Nth camera from `CameraManager::cameras()` |
|
||
| `model:` | `model:imx219` | Exact match on `properties::Model` |
|
||
| `model-substr:` | `model-substr:Logitech` | Substring match on `properties::Model` |
|
||
| `location:` | `location:external` | `front`, `back`, or `external` |
|
||
|
||
The value after `:` runs to the next `;` or end of string. Values may contain
|
||
spaces (e.g. `lcamera-id:foo bar baz`). Leading/trailing whitespace on each
|
||
value is stripped after parsing.
|
||
|
||
### Compound selectors (semicolon-separated)
|
||
|
||
A `deviceSelector` may contain **multiple criteria** separated by `;`. Every
|
||
criterion must match the **same** enumerated camera (**AND** semantics). This
|
||
lets operators combine a stable libcamera ID with a sanity-check on model name,
|
||
or narrow a fuzzy selector with an extra constraint.
|
||
|
||
```text
|
||
lcamera-id:foo bar baz;model:aaaa
|
||
location:external;model-substr:Logitech
|
||
index:0;model:imx219
|
||
```
|
||
|
||
Parsing rules:
|
||
|
||
1. Split the selector on unescaped `;` into one or more clauses (a single clause
|
||
with no `;` is a compound selector of length 1).
|
||
2. Parse each clause as `prefix:value` (or bare opaque ID if no `:` is present
|
||
in that clause only).
|
||
3. A camera is a candidate only if it satisfies **all** clauses.
|
||
4. **Zero candidates** → attach failure with diagnostics.
|
||
5. **More than one candidate** → attach failure (ambiguous); never pick
|
||
arbitrarily.
|
||
6. Exactly one candidate → resolve session by that camera’s `Camera::id()`.
|
||
|
||
If a value must contain a literal `;`, escape it as `\;` in the selector string
|
||
(same backslash-discard convention as whitespace escaping elsewhere in DAP
|
||
tokens).
|
||
|
||
Resolution algorithm (summary):
|
||
|
||
1. Start `CameraManager` (once per process, inside `lcameraDev`).
|
||
2. Enumerate cameras; build an identity record per camera (`id`, `model`,
|
||
`location`, `systemDevices`).
|
||
3. Parse `deviceSelector` into criterion clauses; apply **all** clauses (AND).
|
||
4. **Zero matches** → attach failure with diagnostic listing known cameras.
|
||
5. **Multiple matches** → attach failure (ambiguous selector); never pick
|
||
arbitrarily.
|
||
6. Return or create a refcounted `LcameraDevice` for the winning `Camera::id()`.
|
||
|
||
Discovery helper: `lcameraDev` should expose a list/print entry point (used by
|
||
`lcameraBuff` verbose mode or a small CLI in `libcamera-tools` style) that prints
|
||
each camera’s `id()`, `model`, and `location` so operators can copy a stable
|
||
`lcamera-id:` into DAP specs after first install.
|
||
|
||
## Shared session and refcounting
|
||
|
||
Unlike X11 windows, a camera has no “sub-objects” inside its feed — the selector
|
||
always designates the whole camera stream. Multiple DAP lines for H, S, and V
|
||
are **views** over one capture session, all under the same `dev-identifier`
|
||
(e.g. `cam0`).
|
||
|
||
```text
|
||
dev-identifier cam0 + deviceSelector (resolved) ──> LcameraDeviceSession (refcounted)
|
||
│
|
||
┌─────────────┼─────────────┐
|
||
▼ ▼ ▼
|
||
lcameraBuff lcameraBuff lcameraBuff
|
||
(colour-hsv-h) (colour-hsv-s) (colour-hsv-v + intrins)
|
||
```
|
||
|
||
Rules:
|
||
|
||
* First `getOrCreate` for a resolved camera ID: enumerate, `acquire()` the
|
||
`libcamera::Camera`, negotiate and start the shared stream.
|
||
* Subsequent `getOrCreate` for the same ID: increment refcount; return the same
|
||
session handle.
|
||
* Last `release`: `release()` the libcamera camera, stop streaming, tear down
|
||
buffers.
|
||
* Different `deviceSelector` strings that resolve to the same libcamera ID share
|
||
one session (even if the selector text differs, e.g. one line uses
|
||
`lcamera-id:...` and another uses a compound selector that resolves to the
|
||
same camera).
|
||
* Attachments with the same `dev-identifier` and equivalent resolved camera are
|
||
the same physical device from SMO’s perspective; channel differences come from
|
||
the qualeIface name on each DAP line.
|
||
|
||
## dlopen API (planned)
|
||
|
||
Exported from `liblcameraDev.so` using `extern "C"` symbols (mirroring
|
||
`livoxProto1`). `lcameraBuff` loads the library with `dlopen` + `dlsym` and
|
||
calls through function pointers.
|
||
|
||
### Lifecycle
|
||
|
||
```c
|
||
/* Start CameraManager; idempotent per process. */
|
||
typedef void lcameraDev_mainFn(void);
|
||
|
||
/* Stop manager, release all devices; called on SMO shutdown. */
|
||
typedef void lcameraDev_exitFn(void);
|
||
```
|
||
|
||
### Device acquisition
|
||
|
||
```c
|
||
struct LcameraDevGetOrCreateResult
|
||
{
|
||
bool success;
|
||
/* Opaque session handle; valid until matching release call. */
|
||
void* deviceSession;
|
||
/* Resolved libcamera camera ID (stable session key). */
|
||
const char* resolvedCameraId;
|
||
};
|
||
|
||
typedef LcameraDevGetOrCreateResult lcameraDev_getOrCreateDeviceFn(
|
||
const char* deviceSelector);
|
||
|
||
typedef void lcameraDev_releaseDeviceFn(void* deviceSession);
|
||
```
|
||
|
||
`deviceSession` is opaque to callers. `lcameraBuff` passes it back when
|
||
detaching; it must not call libcamera APIs directly.
|
||
|
||
### Enumeration (discovery)
|
||
|
||
```c
|
||
struct LcameraDevCameraInfo
|
||
{
|
||
const char* id;
|
||
const char* model; /* empty string if unavailable */
|
||
const char* location; /* "front", "back", "external", or "" */
|
||
};
|
||
|
||
struct LcameraDevEnumerateResult
|
||
{
|
||
size_t count;
|
||
LcameraDevCameraInfo* cameras; /* caller frees via lcameraDev_freeEnumerateResult */
|
||
};
|
||
|
||
typedef LcameraDevEnumerateResult lcameraDev_enumerateCamerasFn(void);
|
||
typedef void lcameraDev_freeEnumerateResultFn(LcameraDevEnumerateResult* result);
|
||
```
|
||
|
||
### Frame access (boundary with lcameraBuff)
|
||
|
||
Exact frame-delivery API is TBD and will likely use a callback or a “dequeue
|
||
latest frame buffer” method on the session handle. `lcameraDev` owns libcamera
|
||
`FrameBuffer` allocation and `Request` requeue; `lcameraBuff` copies or maps the
|
||
plane it needs for the active qualeIface channel.
|
||
|
||
Principle: **one negotiated stream format per session** (prefer native YUV such
|
||
as NV12/YUYV). `lcameraBuff` + `rasterStimulus` derive H/S/V greyscale planes
|
||
from that single stream.
|
||
|
||
## Module layout (planned)
|
||
|
||
```text
|
||
commonLibs/lcameraDev/
|
||
CMakeLists.txt
|
||
lcameraDev.h Public C API + C++ internal headers
|
||
lcameraDev.cpp dlopen exports, CameraManager singleton
|
||
cameraSession.cpp Refcounted session, stream negotiation
|
||
selectorResolve.cpp deviceSelector parsing and matching
|
||
cameraEnumerate.cpp Discovery / identity records
|
||
```
|
||
|
||
Build links against `libcamera` (pkg-config). Does **not** link Salmanoff
|
||
`smocore`; stays a standalone `.so` loadable by `lcameraBuff` and test tools.
|
||
|
||
## Relationship to future work
|
||
|
||
| Component | Responsibility |
|
||
|---|---|
|
||
| `lcameraDev` | libcamera lifecycle, selector resolution, shared capture session |
|
||
| `lcameraBuff` | `StimBuffApiDesc`, `StimulusProducer`, per-channel ring buffers, intrins |
|
||
| `rasterStimulus` (future) | YUV↔HSV, plane extraction, threshold masks, stencil geometry |
|
||
| `xcbWindow` / `waylandWindow` | Separate capture path; reuse `rasterStimulus` only |
|
||
|
||
If libcamera IDs prove insufficient in practice, selector policies can gain
|
||
`by-id:` / `serial:` fallbacks that scan udev/USB metadata without changing the
|
||
`lcameraBuff` DAP surface — only `lcameraDev` selector resolution changes.
|
||
|
||
## Open questions
|
||
|
||
1. **Stream format defaults** — prefer first supported YUV format, or allow
|
||
`lcameraBuff(pixfmt=...)` override?
|
||
2. **Hot-unplug** — on camera removal, fail all attached `lcameraBuff` producers
|
||
and drop the session, or attempt re-enumeration by stored `lcamera-id:`?
|
||
3. **Thread model** — libcamera callbacks vs polling in the Body thread
|
||
production daemon; align with `StimulusProducer::productionCDaemon` timing
|
||
(`CONFIG_STIMBUFF_FRAME_PERIOD_MS`).
|
||
4. **IPA packaging** — document per-platform `apt install` requirements in the
|
||
main README when `lcameraBuff` lands (RPi needs `libcamera-ipa`).
|