hayodea/salmanoff
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 2 Wiki Activity

Docs: Add lcameraDev lib notes; add stencil notes

Browse Source
This commit is contained in:
Hayodea Hekol
2026-06-11 23:05:17 -04:00
parent 69a4782e19
commit e383453278
2 changed files with 349 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/lcamera-dev-lib.md
+335
View File
@@ -0,0 +1,335 @@
# liblcameraDev: libcamera device provider library
## Overview
`liblcameraDev.so` (`commonLibs/lcameraDev/`) is a Salmanoff common library that
wraps [libcamera](https://libcamera.org/) and exposes a small, dlopen-able C
API for acquiring refcounted camera device handles. It is the camera analogue of
`libxcbXorg.so` and `liblivoxProto1.so`: it owns transport and device lifecycle,
not stim-buffer semantics.
The eventual `lcameraBuff` stim-buff-api (`stimBuffApis/lcameraBuff/`) loads
this library dynamically (via SMO search paths), resolves cameras from DAP
`deviceSelector` strings, and shares one capture session per physical camera
across multiple DAP attachments (e.g. separate H/S/V channel lines).
```
+edev|cam0|colour-hsv-h()|lcameraBuff()|lcameraDev()|lcamera-id:...||
+edev|cam0|colour-hsv-s()|lcameraBuff()|lcameraDev()|lcamera-id:...||
+edev|cam0|postrin(...)|negtrin(...)|colour-hsv-v()|lcameraBuff()|lcameraDev()|lcamera-id:...||
```
The **`dev-identifier`** (`cam0` above) is the same on all three lines: one
logical camera device, three stim-feature attachments (H, S, V). The
**`deviceSelector`** (provider column) is also typically identical across those
lines so they resolve to the same physical camera and share one capture session.
Layering:
```text
lcameraBuff (stimBuffApi) SMO-facing producers, channel fan-out, intrins
└─ dlopen ─> lcameraDev (commonLib) libcamera manager, device refcounting
└─ libcamera0.2 + libcamera-ipa (system packages)
```
Channel splitting, colourspace conversion, threshold masks, and stencils belong
in a separate shared raster library (`rasterStimulus`, future) and in
`lcameraBuff`; `lcameraDev` stops at “here is a stable, acquired libcamera
`Camera` you can configure and stream from.”
## Why libcamera (for now)
Initial development targets libcamera rather than raw V4L2 so that complex Linux
camera pipelines (Media Controller graphs, platform ISPs, Raspberry Pi CSI
modules) are handled by libcamera’s pipeline handlers. Plain USB UVC webcams are
also supported via libcamera’s `uvcvideo` pipeline on many distributions.
This is a **Linux-only** path. It does not provide cross-Unix portability by
itself; a future `v4l2Linux` provider remains the fallback for generic
`/dev/videoN` access without libcamera.
Runtime packages (Debian/Ubuntu naming):
| Package | Purpose |
|---|---|
| `libcamera0.2` | Core `libcamera.so`, pipeline handlers, `CameraManager` |
| `libcamera-ipa` | IPA plugin modules (required on RPi, Rockchip, IPU3, etc.) |
| `libcamera-dev` | Headers and pkg-config (build only) |
`libcamera-v4l2` (`v4l2-compat.so`) is **not** required for native libcamera
clients; it exists only to wrap legacy V4L2 applications via `LD_PRELOAD`.
## DAP roles
General line shape (see also `docs/deviceAttachmentPipelineSpec.md`):
```text
sensor-type|dev-identifier|postrin-spec|negtrin-spec|quale-iface-api|stim-buff-api|provider|deviceSelector
```
For camera colour channels:
| DAP segment | Value | Role |
|---|---|---|
| `dev-identifier` | e.g. `cam0` | User-defined name for this **device instance**; same across H/S/V lines |
| `quale-iface-api` | `colour-hsv-h()`, `colour-hsv-s()`, `colour-hsv-v()` | Names the stim feature / greyscale channel plane |
| `stim-buff-api` | `lcameraBuff()` | Salmanoff producer plugin; maps qualeIface → channel |
| `provider` | `lcameraDev()` | Selects the libcamera-backed device access path |
| `deviceSelector` | see below | Identifies which physical camera to open (often repeated on each line) |
Intrinsic thresholds on the brightness/value channel live in `postrin(...)` /
`negtrin(...)` segments on the `colour-hsv-v()` line only, following
`docs/design/intrin-thresholds.md`.
Example:
```text
+edev|cam0|colour-hsv-h()|lcameraBuff()|lcameraDev()|lcamera-id:/base/soc/i2c@.../imx219@10||
+edev|cam0|colour-hsv-s()|lcameraBuff()|lcameraDev()|lcamera-id:/base/soc/i2c@.../imx219@10||
+edev|cam0|negtrin(interest-thr=40|distraction-thr=70|intolerable-thr=90)|postrin(interest-thr=30|distraction-thr=50|stupefaction-thr=80)|colour-hsv-v()|lcameraBuff()|lcameraDev()|lcamera-id:/base/soc/i2c@.../imx219@10||
```
Compound selector (multiple criteria on one line):
```text
+edev|cam0|colour-hsv-h()|lcameraBuff()|lcameraDev()|lcamera-id:foo bar baz;model:aaaa||
```
All three lines share one underlying `lcameraDev` capture session keyed by the
resolved libcamera camera ID. `lcameraBuff` must not open the same camera twice.
### `lcameraDev()` provider params
Reserved for future options. Empty parentheses are valid:
```text
lcameraDev()
```
Possible future params (not implemented in v1):
* `ipa-dir=...` — override IPA module search path
* `log-level=...` — map to `LIBCAMERA_LOG_LEVELS` for this provider instance
### `lcameraBuff()` stim-buff-api params
Reserved for per-producer options (frame rate caps, pixel format preference).
Channel selection is driven by the qualeIface name, not by stim-buff-api params.
Possible future params:
* `fps-hz=30`
* `width=640|height=480`
* `pixfmt=NV12` — negotiation hint passed down to `lcameraDev`
## Device selector format
libcamera’s primary stable identifier is `Camera::id()`: an opaque string,
unique per camera on the system, stable across reboot when physical topology is
unchanged. libcamera properties such as `Model` are descriptive but **not**
guaranteed unique; there is no standard `SerialNumber` property in the core
property set.
### Single criterion
`lcameraDev` resolves each criterion using a typed `prefix:value` form. If the
selector contains no `;` and no `prefix:` delimiter, the entire string is
treated as a literal `lcamera-id:` value (opaque libcamera ID).
| Prefix | Example | Match rule |
|---|---|---|
| `lcamera-id:` | `lcamera-id:/base/soc/i2c@1/imx219@10` | Exact `Camera::id()` |
| *(bare string)* | `/base/soc/i2c@1/imx219@10` | Same as `lcamera-id:` |
| `index:` | `index:0` | Nth camera from `CameraManager::cameras()` |
| `model:` | `model:imx219` | Exact match on `properties::Model` |
| `model-substr:` | `model-substr:Logitech` | Substring match on `properties::Model` |
| `location:` | `location:external` | `front`, `back`, or `external` |
The value after `:` runs to the next `;` or end of string. Values may contain
spaces (e.g. `lcamera-id:foo bar baz`). Leading/trailing whitespace on each
value is stripped after parsing.
### Compound selectors (semicolon-separated)
A `deviceSelector` may contain **multiple criteria** separated by `;`. Every
criterion must match the **same** enumerated camera (**AND** semantics). This
lets operators combine a stable libcamera ID with a sanity-check on model name,
or narrow a fuzzy selector with an extra constraint.
```text
lcamera-id:foo bar baz;model:aaaa
location:external;model-substr:Logitech
index:0;model:imx219
```
Parsing rules:
1. Split the selector on unescaped `;` into one or more clauses (a single clause
with no `;` is a compound selector of length 1).
2. Parse each clause as `prefix:value` (or bare opaque ID if no `:` is present
in that clause only).
3. A camera is a candidate only if it satisfies **all** clauses.
4. **Zero candidates** → attach failure with diagnostics.
5. **More than one candidate** → attach failure (ambiguous); never pick
arbitrarily.
6. Exactly one candidate → resolve session by that camera’s `Camera::id()`.
If a value must contain a literal `;`, escape it as `\;` in the selector string
(same backslash-discard convention as whitespace escaping elsewhere in DAP
tokens).
Resolution algorithm (summary):
1. Start `CameraManager` (once per process, inside `lcameraDev`).
2. Enumerate cameras; build an identity record per camera (`id`, `model`,
`location`, `systemDevices`).
3. Parse `deviceSelector` into criterion clauses; apply **all** clauses (AND).
4. **Zero matches** → attach failure with diagnostic listing known cameras.
5. **Multiple matches** → attach failure (ambiguous selector); never pick
arbitrarily.
6. Return or create a refcounted `LcameraDevice` for the winning `Camera::id()`.
Discovery helper: `lcameraDev` should expose a list/print entry point (used by
`lcameraBuff` verbose mode or a small CLI in `libcamera-tools` style) that prints
each camera’s `id()`, `model`, and `location` so operators can copy a stable
`lcamera-id:` into DAP specs after first install.
## Shared session and refcounting
Unlike X11 windows, a camera has no “sub-objects” inside its feed — the selector
always designates the whole camera stream. Multiple DAP lines for H, S, and V
are **views** over one capture session, all under the same `dev-identifier`
(e.g. `cam0`).
```text
dev-identifier cam0 + deviceSelector (resolved) ──> LcameraDeviceSession (refcounted)
│
┌─────────────┼─────────────┐
▼ ▼ ▼
lcameraBuff lcameraBuff lcameraBuff
(colour-hsv-h) (colour-hsv-s) (colour-hsv-v + intrins)
```
Rules:
* First `getOrCreate` for a resolved camera ID: enumerate, `acquire()` the
`libcamera::Camera`, negotiate and start the shared stream.
* Subsequent `getOrCreate` for the same ID: increment refcount; return the same
session handle.
* Last `release`: `release()` the libcamera camera, stop streaming, tear down
buffers.
* Different `deviceSelector` strings that resolve to the same libcamera ID share
one session (even if the selector text differs, e.g. one line uses
`lcamera-id:...` and another uses a compound selector that resolves to the
same camera).
* Attachments with the same `dev-identifier` and equivalent resolved camera are
the same physical device from SMO’s perspective; channel differences come from
the qualeIface name on each DAP line.
## dlopen API (planned)
Exported from `liblcameraDev.so` using `extern "C"` symbols (mirroring
`livoxProto1`). `lcameraBuff` loads the library with `dlopen` + `dlsym` and
calls through function pointers.
### Lifecycle
```c
/* Start CameraManager; idempotent per process. */
typedef void lcameraDev_mainFn(void);
/* Stop manager, release all devices; called on SMO shutdown. */
typedef void lcameraDev_exitFn(void);
```
### Device acquisition
```c
struct LcameraDevGetOrCreateResult
{
bool success;
/* Opaque session handle; valid until matching release call. */
void* deviceSession;
/* Resolved libcamera camera ID (stable session key). */
const char* resolvedCameraId;
};
typedef LcameraDevGetOrCreateResult lcameraDev_getOrCreateDeviceFn(
const char* deviceSelector);
typedef void lcameraDev_releaseDeviceFn(void* deviceSession);
```
`deviceSession` is opaque to callers. `lcameraBuff` passes it back when
detaching; it must not call libcamera APIs directly.
### Enumeration (discovery)
```c
struct LcameraDevCameraInfo
{
const char* id;
const char* model; /* empty string if unavailable */
const char* location; /* "front", "back", "external", or "" */
};
struct LcameraDevEnumerateResult
{
size_t count;
LcameraDevCameraInfo* cameras; /* caller frees via lcameraDev_freeEnumerateResult */
};
typedef LcameraDevEnumerateResult lcameraDev_enumerateCamerasFn(void);
typedef void lcameraDev_freeEnumerateResultFn(LcameraDevEnumerateResult* result);
```
### Frame access (boundary with lcameraBuff)
Exact frame-delivery API is TBD and will likely use a callback or a “dequeue
latest frame buffer” method on the session handle. `lcameraDev` owns libcamera
`FrameBuffer` allocation and `Request` requeue; `lcameraBuff` copies or maps the
plane it needs for the active qualeIface channel.
Principle: **one negotiated stream format per session** (prefer native YUV such
as NV12/YUYV). `lcameraBuff` + `rasterStimulus` derive H/S/V greyscale planes
from that single stream.
## Module layout (planned)
```text
commonLibs/lcameraDev/
CMakeLists.txt
lcameraDev.h Public C API + C++ internal headers
lcameraDev.cpp dlopen exports, CameraManager singleton
cameraSession.cpp Refcounted session, stream negotiation
selectorResolve.cpp deviceSelector parsing and matching
cameraEnumerate.cpp Discovery / identity records
```
Build links against `libcamera` (pkg-config). Does **not** link Salmanoff
`smocore`; stays a standalone `.so` loadable by `lcameraBuff` and test tools.
## Relationship to future work
| Component | Responsibility |
|---|---|
| `lcameraDev` | libcamera lifecycle, selector resolution, shared capture session |
| `lcameraBuff` | `StimBuffApiDesc`, `StimulusProducer`, per-channel ring buffers, intrins |
| `rasterStimulus` (future) | YUV↔HSV, plane extraction, threshold masks, stencil geometry |
| `xcbWindow` / `waylandWindow` | Separate capture path; reuse `rasterStimulus` only |
If libcamera IDs prove insufficient in practice, selector policies can gain
`by-id:` / `serial:` fallbacks that scan udev/USB metadata without changing the
`lcameraBuff` DAP surface — only `lcameraDev` selector resolution changes.
## Open questions
1. **Stream format defaults** — prefer first supported YUV format, or allow
`lcameraBuff(pixfmt=...)` override?
2. **Hot-unplug** — on camera removal, fail all attached `lcameraBuff` producers
and drop the session, or attempt re-enumeration by stored `lcamera-id:`?
3. **Thread model** — libcamera callbacks vs polling in the Body thread
production daemon; align with `StimulusProducer::productionCDaemon` timing
(`CONFIG_STIMBUFF_FRAME_PERIOD_MS`).
4. **IPA packaging** — document per-platform `apt install` requirements in the
main README when `lcameraBuff` lands (RPi needs `libcamera-ipa`).
docs/prompts/stencils-first-pass-with-camera.md
+14
View File
@@ -0,0 +1,14 @@
Hey, have a look at @docs/2d-3d-mathobjs-and-operation-libs.md , and now I need to think about adding a camera stimbuff device which will ultimately present each of the channels of the YUV input as its own greyscale stimbuff; and then I'll also have 2 intrins on the brighness channel -- 1 negtrin and 1 postrin. We need to be forward thinking about our code modules and isolate code which is common so that we can reuse it when creating a future xcbWindow and later, a waylandWindow stimbuffapi lib. AFAICT, the main difference is that the camera stimbuff can give us yuv/hsv directly, whereas the window managers will give us RGB which we'll have to convert to hsv in an extra step. So it seems like we have a common pipeline to some degree: all of them are ultimately aiming to produce hsv/yuv channel-split greyscale stimbuffs.
Now to implement the intrins on the brightness stimbuff, it'll be just a loop in the production pipeline where we look for any channel values that exceed the intrin's thresholds, and we'll create different masks (within each intrin): one mask for those that exceed intolerable/stupefying-thr, one for those between distraction-thr & stupefying, and one for those between interest-thr and distraction.
We'll construct masks (I think?) from those, and those will be our stencils. So it seems like stencil type varies widely across the different stimbuff types. So the stencil type for image channel data will prolly be masks? And we'll attach values to those for comparator-references.
I imagine that other stimfeat types like sound channels, etc will have their own stencil types. So we need to lean into the piolymorphism of our current stencil base class and try to get good abstractions for constructing them as derived type objects from particular data types, and then ensuring that an agnostic component can use them via polymorphic function calls and get the correct results.
Then, we'll need to be albe to create shape vectors for the masks/stencils.
Then we need to be able to create comparators which can compare these shape vectors, and compare the values of two rasterized subregions of of data?
I think the term stencil is itself becoming ambiguous now because it seems I'm trying to denote both the regions of interesting rasterized data, and a notion of a clipping mask that records what that region is? Or maybe the stencil is both of those things together -- i.e, a stencil abstraction is represented by both of those things acting in concert and the stencil notion cannot be algorithmically completely implemented without both together.
Give me recommendations for how you'd approach this, and take into account the need to create and isolate some shared commonLibs/ .so library that uses OpenCV or whatever other abstractions in a way that can eventually be reused for the window manager screen recording stimbuffs we'll eventually implement.