hayodea
aaae3dcbb2
* Updated docs to reflect this. * This was important in allowing us to write the xcbXorg connection code.
165 lines
6.3 KiB
Markdown
165 lines
6.3 KiB
Markdown
# Device manager: attaching sensors and actuators.
|
|
|
|
## Attaching sensors:
|
|
|
|
Sensors are input devices to Harikoff. Harikoff will perceive them as
|
|
perceptual inputs -- like your own sense organs. For example, if you attach a
|
|
camera as a sensor, harikoff will experience it in the same way that you
|
|
experience the visual sense data from your eyes.
|
|
|
|
## Implexors:
|
|
|
|
An implexor is an **Imp**licit **Ex**istent isolat**Or** algorithm. It's
|
|
basically what conventional ML/LLM/ANN developers call an ROI ("Region of
|
|
Interest") extraction algorithm. An Implex algorithm is used to scan a frame
|
|
of input sensor data and detect objects and patterns within it.
|
|
|
|
## Sensor device spec:
|
|
|
|
The general format of a device-spec for a sensor is:
|
|
```
|
|
sensor-type|implexor|api(api-params)|provider(provider-params)|deviceselector
|
|
```
|
|
|
|
* `sensor-type` is always either '`+idev`' (interoceptor), '`+edev`'
|
|
(extrospector), or '`+adev`' (actuator).
|
|
* `implexor` is the name of the implexor algorithm that should be used with
|
|
the data that is provided by the `provider` via the `api`.
|
|
* `api` is the interface that the provider uses to export perceptual data for
|
|
harikoff to read. Harikoff will run the `implexor` algorithm on the data
|
|
from this `api`. The `api-param` in parentheses may be omitted, in which
|
|
case the parentheses will be empty, but the parentheses must always be
|
|
written out.
|
|
* `provider` may be a userspace daemon or an OS kernel that provides perceptual
|
|
data via the `api`. The `provider-params` in parentheses may be omitted, in
|
|
which case the parenthesis will be empty, but the parentheses must always be
|
|
written out.
|
|
* `device selector` is the idiosyncratic label/name used by the `provider` to
|
|
identify the specific device you want to access via that `provider`.
|
|
|
|
## `API-params` and `provider-params` :
|
|
|
|
If there's more than one parameter item in a list of `api-params` or
|
|
`provider-params`, then the individual items in a list of `api-param` or
|
|
`provider-params` should be separated by the h-bar character (`|`). E.g:
|
|
```
|
|
+edev|audio-implexor|alsa(shmem|param2|param3)|alsa()|cardname
|
|
```
|
|
|
|
Each parameter must be in one of these forms:
|
|
* key=value
|
|
* key=
|
|
* key
|
|
|
|
Some examples follow:
|
|
|
|
### To attach a particular window from a window manager:
|
|
```
|
|
+edev|visual-implexor|wayland()|wayland(server-socket)|window0
|
|
```
|
|
Connect to the Wayland server that's listening on `server-socket`, using the
|
|
`wayland` api. Ask that Wayland server to give harikoff read-access to all of
|
|
the frames composited into the window buffer for `window0`. Use harikoff's
|
|
`visual-implexor` to implex from that `window0`'s compositor data.
|
|
|
|
### To attach a window manager's entire rendered desktop:
|
|
```
|
|
+edev|visual-implexor|wayland()|wayland(listen-socket)|all
|
|
```
|
|
In most cases, this is basically the same as attempting to attach all of the
|
|
underlying GFX server's output.
|
|
|
|
Connect to the Wayland server that's listening on `listen-socket`, using the
|
|
`wayland` api. Ask that Wayland server to give harikoff read-access to the
|
|
entire compositor framebuffer. Use harikoff's `visual-implexor` to implex from
|
|
that Wayland server's compositor data.
|
|
|
|
### To attach all of an Xorg server's gfx output to all screens:
|
|
```
|
|
+edev|visual-implexor|x11()|xorg(listen-socket)|all
|
|
```
|
|
|
|
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
|
|
api. Ask that Xorg server to let Harikoff read out all of the frames written
|
|
out to all screens. Use harikoff's `visual-implexor` to implex from the
|
|
server's gfx framebuffer data.
|
|
|
|
In most cases, this is basically the same as attempting to attach all of the
|
|
WM's output.
|
|
|
|
* Implementation note:
|
|
https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
|
|
Seems relevant.
|
|
|
|
### To attach all of an Xorg server's gfx output to a particular screen:
|
|
```
|
|
+edev|visual-implexor|x11()|xorg(listen-socket)|:0
|
|
```
|
|
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
|
|
api. Ask that Xorg server to let Harikoff read out all of the frames written
|
|
out to display `:0`. Use harikoff's `visual-implexor` to implex from display
|
|
`:0`'s framebuffer data.
|
|
|
|
* Implementation note:
|
|
https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
|
|
Seems relevant.
|
|
|
|
### To attach a camera device by connecting directly to its Linux driver:
|
|
```
|
|
+edev|visual-implexor|v4l()|linux()|/dev/video0
|
|
```
|
|
We specify that we want to use the `linux` kernel's loaded driver to connect
|
|
to communicate with `/dev/video0`, via the `Video4Linux` API. We want harikoff
|
|
to use the `visual-implexor` algorithm to implex from `/dev/video0`'s data.
|
|
|
|
If `/dev/video0` is already consumed by another process, this may likely fail.
|
|
|
|
### To attach a microphone that's managed by ALSA server:
|
|
```
|
|
+edev|audio-implexor|alsa(shmem)|alsa()|cardname
|
|
```
|
|
|
|
Connect to the ALSA server via `shmem`, using the `alsa` API. Request access to
|
|
the microphone function of the sound card with the name `cardname`. Use the
|
|
`audio-implexor` algorithm to implex from `cardname`'s microphone data.
|
|
|
|
### To attach a thermal sensor managed by Linux:
|
|
```
|
|
+idev|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0
|
|
```
|
|
|
|
Use the `thermal-zone` SysFS API provided by `linux` to connect to the sensor
|
|
`/sys/class/thermal_zone0`. Use the `thermal-implexor` implexor to implex from
|
|
`thermal_zone0`'s heat data.
|
|
|
|
## Attaching actuators:
|
|
|
|
Actuators are Harikoff's way of enacting changes in the external world.
|
|
They're like your libs, or your mouth. Actuators enable harikoff to write
|
|
outputs to the world outside.
|
|
|
|
### Wilzors:
|
|
|
|
Actuator devices are analogous to your body's limbs. Harikoff controls these
|
|
by using `wilzor` algorithms. Wilzor is a contraction of **Wil**lpower
|
|
Actuat**Or** but with a 'Z' in the middle to make it sound cooler. Different
|
|
types of devices will require different wilzor algorithms. You need to know
|
|
what type of wilzor algorithm needs to be used to enable harikoff to control
|
|
your actuator device.
|
|
|
|
The general format for an actuator's device spec is:
|
|
```
|
|
WIP: TBD.
|
|
```
|
|
|
|
## Device specification files:
|
|
|
|
Inside of a device spec file, you can list any number of device specs.
|
|
Separate individual device specs with two consecutive h-bar characters (`||`),
|
|
like this:
|
|
```
|
|
+edev|visual-implexor|wayland()|wayland(server-socket)|window0
|
|
|| +edev|visual-implexor|x11()|xorg(listen-socket)|all
|
|
|| +idev|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0
|
|
```
|