hayodea
6ef86eea05
This language is used broadly to specify how to attach (and thus also how to detach) devices to/from Salmanoff. The next bit of work we'll do is split off the DSL parsing from the management of the list of parsed binary attached spec objects. We'll be creating a PipeDeviceAttachmentParser, and later on when we support URDF, we'll create a URDFDeviceAttachmentParser.
169 lines
6.7 KiB
Markdown
169 lines
6.7 KiB
Markdown
# Device Attachment Specification DSL: attaching sensors and actuators to SMO.
|
|
|
|
## Attaching sensors:
|
|
|
|
Sensors are input devices to Salmanoff. Salmanoff will perceive them as
|
|
perceptual inputs -- like your own sense organs. For example, if you attach a
|
|
camera as a sensor, salmanoff will experience it in the same way that you
|
|
experience the visual sense data from your eyes.
|
|
|
|
## Implexors:
|
|
|
|
An implexor is an **Imp**licit **Ex**istent isolat**Or** algorithm. It's
|
|
basically what conventional ML/LLM/ANN developers call an ROI ("Region of
|
|
Interest") extraction algorithm. An Implex algorithm is used to scan a frame
|
|
of input sensor data and detect objects and patterns within it.
|
|
|
|
## Sensor device attachment specification:
|
|
|
|
The general format of a device attachment specification for a sensor is:
|
|
```
|
|
sensor-type|dev-identifier
|
|
|implexor|api(api-params)|provider(provider-params)|deviceselector
|
|
```
|
|
|
|
* `sensor-type` is always either '`+idev`' (interoceptor), '`+edev`'
|
|
(extrospector), or '`+adev`' (actuator).
|
|
* `dev-identifier` is a user-defined name for this specific device instance.
|
|
* `implexor` is the name of the implexor algorithm that should be used with
|
|
the data that is provided by the `provider` via the `api`.
|
|
* `api` is the interface that the provider uses to export perceptual data for
|
|
salmanoff to read. Salmanoff will run the `implexor` algorithm on the data
|
|
from this `api`. The `api-param` in parentheses may be omitted, in which
|
|
case the parentheses will be empty, but the parentheses must always be
|
|
written out.
|
|
* `provider` may be a userspace daemon or an OS kernel that provides perceptual
|
|
data via the `api`. The `provider-params` in parentheses may be omitted, in
|
|
which case the parenthesis will be empty, but the parentheses must always be
|
|
written out.
|
|
* `device selector` is the idiosyncratic label/name used by the `provider` to
|
|
identify the specific device you want to attach via that `provider`.
|
|
|
|
## `API-params` and `provider-params`:
|
|
|
|
If there's more than one parameter item in a list of `api-params` or
|
|
`provider-params`, then the individual items in a list of `api-param` or
|
|
`provider-params` should be separated by the h-bar character (`|`). E.g:
|
|
```
|
|
+edev|audio-implexor|alsa(shmem|param2|param3)|alsa()|cardname
|
|
```
|
|
|
|
Each parameter must be in one of these forms:
|
|
* key=value
|
|
* key=
|
|
* key
|
|
|
|
Some examples follow:
|
|
|
|
### To attach a particular window from a window manager:
|
|
```
|
|
+edev|my-window|visual-implexor|wayland()|wayland(server-socket)|window0
|
|
```
|
|
Connect to the Wayland server that's listening on `server-socket`, using the
|
|
`wayland` api. Ask that Wayland server to give salmanoff read-access to all of
|
|
the frames composited into the window buffer for `window0`. Use salmanoff's
|
|
`visual-implexor` to implex from that `window0`'s compositor data.
|
|
|
|
### To attach a window manager's entire rendered desktop:
|
|
```
|
|
+edev|my-desktop|visual-implexor|wayland()|wayland(listen-socket)|all
|
|
```
|
|
In most cases, this is basically the same as attempting to attach all of the
|
|
underlying GFX server's output.
|
|
|
|
Connect to the Wayland server that's listening on `listen-socket`, using the
|
|
`wayland` api. Ask that Wayland server to give salmanoff read-access to the
|
|
entire compositor framebuffer. Use salmanoff's `visual-implexor` to implex from
|
|
that Wayland server's compositor data.
|
|
|
|
### To attach all of an Xorg server's gfx output to all screens:
|
|
```
|
|
+edev|my-xorg-display|visual-implexor|x11()|xorg(listen-socket)|all
|
|
```
|
|
|
|
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
|
|
api. Ask that Xorg server to let Salmanoff read out all of the frames written
|
|
out to all screens. Use salmanoff's `visual-implexor` to implex from the
|
|
server's gfx framebuffer data.
|
|
|
|
In most cases, this is basically the same as attempting to attach all of the
|
|
WM's output.
|
|
|
|
* Implementation note:
|
|
https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
|
|
Seems relevant.
|
|
|
|
### To attach all of an Xorg server's gfx output to a particular screen:
|
|
```
|
|
+edev|my-screen|visual-implexor|x11()|xorg(listen-socket)|:0
|
|
```
|
|
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
|
|
api. Ask that Xorg server to let Salmanoff read out all of the frames written
|
|
out to display `:0`. Use salmanoff's `visual-implexor` to implex from display
|
|
`:0`'s framebuffer data.
|
|
|
|
* Implementation note:
|
|
https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
|
|
Seems relevant.
|
|
|
|
### To attach a camera device by connecting directly to its Linux driver:
|
|
```
|
|
+edev|my-camera|visual-implexor|v4l()|linux()|/dev/video0
|
|
```
|
|
We specify that we want to use the `linux` kernel's loaded driver to connect
|
|
to communicate with `/dev/video0`, via the `Video4Linux` API. We want salmanoff
|
|
to use the `visual-implexor` algorithm to implex from `/dev/video0`'s data.
|
|
|
|
If `/dev/video0` is already consumed by another process, this may likely fail.
|
|
|
|
### To attach a microphone that's managed by ALSA server:
|
|
```
|
|
+edev|my-microphone|audio-implexor|alsa(shmem)|alsa()|cardname
|
|
```
|
|
|
|
Connect to the ALSA server via `shmem`, using the `alsa` API. Request access to
|
|
the microphone function of the sound card with the name `cardname`. Use the
|
|
`audio-implexor` algorithm to implex from `cardname`'s microphone data.
|
|
|
|
### To attach a thermal sensor managed by Linux:
|
|
```
|
|
+idev|my-thermal|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0
|
|
```
|
|
|
|
Use the `thermal-zone` SysFS API provided by `linux` to connect to the sensor
|
|
`/sys/class/thermal_zone0`. Use the `thermal-implexor` implexor to implex from
|
|
`thermal_zone0`'s heat data.
|
|
|
|
## Attaching actuators:
|
|
|
|
Actuators are Salmanoff's way of enacting changes in the external world.
|
|
They're like your libs, or your mouth. Actuators enable salmanoff to write
|
|
outputs to the world outside.
|
|
|
|
### Wilzors:
|
|
|
|
Actuator devices are analogous to your body's limbs. Salmanoff controls these
|
|
by using `wilzor` algorithms. Wilzor is a contraction of **Wil**lpower
|
|
Actuat**Or** but with a 'Z' in the middle to make it sound cooler. Different
|
|
types of devices will require different wilzor algorithms. You need to know
|
|
what type of wilzor algorithm needs to be used to enable salmanoff to control
|
|
your actuator device.
|
|
|
|
The general format for an actuator's device attachment specification is:
|
|
```
|
|
WIP: TBD.
|
|
```
|
|
|
|
## Device attachment specification files:
|
|
|
|
Inside of a device attachment specification file, you can list any number of
|
|
device attachment specifications.
|
|
Separate individual device attachment specifications with two consecutive h-bar
|
|
characters (`||`),
|
|
like this:
|
|
```
|
|
+edev|my-window|visual-implexor|wayland()|wayland(server-socket)|window0
|
|
|| +edev|my-xorg-display|visual-implexor|x11()|xorg(listen-socket)|all
|
|
|| +idev|my-thermal|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0
|
|
```
|