Rename DSL: deviceSpec => deviceAttachmentSpec
This language is used broadly to specify how to attach (and thus also how to detach) devices to/from Salmanoff. The next bit of work we'll do is split off the DSL parsing from the management of the list of parsed binary attached spec objects. We'll be creating a PipeDeviceAttachmentParser, and later on when we support URDF, we'll create a URDFDeviceAttachmentParser.
This commit is contained in:
@@ -0,0 +1,168 @@
|
||||
# Device Attachment Specification DSL: attaching sensors and actuators to SMO.
|
||||
|
||||
## Attaching sensors:
|
||||
|
||||
Sensors are input devices to Salmanoff. Salmanoff will perceive them as
|
||||
perceptual inputs -- like your own sense organs. For example, if you attach a
|
||||
camera as a sensor, salmanoff will experience it in the same way that you
|
||||
experience the visual sense data from your eyes.
|
||||
|
||||
## Implexors:
|
||||
|
||||
An implexor is an **Imp**licit **Ex**istent isolat**Or** algorithm. It's
|
||||
basically what conventional ML/LLM/ANN developers call an ROI ("Region of
|
||||
Interest") extraction algorithm. An Implex algorithm is used to scan a frame
|
||||
of input sensor data and detect objects and patterns within it.
|
||||
|
||||
## Sensor device attachment specification:
|
||||
|
||||
The general format of a device attachment specification for a sensor is:
|
||||
```
|
||||
sensor-type|dev-identifier
|
||||
|implexor|api(api-params)|provider(provider-params)|deviceselector
|
||||
```
|
||||
|
||||
* `sensor-type` is always either '`+idev`' (interoceptor), '`+edev`'
|
||||
(extrospector), or '`+adev`' (actuator).
|
||||
* `dev-identifier` is a user-defined name for this specific device instance.
|
||||
* `implexor` is the name of the implexor algorithm that should be used with
|
||||
the data that is provided by the `provider` via the `api`.
|
||||
* `api` is the interface that the provider uses to export perceptual data for
|
||||
salmanoff to read. Salmanoff will run the `implexor` algorithm on the data
|
||||
from this `api`. The `api-param` in parentheses may be omitted, in which
|
||||
case the parentheses will be empty, but the parentheses must always be
|
||||
written out.
|
||||
* `provider` may be a userspace daemon or an OS kernel that provides perceptual
|
||||
data via the `api`. The `provider-params` in parentheses may be omitted, in
|
||||
which case the parenthesis will be empty, but the parentheses must always be
|
||||
written out.
|
||||
* `device selector` is the idiosyncratic label/name used by the `provider` to
|
||||
identify the specific device you want to attach via that `provider`.
|
||||
|
||||
## `API-params` and `provider-params`:
|
||||
|
||||
If there's more than one parameter item in a list of `api-params` or
|
||||
`provider-params`, then the individual items in a list of `api-param` or
|
||||
`provider-params` should be separated by the h-bar character (`|`). E.g:
|
||||
```
|
||||
+edev|audio-implexor|alsa(shmem|param2|param3)|alsa()|cardname
|
||||
```
|
||||
|
||||
Each parameter must be in one of these forms:
|
||||
* key=value
|
||||
* key=
|
||||
* key
|
||||
|
||||
Some examples follow:
|
||||
|
||||
### To attach a particular window from a window manager:
|
||||
```
|
||||
+edev|my-window|visual-implexor|wayland()|wayland(server-socket)|window0
|
||||
```
|
||||
Connect to the Wayland server that's listening on `server-socket`, using the
|
||||
`wayland` api. Ask that Wayland server to give salmanoff read-access to all of
|
||||
the frames composited into the window buffer for `window0`. Use salmanoff's
|
||||
`visual-implexor` to implex from that `window0`'s compositor data.
|
||||
|
||||
### To attach a window manager's entire rendered desktop:
|
||||
```
|
||||
+edev|my-desktop|visual-implexor|wayland()|wayland(listen-socket)|all
|
||||
```
|
||||
In most cases, this is basically the same as attempting to attach all of the
|
||||
underlying GFX server's output.
|
||||
|
||||
Connect to the Wayland server that's listening on `listen-socket`, using the
|
||||
`wayland` api. Ask that Wayland server to give salmanoff read-access to the
|
||||
entire compositor framebuffer. Use salmanoff's `visual-implexor` to implex from
|
||||
that Wayland server's compositor data.
|
||||
|
||||
### To attach all of an Xorg server's gfx output to all screens:
|
||||
```
|
||||
+edev|my-xorg-display|visual-implexor|x11()|xorg(listen-socket)|all
|
||||
```
|
||||
|
||||
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
|
||||
api. Ask that Xorg server to let Salmanoff read out all of the frames written
|
||||
out to all screens. Use salmanoff's `visual-implexor` to implex from the
|
||||
server's gfx framebuffer data.
|
||||
|
||||
In most cases, this is basically the same as attempting to attach all of the
|
||||
WM's output.
|
||||
|
||||
* Implementation note:
|
||||
https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
|
||||
Seems relevant.
|
||||
|
||||
### To attach all of an Xorg server's gfx output to a particular screen:
|
||||
```
|
||||
+edev|my-screen|visual-implexor|x11()|xorg(listen-socket)|:0
|
||||
```
|
||||
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
|
||||
api. Ask that Xorg server to let Salmanoff read out all of the frames written
|
||||
out to display `:0`. Use salmanoff's `visual-implexor` to implex from display
|
||||
`:0`'s framebuffer data.
|
||||
|
||||
* Implementation note:
|
||||
https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
|
||||
Seems relevant.
|
||||
|
||||
### To attach a camera device by connecting directly to its Linux driver:
|
||||
```
|
||||
+edev|my-camera|visual-implexor|v4l()|linux()|/dev/video0
|
||||
```
|
||||
We specify that we want to use the `linux` kernel's loaded driver to connect
|
||||
to communicate with `/dev/video0`, via the `Video4Linux` API. We want salmanoff
|
||||
to use the `visual-implexor` algorithm to implex from `/dev/video0`'s data.
|
||||
|
||||
If `/dev/video0` is already consumed by another process, this may likely fail.
|
||||
|
||||
### To attach a microphone that's managed by ALSA server:
|
||||
```
|
||||
+edev|my-microphone|audio-implexor|alsa(shmem)|alsa()|cardname
|
||||
```
|
||||
|
||||
Connect to the ALSA server via `shmem`, using the `alsa` API. Request access to
|
||||
the microphone function of the sound card with the name `cardname`. Use the
|
||||
`audio-implexor` algorithm to implex from `cardname`'s microphone data.
|
||||
|
||||
### To attach a thermal sensor managed by Linux:
|
||||
```
|
||||
+idev|my-thermal|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0
|
||||
```
|
||||
|
||||
Use the `thermal-zone` SysFS API provided by `linux` to connect to the sensor
|
||||
`/sys/class/thermal_zone0`. Use the `thermal-implexor` implexor to implex from
|
||||
`thermal_zone0`'s heat data.
|
||||
|
||||
## Attaching actuators:
|
||||
|
||||
Actuators are Salmanoff's way of enacting changes in the external world.
|
||||
They're like your libs, or your mouth. Actuators enable salmanoff to write
|
||||
outputs to the world outside.
|
||||
|
||||
### Wilzors:
|
||||
|
||||
Actuator devices are analogous to your body's limbs. Salmanoff controls these
|
||||
by using `wilzor` algorithms. Wilzor is a contraction of **Wil**lpower
|
||||
Actuat**Or** but with a 'Z' in the middle to make it sound cooler. Different
|
||||
types of devices will require different wilzor algorithms. You need to know
|
||||
what type of wilzor algorithm needs to be used to enable salmanoff to control
|
||||
your actuator device.
|
||||
|
||||
The general format for an actuator's device attachment specification is:
|
||||
```
|
||||
WIP: TBD.
|
||||
```
|
||||
|
||||
## Device attachment specification files:
|
||||
|
||||
Inside of a device attachment specification file, you can list any number of
|
||||
device attachment specifications.
|
||||
Separate individual device attachment specifications with two consecutive h-bar
|
||||
characters (`||`),
|
||||
like this:
|
||||
```
|
||||
+edev|my-window|visual-implexor|wayland()|wayland(server-socket)|window0
|
||||
|| +edev|my-xorg-display|visual-implexor|x11()|xorg(listen-socket)|all
|
||||
|| +idev|my-thermal|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0
|
||||
```
|
||||
Reference in New Issue
Block a user