salmanoff/docs/deviceAttachmentSpec.md

# Device Attachment Specification DSL: attaching sensors and actuators to SMO.

## Attaching sensors:

Sensors are input devices to Salmanoff. Salmanoff will perceive them as
perceptual inputs -- like your own sense organs. For example, if you attach a
camera as a sensor, salmanoff will experience it in the same way that you
experience the visual sense data from your eyes.

## Implexors:

An implexor is an **Imp**licit **Ex**istent isolat**Or** algorithm. It's
basically what conventional ML/LLM/ANN developers call an ROI ("Region of
Interest") extraction algorithm. An Implex algorithm is used to scan a frame
of input sensor data and detect objects and patterns within it.

## Sensor device attachment specification:

The general format of a device attachment specification for a sensor is:
```
sensor-type|dev-identifier
  |implexor|api(api-params)|provider(provider-params)|deviceselector
```

* `sensor-type` is always either '`+idev`' (interoceptor), '`+edev`'
  (extrospector), or '`+adev`' (actuator).
* `dev-identifier` is a user-defined name for this specific device instance.
* `implexor` is the name of the implexor algorithm that should be used with
  the data that is provided by the `provider` via the `api`.
* `api` is the interface that the provider uses to export perceptual data for
  salmanoff to read. Salmanoff will run the `implexor` algorithm on the data
  from this `api`. The `api-param` in parentheses may be omitted, in which
  case the parentheses will be empty, but the parentheses must always be
  written out.
* `provider` may be a userspace daemon or an OS kernel that provides perceptual
  data via the `api`. The `provider-params` in parentheses may be omitted, in
  which case the parenthesis will be empty, but the parentheses must always be
  written out.
* `device selector` is the idiosyncratic label/name used by the `provider` to
  identify the specific device you want to attach via that `provider`.

## `API-params` and `provider-params`:

If there's more than one parameter item in a list of `api-params` or
`provider-params`, then the individual items in a list of `api-param` or
`provider-params` should be separated by the h-bar character (`|`). E.g:
```
+edev|audio-implexor|alsa(shmem|param2|param3)|alsa()|cardname
```

Each parameter must be in one of these forms:
* key=value
* key=
* key

Some examples follow:

### To attach a particular window from a window manager:
```
+edev|my-window|visual-implexor|wayland()|wayland(server-socket)|window0
```
Connect to the Wayland server that's listening on `server-socket`, using the
`wayland` api. Ask that Wayland server to give salmanoff read-access to all of
the frames composited into the window buffer for `window0`. Use salmanoff's
`visual-implexor` to implex from that `window0`'s compositor data.

### To attach a window manager's entire rendered desktop:
```
+edev|my-desktop|visual-implexor|wayland()|wayland(listen-socket)|all
```
In most cases, this is basically the same as attempting to attach all of the
underlying GFX server's output.

Connect to the Wayland server that's listening on `listen-socket`, using the
`wayland` api. Ask that Wayland server to give salmanoff read-access to the
entire compositor framebuffer. Use salmanoff's `visual-implexor` to implex from
that Wayland server's compositor data.

### To attach all of an Xorg server's gfx output to all screens:
```
+edev|my-xorg-display|visual-implexor|x11()|xorg(listen-socket)|all
```

Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
api. Ask that Xorg server to let Salmanoff read out all of the frames written
out to all screens. Use salmanoff's `visual-implexor` to implex from the
server's gfx framebuffer data.

In most cases, this is basically the same as attempting to attach all of the
WM's output.

* Implementation note:
  https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
  Seems relevant.

### To attach all of an Xorg server's gfx output to a particular screen:
```
+edev|my-screen|visual-implexor|x11()|xorg(listen-socket)|:0
```
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
api. Ask that Xorg server to let Salmanoff read out all of the frames written
out to display `:0`. Use salmanoff's `visual-implexor` to implex from display
`:0`'s framebuffer data.

* Implementation note:
  https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
  Seems relevant.

### To attach a camera device by connecting directly to its Linux driver:
```
+edev|my-camera|visual-implexor|v4l()|linux()|/dev/video0
```
We specify that we want to use the `linux` kernel's loaded driver to connect
to communicate with `/dev/video0`, via the `Video4Linux` API. We want salmanoff
to use the `visual-implexor` algorithm to implex from `/dev/video0`'s data.

If `/dev/video0` is already consumed by another process, this may likely fail.

### To attach a microphone that's managed by ALSA server:
```
+edev|my-microphone|audio-implexor|alsa(shmem)|alsa()|cardname
```

Connect to the ALSA server via `shmem`, using the `alsa` API. Request access to
the microphone function of the sound card with the name `cardname`. Use the
`audio-implexor` algorithm to implex from `cardname`'s microphone data.

### To attach a thermal sensor managed by Linux:
```
+idev|my-thermal|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0
```

Use the `thermal-zone` SysFS API provided by `linux` to connect to the sensor
`/sys/class/thermal_zone0`. Use the `thermal-implexor` implexor to implex from
`thermal_zone0`'s heat data.

## Attaching actuators:

Actuators are Salmanoff's way of enacting changes in the external world.
They're like your libs, or your mouth. Actuators enable salmanoff to write
outputs to the world outside.

### Wilzors:

Actuator devices are analogous to your body's limbs. Salmanoff controls these
by using `wilzor` algorithms. Wilzor is a contraction of **Wil**lpower
Actuat**Or** but with a 'Z' in the middle to make it sound cooler. Different
types of devices will require different wilzor algorithms. You need to know
what type of wilzor algorithm needs to be used to enable salmanoff to control
your actuator device.

The general format for an actuator's device attachment specification is:
```
WIP: TBD.
```

## Device attachment specification files:

Inside of a device attachment specification file, you can list any number of
device attachment specifications.
Separate individual device attachment specifications with two consecutive h-bar
characters (`||`),
like this:
```
+edev|my-window|visual-implexor|wayland()|wayland(server-socket)|window0
|| +edev|my-xorg-display|visual-implexor|x11()|xorg(listen-socket)|all
|| +idev|my-thermal|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0
```