salmanoff/docs/deviceAttachmentSpec.md

Device Attachment Specification DSL: attaching sensors and actuators to SMO.

Attaching sensors:

Sensors are input devices to Salmanoff. Salmanoff will perceive them as perceptual inputs -- like your own sense organs. For example, if you attach a camera as a sensor, salmanoff will experience it in the same way that you experience the visual sense data from your eyes.

Implexors:

An implexor is an Implicit Existent isolatOr algorithm. It's basically what conventional ML/LLM/ANN developers call an ROI ("Region of Interest") extraction algorithm. An Implex algorithm is used to scan a frame of input sensor data and detect objects and patterns within it.

Sensor device attachment specification:

The general format of a device attachment specification for a sensor is:

sensor-type|dev-identifier
  |implexor|api(api-params)|provider(provider-params)|deviceselector

• sensor-type is always either '+idev' (interoceptor), '+edev' (extrospector), or '+adev' (actuator).
• dev-identifier is a user-defined name for this specific device instance.
• implexor is the name of the implexor algorithm that should be used with the data that is provided by the provider via the api.
• api is the interface that the provider uses to export perceptual data for salmanoff to read. Salmanoff will run the implexor algorithm on the data from this api. The api-param in parentheses may be omitted, in which case the parentheses will be empty, but the parentheses must always be written out.
• provider may be a userspace daemon or an OS kernel that provides perceptual data via the api. The provider-params in parentheses may be omitted, in which case the parenthesis will be empty, but the parentheses must always be written out.
• device selector is the idiosyncratic label/name used by the provider to identify the specific device you want to attach via that provider.

API-params and provider-params:

If there's more than one parameter item in a list of api-params or provider-params, then the individual items in a list of api-param or provider-params should be separated by the h-bar character (|). E.g:

+edev|audio-implexor|alsa(shmem|param2|param3)|alsa()|cardname

Each parameter must be in one of these forms:

• key=value
• key=
• key

Some examples follow:

To attach a particular window from a window manager:

+edev|my-window|visual-implexor|wayland()|wayland(server-socket)|window0

Connect to the Wayland server that's listening on server-socket, using the wayland api. Ask that Wayland server to give salmanoff read-access to all of the frames composited into the window buffer for window0. Use salmanoff's visual-implexor to implex from that window0's compositor data.

To attach a window manager's entire rendered desktop:

+edev|my-desktop|visual-implexor|wayland()|wayland(listen-socket)|all

In most cases, this is basically the same as attempting to attach all of the underlying GFX server's output.

Connect to the Wayland server that's listening on listen-socket, using the wayland api. Ask that Wayland server to give salmanoff read-access to the entire compositor framebuffer. Use salmanoff's visual-implexor to implex from that Wayland server's compositor data.

To attach all of an Xorg server's gfx output to all screens:

+edev|my-xorg-display|visual-implexor|x11()|xorg(listen-socket)|all

Connect to the Xorg server that's listening on listen-socket, using the x11 api. Ask that Xorg server to let Salmanoff read out all of the frames written out to all screens. Use salmanoff's visual-implexor to implex from the server's gfx framebuffer data.

In most cases, this is basically the same as attempting to attach all of the WM's output.

• Implementation note: https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library Seems relevant.

To attach all of an Xorg server's gfx output to a particular screen:

+edev|my-screen|visual-implexor|x11()|xorg(listen-socket)|:0

Connect to the Xorg server that's listening on listen-socket, using the x11 api. Ask that Xorg server to let Salmanoff read out all of the frames written out to display :0. Use salmanoff's visual-implexor to implex from display :0's framebuffer data.

• Implementation note: https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library Seems relevant.

To attach a camera device by connecting directly to its Linux driver:

+edev|my-camera|visual-implexor|v4l()|linux()|/dev/video0

We specify that we want to use the linux kernel's loaded driver to connect to communicate with /dev/video0, via the Video4Linux API. We want salmanoff to use the visual-implexor algorithm to implex from /dev/video0's data.

If /dev/video0 is already consumed by another process, this may likely fail.

To attach a microphone that's managed by ALSA server:

+edev|my-microphone|audio-implexor|alsa(shmem)|alsa()|cardname

Connect to the ALSA server via shmem, using the alsa API. Request access to the microphone function of the sound card with the name cardname. Use the audio-implexor algorithm to implex from cardname's microphone data.

To attach a thermal sensor managed by Linux:

+idev|my-thermal|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0

Use the thermal-zone SysFS API provided by linux to connect to the sensor /sys/class/thermal_zone0. Use the thermal-implexor implexor to implex from thermal_zone0's heat data.

Attaching actuators:

Actuators are Salmanoff's way of enacting changes in the external world. They're like your libs, or your mouth. Actuators enable salmanoff to write outputs to the world outside.

Wilzors:

Actuator devices are analogous to your body's limbs. Salmanoff controls these by using wilzor algorithms. Wilzor is a contraction of Willpower ActuatOr but with a 'Z' in the middle to make it sound cooler. Different types of devices will require different wilzor algorithms. You need to know what type of wilzor algorithm needs to be used to enable salmanoff to control your actuator device.

The general format for an actuator's device attachment specification is:

WIP: TBD.

Device attachment specification files:

Inside of a device attachment specification file, you can list any number of device attachment specifications. Separate individual device attachment specifications with two consecutive h-bar characters (||), like this:

+edev|my-window|visual-implexor|wayland()|wayland(server-socket)|window0
|| +edev|my-xorg-display|visual-implexor|x11()|xorg(listen-socket)|all
|| +idev|my-thermal|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0