salmanoff/docs/deviceAttachmentPipelineSpec.md

Device Attachment Pipeline (DAP) Specification DSL: attaching sensors and actuators to SMO.

DAP Specs vs DA Specs:

DAP (Device Attachment Pipeline) specs are human-readable DSL specifications that describe a pipeline of steps to connect a particular device role on a particular device-identifier to Salmanoff. This document describes the DAP specification format.

DA (Device Attachment) specs are compiled binary structs used internally by SMO after DAP specs have been parsed into binary format. DA specs are the internal representation that the system actually uses.

Multiple Input Formats: DAP specs can be parsed from multiple human-readable formats. For example, we intend to eventually extend ROS's URDF XML format to specify device attachment specs (URDFDA specs), which would also get compiled into the same DA spec binary format.

Attaching sensors:

Sensors are input devices to Salmanoff. Salmanoff will perceive them as perceptual inputs -- like your own sense organs. For example, if you attach a camera as a sensor, salmanoff will experience it in the same way that you experience the visual sense data from your eyes.

StimIface (Stimulus Interface):

A StimIface is a Stimulus Interface library that connects to a particular stim buffer and allows the mind to process the stim features presented in the device's stim buffers. StimIface libraries replace the previous notion of an implexor. They provide the interface between raw device data and the mind's processing capabilities.

Device Attachment Pipeline (DAP) Specification Format:

The general format of a DAP specification is:

sensor-type|dev-identifier|stim-iface-api|stim-buff-api(api-params)|provider(provider-params)|dev-selector

• sensor-type is always either '+idev' (interoceptor), '+edev' (extrospector), or '+adev' (actuator).
• dev-identifier is a user-defined name for this specific device instance. This represents a logical device that can be accessed through multiple providers and may expose multiple stim features. In a sense it's like a sense organ or sense modality.
• stim-iface-api is the name of the StimIface library that should be used to process the data from the stim buffer. This replaces the previous implexor concept.
• stim-buff-api is the interface that provides access to a specific stim buffer from the device. A single device may have multiple stim buffers (e.g., audio output, microphone input, different data streams). The api-params in parentheses may be omitted, in which case the parentheses will be empty, but the parentheses must always be written out.
• provider may be a userspace daemon or an OS kernel that provides access to the device's I/O functionality; and thereby allows the stim-buff-api to construct and present a stim-buffer to Salmanoff. The provider-params in parentheses may be omitted, in which case the parentheses will be empty, but the parentheses must always be written out.
• dev-selector is the idiosyncratic label/name used by the provider to identify the specific device you want to attach via that provider.

stim-buff-api-params and provider-params:

If there's more than one parameter item in a list of stim-buff-api-params or provider-params, then the individual items in a list of stim-buff-api-param or provider-params should be separated by the h-bar character (|). E.g:

+edev|soundcard0|audio-stimiface|alsa-audio(shmem|param2|param3)|alsa()|cardname

Each parameter must be in one of these forms:

• key=value
• key=
• key

Important Note on stim-buff-api-params:

The stim-buff-api-params should never include options related to the stim buffer's type or format. The stim-buff-api must read and infer such configuration details from the stim-iface-api portion of the DAP spec, and configure itself accordingly to enable connection by the specified stim-iface library in the way that it has been configured.

stim-buff-api-params are for options that are:

• Device-specific (not modality-wide)
• Specific to this particular stim-feature as provided by this device
• Configuration parameters needed by the stim-buff-api to properly interface with the device

Examples of appropriate stim-buff-api-params:

• Buffer size settings
• Device-specific communication parameters
• Hardware-specific configuration options
• Connection timeouts or retry settings

Examples of inappropriate stim-buff-api-params:

• Data format specifications (should be inferred from stim-iface-api)
• Color space settings (should be determined by the stim-iface library)
• Processing algorithm parameters (belong to the stim-iface library)

Logical View and Multiple Access Patterns:

Single Device, Multiple Providers:

A single dev-identifier can unite several dev-selectors from multiple providers. For example, a sound card device soundcard0 could be accessed through:

• ident: soundcard0, provider: alsa - Provides access to the card via ALSA API for audio output
• ident: soundcard0, provider: linux-driver-direct-file-ops - Provides direct connection to Linux driver via read/write posix FD calls for beeper sound output
• ident: soundcard0, provider: alsa - Provides access to the card via ALSA for microphone input

So a single physical device is accessed via multiple providers, each with different selectors.

Single Device, Same Provider, Different Stim-Buff-APIs:

A device could have different stim-buff-apis, possibly provided by different shared libraries:

• ident: soundcard0, provider: alsa, stim-buff-api: alsa-audio - For audio output
• ident: soundcard0, provider: alsa, stim-buff-api: alsa-mic - For microphone input

Different stim-buff-apis may be packaged into the same shared library, or multiple libraries may dlopen a common library behind the scenes.

Stim Features and Buffers:

Logically, a dev-identifier represents a sense modality. Each device can export multiple stim features. For example, an eye can export:

• Color data
• Light intensity data
• Thermal heat data
• Pain input data

Each stim feature is exposed as a stim buffer, provided by a stim-buff-api. Stim-buff-apis rely upon providers to implement the device-specific operations required to effectuate the stim-buff controls.

Examples:

To attach a particular window from a window manager:

+edev|my-window|visual-stimiface|wayland()|wayland(server-socket)|window0

Connect to the Wayland server that's listening on server-socket, using the wayland stim-buff-api. Ask that Wayland server to give salmanoff read-access to all of the frames composited into the window buffer for window0. Use salmanoff's visual-stimiface to process the visual data from that window0's compositor buffer.

To attach a window manager's entire rendered desktop:

+edev|my-desktop|visual-stimiface|wayland()|wayland(listen-socket)|all

In most cases, this is basically the same as attempting to attach all of the underlying GFX server's output.

Connect to the Wayland server that's listening on listen-socket, using the wayland stim-buff-api. Ask that Wayland server to give salmanoff read-access to the entire compositor framebuffer. Use salmanoff's visual-stimiface to process the visual data from that Wayland server's compositor buffer.

To attach all of an Xorg server's gfx output to all screens:

+edev|my-xorg-display|visual-stimiface|x11()|xorg(listen-socket)|all

Connect to the Xorg server that's listening on listen-socket, using the x11 stim-buff-api. Ask that Xorg server to let Salmanoff read out all of the frames written out to all screens. Use salmanoff's visual-stimiface to process the visual data from the server's gfx framebuffer.

In most cases, this is basically the same as attempting to attach all of the WM's output.

• Implementation note: https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library Seems relevant.

To attach all of an Xorg server's gfx output to a particular screen:

+edev|my-screen|visual-stimiface|x11()|xorg(listen-socket)|:0

Connect to the Xorg server that's listening on listen-socket, using the x11 stim-buff-api. Ask that Xorg server to let Salmanoff read out all of the frames written out to display :0. Use salmanoff's visual-stimiface to process the visual data from display :0's framebuffer.

• Implementation note: https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library Seems relevant.

To attach a camera device by connecting directly to its Linux driver:

+edev|my-camera|visual-stimiface|v4l()|linux()|/dev/video0

We specify that we want to use the linux kernel's loaded driver to connect to communicate with /dev/video0, via the Video4Linux stim-buff-api. We want salmanoff to use the visual-stimiface library to process the visual data from /dev/video0's stim buffer.

If /dev/video0 is already consumed by another process, this may likely fail.

To attach a microphone that's managed by ALSA server:

+edev|my-microphone|audio-stimiface|alsa-mic(shmem)|alsa()|cardname

Connect to the ALSA server via shmem, using the alsa-mic stim-buff-api. Request access to the microphone function of the sound card with the name cardname. Use the audio-stimiface library to process the audio data from cardname's microphone stim buffer.

To attach a thermal sensor managed by Linux:

+idev|my-thermal|thermal-stimiface|thermal-zone()|linux()|/sys/class/thermal_zone0

Use the thermal-zone SysFS stim-buff-api provided by linux to connect to the sensor /sys/class/thermal_zone0. Use the thermal-stimiface library to process the thermal data from thermal_zone0's heat stim buffer.

Multiple Provider Examples:

Single Sound Card Device with Multiple Providers:

The same physical sound card soundcard0 can be accessed through different providers:

+edev|soundcard0|audio-stimiface|alsa-audio()|alsa()|card0
||| +edev|soundcard0|audio-stimiface|direct-file-ops()|linux()|/dev/snd/pcmC0D0p
||| +idev|soundcard0|audio-stimiface|alsa-mic()|alsa()|card0

This shows:

• soundcard0 accessed via ALSA provider for audio output (alsa-audio stim-buff-api)
• soundcard0 accessed via Linux provider for direct file operations (direct-file-ops stim-buff-api)
• soundcard0 accessed via ALSA provider for microphone input (alsa-mic stim-buff-api)

Single Camera Device with Multiple Stim-Buff-APIs:

A camera device camera0 might expose different data streams:

+edev|camera0|visual-stimiface|v4l-rgb()|linux()|/dev/video0
||| +edev|camera0|visual-stimiface|v4l-yuv()|linux()|/dev/video0
||| +idev|camera0|thermal-stimiface|v4l-thermal()|linux()|/dev/video0

This shows the same camera device providing:

• RGB color data via v4l-rgb stim-buff-api
• YUV color data via v4l-yuv stim-buff-api
• Thermal data via v4l-thermal stim-buff-api

Attaching actuators:

Actuators are Salmanoff's way of enacting changes in the external world. They're like your libs, or your mouth. Actuators enable salmanoff to write outputs to the world outside.

Wilzors:

Actuator devices are analogous to your body's limbs. Salmanoff controls these by using wilzor algorithms. Wilzor is a contraction of Willpower ActuatOr but with a 'Z' in the middle to make it sound cooler. Different types of devices will require different wilzor algorithms. You need to know what type of wilzor algorithm needs to be used to enable salmanoff to control your actuator device.

The general format for an actuator's device attachment specification follows the same pattern:

+adev|dev-identifier|wilzor-algorithm|stim-buff-api(api-params)|provider(provider-params)|dev-selector

Where wilzor-algorithm is the specific wilzor algorithm needed to control the actuator device.

Device Attachment Pipeline (DAP) specification files:

Inside of a DAP specification file, you can list any number of DAP specifications. Separate individual DAP specifications with two consecutive h-bar characters (||), like this:

+edev|my-window|visual-stimiface|wayland()|wayland(server-socket)|window0
|| +edev|my-xorg-display|visual-stimiface|x11()|xorg(listen-socket)|all
|| +idev|my-thermal|thermal-stimiface|thermal-zone()|linux()|/sys/class/thermal_zone0