Rename DASpec.md=>deviceAttachmentPipelineSpec.md
This commit is contained in:
@@ -0,0 +1,245 @@
|
||||
# Device Attachment Pipeline (DAP) Specification DSL: attaching sensors and actuators to SMO.
|
||||
|
||||
## DAP Specs vs DA Specs:
|
||||
|
||||
**DAP (Device Attachment Pipeline) specs** are human-readable DSL specifications that describe a pipeline of steps to connect a particular device role on a particular device-identifier to Salmanoff. This document describes the DAP specification format.
|
||||
|
||||
**DA (Device Attachment) specs** are compiled binary structs used internally by SMO after DAP specs have been parsed into binary format. DA specs are the internal representation that the system actually uses.
|
||||
|
||||
**Multiple Input Formats**: DAP specs can be parsed from multiple human-readable formats. For example, we intend to eventually extend ROS's URDF XML format to specify device attachment specs (URDFDA specs), which would also get compiled into the same DA spec binary format.
|
||||
|
||||
## Attaching sensors:
|
||||
|
||||
Sensors are input devices to Salmanoff. Salmanoff will perceive them as
|
||||
perceptual inputs -- like your own sense organs. For example, if you attach a
|
||||
camera as a sensor, salmanoff will experience it in the same way that you
|
||||
experience the visual sense data from your eyes.
|
||||
|
||||
## StimIface (Stimulus Interface):
|
||||
|
||||
A StimIface is a **Stim**ulus **I**nter**face** library that connects to a
|
||||
particular stim buffer and allows the mind to process the stim features
|
||||
presented in the device's stim buffers. StimIface libraries replace the
|
||||
previous notion of an implexor. They provide the interface between raw device
|
||||
data and the mind's processing capabilities.
|
||||
|
||||
## Device Attachment Pipeline (DAP) Specification Format:
|
||||
|
||||
The general format of a DAP specification is:
|
||||
```
|
||||
sensor-type|dev-identifier|stim-iface-api|stim-buff-api(api-params)|provider(provider-params)|dev-selector
|
||||
```
|
||||
|
||||
* `sensor-type` is always either '`+idev`' (interoceptor), '`+edev`'
|
||||
(extrospector), or '`+adev`' (actuator).
|
||||
* `dev-identifier` is a user-defined name for this specific device instance.
|
||||
This represents a logical device that can be accessed through multiple
|
||||
providers and may expose multiple stim features.
|
||||
* `stim-iface-api` is the name of the StimIface library that should be used to
|
||||
process the data from the stim buffer. This replaces the previous implexor
|
||||
concept.
|
||||
* `stim-buff-api` is the interface that provides access to a specific stim
|
||||
buffer from the device. A single device may have multiple stim buffers
|
||||
(e.g., audio output, microphone input, different data streams). The
|
||||
`api-params` in parentheses may be omitted, in which case the parentheses
|
||||
will be empty, but the parentheses must always be written out.
|
||||
* `provider` may be a userspace daemon or an OS kernel that provides access to
|
||||
the device's stim buffers via the `stim-buff-api`. The `provider-params` in
|
||||
parentheses may be omitted, in which case the parentheses will be empty, but
|
||||
the parentheses must always be written out.
|
||||
* `dev-selector` is the idiosyncratic label/name used by the `provider` to
|
||||
identify the specific device you want to attach via that `provider`.
|
||||
|
||||
## `stim-buff-api-params` and `provider-params`:
|
||||
|
||||
If there's more than one parameter item in a list of `stim-buff-api-params` or
|
||||
`provider-params`, then the individual items in a list of `stim-buff-api-param` or
|
||||
`provider-params` should be separated by the h-bar character (`|`). E.g:
|
||||
```
|
||||
+edev|soundcard0|audio-stimiface|alsa-audio(shmem|param2|param3)|alsa()|cardname
|
||||
```
|
||||
|
||||
Each parameter must be in one of these forms:
|
||||
* key=value
|
||||
* key=
|
||||
* key
|
||||
|
||||
## Logical View and Multiple Access Patterns:
|
||||
|
||||
### Single Device, Multiple Providers:
|
||||
|
||||
A single `dev-identifier` can unite several `dev-selectors` from multiple providers.
|
||||
For example, a sound card device `soundcard0` could be accessed through:
|
||||
|
||||
* `ident: soundcard0, provider: alsa` - Provides access to the card via ALSA API for audio output
|
||||
* `ident: soundcard0, provider: linux-driver-direct-file-ops` - Provides direct connection to Linux driver via read/write posix FD calls for beeper sound output
|
||||
* `ident: soundcard0, provider: alsa` - Provides access to the card via ALSA for microphone input
|
||||
|
||||
So a single physical device is accessed via multiple providers, each with different selectors.
|
||||
|
||||
### Single Device, Same Provider, Different Stim-Buff-APIs:
|
||||
|
||||
A device could have different `stim-buff-apis`, possibly provided by different shared libraries:
|
||||
|
||||
* `ident: soundcard0, provider: alsa, stim-buff-api: alsa-audio` - For audio output
|
||||
* `ident: soundcard0, provider: alsa, stim-buff-api: alsa-mic` - For microphone input
|
||||
|
||||
Different stim-buff-apis may be packaged into the same shared library, or multiple libraries may dlopen a common library behind the scenes.
|
||||
|
||||
### Stim Features and Buffers:
|
||||
|
||||
Logically, a `dev-identifier` represents a sense modality. Each device can export multiple stim features. For example, an eye can export:
|
||||
- Color data
|
||||
- Light intensity data
|
||||
- Thermal heat data
|
||||
- Pain input data
|
||||
|
||||
Each stim feature is exposed as a stim buffer, provided by a `stim-buff-api`. Stim-buff-apis rely upon providers to implement the device-specific operations required to effectuate the stim-buff controls.
|
||||
|
||||
## Examples:
|
||||
|
||||
### To attach a particular window from a window manager:
|
||||
```
|
||||
+edev|my-window|visual-stimiface|wayland()|wayland(server-socket)|window0
|
||||
```
|
||||
Connect to the Wayland server that's listening on `server-socket`, using the
|
||||
`wayland` stim-buff-api. Ask that Wayland server to give salmanoff read-access to all of
|
||||
the frames composited into the window buffer for `window0`. Use salmanoff's
|
||||
`visual-stimiface` to process the visual data from that `window0`'s compositor buffer.
|
||||
|
||||
### To attach a window manager's entire rendered desktop:
|
||||
```
|
||||
+edev|my-desktop|visual-stimiface|wayland()|wayland(listen-socket)|all
|
||||
```
|
||||
In most cases, this is basically the same as attempting to attach all of the
|
||||
underlying GFX server's output.
|
||||
|
||||
Connect to the Wayland server that's listening on `listen-socket`, using the
|
||||
`wayland` stim-buff-api. Ask that Wayland server to give salmanoff read-access to the
|
||||
entire compositor framebuffer. Use salmanoff's `visual-stimiface` to process the visual data from
|
||||
that Wayland server's compositor buffer.
|
||||
|
||||
### To attach all of an Xorg server's gfx output to all screens:
|
||||
```
|
||||
+edev|my-xorg-display|visual-stimiface|x11()|xorg(listen-socket)|all
|
||||
```
|
||||
|
||||
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
|
||||
stim-buff-api. Ask that Xorg server to let Salmanoff read out all of the frames written
|
||||
out to all screens. Use salmanoff's `visual-stimiface` to process the visual data from the
|
||||
server's gfx framebuffer.
|
||||
|
||||
In most cases, this is basically the same as attempting to attach all of the
|
||||
WM's output.
|
||||
|
||||
* Implementation note:
|
||||
https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
|
||||
Seems relevant.
|
||||
|
||||
### To attach all of an Xorg server's gfx output to a particular screen:
|
||||
```
|
||||
+edev|my-screen|visual-stimiface|x11()|xorg(listen-socket)|:0
|
||||
```
|
||||
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
|
||||
stim-buff-api. Ask that Xorg server to let Salmanoff read out all of the frames written
|
||||
out to display `:0`. Use salmanoff's `visual-stimiface` to process the visual data from display
|
||||
`:0`'s framebuffer.
|
||||
|
||||
* Implementation note:
|
||||
https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
|
||||
Seems relevant.
|
||||
|
||||
### To attach a camera device by connecting directly to its Linux driver:
|
||||
```
|
||||
+edev|my-camera|visual-stimiface|v4l()|linux()|/dev/video0
|
||||
```
|
||||
We specify that we want to use the `linux` kernel's loaded driver to connect
|
||||
to communicate with `/dev/video0`, via the `Video4Linux` stim-buff-api. We want salmanoff
|
||||
to use the `visual-stimiface` library to process the visual data from `/dev/video0`'s stim buffer.
|
||||
|
||||
If `/dev/video0` is already consumed by another process, this may likely fail.
|
||||
|
||||
### To attach a microphone that's managed by ALSA server:
|
||||
```
|
||||
+edev|my-microphone|audio-stimiface|alsa-mic(shmem)|alsa()|cardname
|
||||
```
|
||||
|
||||
Connect to the ALSA server via `shmem`, using the `alsa-mic` stim-buff-api. Request access to
|
||||
the microphone function of the sound card with the name `cardname`. Use the
|
||||
`audio-stimiface` library to process the audio data from `cardname`'s microphone stim buffer.
|
||||
|
||||
### To attach a thermal sensor managed by Linux:
|
||||
```
|
||||
+idev|my-thermal|thermal-stimiface|thermal-zone()|linux()|/sys/class/thermal_zone0
|
||||
```
|
||||
|
||||
Use the `thermal-zone` SysFS stim-buff-api provided by `linux` to connect to the sensor
|
||||
`/sys/class/thermal_zone0`. Use the `thermal-stimiface` library to process the thermal data from
|
||||
`thermal_zone0`'s heat stim buffer.
|
||||
|
||||
## Multiple Provider Examples:
|
||||
|
||||
### Single Sound Card Device with Multiple Providers:
|
||||
|
||||
The same physical sound card `soundcard0` can be accessed through different providers:
|
||||
|
||||
```
|
||||
+edev|soundcard0|audio-stimiface|alsa-audio()|alsa()|card0
|
||||
||| +edev|soundcard0|audio-stimiface|direct-file-ops()|linux()|/dev/snd/pcmC0D0p
|
||||
||| +idev|soundcard0|audio-stimiface|alsa-mic()|alsa()|card0
|
||||
```
|
||||
|
||||
This shows:
|
||||
- `soundcard0` accessed via ALSA provider for audio output (`alsa-audio` stim-buff-api)
|
||||
- `soundcard0` accessed via Linux provider for direct file operations (`direct-file-ops` stim-buff-api)
|
||||
- `soundcard0` accessed via ALSA provider for microphone input (`alsa-mic` stim-buff-api)
|
||||
|
||||
### Single Camera Device with Multiple Stim-Buff-APIs:
|
||||
|
||||
A camera device `camera0` might expose different data streams:
|
||||
|
||||
```
|
||||
+edev|camera0|visual-stimiface|v4l-rgb()|linux()|/dev/video0
|
||||
||| +edev|camera0|visual-stimiface|v4l-yuv()|linux()|/dev/video0
|
||||
||| +idev|camera0|thermal-stimiface|v4l-thermal()|linux()|/dev/video0
|
||||
```
|
||||
|
||||
This shows the same camera device providing:
|
||||
- RGB color data via `v4l-rgb` stim-buff-api
|
||||
- YUV color data via `v4l-yuv` stim-buff-api
|
||||
- Thermal data via `v4l-thermal` stim-buff-api
|
||||
|
||||
## Attaching actuators:
|
||||
|
||||
Actuators are Salmanoff's way of enacting changes in the external world.
|
||||
They're like your libs, or your mouth. Actuators enable salmanoff to write
|
||||
outputs to the world outside.
|
||||
|
||||
### Wilzors:
|
||||
|
||||
Actuator devices are analogous to your body's limbs. Salmanoff controls these
|
||||
by using `wilzor` algorithms. Wilzor is a contraction of **Wil**lpower
|
||||
Actuat**Or** but with a 'Z' in the middle to make it sound cooler. Different
|
||||
types of devices will require different wilzor algorithms. You need to know
|
||||
what type of wilzor algorithm needs to be used to enable salmanoff to control
|
||||
your actuator device.
|
||||
|
||||
The general format for an actuator's device attachment specification follows the same pattern:
|
||||
```
|
||||
+adev|dev-identifier|wilzor-algorithm|stim-buff-api(api-params)|provider(provider-params)|dev-selector
|
||||
```
|
||||
|
||||
Where `wilzor-algorithm` is the specific wilzor algorithm needed to control the actuator device.
|
||||
|
||||
## Device Attachment Pipeline (DAP) specification files:
|
||||
|
||||
Inside of a DAP specification file, you can list any number of
|
||||
DAP specifications.
|
||||
Separate individual DAP specifications with two consecutive h-bar
|
||||
characters (`||`),
|
||||
like this:
|
||||
```
|
||||
+edev|my-window|visual-stimiface|wayland()|wayland(server-socket)|window0
|
||||
|| +edev|my-xorg-display|visual-stimiface|x11()|xorg(listen-socket)|all
|
||||
|| +idev|my-thermal|thermal-stimiface|thermal-zone()|linux()|/sys/class/thermal_zone0
|
||||
```
|
||||
@@ -1,168 +0,0 @@
|
||||
# Device Attachment Specification DSL: attaching sensors and actuators to SMO.
|
||||
|
||||
## Attaching sensors:
|
||||
|
||||
Sensors are input devices to Salmanoff. Salmanoff will perceive them as
|
||||
perceptual inputs -- like your own sense organs. For example, if you attach a
|
||||
camera as a sensor, salmanoff will experience it in the same way that you
|
||||
experience the visual sense data from your eyes.
|
||||
|
||||
## Implexors:
|
||||
|
||||
An implexor is an **Imp**licit **Ex**istent isolat**Or** algorithm. It's
|
||||
basically what conventional ML/LLM/ANN developers call an ROI ("Region of
|
||||
Interest") extraction algorithm. An Implex algorithm is used to scan a frame
|
||||
of input sensor data and detect objects and patterns within it.
|
||||
|
||||
## Sensor device attachment specification:
|
||||
|
||||
The general format of a device attachment specification for a sensor is:
|
||||
```
|
||||
sensor-type|dev-identifier
|
||||
|implexor|api(api-params)|provider(provider-params)|deviceselector
|
||||
```
|
||||
|
||||
* `sensor-type` is always either '`+idev`' (interoceptor), '`+edev`'
|
||||
(extrospector), or '`+adev`' (actuator).
|
||||
* `dev-identifier` is a user-defined name for this specific device instance.
|
||||
* `implexor` is the name of the implexor algorithm that should be used with
|
||||
the data that is provided by the `provider` via the `api`.
|
||||
* `api` is the interface that the provider uses to export perceptual data for
|
||||
salmanoff to read. Salmanoff will run the `implexor` algorithm on the data
|
||||
from this `api`. The `api-param` in parentheses may be omitted, in which
|
||||
case the parentheses will be empty, but the parentheses must always be
|
||||
written out.
|
||||
* `provider` may be a userspace daemon or an OS kernel that provides perceptual
|
||||
data via the `api`. The `provider-params` in parentheses may be omitted, in
|
||||
which case the parenthesis will be empty, but the parentheses must always be
|
||||
written out.
|
||||
* `device selector` is the idiosyncratic label/name used by the `provider` to
|
||||
identify the specific device you want to attach via that `provider`.
|
||||
|
||||
## `API-params` and `provider-params`:
|
||||
|
||||
If there's more than one parameter item in a list of `api-params` or
|
||||
`provider-params`, then the individual items in a list of `api-param` or
|
||||
`provider-params` should be separated by the h-bar character (`|`). E.g:
|
||||
```
|
||||
+edev|audio-implexor|alsa(shmem|param2|param3)|alsa()|cardname
|
||||
```
|
||||
|
||||
Each parameter must be in one of these forms:
|
||||
* key=value
|
||||
* key=
|
||||
* key
|
||||
|
||||
Some examples follow:
|
||||
|
||||
### To attach a particular window from a window manager:
|
||||
```
|
||||
+edev|my-window|visual-implexor|wayland()|wayland(server-socket)|window0
|
||||
```
|
||||
Connect to the Wayland server that's listening on `server-socket`, using the
|
||||
`wayland` api. Ask that Wayland server to give salmanoff read-access to all of
|
||||
the frames composited into the window buffer for `window0`. Use salmanoff's
|
||||
`visual-implexor` to implex from that `window0`'s compositor data.
|
||||
|
||||
### To attach a window manager's entire rendered desktop:
|
||||
```
|
||||
+edev|my-desktop|visual-implexor|wayland()|wayland(listen-socket)|all
|
||||
```
|
||||
In most cases, this is basically the same as attempting to attach all of the
|
||||
underlying GFX server's output.
|
||||
|
||||
Connect to the Wayland server that's listening on `listen-socket`, using the
|
||||
`wayland` api. Ask that Wayland server to give salmanoff read-access to the
|
||||
entire compositor framebuffer. Use salmanoff's `visual-implexor` to implex from
|
||||
that Wayland server's compositor data.
|
||||
|
||||
### To attach all of an Xorg server's gfx output to all screens:
|
||||
```
|
||||
+edev|my-xorg-display|visual-implexor|x11()|xorg(listen-socket)|all
|
||||
```
|
||||
|
||||
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
|
||||
api. Ask that Xorg server to let Salmanoff read out all of the frames written
|
||||
out to all screens. Use salmanoff's `visual-implexor` to implex from the
|
||||
server's gfx framebuffer data.
|
||||
|
||||
In most cases, this is basically the same as attempting to attach all of the
|
||||
WM's output.
|
||||
|
||||
* Implementation note:
|
||||
https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
|
||||
Seems relevant.
|
||||
|
||||
### To attach all of an Xorg server's gfx output to a particular screen:
|
||||
```
|
||||
+edev|my-screen|visual-implexor|x11()|xorg(listen-socket)|:0
|
||||
```
|
||||
Connect to the Xorg server that's listening on `listen-socket`, using the `x11`
|
||||
api. Ask that Xorg server to let Salmanoff read out all of the frames written
|
||||
out to display `:0`. Use salmanoff's `visual-implexor` to implex from display
|
||||
`:0`'s framebuffer data.
|
||||
|
||||
* Implementation note:
|
||||
https://stackoverflow.com/questions/33845447/how-do-i-talk-to-an-x-server-in-c-without-a-graphics-library
|
||||
Seems relevant.
|
||||
|
||||
### To attach a camera device by connecting directly to its Linux driver:
|
||||
```
|
||||
+edev|my-camera|visual-implexor|v4l()|linux()|/dev/video0
|
||||
```
|
||||
We specify that we want to use the `linux` kernel's loaded driver to connect
|
||||
to communicate with `/dev/video0`, via the `Video4Linux` API. We want salmanoff
|
||||
to use the `visual-implexor` algorithm to implex from `/dev/video0`'s data.
|
||||
|
||||
If `/dev/video0` is already consumed by another process, this may likely fail.
|
||||
|
||||
### To attach a microphone that's managed by ALSA server:
|
||||
```
|
||||
+edev|my-microphone|audio-implexor|alsa(shmem)|alsa()|cardname
|
||||
```
|
||||
|
||||
Connect to the ALSA server via `shmem`, using the `alsa` API. Request access to
|
||||
the microphone function of the sound card with the name `cardname`. Use the
|
||||
`audio-implexor` algorithm to implex from `cardname`'s microphone data.
|
||||
|
||||
### To attach a thermal sensor managed by Linux:
|
||||
```
|
||||
+idev|my-thermal|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0
|
||||
```
|
||||
|
||||
Use the `thermal-zone` SysFS API provided by `linux` to connect to the sensor
|
||||
`/sys/class/thermal_zone0`. Use the `thermal-implexor` implexor to implex from
|
||||
`thermal_zone0`'s heat data.
|
||||
|
||||
## Attaching actuators:
|
||||
|
||||
Actuators are Salmanoff's way of enacting changes in the external world.
|
||||
They're like your libs, or your mouth. Actuators enable salmanoff to write
|
||||
outputs to the world outside.
|
||||
|
||||
### Wilzors:
|
||||
|
||||
Actuator devices are analogous to your body's limbs. Salmanoff controls these
|
||||
by using `wilzor` algorithms. Wilzor is a contraction of **Wil**lpower
|
||||
Actuat**Or** but with a 'Z' in the middle to make it sound cooler. Different
|
||||
types of devices will require different wilzor algorithms. You need to know
|
||||
what type of wilzor algorithm needs to be used to enable salmanoff to control
|
||||
your actuator device.
|
||||
|
||||
The general format for an actuator's device attachment specification is:
|
||||
```
|
||||
WIP: TBD.
|
||||
```
|
||||
|
||||
## Device attachment specification files:
|
||||
|
||||
Inside of a device attachment specification file, you can list any number of
|
||||
device attachment specifications.
|
||||
Separate individual device attachment specifications with two consecutive h-bar
|
||||
characters (`||`),
|
||||
like this:
|
||||
```
|
||||
+edev|my-window|visual-implexor|wayland()|wayland(server-socket)|window0
|
||||
|| +edev|my-xorg-display|visual-implexor|x11()|xorg(listen-socket)|all
|
||||
|| +idev|my-thermal|thermal-implexor|thermal-zone()|linux()|/sys/class/thermal_zone0
|
||||
```
|
||||
Reference in New Issue
Block a user