salmanoff/docs/deviceSpec.md

Device manager: attaching sensors and actuators.

Attaching sensors:

Sensors are input devices to Harikoff. Harikoff will perceive them as perceptual inputs -- like your own sense organs. For example, if you attach a camera as a sensor, harikoff will experience it in the same way that you experience the visual sense data from your eyes.

Implexors:

An implexor is an __Imp__licit Ex__istent isolat__Or algorithm. It's basically what conventional ML/LLM/ANN developers call an ROI ("Region of Interest") extraction algorithm. An Implex algorithm is used to scan a frame of input sensor data and detect objects and patterns within it.

The general format of a device-spec for a sensor is:

implexor|api|server(server-params)|deviceselector

• implexor is the name of the implexor algorithm that should be used with the data that is provided by the server via the api.
• api is the interface that the server uses to export perceptual data for harikoff to read. Harikoff will run the implexor algorithm on the data from this api.
• server may be a userspace daemon or an OS kernel that provides perceptual data via the api.
• device selector is the idiosyncratic label/name used by the server to identify the specific device you want to access via that server.

Some examples follow:

To attach a particular window from a window manager:

visual-implexor|wayland|wayland(server-socket)|window0

Connect to the Wayland server that's listening on server-socket, using the wayland api. Ask that Wayland server to give harikoff read-access to all of the frames composited into the window buffer for window0. Use harikoff's visual-implexor to implex from that window0's compositor data.

To attach a window manager's entire rendered desktop:

visual-implexor|wayland|wayland(listen-socket)|all

In most cases, this is basically the same as attempting to attach all of the underlying GFX server's output.

Connect to the Wayland server that's listening on listen-socket, using the wayland api. Ask that Wayland server to give harikoff read-access to the entire compositor framebuffer. Use harikoff's visual-implexor to implex from that Wayland server's compositor data.

To attach all of an Xorg server's gfx output to all screens:

visual-implexor|x11|xorg(listen-socket)|all

Connect to the Xorg server that's listening on listen-socket, using the x11 api. Ask that Xorg server to let Harikoff read out all of the frames written out to all screens. Use harikoff's visual-implexor to implex from the server's gfx framebuffer data.

In most cases, this is basically the same as attempting to attach all of the WM's output.

To attach all of an Xorg server's gfx output to a particular screen:

visual-implexor|x11|xorg(listen-socket)|:0

Connect to the Xorg server that's listening on listen-socket, using the x11 api. Ask that Xorg server to let Harikoff read out all of the frames written out to display :0. Use harikoff's visual-implexor to implex from display :0's framebuffer data.

To attach a camera device by connecting directly to its Linux driver:

visual-implexor|linux|v4l|/dev/video0

We specify that we want to use the linux kernel's loaded driver to connect to communicate with /dev/video0, via the Video4Linux API. We want harikoff to use the visual-implexor algorithm to implex from /dev/video0's data.

If /dev/video0 is already consumed by another process, this may likely fail.

To attach a microphone that's managed by ALSA server:

audio-implexor|alsa|alsa(shmem)|cardname

Connect to the ALSA server via shmem, using the alsa API. Request access to the microphone function of the sound card with the name cardname. Use the audio-implexor algorithm to implex from cardname's microphone data.

Attaching actuators:

Actuators are Harikoff's way of enacting changes in the external world. They're like your libs, or your mouth. Actuators enable harikoff to write outputs to the world outside.

Wilzors:

Actuator devices are analogous to your body's limbs. Harikoff controls these by using wilzor algorithms. Wilzor is a contraction of Wil__lpower Actuat__Or but with a 'Z' in the middle to make it sound cooler. Different types of devices will require different wilzor algorithms. You need to know what type of wilzor algorithm needs to be used to enable harikoff to control your actuator device.

The general format for an actuator's device spec is:

WIP: TBD.