Multimodal Synthesis

Multimodal Synthesis builds a fixed-interval summary of every recorded source — microphone level, camera brightness, telemetry, input and event counts, RF features and RF band power — with explicit missingness, uncertainty and record-level lineage.

Enable Synthesis#

On Recording, expand RTL-SDR survey & Synthesis settings.
Tick Generate Synthesis from recorded sources. This ticks the Multimodal Synthesis source for recording and monitoring. (Ticking that source in the Data sources table has the same effect.)
Set the Epoch seconds and Allowed lateness seconds, edit RF bands and relationship pairs if needed, then start a monitor or recording.

Settings#

Setting	Range (default)	Meaning
Epoch seconds	0.1 – 60 (1)	Length of each summary interval on the session clock.
Allowed lateness seconds	0 – 10 (2)	How long an epoch waits for late-arriving records before it is closed.

RF bands for band-limited power#

Each band produces two features: the bin-weighted mean PSD over the observed bins of the band (only windows acquired inside the epoch) and the observed fraction of the band's bandwidth.

Default band	Range
fm_broadcast	87.5 – 108 MHz
airband	118 – 137 MHz
vhf_144	144 – 148 MHz
ism_433	433.05 – 434.79 MHz
uhf_863_928	863 – 928 MHz
adsb_1090	1089 – 1091 MHz
gnss_l1	1574.42 – 1576.42 MHz

Edit a band's name and edges in place. Add band (up to 32), Remove last band and Defaults manage the list. Names use letters, digits and underscores and must be unique.

Enter pairs of stable feature IDs to have NOESIS compute Pearson, Spearman and covariance with a positive lag search over observed pairs only.

Add pair (up to 16) adds audio.microphone/rms ↔ camera.roi/0; edit both IDs. Remove last pair removes one.
History epochs (4–600, default 60), Minimum observed pairs (default 12) and Maximum lag epochs (0–20, default 5).
Scanning RF aggregates (scan_interval_mean) are excluded; band features (rf.band/…) are allowed.

Feature IDs and operations#

Every feature has a stable ID of the form <source key>/<name>, for example audio.microphone/rms, camera.roi/0 or rf.band/fm_broadcast/mean_psd. Each feature names exactly one operation:

Operation	Computation
`interval_mean`	Time-overlap-weighted mean of observations overlapping the epoch.
`scan_interval_mean`	The same for RF window features whose frequency changes during the epoch.
`interval_rms`	RMS of audio samples inside the epoch.
`interval_sum`	Sum of record values, apportioned by time overlap.
`interval_count`	Number of records or events inside the epoch.
`band_mean_psd`	Bin-weighted mean PSD over a band's observed bins.
`band_coverage`	Observed fraction of a band's bandwidth.

Anything not measured in an epoch is marked MISSING, never zero. No interpolation, carry-forward or normalization is applied.

Live Synthesis panel#

While Synthesis runs, a Synthesis panel appears on Live Monitor, Recording and Analysis:

Epoch interval, observed / total features, dropped work, late inputs and capacity drops.
Missing sources for the epoch.
Lineage: number of sources and records, trials overlapping the epoch (sham trials marked) and the experiment.
A scrolling list of every feature: id = value units [operation n=observations ±SEM], or MISSING.
Relationship results for your selected pairs: r, ρ, lag and number of pairs.

Regenerate Synthesis from a recording#

Synthesis can be rebuilt later from any recorded session, including sessions recorded without Synthesis enabled. The Synthesis settings stored in the session are used. The original session is never modified; a new session is written to the output root (default synthesis-sessions).

noesis-app --synthesize sessions/session-ID --storage synthesis-sessions

The regenerated session records its parent session and refers to the parent's record IDs.