Engineering Record
Problems Encountered. Solutions Discovered. Lessons Documented.
Version 1.0 — April 2026
Eric Becker // FluidFortune.com
This document is a record of problems encountered, diagnosed, and solved across the Fluid Fortune project stack. It is written for two audiences simultaneously: the technical reader who wants to understand the specific engineering decisions and why they were made, and the non-technical reader who wants to understand what was actually accomplished and why it is significant.
Where a technical concept requires deep domain knowledge to understand, a plain English sidebar appears immediately following. These sidebars are not simplifications — they are translations. The technical explanation is accurate. The sidebar makes it accessible.
The projects covered in this document, in the order they are addressed:
- Pisces Moon OS — custom general-purpose operating system for the
- LilyGO T-Deck Plus (ESP32-S3)
- Pisces Moon Linux — the companion tablet platform running Debian
- 13 on a Fujitsu Q508
- The Phantom — local AI agent framework running on home hardware
- Spadra Threat Intelligence System — distributed network security
- monitoring
- WozBot — lightweight local AI proof of concept
- Fluid Fortune Publishing Infrastructure — Punky, Static, and the
- zero-server publishing stack
- VaporwareOS — embedded satirical platform and technical critique
Each section follows the same structure: what the project is, what problems it encountered, what solutions were developed, and what those solutions proved about the technology and the philosophy behind it.
*The problems are not incidental to the story. They are the story.
Every solution documented here is a discovery — something that did
not exist in public documentation before this project encountered the
need and worked out the answer.*
1. Pisces Moon OS
The First Known Documented Dual-Core Persistent Background Tasking OS for Field Intelligence on the ESP32-S3
The LilyGO T-Deck Plus is a \$50 handheld device with a color screen, a physical QWERTY keyboard, a touchscreen, a trackball, and five wireless radios: WiFi, Bluetooth Low Energy, LoRa long-range radio, and GPS. It runs on the ESP32-S3 microcontroller at 240 megahertz across two processor cores simultaneously.
Before Pisces Moon OS, every piece of software ever written for this hardware did one thing. A wardriving tool. A mesh radio platform. A game. When you turned on the device, it did its one function and nothing else.
Pisces Moon OS is a true general-purpose operating system for this hardware — one where you launch different apps, switch between them, and the device's identity is not defined by what is currently running. As of v1.0.0, it ships 47 applications across 7 categories. It was the first known documented dual-core persistent background tasking OS for field intelligence on this hardware class.
+———————————————————————--+
Think of the difference between a microwave oven and a smartphone. A
microwave has one job. A smartphone can run any app. Before Pisces
Moon OS, every ESP32-S3 device was essentially a microwave. This
project turned one into a smartphone — on \$50 of hardware, without
any of the infrastructure that smartphones rely on. That had never
been done on this class of chip before.
+———————————————————————--+
+———————————————————————--+
If you turn the backlight on before the display's video memory is
initialized, the screen shows white noise — a flash of garbage
pixels — before the first real frame renders. Holding the backlight
off means the first thing the user ever sees is the correct image. A
small detail that separates "real device" from "prototype."
+———————————————————————--+
+———————————————————————--+
The WiFi SDK internally uses the same SPI hardware as the display.
When WiFi initializes, it changes SPI state, which corrupts where the
display thinks it is currently drawing. If you start a line of text,
call WiFi init, then finish the line, the text lands in the wrong
place. Fix: always finish the complete display line first, then call
WiFi. Simple once you know the cause. Invisible and maddening before
you do.
+———————————————————————--+
+———————————————————————--+
Core 1 is still finishing boot when Core 0 starts. If Core 0
immediately touches the SD card and WiFi radio, it collides with Core
1's initialization before the Treaty system is fully in place. The
15-second delay is a deliberate safety margin: by then, Core 1 has
finished and the mutex is operational.
+———————————————————————--+
+———————————————————————--+
device**
The T-Deck has two independent processors running simultaneously. One
runs whatever you are looking at on screen. The other one —
invisibly, silently, always — is scanning for WiFi networks,
logging Bluetooth devices, and recording GPS coordinates. Playing
chess, reading a document, doing nothing. The Ghost Engine is always
running. The device is always watching.
+———————————————————————--+
+———————————————————————--+
Imagine four people trying to use the same telephone line
simultaneously. When only one person is on the call, everything
works. When two people try to talk at once, both conversations are
destroyed. The SPI bus is that telephone line. The SD card, LoRa
radio, and display are the people. Without rules about who can talk
and when, everything crashes. Pisces Moon OS needed to invent those
rules — they did not exist for this hardware configuration anywhere
in public documentation.
+———————————————————————--+
⚠ THE PROBLEM: SPI Bus Contention
Multiple hardware components sharing one SPI bus. Any two components
attempting simultaneous access corrupts both transactions and crashes
the device. No documented solution existed for this specific hardware
configuration running a general-purpose OS.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: The SPI Bus Treaty
A formal architectural agreement governing every component of the OS.
Four rules: (1) Hit and run — open a file, write, close, release
immediately. Never hold the bus. (2) No extended holds — no
encryption during writes, no formatting during operation, nothing
that monopolizes the bus. (3) Radio traffic management — a shared
flag (wifi_in_use) tells the wardrive task when WiFi is in use for an
API call, so it pauses its scan window. (4) Metadata-only destructive
functions — the security Nuke sequence deletes index files
(milliseconds) rather than formatting the card (seconds). A FreeRTOS
mutex (spi_mutex) enforces the Treaty in code: every component
touching shared hardware must acquire the mutex first and release it
immediately after.
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
Imagine you have a small desk (320KB SRAM) and a large filing cabinet
in the corner (8MB PSRAM). Every time you need to work on something,
your instinct is to put it on the desk — but the desk fills up fast
when you're working on seven things at once. The filing cabinet is
sitting there empty. The fix: tell the computer to automatically put
large items in the filing cabinet instead of cramming them onto the
desk.
+———————————————————————--+
⚠ THE PROBLEM: SRAM Exhaustion
All heap allocations defaulted to 320KB internal SRAM. Running
multiple simultaneous applications, radio stacks, and background
tasks exhausted available SRAM, causing allocation failures and
crashes. The 8MB PSRAM chip was present but unused.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: PSRAM Heap Redirect
A single build flag — CONFIG_SPIRAM_USE_MALLOC=1 — instructs the
memory management system to automatically route heap allocations
above a threshold to PSRAM. Applications and libraries require no
changes. Internal SRAM is now reserved for small, fast, time-critical
buffers. Everything else goes to PSRAM. This was not a documented
solution for an ESP32-S3 OS context — no previous firmware was
complex enough to require it.
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
Think of two workers sharing one set of tools. If both reach for the
same tool at the same time, you get a conflict. The solution is a
sign-out system: before using a shared tool, you check it out. When
done, you check it back in. Anyone else who needs it waits. Pisces
Moon OS implements this as a software "mutex" — a lock that
ensures only one core touches shared hardware at a time.
+———————————————————————--+
✓ THE SOLUTION: The Dual-Core Architecture with SPI Mutex
The wardrive task is pinned to Core 0 via xTaskCreatePinnedToCore()
with a deliberate 15-second startup delay, allowing Core 1 to
complete boot before Core 0 begins touching shared hardware. The SPI
mutex (part of the Bus Treaty) coordinates SD card access between
cores. A parallel wifi_in_use flag coordinates WiFi radio access.
Neither core needs to know what the other is doing — coordination
happens through shared state flags, not direct coupling.
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
Imagine you're taking notes every time someone walks past you. In a
quiet hallway, this is easy. In a crowded train station, people are
passing faster than you can write. You run out of notepad space and
drop everything. The solution: instead of writing everything down
immediately, hand notes to a queue and process them at a manageable
pace.
+———————————————————————--+
✓ THE SOLUTION: ISR-Safe BLE Queue
BLE advertisement callbacks now do the minimum possible work: they
push a small record into a lock-free circular queue and return
immediately. A separate task on Core 0 drains the queue at a
controlled pace. The ISR never overflows because it does almost
nothing. The queue absorbs bursts. Validated in real-world testing:
40+ WiFi access points and 100+ BLE devices, no crash.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: Auto-Baud GPS Detection
The GPS initialization routine tries 9600 baud first. If no valid
NMEA sentences are received within a timeout, it switches to 38400.
The RX buffer was also increased from the default 128 bytes to 512
bytes — the default overflows during the 2-4 second WiFi scan
windows that block the CPU, causing GPS fix acquisition to stall.
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
The PSRAM heap redirect from Problem 1.3 was a great solution — but
it only works for some kinds of memory. The audio system uses a
different kind of memory allocation that bypasses the redirect
entirely and always takes from the small desk (SRAM). With the desk
already crowded, there was no room for audio. The fix: make the audio
system ask for less desk space.
+———————————————————————--+
✓ THE SOLUTION: I2S DMA Budget Rule
DMA buffers capped at 4 buffers × 512 bytes = 4KB (down from 16KB).
Read buffers moved to ps_malloc() — PSRAM-allocated on launch,
freed on exit. Warmup buffers reduced from 8KB stack allocations to
512-byte stack arrays. This constraint is now documented as a hard
rule for all future I2S users in the OS.
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
Imagine every app leaving its furniture in the hallway even when the
app is closed. The hallway fills up. The PSRAM redirect was supposed
to send large items to the storage room — but it only works for
items you specifically request storage for. If you declare something
as always-available-in-the-hallway, it stays in the hallway. The fix:
stop leaving furniture in the hallway. Bring it out when the app
opens, put it away when it closes.
+———————————————————————--+
⚠ THE PROBLEM: BSS Overflow — 70,120 Bytes
Static global arrays across 8 new apps loaded permanently into
internal SRAM at boot: RF spectrum waterfall buffer (105KB), probe
device arrays (27KB), packet analysis structures (9KB), and others.
Total BSS pressure exceeded available SRAM by 70KB.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: PSRAM Dynamic Allocation Pattern
All large data structures converted from static globals to
ps_malloc() allocated on app launch and freed on exit. The RF
waterfall buffer (105KB) alone moved to PSRAM. Total moved from BSS
to PSRAM: \~137KB, resolving the 70KB deficit with margin. This
pattern is now documented as the required approach for any large
buffer in the OS — no large arrays in global scope.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: NimBLE Ownership Protocol
wardrive.cpp owns NimBLE permanently for the lifetime of the session.
Any app requiring BLE access calls wardrive_ble_stop() before using
the scan object — which halts the active scan and clears callbacks
cleanly — and wardrive_ble_resume() when finished. The app in the
middle has exclusive, clean access. No app may call
NimBLEDevice::init() or deinit() under any circumstances.
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
A USB device tells the computer what it is when it plugs in. It can
say "I'm a serial port" or "I'm a keyboard" — but not both at
once. The T-Deck needs to be a serial port for development work. For
keyboard injection testing, it needs to be a keyboard. These are
physically the same socket but logically incompatible identities.
+———————————————————————--+
✓ THE SOLUTION: Dual Build System
Two separate build environments in platformio.ini. The standard build
(esp32s3) uses CDC serial mode — full development capability,
PlatformIO flashing, stable GPIO1. The HID build (esp32s3_hid) uses
USB HID mode — keyboard injection, no serial console. The USB Ducky
app detects which build is running and displays appropriate UI for
each case. BLE Ducky provides keyboard injection in the standard
build without any reflashing required.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: Split Trackball API
update_trackball() retains the 250ms lockout for all UI navigation
— unchanged. A new update_trackball_game() provides an 80ms lockout
for game loops. All existing UI code is unaffected. All game loops
call the game variant. Future games must use update_trackball_game().
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
Imagine a briefcase with a hidden compartment. The visible section
has innocent contents — maps, a calculator, some notes. The hidden
section requires two separate keys to open. If you enter the wrong
combination, the briefcase shows you the innocent section anyway,
with no indication that a hidden section exists. If you need to
destroy the hidden contents quickly, one code wipes the filing system
— leaving the data physically present but permanently unreachable
through normal means. This is the Ghost Partition.
+———————————————————————--+
✓ THE SOLUTION: Fresh Minimal Install Protocol
Decision: wipe and reinstall from scratch with Debian 13 minimal XFCE
— only XFCE checked during installation, no LibreOffice, no extras,
nothing. A minimal install has no mystery leftovers. If something
breaks, it is attributable to the current install script, not to
historical contamination. This is now the documented baseline for the
Pisces Moon Linux install.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: Direct Question Over Silent Detection
When no \$DISPLAY is set, the installer asks directly: "Is this a
Fujitsu Q508?" The user types yes or no. No clever detection
required. Simple input beats clever inference when the clever
inference fails on the most common use case.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: Correct Menu Registration Path
Menu file moved to
/etc/xdg/menus/applications-merged/pisces-moon.menu.
update-desktop-database run after installation. App launch method
changed from xdg-open (which requires a browser to be configured) to
chromium \--app= (which opens each HTML app as a standalone
borderless window — a functional proto-Trojan Horse wrapper).
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
When you ask a local AI "what was the Dodgers score last night?",
the model has no idea — it was trained months ago and cannot access
the internet. The Phantom solves this by checking the score BEFORE
asking the model. It finds the answer, writes it into the question,
and hands the model a message that says "given that the Dodgers won
4-2, tell me about the game." The model looks like it knows. The
Phantom actually knew.
+———————————————————————--+
⚠ THE PROBLEM: Local AI Has No Memory
Each conversation session starts completely fresh. Context
established in previous sessions is lost. The model cannot recall
corrections, preferences, or ongoing projects. This makes local AI
feel significantly less capable than cloud alternatives, regardless
of actual model quality.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: ChromaDB Vector Memory
ChromaDB runs entirely locally — no cloud, no API key. After every
conversation turn, each message is converted to a mathematical vector
representing its meaning and stored. Before every turn, the system
retrieves the five most semantically similar past exchanges — not
by keyword search but by meaning. "How has Miguel Rojas been
performing?" retrieves conversations about Dodgers players even if
they used different words. Combined with automatic conversation
distillation (after 60 turns, older exchanges are summarized into a
compact memory snapshot), The Phantom maintains genuine continuity
across sessions.
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
Keyword search asks "does this document contain the word
'baseball'?" Vector search asks "is this document about the same
subject as the question I'm asking?" It finds related content even
when the words are completely different. This is how human memory
actually works — you remember things by association, not by exact
word match. ChromaDB gives a local AI something close to that.
+———————————————————————--+
✓ THE SOLUTION: Two-Stage Web Intelligence
Before every conversation turn, two detection stages run. First: a
keyword check for trigger phrases ("search", "score", "who is",
"latest"). If keywords are present, web search runs. Second: if
keywords miss, the model itself is asked in a single 5-token query
whether it needs web access for this question. This catches queries
that keyword matching would miss — "How has Miguel Rojas performed
this season?" contains no trigger keyword but the model correctly
identifies that it needs current data. Web results are scraped in
full (not just snippet summaries), with a paywall bypass cascade
using 12ft.io.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: MLB Stats API Integration
The Phantom integrates directly with the official MLB Stats API —
free, no key required — for real-time scores, full season
statistics, and player biographical data. Every player lookup
auto-writes to phantom_databases/mlb_players.json in a format
compatible with Pisces Moon apps on the tablet and T-Deck. The
database export function minifies JSON for device transfer. The
distinction between at-bats and plate appearances is correctly
maintained.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: The Log Aggregator
A single aggregation service pulls from all nodes on a schedule,
merges events into one chronological timeline, and runs correlation
checks automatically. The question "did the IP that hit our cloud VM
at 5:38 AM also appear in local network traffic?" was unanswerable
before the Aggregator. Now it is answered automatically every 30
minutes without human intervention.
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
Nobody shares a productivity assistant with their friends. But a
boisterous AI that tells terrible 1970s hardware puns and recounts
Apple Computer history from the inside? That gets posted on Reddit.
When someone finds the GitHub repo, they find Fluid Fortune, and they
find The Phantom, and they think: hm. The joke bot is the Trojan
Horse. The Phantom is what's inside.
+———————————————————————--+
✓ THE SOLUTION: The Minimal Proof of Concept
WozBot demonstrates the core architecture — local model + system
prompt + web interface + server — in a form anyone can experience
without technical knowledge. It is live at wozbot.fluidfortune.com,
running on bare metal in the Fluid Fortune private perimeter,
tunneled to the public internet via Cloudflare. No cloud compute. No
corporate data center. No ongoing cost beyond electricity. The joke
is the point. The architecture behind the joke is also the point.
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
WordPress is bloat. Not a little bloat — architectural bloat. A
blog post is a text file. WordPress requires a database server, a PHP
runtime, a web server, a plugin ecosystem with its own security
vulnerabilities, and a hosting provider. Punky replaces the entire
stack with one HTML file and a GitHub account. The blog that results
is faster, cheaper (free), more durable, and owned completely by the
person who wrote the posts.
+———————————————————————--+
✓ THE SOLUTION: Punky — GitHub as the Server
Punky is a single HTML file. Open it in Chrome. Write a post using a
rich text editor. Click publish. Punky calls the GitHub API directly
from your browser, creates the post file in your repository, updates
the JSON manifest, regenerates the RSS feed and sitemap. Your GitHub
token stays in your browser. Your posts go to your repository. The
cost of running a blog on Punky: \$0/month.
+———————————————————————--+ +———————————————————————--+
✓ THE SOLUTION: Static — Archive.org as the Host
Static is Punky's sibling, built for audio. Audio files go to
archive.org — the Internet Archive, a non-profit digital library
that has been preserving the web since 1996 and has no business model
that requires monetizing your audience. Episode pages and RSS feeds
are generated by Static and served by GitHub Pages. The shows cannot
be deplatformed by a hosting company changing its terms. Total cost:
\$0/month.
+———————————————————————--+ +———————————————————————--+
+———————————————————————--+
The most credible criticism of the technology industry comes from
inside it. A developer who has shipped production code earns the
right to criticize production code. VaporwareOS is built with
professional-grade embedded engineering — PSRAM framebuffers, BLE
HID, passive RF monitoring — so the critique cannot be dismissed as
the work of someone who does not understand the engineering. The joke
is built from the same materials as the thing it is joking about.
That is what makes it land.
+———————————————————————--+
+———————————————————————--+
The firmware that mocks progress bars that lie has its own serial
monitor reporting "init complete" while the display shows nothing.
This is not a failure. It is the most honest thing VaporwareOS does.
The court jester is lying on the floor of the throne room. The bit
landed anyway. When the screen eventually works, the first image will
be VAPOR OS, then EVERYTHING, then a subtitle: "Alpha Version:
Whatever Your Mind Believes It To Be." This is the most honest thing
the device will ever say.
+———————————————————————--+