Engineering History

Structured development record for the bot and its supporting portal.

Updated 2026-04-20 16:13:21 UTC

F1 Bot Engineering History

Purpose: this document records the meaningful evolution of the trading bot and its supporting portal. It is intended to capture not just what changed, but why the change was needed, what was implemented, and what operational effect followed.

Scope:

Include architecture changes, behavior changes, bug fixes, reliability work, observability changes, performance work, permissions changes, and data-flow changes.
Exclude formatting-only edits, trivial refactors, variable renames, and comment-only changes unless they fixed a real operational issue.
Record only changes that affected how the bot behaves, fails, recovers, scales, or is maintained.

Versioning policy:

Versions are listed in reverse chronological order within the active history.
Patch releases cover bug fixes and narrowly scoped operational corrections.
Minor releases cover behavior additions, workflow changes, and meaningful quality-of-life improvements.
Major releases cover foundational or breaking architectural changes.

Entry standard:

Each entry should explain the problem, the decision, the implementation, the impact, any tradeoffs, and any follow-up considerations.
Write at the system level, not the line-of-code level.
Treat this file as interpreted engineering history, not as a substitute for git.

v0.8.0 - 2026-04-20 - Trade accounting normalized around locked net, open gross, and archive-level traceability

Problem / context

The live trade dashboard had started to blur together several different accounting states: realized profit, unrealized mark-to-market, stop-out outcome, and profit-protection state. That made it hard to answer the simplest operational question: if the trade stops now, does the whole position end green or red, and why? The archive had a related gap. Older records showed the final result, but not the full sequence of fills, stop moves, or rule-triggered partials needed to reconstruct a trade cleanly from history alone.

Decision

Separate the accounting model into explicit buckets and make the archive event-ledger-driven. The live view should distinguish locked net P/L, open gross P/L, remaining-position breakeven, whole-trade scratch, and the net result if the stop fills now. The archive should preserve the fill sequence and rule events so a trade can be explained later without relying on memory or external logs.

Implementation

Reworked the live trade view so the portal now reports locked net P/L, open gross P/L, trade net if flat now, and if-stopped-now outcome as separate values.
Added explicit labels for remaining-position breakeven and whole-trade scratch so "breakeven" no longer means two different things at once.
Added a trade-state banner and rule-effects ledger so each partial, stop move, and final exit can be read as a sequence rather than a summary.
Changed target sizing so ladder sizes are calculated from the current remaining position after profit-protection reductions, not always from the original size.
Upgraded the archive format to store entry fills, exit fills, profit-rule metadata, and structured events under a newer schema version.
Began recording live entry fills, profit-protection partials, target fills, and stop moves as explicit events so future archive records can be replayed step by step.
Backfilled the most recent closed trade so the archive now contains the exact event sequence for the current accounting model.

Impact / outcome

The dashboard now answers the real operational question directly: what is already locked in, what is still at risk, and what happens if the stop hits now.
Profit-protection and target logic no longer look mathematically contradictory because the UI shows which size base each rule is using.
Archived trades are much closer to forensic records, which means future reviews can reconstruct a trade sequence without depending on the live log.

Tradeoffs / side effects

The live state model is more explicit, which means more fields have to stay in sync across the bot, the status JSON, and the portal.
Older archived trades still only have summary-level data, so the new traceability standard applies fully only to trades recorded after the schema upgrade.

Follow-up / future considerations

Backfill older closed trades when practical so the archive becomes uniformly queryable over time.
Keep the remaining-position and whole-trade labels stable; changing those names again would reintroduce the same ambiguity this update was meant to remove.
If future profit-protection ladders change, keep the event ledger format stable so the history remains comparable across versions.

v0.7.1 - 2026-04-20 - Live alert surface narrowed to EURUSD with explicit operator visibility

Problem / context

The live alert surface had become too broad for the next operating model. The user wanted one focused alert stream for live execution, with EURUSD as the only active market in the new strategy path, while S&P500 alerts needed to be disabled. At the same time, the operator needed a simple page on the 8001 service that showed the live strategy state in plain terms, and the alert text itself needed to be readable enough to log and audit without interpretation.

Decision

Separate the live strategy display from the execution logic, keep the 8001 page as a pure information surface, and make the live webhook relay reflect only the EURUSD strategy. Standardize the alert wording and time formatting so the Telegram output reads like an operator note rather than a raw signal dump.

Implementation

Added a dedicated 8001 market-watch page that shows only the live prices needed for the current strategy workflow.
Switched the page to a live market-data feed rather than a pull-based refresh model.
Kept the page informational only, with no trading controls or execution actions on the UI.
Updated the live alert text to use the EMA 20/50 Cross Inspection title and clearer human-readable timestamps.
Narrowed the live webhook output to the EURUSD strategy and disabled the S&P500 webhook path for the new setup.
Kept the alert relay aimed at Telegram so the operator can receive the bot’s findings in a consistent downstream format.

Impact / outcome

The operator now has a single live market-facing page that matches the current strategy scope.
Telegram alerts read more cleanly and are easier to correlate with later logs or manual review.
The live execution surface is less ambiguous because the strategy scope is now market-specific rather than generic.

Tradeoffs / side effects

The live strategy surface is simpler, but it is also narrower: the S&P500 path is intentionally inactive.
A single informational page is easier to maintain, but it pushes more responsibility into the alert text and history log for context.

Follow-up / future considerations

Keep the Telegram format stable so future strategy additions can be compared against the current baseline.
If another live market is added later, record it as a separate entry instead of broadening the current EURUSD path in place.

v0.7.0 - 2026-04-20 - Transcript-driven backtesting harness and cached market history

Problem / context

The bot needed a way to answer a different class of question: not "what is the current signal?" but "what would have happened if we had run this trade on that exact date?" Before this work, transcript ideas were still being checked manually against charts, which made the process easy to misread and hard to repeat. There was also no durable cache of the historical candles needed to replay those setups without redownloading data every time.

Decision

Create a standalone backtesting layer that stores both the market history and the trade plans, then replays each setup forward in time as if the bot had been running live on that date. Use cached candle data, explicit plan JSON files, and generated reports so each simulation is repeatable and auditable.

Implementation

Added a local trade-backtester workspace under apps/trade-backtester.
Built a Binance candle cache that stores BTCUSDT 1-minute history in SQLite and reuses monthly bulk files when available.
Added Coinbase BTC-USD daily caching for transcript plans that require a separate venue and timeframe.
Stored repeated trade ideas as versioned JSON plans so each backtest has an explicit, machine-readable contract.
Implemented replay logic for confirmed breakouts, retests, touch-and-close setups, sweep/reclaim behavior, pending limits, ladder entries, close-based invalidations, stop movement after targets, and time stops.
Generated text and JSON reports for each run so outcomes can be reviewed without rerunning the simulation.
Preserved the main bot and portal work while keeping the backtest system isolated as a research and validation layer.

Impact / outcome

Transcript ideas can now be simulated against actual historical candles instead of being checked by hand.
Historical data is cached locally, which makes repeated trade checks much faster and removes the need to redownload candles for every run.
The output is now structured enough to compare different trade ideas consistently across dates and setups.

Tradeoffs / side effects

The backtester uses OHLCV candles, not tick-by-tick order book data, so intrabar ambiguity still needs conservative handling.
Some setups require explicit assumptions where the transcript language was directional but not numeric.
Maintaining the historical cache consumes local disk, though it is far cheaper than redownloading or reprocessing data every time.

Follow-up / future considerations

Add more historical markets if future transcript work expands beyond BTCUSDT and BTC-USD.
Keep converting vague transcript wording into objective rules before running simulations so the history stays honest.
If a setup depends on exact order cancellation behavior, keep recording the cancellation rule directly in the plan JSON.

v0.6.0 - 2026-04-20 - On-demand BTC transcript intake portal

Problem / context

The bot workflow now depends on outside BTC analysis signals being turned into structured setups, but there was no first-class way to run that intake pipeline from the portal itself. The transcript bundle had to be produced manually and then moved around as a local file, which made the analysis-to-setup loop slower and less repeatable than the rest of the system.

Decision

Make the transcript ingestion pipeline a portal-backed workflow: one page, one button, background execution, status tracking, and a downloadable text artifact when the run completes.

Implementation

Added a dedicated youtube-analysis portal page with a start button, live status, query list, and download links for the generated transcript bundles.
Implemented a server-side YouTube search-and-transcript pipeline that runs the five BTC search queries, filters to fresh videos, deduplicates candidates, and concatenates up to 15 transcripts into a single .txt bundle.
Added JSON output alongside the text bundle so runs can be inspected or reused without parsing the transcript file itself.
Added a background-job status model and a dedicated pipeline log so long-running runs can be monitored without blocking the web request.
Kept the intake workflow resilient by treating zero-candidate or zero-transcript runs as failures instead of as successful bundles.
Added a transcript source fallback so the pipeline can still produce bundles when YouTube watch-page captions are blocked from the server.

Impact / outcome

The analysis intake loop became a repeatable operator action instead of a one-off shell process.
Transcript bundles are now generated in a controlled, inspectable way with clear completion state and downloadable artifacts.
The portal can be used to bootstrap future setup generation without switching away from the site.

Tradeoffs / side effects

The pipeline depends on external transcript availability and can still skip videos that expose no usable captions.
The background job introduces another long-running portal task that needs status handling and a dedicated log file.

Follow-up / future considerations

Consider persisting richer per-run metadata if transcript intake becomes a regular workflow.
If analysis throughput increases, add queueing or rate controls around the transcript job so it does not compete with other portal work.

v0.5.2 - 2026-04-20 - Breakdown short now closes the long before entering

Problem / context

The confirmed breakdown short at 73,230 needed to supersede the existing long runner when BTC broke down, but the setup engine was still blocking against open long exposure before it had a chance to execute the reversal logic. That left the strategy in an awkward state where the long stayed open even when the short thesis had already invalidated it.

Decision

Make the BTC_BREAKDOWN_SHORT_73230 setup an explicit long-first reversal path: when the 5m close below 73,230 fires, close the open long setup first, then allow the short entry only if that close succeeds.

Implementation

Added a setup-level close_long_first conflict flag for the breakdown short.
Introduced a targeted reversal helper that closes any open long setup before the short entry is evaluated.
Marked the closed long as reversed so the portal and bot treat it as terminal instead of pending or live.
Tightened the close-size handling so dust-level margin rounding does not leave a tiny residual position behind and block the next state transition.
Updated the portal pending-state messaging so the breakdown setup is described as a close-first short rather than a permanently blocked idea.

Impact / outcome

The short can now supersede the long deterministically when the confirmed breakdown occurs.
The bot no longer needs a manual intervention step to clear the long before it can follow the new bearish thesis.
The setup state now reflects the actual regime change rather than hiding it behind a generic block reason.

Tradeoffs / side effects

The reversal path is more stateful than a simple short entry because it depends on the long close filling cleanly.
If the long exit fails, the short remains safely unarmed rather than forcing a conflicting position.

Follow-up / future considerations

Track reversal-linked trades more explicitly in history if this pattern becomes common.
Monitor whether the long-close first path needs additional guardrails for partial fills or future margin edge cases.

v0.5.1 - 2026-04-20 - Portal visibility and trade history surfaced directly

Problem / context

The operator experience was still too fragmented. Key information such as current equity, live position state, active versus pending setups, and trade history was not visible enough in the portal, which forced repeated context switching back to logs and support files. The active setup count also risked looking like "all setups are live" when in reality only one had a running position.

Decision

Turn the portal into the operational surface for the bot: show the current account and setup state directly, distinguish live trades from pending setups, and expose historical trades from the current day onward so the operator can understand what the bot is doing without leaving the site.

Implementation

Added current equity and max equity reporting to the bot status payload.
Changed the setup summary logic so active counts reflect actual live positions and not just the number of configured ideas.
Split the dashboard into live-trade and pending-setup sections so a running position is visually distinct from queued or blocked setups.
Added a daily trade-history store that starts from the current day and only records the current operational window.
Surfaced live and historical trade details in the portal so the user can see entry, exit, P/L, and related metrics in one place.
Added sitemap navigation across the pages so the portal feels like one navigable product instead of isolated screens.
Kept the temporary F1-bot disable path available so the old workflow can be grayed out without deleting the page outright.

Impact / outcome

The portal now answers the common operator questions directly: what is open, what is pending, what has already completed, and what equity the margin account is actually showing.
The active setup count stopped implying that every configured setup was currently live.
The bot became easier to supervise because the operator no longer needs to cross-reference the portal, logs, and external notes for basic state.

Tradeoffs / side effects

The portal now depends more heavily on the freshness of live equity and setup snapshots.
More state is mirrored into the UI, which makes consistency between bot memory, JSON files, and rendered views more important.

Follow-up / future considerations

Consider adding persistence or export for richer trade history if the current-day-only window becomes too limiting.
If the dashboard grows further, separate strategic overview from execution detail more aggressively so the page stays readable.

v0.5.0 - 2026-04-20 - Deprecated rejection core and made trade setups the sole live strategy path

Problem / context

The bot had accumulated two different decision systems: the original rejection/reclaim state machine and the newer operator-defined trade-setup pipeline. That split created ambiguity about which mechanism was authoritative in live operation. It also made it possible for the deprecated core strategy to fire a live short even when the active trade-setup ledger only contained long plans, which was no longer acceptable for the intended operating model.

Decision

Treat trade_setups.json as the only live strategy source and disable the legacy rejection/reclaim entry path by default. The remaining core code stays in place for historical reference and future comparison, but it is no longer allowed to arm or enter trades unless explicitly re-enabled for a controlled experiment.

Implementation

Added a REJECTION_CORE_ENABLED configuration gate, defaulting to false.
Wrapped the legacy arming and confirmation flow so the rejection/reclaim state machine no longer produces live entries when the core path is disabled.
Kept the trade-setup pipeline fully dynamic so any future setups added to trade_setups.json are processed automatically without hardcoding a count or list.
Updated the bot startup message to state that the rejection core is disabled and that only trade setups are being processed.
Preserved the setup ledger behavior so the bot still evaluates all configured active setups on each cycle.

Impact / outcome

The live trading authority is now unambiguous: the bot only acts on operator-defined trade setups.
The risk of an unexpected rejection/reclaim short or long being placed outside the setup ledger was removed from normal operation.
The strategy model became easier to reason about because setup management and execution are now aligned.

Tradeoffs / side effects

The legacy rejection/reclaim logic is still present in code, so the repository carries one additional disabled path until it is either removed or intentionally revived.
Any future decision to test the deprecated core again now requires an explicit configuration change, which adds a small amount of operational friction by design.

Follow-up / future considerations

Decide whether the deprecated core should eventually be deleted entirely or retained behind the feature flag as a fallback research path.
Keep the setup ledger format stable so future additions can be introduced without changing the bot’s execution model.
If more than three setups are added later, the current dynamic loader already supports that growth without further code changes.

v0.4.0 - 2026-04-20 - Operator-defined trade setup pipeline and market-data fallback

Problem / context

The bot had matured past a single fixed rejection/reclaim strategy, but the operational workflow was still too narrow. There was no structured way to load multiple trade ideas, monitor which setup was currently armed or running, or retire a stale setup without touching the bot itself. In parallel, the Binance websocket manager proved unstable in this environment, which created a reliability gap between market conditions and live execution.

Decision

Separate strategy expression from execution by introducing a trade-setup layer that can hold multiple operator-defined plans at once, each with its own entry logic, stop, targets, and cancellation state. At the same time, reduce dependence on Binance's websocket manager by switching to direct market-data sockets with fallback providers and a polling safety net.

Implementation

Added a persistent trade-setup ledger in data/f1bot/trade_setups.json.
Loaded three active setups covering deep-flush, panic-wick, and breakout-confirmation behavior.
Added setup-level state tracking for arming, entry fills, target hits, trailing behavior, and cancel/reset handling.
Added risk-based sizing from max equity so each setup can size itself consistently from the configured risk percentage.
Extended the bot status payload to report setup counts, active/running state, last actions, and market-data health.
Exposed setup state in the web portal so the current active or pending setup can be seen without reading logs.
Added a portal delete path that marks a setup cancelled and resets it out of the active workflow.
Replaced the fragile Binance websocket-manager startup path with direct market websockets and fallback support across Binance, Bybit, and OKX, with polling available as a backup when streaming becomes stale.

Impact / outcome

The bot can now hold multiple trade plans at once instead of being locked to one hard-coded strategy.
Operators can see which setup is armed, running, or idle from the web UI instead of inferring state from logs.
Stale setups can be removed cleanly without redeploying the bot.
Live market-data ingestion became more resilient, which improved the chance that strategy conditions are actually observed in real time.

Tradeoffs / side effects

The system now carries more state, which increases the cost of recovery and the need for clear status reporting.
Different market-data providers may not match Binance fills perfectly, so feed selection matters operationally.
The portal and bot are more tightly coupled around shared setup state, which makes schema and file-format stability more important.

Follow-up / future considerations

Add per-setup order IDs and fill history to the portal so execution can be audited more precisely.
Consider a stronger distinction between legacy strategy pages and operator-defined setups so the UI is less ambiguous.
If the multi-provider market feed becomes the default path, add provider selection and alerting so fallback behavior is visible before it becomes a problem.

v0.3.0 - 2026-04-20 - Isolated margin execution for rejection/reclaim

Problem / context

The rejection/reclaim strategy needed to trade against the isolated margin balance that was actually available to the account. Futures endpoints were not usable for this setup, and the bot's earlier execution path did not support isolated margin borrow/repay behavior.

Decision

Move the live execution path to isolated margin and make the bot manage the borrow, entry, exit, and repayment flow itself. Keep the entry logic event-driven so the system still waits for a price touch and then a confirmation close before it enters.

Implementation

Changed the live execution mode to BINANCE_PRODUCT=MARGIN with isolated margin enabled.
Added margin-specific order flow for both directions.
For shorts, the bot borrows base asset, sells into the position, and places OCO protection with auto-repay behavior.
For longs, the bot buys from isolated quote balance and places OCO protection on the exit.
Added isolated-margin websocket subscription support so fills and order updates can be tracked in real time.
Kept the risk model tied to stop distance and RISK_USD rather than allocating the full account balance.

Impact / outcome

The bot can now operate against the isolated margin funds that are actually present on the account.
The execution model matches the intended trading venue instead of assuming futures access.
Risk remains bounded by design rather than by available balance.

Tradeoffs / side effects

Margin trading adds borrow/repay complexity and more failure modes than spot-only execution.
Shorts depend on borrow availability and on the isolated margin symbol being enabled correctly on the exchange side.
Exit handling is more operationally sensitive because the bot now manages paired protection orders and repayment flow.

Follow-up / future considerations

Monitor whether borrow limits, order fill behavior, or repayment timing require tighter safeguards.
If isolated margin proves too restrictive operationally, consider a clearer separation between long-only and short-capable paths.

v0.2.0 - 2026-04-20 - Event-driven rejection/reclaim state machine

Problem / context

The strategy logic needed more than static order placement. The desired behavior was conditional: wait for BTC to touch a zone, then confirm rejection or reclaim on a close back through a threshold. Native exchange orders cannot express that full sequence on their own.

Decision

Implement the entry logic as a websocket-driven state machine with explicit arming, confirmation, and timeout rules. Use candle closes for confirmation to reduce noise and avoid reacting to transient spikes.

Implementation

Added a dedicated rejection/reclaim bot process.
Subscribed to trade and kline streams for fast touch detection and close-based confirmation.
Introduced the NEUTRAL, SHORT_ARMED, LONG_ARMED, ENTERING, and IN_TRADE state flow.
Defined resistance and support zones from the transcript-derived levels.
Added risk-based position sizing from RISK_USD and stop distance.
Added immediate protection orders after entry.
Added bot-specific status and log output files so the strategy could be monitored independently.

Impact / outcome

The bot gained a precise entry model instead of relying on native order conditions that could not describe the sequence.
Execution became more deterministic and easier to reason about during live operation.
The strategy now exposes a clear operational state, which makes the bot easier to supervise.

Tradeoffs / side effects

The bot became dependent on websocket health and timing.
More state has to be maintained in-process, which makes restart and recovery behavior more important.
The implementation is more complex than a simple order-only bot.

Follow-up / future considerations

Strengthen reconnect and stale-feed handling if websocket reliability becomes a practical issue.
Keep monitoring whether 1m close confirmation is the right tradeoff between responsiveness and noise.

v0.1.0 - 2025-11-19 - Operational logging and dashboard foundation

Problem / context

The bot did not have durable enough operational visibility. It needed a persistent status model and a simple dashboard so the live process could be understood without attaching to the runtime directly.

Decision

Persist bot status to disk and expose a lightweight Flask/Gunicorn dashboard that could surface the status and logs in a stable way.

Implementation

Added status.json style persistence for live bot state.
Added richer log payloads so the bot’s status could be read after the fact.
Built a Flask/Gunicorn status server to expose the bot state, JSON endpoint, and log view.
Wired the dashboard into the existing runtime so it could stay available alongside the bot process.
Confirmed the operational schedule and supporting dependencies needed for the bot to keep running.

Impact / outcome

The bot became much easier to monitor and debug.
Operational state could be inspected without digging through process output.
The supporting dashboard created a stable base for later strategy-specific pages.

Tradeoffs / side effects

Added another long-lived service to maintain.
Introduced file-backed state that must remain consistent with the running bot.

Follow-up / future considerations

Use the dashboard as the primary place to surface new strategy state.
Keep the status payload structured so future changes remain readable.