BS Detection Phase 0: Human-in-the-Loop Approach

The Problem: Subtle BS in Complex Systems

As our Mercury trading system has grown more sophisticated, we've encountered an increasing number of subtle "BS" issues - data inconsistencies, semantic misrepresentations, and logical contradictions that can lead to poor decision-making. These issues are particularly challenging because they often involve data that is technically correct but semantically misleading.

A recent example highlighted this problem: volume slope (a percentage change value) was being formatted with volume suffixes (M for millions), completely changing its meaning from "439.74%" to "439.74 million" - a representation that could lead to dramatically wrong interpretations.

The Pragmatic Solution: Phase 0

While our long-term vision includes the MAAT validation system (named after the Egyptian goddess of truth), we need an immediate, pragmatic approach to catch these issues. Enter "Phase 0" - a human-in-the-loop BS detection system that leverages the most capable neural network available: the human brain.

Core Concept

Phase 0 is built around systematic sampling of data throughout our tournament system pipeline, with human expertise applied to identify patterns of BS. The approach is deliberately lightweight and focused on building a knowledge base rather than implementing complex automated checks prematurely.

How It Works

1. Strategic Sampling Points

   • TA API Data Fetch: Raw data from technical analysis endpoints
   • Data Transformations: Before/after transformation samples
   • LLM Interactions: Prompts and responses from our models
   • Market Comparisons: Inputs and decisions from market comparisons
   • Tournament Decisions: Final bracket decisions and reasoning

2. Daily Gist Generation

   • Samples are collected with a low probability (e.g., 5%)
   • A daily gist is generated with categorized samples
   • Each sample includes clear input/output data
   • Samples are formatted for easy analysis with OpenAI's O1

3. O1-Assisted Analysis

   • Samples are structured for direct pasting into O1
   • O1's reasoning capabilities and large context window make it ideal for BS detection
   • Analysis focuses on inconsistencies, misrepresentations, and logical contradictions
   • Findings are documented for future reference

4. Iterative Improvement

   • Patterns of BS are identified over time
   • Most critical issues are addressed first
   • Knowledge gained informs the design of automated checks
   • Gradually transition from human-in-the-loop to automated validation

Sample Format for O1 Analysis

Each sample in the daily gist follows a structured format designed for O1 analysis:

# [Category] Sample [Number]

## Input
[Input data in a clear format]

## Output
[Output data in a clear format]

## Analysis Questions
1. Are there any inconsistencies in how values are formatted?
2. Do any values seem to be represented incorrectly?
3. Are units being applied appropriately to each value?
4. Would this representation mislead a human or AI analyzing the data?

This format allows for efficient analysis by O1, which can identify subtle issues that might be missed by automated checks.

Benefits of Phase 0

1. Low Engineering Overhead: Minimal implementation effort required
2. Human Expertise Leverage: Uses human judgment where it excels
3. Knowledge Base Building: Creates a repository of BS patterns
4. Targeted Improvements: Focuses resources on the most critical issues
5. O1 Integration: Leverages OpenAI's O1 capabilities for deep analysis
6. Evolutionary Approach: Builds toward automated validation incrementally

Path to Full Automation

Phase 0 is the first step in a journey toward comprehensive automated BS detection:

• Phase 1: Field metadata registry and basic type/unit validation
• Phase 2: Cross-field consistency rules
• Phase 3: Temporal consistency validation
• Phase 4: LLM-specific validation
• Phase 5: Learning system to improve detection based on historical issues

By starting with Phase 0, we build a solid foundation of knowledge about where and how BS manifests in our system, ensuring that our eventual automated solution addresses real-world issues rather than theoretical concerns.

Conclusion

The Phase 0 approach to BS detection represents a pragmatic balance between immediate value and long-term vision. By leveraging human expertise and O1's capabilities, we can quickly identify and address critical issues while building toward a more comprehensive automated solution.

In the world of algorithmic trading, where small inconsistencies can lead to significant financial consequences, this approach ensures that we maintain data integrity throughout our system without overengineering solutions prematurely.