Skip to main content

Queue Types Refactoring

· 4 min read

Problem Statement​

The current job queue system has several issues:

  • Job names and data types are not properly typed
  • Job data types are not enforced at compile time
  • Queue and job names are scattered across different files
  • No clear relationship between queues and their valid job types
  • Type safety is not enforced when adding jobs to queues

Solution Architecture​

1. Module-Level Types​

Each module that uses queues will have its own types/queue.types.ts file:

// apps/mercury/src/atlas/types/queue.types.ts
export enum AtlasQueueName {
  ATLAS = 'atlas',
  TA_AVAILABILITY = 'ta_availability',
}

export enum AtlasJobName {
  CASSANDRA_SAMPLE_ANALYSIS = 'cassandra_sample_analysis',
  CASSANDRA_MARKET_ANALYSIS = 'cassandra_market_analysis',
  // ...
}

// Job data type definitions
export interface CassandraSampleAnalysisData {
  symbol: string;
  // ...
}

export interface CassandraMarketAnalysisData {
  markets: string[];
  // ...
}

// Map job names to their data types
export interface AtlasJobDataMap {
  [AtlasJobName.CASSANDRA_SAMPLE_ANALYSIS]: CassandraSampleAnalysisData;
  [AtlasJobName.CASSANDRA_MARKET_ANALYSIS]: CassandraMarketAnalysisData;
}

// Define valid jobs for each queue
export const AtlasQueueJobs = {
  [AtlasQueueName.ATLAS]: [
    AtlasJobName.CASSANDRA_SAMPLE_ANALYSIS,
    AtlasJobName.CASSANDRA_MARKET_ANALYSIS,
  ],
  [AtlasQueueName.TA_AVAILABILITY]: [
    // ...
  ],
} as const;

// Type to get valid jobs for a queue
export type AtlasQueueJobMap = {
  [Q in keyof typeof AtlasQueueJobs]: (typeof AtlasQueueJobs)[Q][number];
};

2. Root Types​

The root types file will combine all module-specific types:

// apps/mercury/src/types/queue.types.ts
import {
  AtlasQueueName,
  AtlasJobName,
  AtlasJobDataMap,
  AtlasQueueJobMap
} from '../atlas/types/queue.types';
import {
  DikeQueueName,
  DikeJobName,
  DikeJobDataMap,
  DikeQueueJobMap
} from '../dike/types/queue.types';
// ... other imports

// Combine all queue names
export type QueueName = AtlasQueueName | DikeQueueName | /* ... */;

// Combine all job names
export type JobName = AtlasJobName | DikeJobName | /* ... */;

// Map queues to their jobs
export type QueueJobMap = AtlasQueueJobMap & DikeQueueJobMap & /* ... */;

// Map jobs to their data
export type JobDataMap = AtlasJobDataMap & DikeJobDataMap & /* ... */;

// Helper to get job data type
export type JobData<
  Q extends QueueName,
  J extends JobName
> = J extends keyof JobDataMap ? JobDataMap[J] : never;

// Helper to get queue type
export type Queue<Q extends QueueName> = BullMQ.Queue<
  JobData<Q, QueueJobMap[Q]>,
  any,
  QueueJobMap[Q]
>;

3. Usage in Consumers​

Example of a type-safe consumer:

@Injectable()
@Processor(AtlasQueueName.ATLAS)
export class AtlasConsumer extends WorkerHost {
  constructor(
    @InjectQueue(AtlasQueueName.ATLAS)
    private readonly queue: Queue<AtlasQueueName.ATLAS>,
  ) {
    super();
  }

  async process(job: Job<JobData<AtlasQueueName.ATLAS, AtlasJobName>>) {
    switch (job.name) {
      case AtlasJobName.CASSANDRA_SAMPLE_ANALYSIS:
        // TypeScript knows job.data is CassandraSampleAnalysisData
        return this.handleCassandraSampleAnalysis(job.data);
      case AtlasJobName.CASSANDRA_MARKET_ANALYSIS:
        // TypeScript knows job.data is CassandraMarketAnalysisData
        return this.handleCassandraMarketAnalysis(job.data);
    }
  }
}

4. Usage in Producers​

Example of a type-safe producer:

@Injectable()
export class AtlasProducer {
  constructor(
    @InjectQueue(AtlasQueueName.ATLAS)
    private readonly queue: Queue<AtlasQueueName.ATLAS>,
  ) {}

  async addCassandraSampleAnalysis(
    data: JobData<AtlasQueueName.ATLAS, AtlasJobName.CASSANDRA_SAMPLE_ANALYSIS>,
  ) {
    return this.queue.add(AtlasJobName.CASSANDRA_SAMPLE_ANALYSIS, data);
  }
}

Implementation Plan​

  1. Discovery Phase

    • Find all queue consumers using grep: @Processor
    • Find all job name usages: JobName.
    • Find all queue name usages: QueueName.
    • Document all job data types currently in use

  2. Per-Module Implementation

    • Create module's types/queue.types.ts
    • Move queue and job names to module-specific enums
    • Create job data interfaces
    • Create job data map
    • Define valid jobs for each queue
    • Update imports in module files
    • Update consumer to use new types
    • Update producer to use new types
    • Verify type safety with tsc

  3. Root Types Integration

    • Create/update root types file
    • Import and combine all module types
    • Create helper types
    • Update cross-module references

  4. Validation

    • Run TypeScript compiler
    • Check for any implicit any types
    • Verify job data types are enforced
    • Test queue operations

Benefits​

  1. Type Safety

    • Queue names are typed
    • Job names are typed and queue-specific
    • Job data is typed
    • TypeScript enforces correct job data structure

  2. Developer Experience

    • IDE autocompletion for job names and data
    • Refactoring support
    • Clear relationship between queues and jobs
    • Self-documenting code

  3. Maintainability

    • Modular type definitions
    • Easy to add new queues and jobs
    • Type errors caught at compile time
    • Clear separation of concerns

Migration Strategy​

  1. Start with one module as proof of concept (e.g., Atlas)
  2. Get team feedback on the approach
  3. Iterate on the design if needed
  4. Roll out to other modules
  5. Update root types incrementally
  6. Full type safety check after all modules are migrated

Rollback Plan​

Since this is a TypeScript-only change that doesn't affect runtime behavior:

  1. Keep old type definitions until migration is complete
  2. Can roll back individual modules if issues are found
  3. No impact on production runtime

Next Steps​

  1. Choose first module for proof of concept
  2. Create module types
  3. Update module consumers/producers
  4. Review with team
  5. Plan full rollout

Progress Tracking​

Atlas Module (✓)​

  • ✓ atlas.consumer.ts
  • ✓ atlas.producer.ts
  • ✓ atlas.module.ts
  • ✓ types/queue.types.ts

Artifacts Module (✓)​

  • ✓ artifact-update-github.consumer.ts
  • ✓ artifact-update-tg.consumer.ts
  • ✓ artifacts.module.ts
  • ✓ types/queue.types.ts

Minerva Module (✓)​

  • ✓ ta.consumer.ts
  • ✓ ta.producer.ts
  • ✓ ta.module.ts
  • ✓ types/queue.types.ts

Morpheus Module (✓)​

  • ✓ shadow-portfolio.consumer.ts
  • ✓ shadow-portfolio.producer.ts
  • ✓ shadow-portfolio.module.ts
  • ✓ types/queue.types.ts

Dike Module (✓)​

  • ✓ tournament.consumer.ts
  • ✓ tournament.producer.ts
  • ✓ tournament.module.ts
  • ✓ types/queue.types.ts

Thoth Module (Pending)​

  • ⏳ rag.consumer.ts
  • ⏳ rag.producer.ts
  • ⏳ rag.module.ts
  • ⏳ types/queue.types.ts

Hermes Module (Pending)​

  • ⏳ telegram.consumer.ts
  • ⏳ telegram.producer.ts
  • ⏳ telegram.module.ts
  • ⏳ types/queue.types.ts

Core Types (✓)​

  • ✓ queue.types.ts (updated with re-exports)
  • ✓ queue.config.ts (using ENV_DEFAULTS)
  • ✓ env.defaults.ts (with critical safety warning)
dev@redmax:~/ton-arcana$ echo "Find all job name usages:" && find apps -type f -name "*.ts" -exec grep -l "JobName\." {} \; && echo -e "\nFind all queue name usages:" && find apps -type f -name "*.ts" -exec grep -l echo "Find all job name usages:" && find apps -type   -name "*.ts" -exec      -l "JobName\." {}    && echo -e "\nFind all queue name usages:" && find apps -type   -name "*.ts" -exec      -l "QueueName\." {}
Find all job name usages:
apps/arcana-bot/src/image/image.consumer.ts
apps/arcana-bot/src/teachings/teaching.consumer.ts
apps/arcana-bot/src/teachings/teaching.service.ts
apps/arcana-bot/src/telegram/telegram.consumer.ts
apps/arcana-bot/src/telegram/telegram.service.ts
apps/arcana-bot/src/types/queue.types.ts
apps/arcana-bot/src/worker/tarot.consumer.ts
apps/arcana-bot/src/worker/worker.service.ts
apps/mercury/src/artifacts/artifacts.service.ts
apps/mercury/src/atlas/orchestrators/ta-availability.orchestrator.ts
apps/mercury/src/atlas/orchestrators/daily-report.orchestrator.ts
apps/mercury/src/atlas/orchestrators/tournament.orchestrator.ts
apps/mercury/src/atlas/atlas.module.ts
apps/mercury/src/telegram/handlers/admin/daily-report.handler.ts
apps/mercury/src/telegram/handlers/admin/ta-availability.handler.ts
apps/mercury/src/telegram/telegram.consumer.ts
apps/mercury/src/telegram/telegram.service.ts
apps/mercury/src/apollo/report/report.consumer.ts
apps/mercury/src/dike/tournament.service.ts
apps/mercury/src/dike/tournament.consumer.ts
apps/mercury/src/dike/tournament.producer.ts

Find all queue name usages:
apps/arcana-bot/src/image/image.consumer.ts
apps/arcana-bot/src/image/image.module.ts
apps/arcana-bot/src/teachings/teaching.consumer.ts
apps/arcana-bot/src/teachings/teaching.module.ts
apps/arcana-bot/src/teachings/teaching.service.ts
apps/arcana-bot/src/telegram/telegram.consumer.ts
apps/arcana-bot/src/telegram/telegram.service.ts
apps/arcana-bot/src/types/queue.types.ts
apps/arcana-bot/src/worker/tarot.consumer.ts
apps/arcana-bot/src/worker/worker.module.ts
apps/arcana-bot/src/worker/worker.service.ts
apps/mercury/src/admin/admin.consumer.ts
apps/mercury/src/admin/admin.module.ts
apps/mercury/src/artifacts/artifact-update-github.consumer.ts
apps/mercury/src/artifacts/artifact-update-tg.consumer.ts
apps/mercury/src/artifacts/artifacts.module.ts
apps/mercury/src/artifacts/artifacts.service.ts
apps/mercury/src/atlas/orchestrators/cassandra.orchestrator.ts
apps/mercury/src/atlas/orchestrators/ta-availability.orchestrator.ts
apps/mercury/src/atlas/orchestrators/daily-report.orchestrator.ts
apps/mercury/src/atlas/orchestrators/tournament.orchestrator.ts
apps/mercury/src/atlas/atlas.consumer.ts
apps/mercury/src/atlas/consumers/ta-availability.consumer.ts
apps/mercury/src/atlas/atlas.module.ts
apps/mercury/src/telegram/telegram.consumer.ts
apps/mercury/src/telegram/telegram.module.ts
apps/mercury/src/telegram/telegram.service.ts
apps/mercury/src/types/queue.types.ts
apps/mercury/src/cassandra/cassandra.module.ts
apps/mercury/src/apollo/report/report.consumer.ts
apps/mercury/src/apollo/report/report.module.ts
apps/mercury/src/dike/tournament.service.ts
apps/mercury/src/dike/dike.module.ts
apps/mercury/src/dike/tournament.consumer.ts
apps/mercury/src/dike/tournament.producer.ts
apps/mercury/src/minerva/ta-availability.consumer.ts
apps/mercury/src/minerva/minerva.module.ts
apps/mercury/src/morpheus/shadow-portfolio.consumer.ts
apps/mercury/src/morpheus/shadow-portfolio.producer.ts
apps/mercury/src/morpheus/shadow-portfolio.service.spec.ts
apps/mercury/src/morpheus/morpheus.module.ts
apps/mercury/src/morpheus/shadow-portfolio.module.ts
apps/mercury/src/tyche/test/market-comparison.service.e2e-spec.ts
apps/mercury/src/tyche/test/market-ranking.service.e2e-spec.ts
apps/mercury/src/tyche/tyche.module.ts
apps/mercury/src/tyche/market-ranking.service.ts
dev@redmax:~/ton-arcana$