Stop Using Just One AI Model in Production

“Why Model Redundancy Beats Optimization”

---

The Problem: A Rate-Limited Bottleneck

It was a frustrating Thursday afternoon. Our code analysis service kept hitting rate limits, and I did what any logical engineer would: optimize token usage, implement better queuing, and squeeze every bit of performance from our AI model.

Nothing worked. Or rather, everything worked a little—but not enough.

The Discovery: A Happy Accident

We were using Amazon Bedrock’s Nova-Pro model for code analysis. With a quota of 100,000 tokens per minute, it seemed more than sufficient—until reality proved otherwise.

The issue? Code analysis isn’t linear. A developer might push 50 files at once, creating unpredictable bursts. Rate limiting punished us by dropping entire batches, forcing retries, and slowing everything down.

Then, during one particularly frustrating debugging session, I switched to a supposedly weaker model—Llama-3—expecting nothing. Instead, our throughput increased.

The Math Behind the Solution

Our quotas:

• Nova-Pro: 100,000 tokens/minute

• Llama-3: 60,000 tokens/minute

On paper, it made no sense to mix models. But in practice, treating them as complementary rather than primary/backup changed everything.

The Old Playbook (That We Ditched)

Conventional wisdom says:

1. Pick the best model for your use case.

2. Optimize usage with smarter queuing.

3. Implement strict rate limiting.

4. Maybe keep a backup model for failover.

We threw that out the window. Instead, we:

• Used both models simultaneously.

• Routed larger batches to Nova-Pro.

• Sent smaller, simpler batches to Llama-3.

• Let each model handle its own rate limits.

Why This Shouldn’t Work (But Does)

1. Rate Limits Are Not Linear

Hitting a rate limit doesn’t just drop the extra tokens—it can delay entire batches, forcing retries and increasing latency.

2. Context Switching Is Expensive

Retrying batches later means losing momentum. Keeping two models running in parallel maintains workflow continuity.

3. Different Models, Different Strengths

Surprisingly, Llama-3 performed better for smaller files, while Nova-Pro excelled at analyzing large, complex dependencies.

The Implementation: Smarter Model Orchestration

Instead of complex rate-limiting logic, we built a model-aware batching strategy:

• Large batches → Nova-Pro.

• Smaller batches → Llama-3.

• Let each model manage its own limits.

The Code

Here’s how we implemented this approach in JavaScript/TypeScript:

// Model-specific quotas and characteristics
const MODEL_QUOTAS = {
  'nova-pro': {
    tokensPerMinute: 100_000,
    maxKbPerBatch: 350,  // Handles large, complex files
  },
  'llama-3': {
    tokensPerMinute: 60_000,
    maxKbPerBatch: 200,  // Better for smaller, simpler files
  }
} as const;

/**
 * Assigns models to file batches based on complexity.
 */
function createModelAwareBatches(fileGroups: FileGroup[]): { model: ModelId, batch: FileGroup[] }[] {
  const groupsWithModels = fileGroups.map(group => ({
    group,
    model: selectModelForGroup(group)
  }));

  const batches: { model: ModelId, batch: FileGroup[] }[] = [];
  let currentBatch: FileGroup[] = [];
  let currentModel: ModelId | null = null;
  let currentBatchSize = 0;

  groupsWithModels.forEach(({ group, model }) => {
    const groupSizeKb = group.totalSize / 1024;

    if (
      currentModel === null ||
      model !== currentModel ||
      currentBatchSize + groupSizeKb > MODEL_QUOTAS[model].maxKbPerBatch
    ) {
      if (currentBatch.length > 0) {
        batches.push({ model: currentModel, batch: currentBatch });
      }
      currentBatch = [group];
      currentModel = model;
      currentBatchSize = groupSizeKb;
    } else {
      currentBatch.push(group);
      currentBatchSize += groupSizeKb;
    }
  });

  if (currentBatch.length > 0 && currentModel) {
    batches.push({ model: currentModel, batch: currentBatch });
  }

  return batches;
}

/**
 * Determines the best model for a file group.
 */
function selectModelForGroup(group: FileGroup): ModelId {
  const COMPLEXITY_THRESHOLD = 0.7;
  const complexityScore = calculateComplexityScore(group);
  return complexityScore > COMPLEXITY_THRESHOLD ? 'nova-pro' : 'llama-3';
}

/**
 * Computes a complexity score (0-1) based on file size, dependencies, and count.
 */
function calculateComplexityScore(group: FileGroup): number {
  const sizeFactor = Math.min(group.totalSize / (20 * 1024), 1);
  const dependencyFactor = group.complexity;
  const fileCountFactor = Math.min(group.files.length / 10, 1);

  return (
    sizeFactor * 0.3 +
    dependencyFactor * 0.5 +
    fileCountFactor * 0.2
  );
}

/**
 * Processes file batches, handling rate limits dynamically.
 */
async function processBatches(batches: { model: ModelId, batch: FileGroup[] }[]): Promise<Result[]> {
  const results: Result[] = [];

  for (const { model, batch } of batches) {
    try {
      const result = await processWithModel(batch, model);
      results.push(result);
    } catch (error) {
      if (isRateLimitError(error)) {
        const alternateModel: ModelId = model === 'nova-pro' ? 'llama-3' : 'nova-pro';
        const result = await processWithModel(batch, alternateModel);
        results.push(result);
      } else {
        throw error;
      }
    }
  }
  return results;
}

The Results

• 1.8x improvement in throughput.

• 40% reduction in rate limit errors.

• More consistent performance under load.

• Faster results = happier developers.

Why This Matters

AI is no longer a nice-to-have—it’s core to our applications. But the old “single model” mindset is holding us back. Instead of finding the perfect model, the real advantage comes from orchestrating multiple models efficiently.

The Trade-Offs

• Higher costs (but offset by reduced latency).

• Slightly more complex routing logic.

• Monitoring overhead to track both models.

Looking Ahead

The future isn’t about choosing one AI model. It’s about leveraging multiple models in harmony.

What do you think? Have you tried similar approaches? Let’s discuss AI service architecture!