AI Architecture

​

Overview

​

Technology Stack

​

Core Components

​

Supported Models

​

Capabilities

​

OpenRouter Integration

​

Why OpenRouter?

• Single API: One integration for multiple providers
• Unified Billing: Consistent pricing across models
• Automatic Failover: Fallback to alternative models
• Rate Limiting: Built-in rate limit handling

​

Request Flow

​

Chat Completions

​

Request Processing

1. Authentication: Verify JWT token
2. Configuration: Load project AI settings
3. System Prompt: Prepend configured prompt
4. Model Selection: Use specified or default model
5. OpenRouter Call: Forward to OpenRouter
6. Response Handling: Stream or batch response
7. Usage Tracking: Record token usage

​

Streaming Architecture

// Server-Sent Events (SSE) for streaming
async function* streamChat(messages, options) {
  const stream = await openRouter.chat.completions.create({
    model: options.model,
    messages: messages,
    stream: true,
    temperature: options.temperature,
    max_tokens: options.maxTokens
  });

  for await (const chunk of stream) {
    if (chunk.choices[0]?.delta?.content) {
      yield { chunk: chunk.choices[0].delta.content };
    }
  }
  
  yield { done: true, tokenUsage: {...} };
}

​

Tool Calling

​

Response Formats

{
  "response": "AI generated response text",
  "model": "anthropic/claude-3.5-haiku",
  "usage": {
    "promptTokens": 150,
    "completionTokens": 200,
    "totalTokens": 350
  }
}

data: {"chunk": "The "}
data: {"chunk": "answer "}
data: {"chunk": "is..."}
data: {"done": true, "tokenUsage": {...}}

​

Image Generation

​

Generation Flow

1. Prompt Processing: Validate and enhance prompt
2. Model Selection: Choose appropriate image model
3. Size Configuration: Set dimensions and quality
4. OpenRouter Request: Send generation request
5. URL Generation: Receive image URLs
6. Storage Integration: Optional save to storage
7. Response Delivery: Return URLs to client

​

Image Parameters

​

Configuration Management

​

Database Schema

CREATE TABLE _ai_configs (
  id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
  modality VARCHAR(255) NOT NULL,     -- 'text' or 'image'
  provider VARCHAR(255) NOT NULL,      -- 'openrouter'
  model_id VARCHAR(255) UNIQUE NOT NULL,
  system_prompt TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

​

System Prompts

• Configured per project
• Applied to all chat requests
• Cannot be overridden by client
• Support for multiple configurations

​

Usage Tracking

​

Metrics Collected

CREATE TABLE _ai_usage (
  id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
  config_id UUID NOT NULL,
  input_tokens INT,
  output_tokens INT,
  image_count INT,
  image_resolution TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  FOREIGN KEY (config_id) REFERENCES _ai_configs(id) ON DELETE NO ACTION
);

​

Security & Rate Limiting

​

Error Handling

​

Error Types

​

Retry Strategy

async function retryableRequest(fn, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (error.status === 429) { // Rate limited
        await sleep(Math.pow(2, i) * 1000); // Exponential backoff
        continue;
      }
      throw error;
    }
  }
}

​

Performance Optimizations

​

Streaming Optimizations

• Server-Sent Events: Real-time response streaming
• Chunked Transfer: Efficient data streaming
• Keep-Alive: Persistent connections for SSE
• Low Latency: Direct OpenRouter integration

​

Future Optimizations

• Response Caching: Cache for identical requests (coming soon)
• Batch Processing: Multiple requests in parallel (coming soon)
• Embeddings Cache: Store computed embeddings (coming soon)

​

API Endpoints

​

Chat Endpoints

​

Image Endpoints

​

Configuration Endpoints

​

Usage Tracking Endpoints

​

Gateway Credentials & BYOK

​

Managing BYOK Keys

​

Usage Tracking

​

API Endpoints

​

Environment Variables

• In cloud environments, the API key is fetched dynamically from the cloud API unless a BYOK key is configured
• A BYOK key configured from the dashboard always takes precedence over other sources
• Model configuration is stored in the database (_ai_configs table)
• No other AI-related environment variables are used

​

Best Practices

​

Comparison with Direct Integration

​

SDK References