> ## Documentation Index
> Fetch the complete documentation index at: https://docs.insforge.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Storage Architecture

> Enterprise-grade storage system powered by S3-compatible storage

## Overview

InsForge provides a high-performance, scalable storage system built on any S3-compatible endpoint, delivering enterprise-grade reliability. Actual durability depends on your storage provider (e.g., AWS S3 offers 99.999999999% durability).

<Info>
  **Bring your own S3** — Enterprise customers can use their own S3-compatible storage endpoint. [Contact us](mailto:tony.chang@insforge.dev) to learn more.
</Info>

## Technology Stack

```mermaid theme={null}
graph TB
    Client[Client Application] --> SDK[InsForge SDK]
    SDK --> StorageAPI[Storage API]
    
    StorageAPI --> S3[AWS S3]
    
    StorageAPI --> DB[(PostgreSQL)]
    DB --> Metadata[File Metadata]
    DB --> Buckets[Bucket Configuration]
    
    S3 --> DirectUpload[Presigned URLs]
    S3 --> SecureAccess[IAM Policies]
    
    style Client fill:#1e293b,stroke:#475569,color:#e2e8f0
    style SDK fill:#1e40af,stroke:#3b82f6,color:#dbeafe
    style StorageAPI fill:#166534,stroke:#22c55e,color:#dcfce7
    style S3 fill:#ea580c,stroke:#f97316,color:#fed7aa
    style DB fill:#0e7490,stroke:#06b6d4,color:#cffafe
    style Metadata fill:#0e7490,stroke:#22d3ee,color:#cffafe
    style Buckets fill:#0e7490,stroke:#22d3ee,color:#cffafe
    style DirectUpload fill:#ea580c,stroke:#fb923c,color:#fed7aa
    style SecureAccess fill:#ea580c,stroke:#fb923c,color:#fed7aa
```

## Core Components

| Component           | Technology              | Purpose                                     |
| ------------------- | ----------------------- | ------------------------------------------- |
| **Storage Backend** | S3-compatible           | Enterprise-grade object storage             |
| **Metadata Store**  | PostgreSQL              | File metadata, bucket configuration         |
| **Upload Handler**  | Multer                  | Multipart form data parsing (both backends) |
| **URL Strategy**    | Presigned URLs          | Secure direct uploads/downloads (S3 only)   |
| **Access Control**  | JWT + Bucket visibility | Public/private bucket permissions           |

## AWS S3 Architecture

### Enterprise Features

* **Direct Uploads**: Presigned URLs bypass API server for unlimited scale
* **IAM Security**: Role-based authentication without credential management
* **Multi-Tenancy**: Secure isolation between projects using app key prefix
* **Bucket Policies**: Public and private bucket configurations
* **Automatic Cleanup**: Configurable lifecycle policies
* **Metadata Tracking**: Rich file metadata stored in PostgreSQL

## Upload Strategies

### Direct Upload to S3

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant API
    participant S3
    participant DB
    
    Client->>API: POST/PUT with multipart/form-data
    API->>S3: Upload file
    S3-->>API: Success
    API->>DB: Store metadata
    API-->>Client: {url, key, size}
```

### Presigned URL Upload (Recommended)

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant API
    participant S3
    participant DB
    
    Client->>API: GET upload strategy
    API->>S3: Generate presigned URL
    S3-->>API: Presigned URL
    API-->>Client: {uploadUrl, key}
    Client->>S3: Direct upload
    Client->>API: Confirm upload
    API->>DB: Store metadata
    API-->>Client: {url, key, size}
```

## Bucket Configuration

### Bucket Types

| Type        | Access                           | Use Case                            |
| ----------- | -------------------------------- | ----------------------------------- |
| **Public**  | No auth required for downloads   | Public assets, images, static files |
| **Private** | Auth required for all operations | User files, sensitive documents     |

## File Operations

### Upload Flow

1. **Request Upload**: Client requests upload permission
2. **Validation**: Check auth, bucket permissions, file size
3. **Strategy Selection**: Choose direct or presigned upload
4. **Upload**: Client uploads via selected method
5. **Confirmation**: Verify upload and store metadata
6. **Response**: Return file URL and metadata

### Download Flow

1. **Request File**: Client requests file access
2. **Permission Check**: Verify bucket/object permissions
3. **Strategy Selection**: Direct serve or presigned URL
4. **Delivery**: Stream file or redirect to URL

## Security Features

<CardGroup cols={2}>
  <Card title="Bucket Policies" icon="shield">
    Public, private, or protected bucket access control
  </Card>

  <Card title="JWT Authentication" icon="key">
    Token-based access for private resources
  </Card>

  <Card title="Presigned URLs" icon="link">
    Time-limited URLs for secure S3 access
  </Card>

  <Card title="MIME Type Validation" icon="file-check">
    Restrict uploads to allowed file types
  </Card>

  <Card title="Size Limits" icon="weight">
    50MB default, configurable via MAX\_FILE\_SIZE
  </Card>

  <Card title="App Key Isolation" icon="key">
    Multi-tenant isolation using app key prefix in S3
  </Card>
</CardGroup>

## Intelligent Metadata Management

InsForge maintains optimized metadata in PostgreSQL for instant queries:

* **Fast Search**: Indexed metadata for sub-millisecond lookups
* **Rich Metadata**: MIME types, sizes, timestamps, custom tags
* **Usage Analytics**: Track downloads, bandwidth, popular files
* **Access Control**: Fine-grained permissions per file or bucket

### Secure URL Generation

| Type               | Use Case              | Security                 |
| ------------------ | --------------------- | ------------------------ |
| **Public Access**  | Static assets, images | Direct S3 URLs           |
| **Presigned GET**  | Private file access   | Time-limited, single-use |
| **Presigned POST** | Direct uploads        | Validated, size-limited  |

## Performance Optimizations

### Performance Features

* **Direct S3 Access**: Bypass API server for uploads/downloads
* **Browser Caching**: Cache-Control headers
* **ETag Support**: Conditional requests for S3
* **Parallel Uploads**: Support for multipart uploads

### Upload Methods

**1. Direct Upload through API Server:**

* Client sends file to `/api/storage/buckets/{bucket}/objects`
* File passes through API server memory
* Server uploads to S3
* Limited by server memory (10MB default)

**2. Presigned URL Upload (Recommended):**

* Client requests upload URL from `/api/storage/buckets/{bucket}/upload-strategy`
* Server returns presigned POST URL
* Client uploads directly to S3 (bypasses API server)
* No server memory constraints
* Client confirms upload via `/api/storage/buckets/{bucket}/objects/{key}/confirm-upload`

## Configuration

### Environment Variables

| Variable        | Description                  | Example          |
| --------------- | ---------------------------- | ---------------- |
| `AWS_S3_BUCKET` | S3 bucket name               | `my-app-storage` |
| `AWS_REGION`    | AWS region                   | `us-east-2`      |
| `APP_KEY`       | App key for S3 multi-tenancy | `my-app-key`     |

### S3 Configuration

```javascript theme={null}
// S3 client configuration
const s3Client = new S3Client({
  region: this.region, // e.g., 'us-east-2'
  // IAM role credentials are automatically used on EC2
  // No explicit credentials needed in production
});

// File paths use app key prefix for multi-tenancy
const s3Key = `${this.appKey}/${bucket}/${key}`;
```

## Best Practices

<CardGroup cols={2}>
  <Card title="Use Buckets" icon="folder">
    Organize files logically in buckets
  </Card>

  <Card title="Set Limits" icon="gauge">
    Configure appropriate size/type limits
  </Card>

  <Card title="Clean URLs" icon="link">
    Use consistent, SEO-friendly key naming
  </Card>

  <Card title="Metadata" icon="tags">
    Store searchable metadata in database
  </Card>

  <Card title="Backup Strategy" icon="cloud-arrow-up">
    Implement regular backups for production
  </Card>

  <Card title="Monitor Usage" icon="chart-line">
    Track storage costs and usage patterns
  </Card>
</CardGroup>

## SDK References

Choose the storage SDK guide for your platform:

<CardGroup cols={2}>
  <Card title="TypeScript" icon="js" href="/sdks/typescript/storage">
    Web and Node.js applications with upload, download, and presigned URL support
  </Card>

  <Card title="Swift" icon="swift" href="/sdks/swift/storage">
    iOS, macOS, tvOS, and watchOS with async/await file operations
  </Card>

  <Card title="Kotlin" icon="android" href="/sdks/kotlin/storage">
    Android applications with Coroutines-based file handling
  </Card>
</CardGroup>