​
Overview

InsForge provides a high-performance, scalable storage system built on AWS S3, delivering enterprise-grade reliability with 99.999999999% (11 9’s) durability.

​
Technology Stack

​
Core Components

ComponentTechnologyPurpose
Storage BackendAWS S3Enterprise-grade object storage
Metadata StorePostgreSQLFile metadata, bucket configuration
Upload HandlerMulterMultipart form data parsing (both backends)
URL StrategyPresigned URLsSecure direct uploads/downloads (S3 only)
Access ControlJWT + Bucket visibilityPublic/private bucket permissions

​
AWS S3 Architecture

​
Enterprise Features

  • Direct Uploads: Presigned URLs bypass API server for unlimited scale
  • IAM Security: Role-based authentication without credential management
  • Multi-Tenancy: Secure isolation between projects using app key prefix
  • Bucket Policies: Public and private bucket configurations
  • Automatic Cleanup: Configurable lifecycle policies
  • Metadata Tracking: Rich file metadata stored in PostgreSQL

​
Upload Strategies

​
Direct Upload to S3

​
Bucket Configuration

​
Bucket Types

TypeAccessUse Case
PublicNo auth required for downloadsPublic assets, images, static files
PrivateAuth required for all operationsUser files, sensitive documents

​
File Operations

​
Upload Flow

  1. Request Upload: Client requests upload permission
  2. Validation: Check auth, bucket permissions, file size
  3. Strategy Selection: Choose direct or presigned upload
  4. Upload: Client uploads via selected method
  5. Confirmation: Verify upload and store metadata
  6. Response: Return file URL and metadata

​
Download Flow

  1. Request File: Client requests file access
  2. Permission Check: Verify bucket/object permissions
  3. Strategy Selection: Direct serve or presigned URL
  4. Delivery: Stream file or redirect to URL

​
Security Features

Bucket Policies

Public, private, or protected bucket access control

JWT Authentication

Token-based access for private resources

Presigned URLs

Time-limited URLs for secure S3 access

MIME Type Validation

Restrict uploads to allowed file types

Size Limits

10MB default, configurable via MAX_FILE_SIZE

App Key Isolation

Multi-tenant isolation using app key prefix in S3

​
Intelligent Metadata Management

InsForge maintains optimized metadata in PostgreSQL for instant queries:
  • Fast Search: Indexed metadata for sub-millisecond lookups
  • Rich Metadata: MIME types, sizes, timestamps, custom tags
  • Usage Analytics: Track downloads, bandwidth, popular files
  • Access Control: Fine-grained permissions per file or bucket

​
Secure URL Generation

TypeUse CaseSecurity
Public AccessStatic assets, imagesDirect S3 URLs
Presigned GETPrivate file accessTime-limited, single-use
Presigned POSTDirect uploadsValidated, size-limited

​
Performance Optimizations

​
Performance Features

  • Direct S3 Access: Bypass API server for uploads/downloads
  • Browser Caching: Cache-Control headers
  • ETag Support: Conditional requests for S3
  • Parallel Uploads: Support for multipart uploads

​
Upload Methods

1. Direct Upload through API Server:
  • Client sends file to /api/storage/buckets/{bucket}/objects
  • File passes through API server memory
  • Server uploads to S3
  • Limited by server memory (10MB default)
2. Presigned URL Upload (Recommended):
  • Client requests upload URL from /api/storage/buckets/{bucket}/upload-strategy
  • Server returns presigned POST URL
  • Client uploads directly to S3 (bypasses API server)
  • No server memory constraints
  • Client confirms upload via /api/storage/buckets/{bucket}/objects/{key}/confirm-upload

​
Configuration

​
Environment Variables

VariableDescriptionExample
AWS_S3_BUCKETS3 bucket namemy-app-storage
AWS_REGIONAWS regionus-east-2
APP_KEYApp key for S3 multi-tenancymy-app-key

​
S3 Configuration

// S3 client configuration
const s3Client = new S3Client({
  region: this.region, // e.g., 'us-east-2'
  // IAM role credentials are automatically used on EC2
  // No explicit credentials needed in production
});

// File paths use app key prefix for multi-tenancy
const s3Key = `${this.appKey}/${bucket}/${key}`;

​
Best Practices

Use Buckets

Organize files logically in buckets

Set Limits

Configure appropriate size/type limits

Clean URLs

Use consistent, SEO-friendly key naming

Metadata

Store searchable metadata in database

Backup Strategy

Implement regular backups for production

Monitor Usage

Track storage costs and usage patterns