How We Built Blazing Fast CDN Delivery for 1000+ Websites (S3 + CloudFront)

TL;DR: We built a multi-tenant CDN architecture that serves 1000+ websites from S3 via CloudFront edge locations worldwide. Every website loads in <500ms globally. The secret: CloudFront Functions for subdomain routing, S3 for static hosting, and intelligent cache invalidation. Zero servers, infinite scale.

The Problem: Traditional Hosting is Slow

When we started WebZum, we generated websites dynamically. Every request hit our Next.js server.

The results were terrible:

• Load times: 2-4 seconds (unacceptable)
• Server costs: $500/month for 100 websites
• Scaling issues: CPU spikes during traffic bursts
• Geographic latency: Users in Australia waited 3+ seconds

The insight: Generated websites are static HTML. Why serve them from a server?

Bad: User → App Runner → Generate Response → User (2-4 seconds) Good: User → CloudFront Edge → S3 → User (<500ms)

The Breakthrough: Multi-Tenant CDN Architecture

The breakthrough came when we realized: Every website is just static files. Put them on a CDN.

But there’s a challenge: How do you serve 1000+ websites from one S3 bucket?

Traditional approach: One S3 bucket per website (doesn’t scale, expensive) Our approach: One S3 bucket, subdomain-based routing via CloudFront Functions

The architecture:

User requests: business.webzum.com
  ↓
CloudFront Edge Location (worldwide)
  ↓
CloudFront Function rewrites: business.webzum.com → /generated/business/latest/
  ↓
S3 Bucket: refresh-websites-146293675031-us-west-2
  ↓
Cached at edge for 24 hours

Result: Every website loads from the nearest CloudFront edge location. Blazing fast, globally.

The Technical Architecture

1. S3 Bucket Structure

One bucket, organized by business ID:

refresh-websites-146293675031-us-west-2/
├── generated/
│   ├── bairddrainservice/
│   │   ├── latest/                    # Live production version
│   │   │   ├── index.html
│   │   │   ├── styles.css
│   │   │   ├── images/
│   │   │   └── ...
│   │   ├── v-uuid-1/                  # Version history
│   │   ├── v-uuid-2/
│   │   └── v-uuid-3/
│   ├── pizzapalace/
│   │   ├── latest/
│   │   └── ...
│   └── [1000+ more businesses]/

Key insight: The /latest/ folder is the live version. Deployments copy files here.

2. CloudFront Function for Subdomain Routing

CloudFront Functions run at edge locations (not Lambda@Edge). Faster, cheaper, more scalable.

function handler(event) {
  var request = event.request;
  var host = request.headers.host.value;
  
  console.log('Original request:', {
    host: host,
    uri: request.uri
  });
  
  // Extract subdomain from host
  var subdomain = getSubdomain(host);
  
  if (subdomain) {
    // Rewrite path to S3 structure
    var newUri = '/generated/' + subdomain + '/latest' + request.uri;
    
    // Handle directory requests
    if (newUri.endsWith('/')) {
      newUri += 'index.html';
    }
    
    console.log('Rewritten URI:', newUri);
    request.uri = newUri;
  }
  
  return request;
}

function getSubdomain(host) {
  // Extract subdomain from host (e.g., 'business.webzum.com' → 'business')
  var parts = host.split('.');
  
  // Handle different domain formats
  if (parts.length >= 3) {
    // Format: subdomain.webzum.com
    return parts[0];
  }
  
  return null;
}

What this does:

• User requests: business.webzum.com/about
• Function rewrites to: /generated/business/latest/about
• S3 serves: s3://bucket/generated/business/latest/about/index.html
• CloudFront caches at edge

Cost: $0.10 per 1M requests (vs $0.60 for Lambda@Edge)

3. CloudFront Distribution Configuration

Two distributions for clean separation:

Distribution 1: Subdomains (*.webzum.com)

const subdomainDistribution = new cloudfront.Distribution(this, 'SubdomainDist', {
  comment: 'Subdomain to S3 path mapping',
  
  // Default behavior - S3 for all subdomains
  defaultBehavior: {
    origin: new origins.S3Origin(s3Bucket),
    viewerProtocolPolicy: cloudfront.ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
    compress: true, // Enable gzip/brotli compression
    
    // Aggressive caching for static content
    cachePolicy: cloudfront.CachePolicy.CACHING_OPTIMIZED,
    
    // Attach subdomain routing function
    functionAssociations: [
      {
        function: subdomainMappingFunction,
        eventType: cloudfront.FunctionEventType.VIEWER_REQUEST,
      }
    ],
  },
  
  // Domain configuration
  domainNames: ['*.webzum.com'],
  certificate: acm.Certificate.fromCertificateArn(this, 'Cert', certificateArn),
  
  // Performance optimizations
  httpVersion: cloudfront.HttpVersion.HTTP2_AND_3, // HTTP/3 support
  priceClass: cloudfront.PriceClass.PRICE_CLASS_100, // US, Canada, Europe
  enableIpv6: true,
});

Distribution 2: Main Site (webzum.com)

const mainDistribution = new cloudfront.Distribution(this, 'MainDist', {
  comment: 'Main site - App Runner',
  
  // App Runner origin for dynamic content
  defaultBehavior: {
    origin: new origins.HttpOrigin(appRunnerUrl),
    viewerProtocolPolicy: cloudfront.ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
    
    // No caching for dynamic content
    cachePolicy: cloudfront.CachePolicy.CACHING_DISABLED,
  },
  
  domainNames: ['webzum.com', 'www.webzum.com'],
  certificate: acm.Certificate.fromCertificateArn(this, 'Cert', certificateArn),
  
  httpVersion: cloudfront.HttpVersion.HTTP2_AND_3,
  priceClass: cloudfront.PriceClass.PRICE_CLASS_100,
});

Why two distributions?

• Clean separation: Subdomains → S3, Main site → App Runner
• No path conflicts: No need to manage cache behaviors
• Easier debugging: Clear separation of concerns
• Better performance: Subdomains never touch App Runner

4. Deployment Pipeline

When a website is generated or updated:

export async function deployToLatest(
  outputDir: string,
  progressTracker?: ProgressTracker
): Promise<void> {
  console.log(`📂 [DEPLOY] Starting deployment from ${outputDir}`);
  
  // Step 1: Clear old files from /latest/
  const latestDir = path.join(parentDir, 'latest');
  await clearLatestDirectory(latestDir);
  
  // Step 2: Copy new files to /latest/ (strip editor code)
  const copyResult = await copyAndCleanFiles(outputDir, latestDir);
  console.log(`✅ [DEPLOY] Copied ${copyResult.copiedFiles.length} files`);
  
  // Step 3: Invalidate CloudFront cache
  await cloudfrontInvalidation.invalidateSubdomain({
    domainPath: businessId,
    progressTracker
  });
  
  // Step 4: If custom domain exists, invalidate that too
  const business = await BusinessRegistry.findByBusinessId(businessId);
  if (business?.customDomain?.cloudFrontDistributionId) {
    await cloudfrontInvalidation.invalidateCustomDomain({
      domainPath: businessId,
      distributionId: business.customDomain.cloudFrontDistributionId
    });
  }
  
  console.log(`✅ [DEPLOY] Deployment complete for ${businessId}`);
}

Deployment flow:

1. Generate website → /generated/business/v-uuid/
2. Copy to /generated/business/latest/
3. Invalidate CloudFront cache
4. New version live in <30 seconds

5. Intelligent Cache Invalidation

CloudFront caches aggressively (24 hours). When we deploy, we need to invalidate:

export class CloudFrontInvalidationService {
  /**
   * Invalidate CloudFront cache for a specific subdomain
   */
  public async invalidateSubdomain(options: {
    domainPath: string;
    progressTracker?: ProgressTracker;
  }): Promise<void> {
    const { domainPath } = options;
    
    // Skip in development
    if (!this.isProduction()) {
      console.log('🔄 [INVALIDATION] Skipping - not in production');
      return;
    }
    
    const client = new CloudFrontClient({ region: 'us-east-1' });
    
    // Generate targeted invalidation paths
    const invalidationPaths = [
      `/generated/${domainPath}/latest/*`,           // All files
      `/generated/${domainPath}/latest/index.html`,  // Specific index
      `/generated/${domainPath}/latest/`,            // Directory
    ];
    
    console.log(`🎯 [INVALIDATION] Invalidating paths:`, invalidationPaths);
    
    const command = new CreateInvalidationCommand({
      DistributionId: process.env.CLOUDFRONT_DISTRIBUTION_ID,
      InvalidationBatch: {
        CallerReference: `webzum-${domainPath}-${Date.now()}`,
        Paths: {
          Quantity: invalidationPaths.length,
          Items: invalidationPaths
        }
      }
    });
    
    const result = await client.send(command);
    
    console.log(`✅ [INVALIDATION] Invalidation created:`, {
      invalidationId: result.Invalidation?.Id,
      status: result.Invalidation?.Status,
    });
  }
}

Key insights:

• Targeted invalidation: Only invalidate changed paths (not /*)
• Non-blocking: Invalidation happens asynchronously
• Graceful degradation: If invalidation fails, deployment still succeeds
• Cost optimization: Targeted paths = fewer invalidation units

Cost: First 1,000 invalidation paths/month are free, then $0.005 per path

6. Custom Domain Support

Custom domains get their own CloudFront distribution:

async createCloudFrontDistribution(params: {
  domain: string;
  certificateArn: string;
  businessId: string;
}): Promise<Distribution> {
  const cloudfront = new CloudFrontClient({ region: 'us-east-1' });
  
  const response = await cloudfront.send(new CreateDistributionCommand({
    DistributionConfig: {
      // Origin: S3 bucket with business-specific path
      Origins: {
        Quantity: 1,
        Items: [{
          Id: 'S3-webzum-generated-sites',
          DomainName: 'webzum-generated-sites.s3.amazonaws.com',
          S3OriginConfig: {
            OriginAccessIdentity: `origin-access-identity/cloudfront/${OAI_ID}`
          },
          OriginPath: `/${params.businessId}/latest` // Business-specific path
        }]
      },
      
      // Custom domain
      Aliases: {
        Quantity: 2,
        Items: [params.domain, `www.${params.domain}`]
      },
      
      // SSL certificate
      ViewerCertificate: {
        ACMCertificateArn: params.certificateArn,
        SSLSupportMethod: 'sni-only',
        MinimumProtocolVersion: 'TLSv1.2_2021'
      },
      
      // Aggressive caching
      DefaultCacheBehavior: {
        TargetOriginId: 'S3-webzum-generated-sites',
        ViewerProtocolPolicy: 'redirect-to-https',
        CachedMethods: {
          Quantity: 2,
          Items: ['GET', 'HEAD']
        },
        MinTTL: 0,
        DefaultTTL: 86400, // 1 day
        MaxTTL: 31536000 // 1 year
      }
    }
  }));
  
  return {
    Id: response.Distribution.Id,
    DomainName: response.Distribution.DomainName
  };
}

Result: Custom domains get the same CDN performance as subdomains.

The Challenges We Solved

Challenge 1: Cache Invalidation Timing

Problem: Deployments take 30 seconds, but cache invalidation takes 5-10 minutes

Solution: Optimistic updates + background invalidation

// Deploy immediately
await deployToLatest(outputDir);

// Invalidate in background (non-blocking)
cloudfrontInvalidation.invalidateSubdomain({ domainPath }).catch(error => {
  console.warn('Invalidation failed (non-critical):', error);
});

// Return success immediately
return { success: true, url: `https://${businessId}.webzum.com` };

Result: Users see “Deployed!” immediately, cache clears in background

Challenge 2: Editor Code in Production

Problem: Generated websites include editor toolbar code (for live editing)

Solution: Strip editor code during deployment

async function copyAndCleanFiles(sourceDir: string, destDir: string) {
  const files = await refreshStorage.listFiles(sourceDir, { recursive: true });
  
  for (const fileName of files) {
    if (fileName.endsWith('.html')) {
      // Read HTML
      const htmlContent = await refreshStorage.readText(sourceFilePath);
      
      // Strip editor code (keep core functionality)
      const cleanedHtml = stripEditorCode(htmlContent);
      
      // Write cleaned HTML to /latest/
      await refreshStorage.writeText(destFilePath, cleanedHtml);
      
      console.log(`✅ Cleaned: ${fileName} (${htmlContent.length} → ${cleanedHtml.length} bytes)`);
    } else {
      // Copy non-HTML files directly
      const fileBuffer = await refreshStorage.readBuffer(sourceFilePath);
      await refreshStorage.writeBuffer(destFilePath, fileBuffer);
    }
  }
}

Result: Production sites are clean, no editor code bloat

Challenge 3: Version History Storage

Problem: Users want to roll back to previous versions

Solution: Keep all versions in S3, only deploy /latest/

generated/business/
├── latest/           # Live version (deployed)
├── v-uuid-1/        # Version 1 (archived)
├── v-uuid-2/        # Version 2 (archived)
└── v-uuid-3/        # Version 3 (archived)

Rollback process:

async function rollbackToVersion(businessId: string, versionId: string) {
  // Copy archived version to /latest/
  await copyDirectory(
    `generated/${businessId}/${versionId}`,
    `generated/${businessId}/latest`
  );
  
  // Invalidate cache
  await cloudfrontInvalidation.invalidateSubdomain({ domainPath: businessId });
  
  // Update database
  await VersionHistory.markVersionAsLive(businessId, versionId);
}

Storage cost: ~$0.023/GB/month (cheap for version history)

Challenge 4: Multi-Region Performance

Problem: S3 is in us-west-2, but users are worldwide

Solution: CloudFront edge locations cache content globally

User in Tokyo → CloudFront Tokyo Edge → (first request) → S3 us-west-2
User in Tokyo → CloudFront Tokyo Edge → (cached) → <50ms response

Result: First request: ~500ms, Subsequent requests: <50ms

The Results: 10x Faster, 5x Cheaper

Before (App Runner serving websites):

• Load time: 2-4 seconds (global average)
• Server costs: $500/month for 100 websites
• Scaling: Manual, painful
• Uptime: 99.5% (server crashes)

After (S3 + CloudFront CDN):

• Load time: <500ms (global average)
• CDN costs: $50/month for 1000+ websites
• Scaling: Automatic, infinite
• Uptime: 99.99% (AWS SLA)

User feedback:

“My website loads instantly now. I thought something was broken.” - Plumber in Texas

“I’m in Australia and it’s faster than my old WordPress site hosted locally.” - Consultant in Sydney

Cost Breakdown: $50/Month for 1000+ Websites

S3 Storage:

• 1000 websites × 5MB average = 5GB
• Cost: 5GB × $0.023/GB = $0.12/month

S3 Requests:

• Deployments: 1000/month × 100 files = 100,000 PUT requests
• Cost: 100,000 × $0.005/1000 = $0.50/month

CloudFront Data Transfer:

• 1000 websites × 1000 visitors/month × 5MB = 5TB
• Cost: 5TB × $0.085/GB = $425/month (first 10TB)
• Actual cost: ~$50/month (most sites <1000 visitors)

CloudFront Requests:

• 1M requests/month × $0.0075/10,000 = $0.75/month

CloudFront Functions:

• 1M invocations × $0.10/1M = $0.10/month

Cache Invalidations:

• 1000 deployments × 3 paths = 3,000 paths
• First 1,000 free, then 2,000 × $0.005 = $10/month

Total: ~$50-60/month for 1000+ websites

Compare to:

• Vercel: $20/month per website = $20,000/month
• Netlify: $19/month per website = $19,000/month
• App Runner: $500/month for 100 websites = $5,000/month for 1000

Savings: 99% cheaper than competitors

Why This Matters for SaaS Products

Most SaaS products serve content from servers. We learned:

Bad: Serve from App Runner → slow, expensive, doesn’t scale Good: Generate static files → serve from CDN → fast, cheap, infinite scale

The startup lesson: If your content is static (or can be pre-generated), use a CDN. Don’t pay for servers.

Key Insights

1. CloudFront Functions > Lambda@Edge: 6x cheaper, faster, simpler
2. Multi-tenant S3 structure: One bucket, subdomain-based routing
3. Aggressive caching: 24-hour TTL, targeted invalidation
4. Optimistic updates: Deploy immediately, invalidate in background
5. Version history: Keep all versions, deploy /latest/

What’s Next

We’re exploring:

• Edge computing: Run custom code at CloudFront edge (CloudFront Functions)
• Image optimization: Automatic WebP conversion at edge
• Prerendering: Generate pages on-demand, cache forever
• Multi-region S3: Replicate to multiple regions for <100ms global load times

But the core insight remains: CDN delivery is 10x faster and 10x cheaper than servers.

---

Try it yourself: Generate a website with WebZum. It’s served from CloudFront edge locations worldwide. Check the response headers—you’ll see x-cache: Hit from cloudfront.

Building a multi-tenant SaaS? Key takeaway: Use S3 + CloudFront + Functions for subdomain routing. One bucket, infinite websites, blazing fast.

The future of web hosting isn’t servers—it’s edge delivery.