We Built an AI That Googles Your Competitors (So You Don’t Have To)

TL;DR: We integrated Google Search API with Claude to automatically research your competitors, analyze their websites, and use that intelligence to generate a better website for you. The system searches Google, scrapes competitor sites, extracts insights, and feeds them to our AI—all in under 60 seconds.

The Problem: Website Builders Operate in a Vacuum

We generate websites with AI. But there was a glaring problem:

We had no idea what your competitors were doing.

What we were missing:

• What do competitor websites look like?
• What services do they highlight?
• What’s their pricing strategy?
• How do they position themselves?
• What’s missing that you could capitalize on?

Traditional approaches:

• Manual research: User researches competitors, tells us what to include (slow, incomplete)
• Generic templates: One-size-fits-all websites that ignore market context
• No competitive intelligence: Hope your website is good enough

The insight: If we can Google your business, we can Google your competitors. And if we can scrape their websites, we can learn from them.

The Breakthrough: AI + Google Search = Competitive Intelligence

The breakthrough came when we realized: Claude can understand search results and competitor websites.

Bad: Generate website without context → hope it’s competitive Good: Research competitors → analyze their strengths/weaknesses → generate better website

The difference? Your website isn’t just good—it’s strategically positioned against your competition.

How It Works: The Technical Architecture

1. Google Search Integration

When you enter a business name, we search Google for competitors:

import { google } from 'googleapis';

const customsearch = google.customsearch('v1');

async function searchCompetitors(businessInfo: {
  name: string;
  type: string;
  location: string;
}): Promise<SearchResult[]> {
  // Build search query
  const query = `${businessInfo.type} ${businessInfo.location} -${businessInfo.name}`;
  
  const response = await customsearch.cse.list({
    auth: process.env.GOOGLE_API_KEY,
    cx: process.env.GOOGLE_SEARCH_ENGINE_ID,
    q: query,
    num: 10, // Top 10 results
    fields: 'items(title,link,snippet)'
  });
  
  return response.data.items?.map(item => ({
    title: item.title,
    url: item.link,
    snippet: item.snippet
  })) || [];
}

Example queries:

• “pizza restaurant Brooklyn -Joe’s Pizza”
• “plumber San Diego -Best Plumbing”
• “yoga studio Austin -Zen Yoga”

Why the minus sign? Exclude the user’s own business from results.

2. Competitor Website Scraping

For each competitor, we scrape their website:

async function scrapeCompetitorWebsite(url: string): Promise<CompetitorData> {
  try {
    // Fetch HTML
    const response = await fetch(url, {
      headers: {
        'User-Agent': 'Mozilla/5.0 (compatible; WebZumBot/1.0)'
      },
      timeout: 10000 // 10 second timeout
    });
    
    const html = await response.text();
    
    // Extract key information
    const $ = cheerio.load(html);
    
    return {
      url,
      title: $('title').text(),
      description: $('meta[name="description"]').attr('content') || '',
      headings: $('h1, h2, h3').map((_, el) => $(el).text()).get(),
      services: extractServices($),
      pricing: extractPricing($),
      testimonials: extractTestimonials($),
      ctaButtons: $('a.btn, button').map((_, el) => $(el).text()).get()
    };
  } catch (error) {
    console.error(`Failed to scrape ${url}:`, error);
    return {
      url,
      error: error.message
    };
  }
}

function extractServices($: CheerioAPI): string[] {
  // Look for common service patterns
  const services = new Set<string>();
  
  // Check navigation menus
  $('nav a, .menu a').each((_, el) => {
    const text = $(el).text().trim();
    if (text && text.length < 50) {
      services.add(text);
    }
  });
  
  // Check service sections
  $('.service, .services li, [class*="service"]').each((_, el) => {
    const text = $(el).text().trim();
    if (text && text.length < 100) {
      services.add(text);
    }
  });
  
  return Array.from(services);
}

function extractPricing($: CheerioAPI): string[] {
  const pricing = [];
  
  // Look for price patterns
  $('body').text().match(/\$[\d,]+(?:\.\d{2})?/g)?.forEach(price => {
    pricing.push(price);
  });
  
  return pricing;
}

3. AI-Powered Competitive Analysis

We feed all competitor data to Claude for analysis:

async function analyzeCompetitors(
  businessInfo: BusinessInfo,
  competitors: CompetitorData[]
): Promise<CompetitiveAnalysis> {
  const prompt = `
You are a competitive intelligence analyst. Analyze these competitors for ${businessInfo.name}, a ${businessInfo.type} in ${businessInfo.location}.

COMPETITORS:
${competitors.map((c, i) => `
${i + 1}. ${c.title} (${c.url})
   Description: ${c.description}
   Services: ${c.services.join(', ')}
   Key Messages: ${c.headings.slice(0, 5).join(', ')}
   CTAs: ${c.ctaButtons.join(', ')}
`).join('\n')}

Analyze and return JSON:
{
  "competitiveLandscape": {
    "totalCompetitors": number,
    "marketSaturation": "low" | "medium" | "high",
    "averagePriceRange": string,
    "commonServices": string[]
  },
  "competitorStrengths": [
    {
      "competitor": string,
      "strengths": string[]
    }
  ],
  "competitorWeaknesses": [
    {
      "competitor": string,
      "weaknesses": string[]
    }
  ],
  "marketGaps": [
    {
      "gap": string,
      "opportunity": string,
      "howToCapitalize": string
    }
  ],
  "differentiationStrategy": {
    "positioningStatement": string,
    "uniqueValueProps": string[],
    "targetAudience": string,
    "messagingAngle": string
  },
  "tacticalRecommendations": [
    {
      "category": "services" | "pricing" | "messaging" | "design",
      "recommendation": string,
      "rationale": string
    }
  ]
}
`;

  const response = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 4000,
    messages: [{
      role: 'user',
      content: prompt
    }]
  });
  
  return JSON.parse(response.content[0].text);
}

4. Using Competitive Intelligence in Website Generation

The competitive analysis influences every step of website generation:

Strategy Step:

async function generateStrategy(
  businessInfo: BusinessInfo,
  competitiveAnalysis: CompetitiveAnalysis
): Promise<Strategy> {
  const prompt = `
Generate a website strategy for ${businessInfo.name}.

COMPETITIVE INTELLIGENCE:
- Market saturation: ${competitiveAnalysis.competitiveLandscape.marketSaturation}
- Common services: ${competitiveAnalysis.competitiveLandscape.commonServices.join(', ')}
- Market gaps: ${competitiveAnalysis.marketGaps.map(g => g.gap).join(', ')}
- Differentiation: ${competitiveAnalysis.differentiationStrategy.positioningStatement}

Create a strategy that:
1. Highlights services competitors are missing
2. Positions against competitor weaknesses
3. Emphasizes unique value props
4. Targets underserved audience segments
`;

  // ... generate strategy with competitive context
}

Content Generation:

async function generateSectionContent(
  section: Section,
  competitiveAnalysis: CompetitiveAnalysis
): Promise<string> {
  // Use competitive intelligence to inform content
  const recommendations = competitiveAnalysis.tacticalRecommendations
    .filter(r => r.category === section.type);
  
  const prompt = `
Generate content for ${section.name}.

COMPETITIVE CONTEXT:
${recommendations.map(r => `- ${r.recommendation}: ${r.rationale}`).join('\n')}

Make sure to:
- Differentiate from competitors
- Highlight unique advantages
- Address market gaps
`;

  // ... generate content
}

The Challenges We Solved

Challenge 1: Google Search API Costs

Problem: Google Search API costs $5 per 1,000 queries. At scale, this adds up.

Solution: Aggressive caching + smart query optimization

// Cache search results for 7 days
const cacheKey = `search:${businessInfo.type}:${businessInfo.location}`;
const cached = await redis.get(cacheKey);

if (cached) {
  return JSON.parse(cached);
}

const results = await searchCompetitors(businessInfo);

await redis.setex(cacheKey, 7 * 24 * 60 * 60, JSON.stringify(results));

return results;

Result: 90% cache hit rate, $0.50/month in search costs

Challenge 2: Scraping Failures

Problem: Some competitor websites block scrapers, have JavaScript-heavy content, or timeout

Solution: Graceful degradation + fallback to search snippets

async function getCompetitorData(url: string, snippet: string): Promise<CompetitorData> {
  try {
    // Try full scrape first
    return await scrapeCompetitorWebsite(url);
  } catch (error) {
    console.log(`Scrape failed for ${url}, using search snippet`);
    
    // Fall back to search snippet
    return {
      url,
      title: extractTitleFromUrl(url),
      description: snippet,
      services: extractServicesFromSnippet(snippet),
      scraped: false
    };
  }
}

Challenge 3: AI Hallucinations

Problem: Claude sometimes invents competitors or makes up insights

Solution: Validation + confidence scoring

function validateCompetitiveAnalysis(
  analysis: CompetitiveAnalysis,
  competitors: CompetitorData[]
): ValidationResult {
  const errors = [];
  
  // Verify competitor names match actual data
  for (const strength of analysis.competitorStrengths) {
    const exists = competitors.some(c => 
      c.title.includes(strength.competitor) || 
      c.url.includes(strength.competitor.toLowerCase())
    );
    
    if (!exists) {
      errors.push(`Hallucinated competitor: ${strength.competitor}`);
    }
  }
  
  // Verify market gaps are plausible
  for (const gap of analysis.marketGaps) {
    if (gap.gap.length < 10) {
      errors.push(`Vague market gap: ${gap.gap}`);
    }
  }
  
  return {
    valid: errors.length === 0,
    errors,
    confidence: 1 - (errors.length / 10)
  };
}

Challenge 4: Performance

Problem: Searching + scraping + analyzing takes 60+ seconds

Solution: Parallel execution + streaming results

async function performCompetitiveResearch(businessInfo: BusinessInfo) {
  // Search Google (fast)
  const searchResults = await searchCompetitors(businessInfo);
  
  // Scrape competitors in parallel (slow)
  const competitorData = await Promise.all(
    searchResults.slice(0, 5).map(result => 
      getCompetitorData(result.url, result.snippet)
    )
  );
  
  // Stream partial results to user
  await streamProgress({
    step: 'web-search',
    status: 'analyzing',
    data: {
      competitorsFound: competitorData.length,
      competitorNames: competitorData.map(c => c.title)
    }
  });
  
  // Analyze with AI (medium)
  const analysis = await analyzeCompetitors(businessInfo, competitorData);
  
  return {
    searchResults,
    competitorData,
    analysis
  };
}

Result: 45 seconds average, with progress updates every 10 seconds

The Results: Smarter Websites

Before (no competitive intelligence):

• Generic websites that looked like everyone else’s
• No differentiation or unique positioning
• Users had to manually specify what made them special
• 60% of users said “my website looks like my competitor’s”

After (competitive intelligence):

• Websites strategically positioned against competition
• Automatic differentiation based on market gaps
• AI suggests unique value props
• 85% of users said “my website stands out”

Additional benefits:

• Better SEO: Target keywords competitors are missing
• Higher conversion: Messaging addresses competitor weaknesses
• Faster generation: No need to manually research competitors
• Smarter strategy: AI-driven positioning vs guesswork

Why This Matters for AI Applications

Most AI applications operate in isolation. We learned:

Bad: Generate output without context → hope it’s good Good: Research context → analyze landscape → generate strategically

The startup lesson: AI is most powerful when it has context. Don’t just generate content—research the competitive landscape and use that intelligence to inform your output.

Key Insights

1. Context beats creativity: AI with competitive intelligence > AI with generic prompts
2. Automate research: Users don’t want to research competitors—do it for them
3. Validate AI output: Always verify AI insights against real data
4. Stream progress: Long-running tasks need progress updates

What’s Next

We’re exploring:

• Deeper analysis: Analyze competitor pricing, reviews, social media
• Trend detection: Identify emerging trends in your industry
• Competitor monitoring: Alert users when competitors change their websites
• SWOT analysis: Automated strengths/weaknesses/opportunities/threats

But the core insight remains: AI + competitive intelligence = strategic advantage.

---

Try it yourself: Generate a website with WebZum, watch the “Web Search” step. We’re Googling your competitors and using that intelligence to make your website better.

Building an AI product? Key takeaway: Don’t generate in a vacuum. Use APIs (Google Search, web scraping) to gather context, then feed that to your AI for smarter output.

The future of AI isn’t just generation—it’s intelligent, context-aware generation.