LLM Generator
Generate llms.txt files for websites to improve their visibility to large language models
Project Overview
LLMGenerator is a service that generates llms.txt files for websites to improve their visibility to large language models. The service uses Firecrawl's map + scrape approach with OpenAI enhancement to create high-quality, LLM-optimized content.
What is llms.txt?
llms.txt is a standardized format for making website content more accessible to Large Language Models (LLMs):
- llms.txt: A concise index of all pages with AI-enhanced titles and descriptions
- llms-full.txt: Complete content of all pages for comprehensive access
Key Features
Fast Generation
Direct map + scrape with background processing
Reliable Jobs
Trigger.dev integration with automatic retries
AI-Enhanced
OpenAI-powered titles and descriptions
Cost-Optimized
95% cheaper than competitors
Analytics
Track generation stats and usage
Real-time Updates
Live progress tracking
API Docs
OpenAPI/Swagger 3.0 interactive docs
Flexible Pricing
Credits (never expire) + subscriptions
Generation Methods
Simple Generation (1x Credits)
- Fast generation (15-30 seconds)
- Uses Firecrawl's /llmstxt endpoint
- Cost-effective for free tier users
- Locked for free plans, optional for paid
Enhanced Generation (2x Credits)
- AI-enhanced titles and descriptions
- Uses map + scrape + OpenAI integration
- Premium quality for paid plans
- Default for paid users
Tech Stack
Frontend
- React 19 with TypeScript
- Vite for fast builds
- Tanstack Router & Query
- Tailwind CSS & shadcn/ui
- React Hook Form with Zod
Backend
- Hono on Cloudflare Workers
- Cloudflare D1 (SQLite) with Drizzle ORM
- Better Auth for authentication
- Trigger.dev for background jobs
- tRPC for type-safe APIs
Integrations
- Firecrawl API for web scraping
- OpenAI API for content enhancement
- Stripe for payments
- Cloudflare KV for session caching
- Sentry for error tracking
Development
- Biome for linting/formatting
- Bun for package management
- Wrangler for deployments
- Vitest with 135 tests
- OpenAPI/Swagger documentation