The llms-text API lets you generate, store, and retrieve llms.txt files for any public website. All endpoints return JSON (or plain text with ?format=text) and include full CORS headers for browser use.
Base URL
https://www.llms-text.com/api/v1
Format
All requests and responses are application/json unless ?format=text is set.
The API is free to use without a key, subject to rate limits. Pass an API key to bypass limits.
X-API-Key: your-api-key
Getting a key
API keys are issued manually. Contact us to request access.
Limits are per IP per 10-minute window. Authenticated requests bypass all limits.
| Endpoint | Limit (unauthenticated) | Authenticated |
|---|---|---|
POST /generate (standard) | 5 / 10 min | Unlimited |
POST /generate (fullVersion) | 2 / 10 min | Unlimited |
GET /generate/:domain | Unlimited | Unlimited |
Rate-limited responses return 429 with a Retry-After header.
All errors use standard JSON error responses.
| HTTP | code | When |
|---|---|---|
| 400 | INVALID_JSON | Body is not valid JSON |
| 400 | MISSING_URL | url field absent from body |
| 403 | SSRF_BLOCKED | URL resolves to a private address |
| 404 | NOT_FOUND | No stored generation for this domain |
| 422 | INVALID_URL | URL is not a valid http/https address |
| 429 | RATE_LIMITED | Too many requests in window |
| 502 | CRAWL_FAILED | Target site unreachable |
{ "status": "ok", "api": "llms-text", "version": "v1" } Crawls a URL, generates a structured llms.txt, and stores it in the database.
| Field | Type | Description |
|---|---|---|
| urlrequired | string | Full http/https URL of the website to generate for |
| fullVersion | boolean | Crawl up to 50 pages (vs 15 standard). Default: false |
| sitemapUrl | string | Explicit sitemap URL. Skips autodiscovery if provided |
| excludePaths | string | Comma-separated path fragments to exclude |
curl -X POST https://www.llms-text.com/api/v1/generate -H "Content-Type: application/json" -d '{"url":"https://www.example.com"}'
{ "llmsTxt": "# Example...", "generationId": 42, "domain": "example.com", "crawledPages": 15 } // Response will appear here...
Fetches the most recent stored generation for a domain.
| Param | Type | Description |
|---|---|---|
| domainrequired | string | Domain name (e.g. michaelvereb.com) |
curl https://www.llms-text.com/api/v1/generate/michaelvereb.com
{ "llmsTxt": "# Michael Vereb...", "domain": "michaelvereb.com", "generationId": 30 } Machine-readable OpenAPI 3.1 spec. Import into Postman or OpenAPI tooling.
Download openapi.json