How MarkdownMe runs 79 AI tools in production

The site is a static Next.js export on S3 and CloudFront. All AI tools share one backend: an API Gateway HTTP API in front of one Python 3.11 Lambda function. Each AI tool is one entry in a Python dictionary with a system prompt, a user template, an input limit, an output limit, and an optional model. A new AI tool is a new entry, not a new service.

The model call times out at 25 seconds, because API Gateway stops waiting at 29 seconds. A slow model returns HTTP 502 to the tool page instead of a hung request.

The site is free and has no accounts, so I cap spend in the backend, not per user.

• Circuit breaker. I cap all AI tools together at 1,000 requests and $5.00 of estimated model cost per UTC day. Before each model call, the Lambda takes a request slot with a conditional DynamoDB update, so parallel calls cannot pass the cap. It also reads the cost counter. When either limit is reached, the API returns HTTP 503 with the time until midnight UTC.
• Per-tool caps. 2 tools have their own daily cap: LinkedIn Post Enhancer at 1,000 calls and voice-profile at 500. They use the same atomic slot, so one busy tool cannot use the whole shared budget.
• Input and output limits. Each tool sets a maximum input between 800 and 20,000 characters. Larger input gets HTTP 413 before any model call. Each tool also sets a maximum output between 200 and 3,000 tokens.
• Cost per call. The Lambda prices every call from its token counts and a per-model price table, then adds the result to the daily counter. The estimate covers model tokens only, not Lambda, DynamoDB, or CloudFront.

Real numbers from my 30-day audit (August 22 to September 20, 2026): 535 AI calls across 40 tools, for $0.19 of estimated model cost. The 535 log rows matched the 535 counter increments exactly. That window is before the move to GPT-6 Luna.

Until September 2026 the tools ran on Amazon Nova and Claude Haiku on Bedrock. On September 23, 2026 I ran two blind evals to decide whether to move them.

• Bake-off. The top 8 tools by 30-day use, 6 test cases each. The cases follow the shape of real inputs, but no user text left the system. Mean score: GPT-6 Luna 7.33, DeepSeek V4 Flash 6.96, the production models 4.38.
• All other tools. The remaining 71 tools, 3 cases each: a typical input, a hard input, and a non-English, vague, or unsafe input. 753 model calls.
• Judge. Claude Sonnet graded each output from 1 to 10 against a rubric for that tool. It saw the candidates in shuffled order with no model names, and it penalized truncated output.
• Rule. I fixed the rule before I saw the scores. Luna by default. Nova Micro if it is within 0.5 of the best mean with no more bad scores than Luna. Any other model only if it beats Luna by 1.5 or more.

DeepSeek V4 Flash won 3 tools on score. I did not ship it: 88 of its 213 calls took over 25 seconds, which is the production timeout. Claude Haiku 4.5 stays on Tweet Hook Generator, where it scored 7.33 against 5.0 for Luna. The Markdown Editor's AI fix-and-format action stays on Nova Micro, which scored 6.2 against 6.8 for Luna in the bake-off and answers faster.

Generation for both evals cost $0.26 on OpenRouter. A crashed first attempt cost about $0.27 more. The judge ran on my Claude subscription, not a metered API. The eval scripts are in the repo and rerun with 3 commands.

The guardrails are my own Python module, not a managed service. They add no model call and no per-request fee. They fail open: a guardrail bug must not take a tool offline.

• Credentials: redacted. The Lambda looks for 8 credential formats (AWS keys, private keys, OpenAI, Anthropic, GitHub, Slack, and Google keys, bearer tokens) and card numbers that pass a Luhn check. It replaces them with placeholders before the model call and before the log write. This is the only check that changes the input.
• Prompt injection: detected and logged, not blocked. 10 phrase patterns (for example "ignore previous instructions" and system prompt probes) and base64 text payloads set an issue flag on the log row. The request still runs.
• Spotlighting: 1 of 79 tools. The AI Prompt Optimizer wraps user text in a random per-request marker and tells the model the block is data. It changes the prompt, so each tool needs its own quality check before I turn it on.
• Canary token. Every system prompt carries a random token. If the output contains it, the system prompt leaked. The Lambda removes that text and flags the row. It also removes credentials that appear in the output.

Pattern matching stops copy-pasted attacks, not a determined attacker who rephrases. The real limit is the blast radius: these tools have no tool access, no memory, no private data, and no side effects. A successful injection returns text to the person who sent it.

• Each AI call writes a log row with the tool, model, token counts, estimated cost, latency, guardrail flags, browser family, country, and a salted hash of the IP address. The row keeps up to 2,000 characters of input and 4,000 of output, and expires after 30 days.
• Each row also names the exact prompt and model. The prompt version is a hash of the system prompt, the user template, the model ID, the output limit, and the temperature. A change to any of them gives a new version. When quality drops, the version tells me if a prompt deploy caused it or if the provider changed the model under the same name.
• Each row records two times: the model call alone, and the full request with caps, guardrails, and counter writes. The daily email and the admin metrics show p50, p95, and p99 for both, per tool. At low volume p99 is the slowest call of the day, and I report it that way.
• A canary runs every Monday. 6 fixed inputs go through the production path, each on the model its tool uses. Code checks the result, not a judge: valid JSON, a kept ticket ID, a correct commit format, and a latency budget. A failure emails me. It catches a provider change that no deploy of mine caused. The canary calls count against the daily cap but not on the public stats.
• A second record keeps the output and the call details with no end date. It keeps the input only when "Improve the tools for everyone" is on. Users turn it off on the privacy page.
• Daily totals per tool (calls, tokens, cost, no text) have no end date.
• EventBridge runs a report at 00:05 UTC every day. It emails me yesterday's requests against the 1,000 cap, token totals, estimated cost against the $5.00 cap, latency percentiles, and calls per tool.
• A public, counts-only endpoint feeds the stats page and the numbers at the top of this page.
• A weekly loop maps search demand to tools, so I build what people search for.

From launch on April 19, 2026 to September 20, 2026, the AI Lambda ran 1,497 times (CloudWatch). Before September 21, 2026 I kept only 30 days of call logs, so I cannot count every successful call since launch.

• Every push runs unit tests plus a browser smoke test over every tool page before deploy.
• Every build first runs audit scripts over the guide pages and the tool registry.
• I cut the catalog from 214 tools to 96 with Search Console data. A tool is hidden when it is not a core Markdown tool, had fewer than 10 search clicks in 90 days, and had fewer than 10 AI calls. Hidden pages still load but carry noindex, and the build audits fail if a hidden tool leaks into the sitemap, llms.txt, or the tool index.
• The backend tests cover the guardrails, model routing, the usage archive, prompt versions, latency percentiles, and the daily cap. The cap test runs 50 parallel callers against a cap of 10 and expects exactly 10 to pass. The guardrail tests also check that normal developer input (JWTs, hashes, "decode this base64") does not get flagged. A backend deploy runs these tests first and stops if one fails.
• A push to main deploys through GitHub Actions with OIDC, so no long-lived AWS keys exist. Frontend changes go to S3 and CloudFront. Backend changes, including prompt changes, deploy the Lambda stack.

• Block, not only log. Injection detection flags rows but blocks nothing. I want a few weeks of flag rates and false positives first, then a block for high-confidence patterns.
• Spotlighting on more tools. It runs on 1 of 79 tools. Each new tool needs an eval run to prove the output quality holds.
• Check the judge. One LLM judge scored the evals, and no human checked it. I am grading a blind sample of 30 outputs by hand, pass or fail. The script reports how often the judge agrees with me (Cohen's kappa). If the agreement is low, the model choice above needs a second look.
• Real inputs as regression cases. The evals use synthetic cases, and I run them by hand. A script already lists which tools' prompts changed since the last deploy. Next: a regression set per tool, built from real inputs where users left "Improve the tools" on, that runs on every changed prompt.