Book a crash test

METHODOLOGY · v0.1 · UPDATED 15 NOV 2026

How we crash-test an app

A crash test is only useful if you know what was tested. This page lists every check, the exact rule behind the Safety Rating, and what we never do. If we change any of it, we'll update this page and note the change at the bottom.

The short version

  • We only test apps whose owner has verified them. No exceptions.
  • We test the live app from the outside, the way a user or a stranger reaches it. We don't need your code or your builder login.
  • There are two levels:
    • The Stranger (Dummy Run) has no account and sees what anyone on the internet sees.
    • Two dummies (Full Crash Test) are two test accounts you create. They check whether one user can get at the other's data.
  • Every finding comes with evidence. If we can't show it, it doesn't count against you.
  • The Safety Rating follows a fixed, published rule. It's the same rule for every app.

Step 0: Verification and scope

Before anything runs:

  1. You prove ownership in one of three ways: a <meta name="crashdummy-verify"> tag on your site, a /.well-known/crashdummy.txt file, or a DNS TXT record.
  2. We set the scope. We test your verified address, plus the backend services your own app's code connects to, for example your Supabase project. Requests to anything else are blocked.
  3. Dummy Run: you accept our testing terms when you order. The Stranger never logs in, so the verification tag and that acceptance are enough.
  4. Full Crash Test: you also create two test accounts and sign a one-page testing authorization, because the dummies go inside your app.

What we test

Level 1: The Stranger (Dummy Run, and the start of every Full Crash Test)

No account, no password. The Stranger only sees what any visitor could see.

#CheckWhat we actually doWhat we record
S1Look aroundA real browser opens your app, visits its pages and notes which services it talks to (for example Supabase or Stripe)The services and public keys found. Public keys are expected, so they aren't a finding on their own
S2Secrets in your public codeWe scan the JavaScript your app sends to every visitor for known secret formats: Supabase secret or service_role keys, Stripe live secret keys, AI provider keys and similar. Any token we find is decoded to see what role it carriesKey type plus a masked prefix (sk_live_ab12…), never the full key
S3Tables anyone can readUsing only your app's public key, we ask each table whether it will answer someone who isn't logged inTable name and row count, never row contents
S4Files anyone can reachWe check whether storage buckets are public and whether their contents can be listedBucket name, public or private, a file count
S5Functions anyone can callWe list database functions exposed to logged-out visitorsFunction names
S6Web hygieneWe check security headers and CORS, and whether /.env, /.git or source maps are reachablePresent or absent, plus response codes

Level 2: Two dummies (Full Crash Test only)

Dummy A and Dummy B are two test accounts you create in your own app.

#CheckWhat we actually doWhat we record
D1Dummy A vs Dummy BDummy A uses the app normally while we record its requests. We then replay the requests that refer to A's records using Dummy B's session, and try to read, change and delete themThe request shape, the status code, and whether the response matched A's data. Matching uses hashes, and we never store the content
D2Self-upgradesDummy B tries to edit its own profile fields that control privileges, like is_pro, plan or roleWhether the change was accepted
D3Locked doorsDummy B (a regular user) tries to open admin pages and call admin-only actionsStatus codes and screenshots of what B saw
D4Sign-up and sign-inWe check whether sign-up is open, whether email is confirmed, and what the password rules are. We send a small burst to email-sending features: at most 10 requestsSettings observed, and whether the burst was stopped
D5PaymentsWe try to unlock a paid feature from the browser without paying. Stripe test mode onlyWhether the server refused
D6Critical flowsWe run your sign-up, sign-in and main action end to end, on a mobile and a desktop screen sizePass or fail, plus screenshots
D7Human reviewA human tester reviews every finding before you get it, and checks the repair prompts—

What we don't test (v0.1)

To be clear about the limits: we don't run full penetration tests, fuzzing or injection testing beyond the basics. We don't test native mobile apps, or anything you haven't put in scope. A crash test is focused on the problems AI-built apps hit most often, as listed above. It isn't a complete security audit.

Evidence: "evidence or it didn't happen"

  • Every finding includes evidence you can check: the request we sent, the answer we got, and a timeline of what each dummy did (the Crash Replay).
  • Confirmed vs likely: a finding is confirmed when the evidence directly proves it, for example Dummy B received a row that matched Dummy A's data. Anything weaker is marked likely. Only confirmed findings affect your Safety Rating. Likely findings still appear in the report, so you can judge them yourself.
  • Accuracy target: we test every check against our own deliberately broken demo apps and their fixed versions. Each check must catch the planted problem, and stay quiet on the fixed version. Our target is under 5% false positives on that test suite.

The Safety Rating

Severity levels

LevelMeaningExamples
Totaled (critical)Anyone, or any user, can read or change other users' data, or a privileged secret has leakedA table of user profiles readable without login. Dummy B reads Dummy A's records. A service_role key in public code
Crash (high)Privilege escalation, private files that are public, or payment bypassA user can set is_pro=true. Invoices in a public bucket. A paid feature unlocked from the browser
Dent (medium)Abuse vectors and weak sign-in flowsAn email-sending feature with no limit. No email confirmation where one is needed
Scratch (low)Hygiene issuesMissing security headers. A reachable source map

Severity is scored as impact × exploitability. In other words: how bad would it be, and how easy is it to do?

The rule

Every app starts at 5 stars. Then, in this order:

  1. Any Totaled finding caps the rating at 1 star.
  2. Any Crash finding caps it at 2 stars.
  3. Two or more Dents cap it at 3 stars.
  4. One Dent caps it at 4 stars.
  5. Scratches then lower it in half-star steps: each Scratch removes half a star from the cap, Scratches together remove at most one star, and the rating never drops below 1 star.

Only confirmed findings count. The same findings always produce the same rating, with no judgment calls and no hidden weighting.

Worked examples

Confirmed findingsRatingWhy
None★★★★★ 5Nothing found in the areas we test
1 Crash, 1 Dent★★ 2A Crash caps the rating at 2
2 Dents★★★ 3Two or more Dents cap the rating at 3
1 Dent★★★★ 4One Dent caps the rating at 4
1 Scratch★★★★½ 4.5Each Scratch removes half a star
4 Scratches★★★★ 4Scratches together remove at most one star
1 Dent, 1 Scratch★★★½ 3.5One Dent caps the rating at 4, then the Scratch removes half a star
1 Totaled, 3 Scratches★ 1Never below 1 star

What the rating means, and what it doesn't

  • It describes the app on the test date, across the checks listed above. It isn't a guarantee, a certification or a promise that your app is secure.
  • 5 stars means: in the areas we test, our dummies found nothing confirmed.
  • Any AI edit can change it. That's why every report is dated, and why Re-crash and Airbag exist.

What we never do

We never:

  1. Test an app whose owner hasn't verified it. That includes "just a quick look" at someone else's app, a competitor's, or a well-known app.
  2. Store other people's data. We keep counts, hashes, structure and redacted snippets. We don't keep content.
  3. Keep secrets. Any key we find is shown masked, and is never stored or used.
  4. Try to knock your app over. No load tests, no stress tests, no denial of service. We send at most 5 requests per second, with a cap per test and timeouts on every check.
  5. Change real users' data. Writes happen only in the test accounts you gave us, and we clean them up afterwards.
  6. Touch real money. Payment checks run in Stripe test mode only.
  7. Contact your users or staff, or try phishing or social engineering.
  8. Test the platforms themselves. We don't probe Lovable, Supabase, Vercel or anyone else's infrastructure, or any other customer's project hosted there.
  9. Keep access. We don't keep access after the test, install anything, or move beyond the scope you approved.
  10. Publish your results. Reports are private. You decide whether to share a public summary, and it never includes evidence.

And we always:

  • Keep a log of every request we send: method, address, status and which check sent it.
  • Can stop instantly: any test can be halted at any moment.
  • Delete evidence 30 days after your report, unless you ask in writing for a different period.
  • Tell you right away if we find something critical during a Full Crash Test. We don't wait for the report.

Changelog

VersionDateChange
v0.119 Nov 2026First public version