Why websites should worry about path scanning: frequent 404s and guessed APIs are not harmless noise

Open your access logs and the most conspicuous traffic is often not real users, but a long string of addresses that do not exist. They arrive in a short window, each with a different path, and most of them end in a 404. The first reaction is usually: if the page does not open, nothing happened. For a public site, the scan itself is a risk, and a flood of 404s is not background noise you can ignore.

What the scan is looking for

Path scanning is when an automated program requests common directories, file names, and API names one after another, to see whether an address exists, which software handles it, and what it returns. It does not follow your public navigation. It guesses: admin entries, leftover installers, backup files, debug pages, old directories, and APIs named like “resource plus action.”

Guessing paths and guessing APIs are two sides of the same activity. Page scans look for entrances a person could open. API scans look for entrances programs call. The latter often have no menu and no captcha, and their access control is easier to leave as “anyone who knows the address can use it.”

A lot of 404s still leak information

A single 404 only means that address has no content right now. Thousands of patterned 404s mean someone is mapping the site and already sorting “what looks real” from “what was only a guess.”

The responses themselves reveal differences. One address returns a uniform “not found.” Another returns the framework’s default error page. Another returns “not logged in” or “permission denied.” Another returns a parameter error. Those differences tell the other side: there is a real function behind this path; it is just blocked for now, or the parameters are wrong. Scanning software separates “does not exist” from “exists but was refused,” and the next round concentrates on the refused addresses.

If an error page includes a framework version, file path, stack trace, or database error, the technology stack and directory layout are handed over as well. The other side does not need to break anything. Your own error messages narrow the search.

One correct guess lasts a long time

After a site goes live, it keeps many entrances that people never use but the software still serves: campaign pages that were taken down, test directories copied into production, install scripts that were never removed, downloadable backups and exports, internal APIs left without authentication, and old API versions still mounted on their original paths. Normal users never walk there. Scans look for them on purpose.

A guessed admin entrance leads to credential stuffing and unauthorized actions. A guessed backup or config file leads to leaked accounts, keys, and database connection details. A guessed debug or status endpoint leads to internal structure, user data, and runtime state being read directly. APIs are especially easy to underestimate: the page requires login, but the API only checks that “the request looks like a legitimate call.” Once the address is known, it can bypass the restriction you added on the website and read or change data in bulk.

These incidents rarely become news on the first scan. The usual sequence is: use 404s to filter out fake addresses, then confirm the ones that return 401, 403, 405, or a parameter error, and finally hit only the few paths that really exist. Early logs are almost all failures. The actual loss is concentrated in a handful of times the guess was right.

Frequent 404s hurt the business on their own

Even if nothing is guessed correctly, a high rate of useless requests has a cost.

Servers, bandwidth, the CDN, and the firewall are all working on these requests. Scans often come from many addresses and carry different headers. Loose defenses let normal traffic and junk traffic through together. The origin has to route every 404, run the application, and write a log. Once the volume rises, CPU, connections, and disk are occupied, and real users open pages more slowly.

Logs get buried. If security tools alert on every 404, the people on duty soon treat alerts as normal. If 404s are ignored to save effort, real credential stuffing, privilege abuse, and data pulls disappear into the same noise. An attacker can even fill the alerts with path scanning first, then do something else at the same time.

The bill goes up too. CDNs, WAFs, and log services that charge per request count invalid scans as usage. A few thousand extra lines a day may not matter to a small site. For a public business API hit for several days in a row, storage and cleanup costs become concrete.

Search engines and monitoring are affected as well. A mass of abnormal status codes raises the error rate and makes availability charts look as if the site is constantly unhealthy. If error pages are cached by mistake, legitimate addresses can also be treated as missing for a while.

What happens when an API is guessed correctly

Once an API is confirmed to exist, the next step is no longer “see whether this address is there.” An unauthenticated read API exposes user profiles, orders, inventory, or internal configuration. An unauthenticated write API lets data be changed, deleted, or filled with junk. Old API versions often keep old validation, so a bug fixed on the new version still works on the old path.

For teams that must meet security baselines, privacy rules, or public availability commitments, an incident like this raises three questions at once: whether data has already left the system, whether the business was tampered with, and whether logs can prove the scope. If the scanning phase left no searchable record, the aftermath can only be estimated, not bounded.

What the site should close off first

The point is not to outguess the scanner. It is to make guessing unrewarding, and to keep abnormal traffic from covering real problems.

Expose only pages and APIs that are still in use. When a feature is retired, remove the route as well; do not merely hide it from the menu. Keep test directories, leftover installers, backups, and export files off any path the website can serve. Every API should require identity and permission by default. “The address was never published” is not protection. Admin entrances, debug endpoints, and status pages should not sit on the public internet. If one must remain, restrict its source and audit it separately.

Keep error responses consistent. Missing, logged-out, and forbidden can return different business statuses, but the page or JSON should not include physical paths, version numbers, stack traces, or SQL. A scanner should not be able to tell from the error that “this API is real.”

Rate-limit and temporarily block a single source that requests many unknown paths, and stop obvious scans before they reach the application. Aggregate logs by path. Watch for bursts of many different paths from one source, and for spikes in 401, 403, and 500, rather than sending one alert per 404. Keep an inventory of real APIs. A path in the access log that is not on that list is an anomaly, not a user who took a wrong turn.

A CDN and a firewall are only suited to obvious junk traffic. They do not replace authorization. A guessed API with no authentication will be allowed through, because it looks like an ordinary API call.

How to tell that you are being scanned

Look for a few patterns: many requests from a small set of address ranges, with paths that do not resemble each other, arriving evenly over a short time; empty or very old user agents, or one source using a large number of different agents; a few 401s, 403s, 405s, and 500s mixed in with the 404s, on addresses that never appear in your pages or documentation; one nonexistent directory requested over and over, with only the file name changing.

These signs do not mean the site has already been taken over. They mean someone is measuring it. Check whether response differences include technical details, whether the non-404 addresses are real APIs, and whether those APIs return data when no one is logged in.

Closing

The harm of path scanning is not the 404 status code. It is that scanning probes the whole site at very low cost: which addresses are alive, which errors talk too much, and which APIs were left unlocked. Frequent 404s raise cost and bury alerts. The one correct guess can land directly on a backup, an admin entrance, or an unauthenticated API.

Remove unused paths, strip detail out of errors, and treat every API as a public entrance that needs permission. Then a scan stays a long list of failures, instead of becoming a data or access incident.