How an application is protected from automated scans

An automated scan is a program that visits addresses in bulk and asks the same question of all of them: is there a forgotten file here, an old admin page, a known weakness? It does not know who you are. It only knows that the address exists. The test costs the sender almost nothing, which is why every public site receives it.

OWASP, the open security community, describes it as unwanted automated use that goes on "day after day". It also notes that this often abuses normal functions of an application, not some broken part. That matches what the logs of a server show. A new application can receive tests shortly after it opens, long before its audience knows it.

So the useful question is "what happens when we get scanned?". For a simple business site the answer is often "nothing important". For a platform that holds customer data, protection designed from the start is worth it.

Scanners look for mistakes common enough to make the test worthwhile. Their requests are simple and repeated, so they are easy to recognise. The table shows typical targets, with what the scanner hopes to find in each. It is a list that every scanner runs, whatever the site is about.

In a properly built application these files do not exist, so no such request succeeds. The problem is volume. Thousands of these requests fill the logs, use up resources and hide the one request that matters. When they are stopped early, the picture stays clean.

A filter in front of the application examines every request and stops the harmful ones before they reach it. On AWS this is AWS WAF, a web application firewall. In simple terms, it is a filter that stops harmful traffic based on rules that you define.

AWS describes it as a service that watches the HTTP and HTTPS requests sent to the protected resources. It controls access by criteria such as the IP address a request comes from or the values in the URL. When a request matches a blocking rule, it gets an HTTP 403 response, which means "forbidden", or a custom response.

We can write the rules ourselves. There are also ready-made rule groups maintained by AWS. In the infrastructure we describe in the case study for a healthcare platform in the US, the filter had 23 rules: limits on requests per visitor, geographic restrictions, ready-made groups for common bad inputs, and explicit blocks for requests that ask for `.env`, `.git`, phpMyAdmin or PHP files.

Persistent scanners are slowed down by two mechanisms that work together. A rate-based rule counts requests and limits those that arrive too fast. A small automated script then reads the logs, spots those who keep scanning, and blocks them for longer each time they return. The first covers bursts, the second covers persistence.

AWS describes the first as a rule that counts incoming requests, groups them by criteria of your choice, and applies an action when a visitor goes over the limit within the time window. A person browsing never gets near such a limit. A scanner sending hundreds of requests a minute hits it at once.

For the second mechanism we use AWS Lambda. It runs code in response to events or on a schedule, without us managing servers. In the case study we mentioned, a Python script ran every three minutes, found scanners in the logs and blocked them for a growing length of time. The escalation matters: the first violation brings a short block, persistence brings a longer one. A block applied by mistake soon expires on its own.

The order of work for such a system is short:

1. We define what is legitimate. We write down the real paths, the methods, and the countries your users come from.

2. We add the ready-made rule groups. They cover widely known bad inputs without us writing every pattern.

3. We add explicit blocks for the files and pages that your application never serves.

4. We add a rate limit per visitor, with exceptions only for the few places that rightly receive many requests, such as sign-up.

5. We run the log check on a schedule, so repeat visitors get longer blocks.

6. We review the logs every week at the start, to find legitimate users who were blocked by mistake.

You see it by reading the logs, which are organised in advance so they are easy to read. Amazon CloudWatch Logs Insights lets you search and analyse logs with queries. We prepare saved queries for the usual questions: which requests returned 4xx or 5xx errors, which visitors were redirected, which addresses look suspicious, what the filter has blocked.

AWS documentation notes that queries can be saved, run again and placed on dashboards. This turns a one-off investigation into a routine. A filter needs a regular look to stay current. A saved query for "blocked requests this week" takes a minute to read. It shows whether the rules still match the real traffic.

Logs also answer questions after the fact. If a customer says "I could not sign in yesterday", you can see whether the filter blocked them and why. Then you fix the rule based on evidence.

The application must check on the server everything it receives, whatever the filter let through. The filter does not know the rules of your business. The OWASP Input Validation Cheat Sheet recommends accepting only what is meant for each field, instead of listing every harmful string.

The same guide explains that checks in the browser are easy to bypass. They are useful for instant feedback to the user, but the server must repeat them. That is why the filter is only one layer of protection. The OWASP Top 10, the best-known list of the most critical risks for web applications, describes risks in the way applications are built. Good design of the application solves these, beyond any firewall.

Other good practices belong here too: up-to-date software, only the access each user needs, secrets stored outside the code. They are part of how we build. The infrastructure side, meaning the filter, the logs and the access settings, is covered by the AWS cloud setup service.

Other layers are needed for what escapes mass scans. The filter does not fix errors in the code, and it does not recognise a stolen password. It also cannot judge a request that looks perfectly normal but is made with bad intent. Targeted attacks need their own protection.

Each layer has its own job. AWS GuardDuty, for example, is a threat detection service. It watches activity and logs inside your AWS account for signs of unusual behaviour. Backups, access control, and a plan for who acts when an alert arrives matter as much as any rule.

What you gain from this setup is concrete: most of the automated traffic is stopped at low cost, the logs stay readable, and the rarer serious attempts are easier to see. If you are planning a platform and want its protection from the start, describe it in a project request.