INDEPENDENT SYSTEMS JOURNALEDITION 01 / 2026
OPEN SYSTEMS
FIELDBOOK RSS
WEB OPERATIONS
WEB OPERATIONS • RESEARCHED 2026-09-24

Bot traffic on a small site: measure before you block

A practical way to separate crawlers, broken clients and abuse using request logs, rate patterns and a clear response policy.

≈ 5 MIN READUPDATED 2026-10-16

A busy access log does not tell you who is reading. One crawler can generate more requests than a thousand human visitors, while a single address can represent many people behind a shared network. Begin with a measurement window and a question: what resource is actually under pressure?

Bot-traffic triage showing discovery, looping requests, and load pressure as distinct patterns.
Three log patterns call for different responses; a bot label alone is not enough.

1. Establish a baseline

Measure requests per minute, distinct paths, error rate, response size and bandwidth over a representative week. Segment by status code and user agent, but do not treat the user-agent string as proof of identity. Compare a normal period with the suspected spike. If the site stays responsive, the cost may be more important than the request count.

2. Classify behavior, not labels

A bot may fetch robots.txt and a few changed pages; a scraper may sweep the entire archive repeatedly; a broken client may loop on one missing URL. Look for path sequences, intervals and concentration. A repeated 404 on an old page suggests a link or redirect problem, not automatically a hostile actor.

A SIMPLE TRIAGE
Discovery

Many distinct paths, slow pace → check crawl policy and cache.

Loop

One path, high frequency → investigate client or redirect.

Pressure

High rate plus errors or load → rate limit or challenge carefully.

3. Choose the lightest intervention

Cache static assets, fix accidental infinite redirects and return an honest 404 for missing pages. Use robots.txt to state a crawl preference. RFC 9309 makes clear that robots rules are requests to crawlers, not access control. When necessary, apply rate limits or challenges to the observed pattern, then check that legitimate readers and search crawlers still reach the site.

4. Measure the result

Record the rule, start time, target path and expected change. Compare origin load, error rate and real page delivery afterward. Revert a rule that reduces legitimate access without meaningfully reducing cost. Keep logs only as long as operationally necessary and avoid publishing raw IP addresses.

For Caddy operators

Caddy's log directive enables HTTP access logging and supports JSON output. Verify what your current configuration records before making claims about bot share or bandwidth.

Count a synthetic Nginx combined sample

The article above discusses Caddy. This separate exercise uses Nginx’s predefined combined layout; Caddy JSON and custom log formats are different inputs. It counts status classes and known response-body bytes, not crawler identity or server cost.

The default four rows are invented, using documentation-only addresses. Use only synthetic or anonymised records. The parser does not redact what you paste. It keeps input in this page, sends and saves none of it, and excludes addresses, paths, referrers and agents from the summary.

This parser needs JavaScript. Its controls stay unavailable until it loads. The synthetic example and worked totals remain readable.

Parse the sample to inspect its totals.

The strict supported layout is: address, dash, one username token, bracketed common-log timestamp, quoted request, status 100–599, body-byte count, quoted referrer and quoted agent. Default hexadecimal escapes such as \x22 are accepted inside quoted fields. JSON escaping, extra columns, internal blank rows and usernames with literal spaces are not supported. Unsupported input rejects the whole batch, with row numbers only; no partial total is presented as complete.

A dash byte value is unknown, not zero. The synthetic example has 4 responses, one in each of 2xx–5xx, 1,200 known body bytes and one unknown byte count. Nginx combined uses body_bytes_sent; it does not measure full wire traffic, CPU or request duration. Safe-integer limits are checked. Counting responses does not deduplicate people, authenticate a User-Agent or establish abuse.

Primary reference: Nginx log module: combined layout and escaping. This is original browser code, tested with synthetic fixtures; no production access log or server configuration is read or changed by it.

REFERENCE DESK

Sources & scope

These sources support the technical principles above. Commands and compatibility can change; verify the current version before changing a machine.

  1. RFC 9309: Robots Exclusion Protocol
  2. Caddy documentation: log directive
  3. Caddy documentation: how logging works
A note on this domain

This is this website. Its articles are original work inspired by the domain's technical history; we are not the former author. Meet the author.

Many requests and costly requests are different questions

The combined-log parser above can count responses and known body bytes. It cannot recover processing time that was never logged, identify a person from a User-Agent, or attribute CPU cost to a crawler.

  1. Fix one observation window and the exact log format. Keep only the fields needed for the comparison; do not publish raw visitor identifiers or query strings.
  2. Compare request count, known response bytes and status groups separately. Preserve missing bytes as unknown rather than zero.
  3. If the existing log includes $request_time, compare durations on the same routes and window. Nginx measures elapsed request processing through sending the response; this is not CPU time.
  4. Check cache and upstream evidence, if actually recorded, before attributing a bottleneck. Without it, write an investigation question rather than a blocking rule.

Synthetic comparison: group A has 100 requests, each 0.020 seconds; group B has ten requests, each 2.000 seconds. Their summed durations are 2 and 20 seconds respectively. B has fewer requests yet a larger sum. Concurrent requests overlap: 22 summed seconds is not 22 seconds of wall-clock delay and does not measure processor consumption.

Use a record: window __; grouping rule __; route __; requests __; known/unknown bytes __; timing field present __; duration summary __; unanswered cause __. The existing combined sample lacks timing, so its timing result must remain unavailable. These invented groups are not claims about a named bot or this server.

Many requests and costly requests are different questions
A: 100 × 0.020 = 2 s · B: 10 × 2.000 = 20 s · Elapsed sum ≠ CPU.

Source: nginx.org.

Choose between crawler instructions, access control and rate limits

Start with the outcome you need. “Please do not crawl this path”, “only authenticated users may read it” and “slow down these requests” are three different requirements. A user-agent label does not prove identity. Use the following fictional response card to review a policy before changing a production configuration.

  1. Public material, excluded from cooperative crawling: publish the intended robots rule, then request the path directly in your own test environment. A normal public response can still be 200. If the path contains private information, this design has failed the privacy requirement even if a cooperative crawler obeys it.
  2. Protected material: an unauthenticated test request must not receive its body. A 401 response challenges the client and must include an applicable WWW-Authenticate header. Check the response body as well as the status. A login-looking page that returns confidential content elsewhere has not established protection.
  3. Excessive request rate: record the controlled test boundary, counting window and observed response. A 429 response means too many requests; it may include Retry-After. It does not establish the caller’s identity, and a missing 429 does not prove that a server has no rate control.

Worked policy review: a fictional public guide is excluded in robots.txt but direct GET returns 200 with the guide. That matches a crawling preference, not a confidentiality policy. A separate private draft responds 401 with a challenge and no draft text: this passes only the unauthenticated-denial case, not the complete login lifecycle. A local rate-limit test responds 429 and Retry-After: 60; the client pauses rather than retrying repeatedly. The 60 seconds are an invented example, not a recommended universal threshold.

Record to copy: requirement ___; own test URL ___; test identity or anonymous ___; controlled request count/window ___; status ___; relevant response headers ___; sensitive body absent/present ___; next permitted request time ___; result and remaining tests ___. Use synthetic paths and keep credentials and raw visitor identifiers out of the record.

These are response expectations, not a claim that this website deployed a new authentication or rate-limiting system. Test only systems you control and avoid production load tests without an agreed limit. The standards explain the separation: RFC 9110 §15.5.2 specifies 401, while RFC 6585 §4 defines 429.

A robots rule can coexist with a public 200 response. An unauthenticated protected draft is denied with 401 and a challenge. Too many requests can receive 429 and a retry delay. None of these controls substitutes for the other two.
Choose the response from the requirement, then inspect both headers and body. © The Holy Tachanka · CC BY 4.0.

RFC 9309, section 3: robots rules are not access authorisation

KEEP EXPLORINGBrowse all field notes

Verify a request that says it came from Googlebot

Keep the claimed crawler name and the verification result in separate log columns. Start with the client address recorded by a trusted ingress. If the origin records only the proxy address and you cannot establish a trustworthy client address, leave the request unverified.

  1. Look up the client IP in reverse DNS, for example with host 192.0.2.41. Use the exact hostname from the result and ignore a trailing DNS root dot when comparing its suffix.
  2. For a claimed common Google crawler, compare against the common-crawler hostname patterns in Google’s table. Check the category you actually intend to verify. A lookalike such as crawl-192-0-2-41.googlebot.com.example.net belongs under example.net.
  3. Resolve the accepted hostname forward with host HOSTNAME and confirm the original IP is among the addresses returned. For automated checks, use the published address list for the specific crawler category, and retain the date of the list used.

Worked log entry: the synthetic request claims Googlebot from 192.0.2.41. The teaching reverse result is the lookalike hostname above. The hostname check fails, so the result is “unverified” even if the forward result returns 192.0.2.41. If the lookup times out, record “unverified — DNS timeout” so it is distinguishable from a suffix mismatch.

Log record: timestamp ___ · claimed identity ___ · trusted client IP ___ · reverse hostname ___ · category checked ___ · forward addresses ___ · result and reason ___.

Start with the trustworthy client IP. Reverse lookup gives a hostname. Check the documented hostname category, then resolve it forward and look for the original IP.
A crawler claim needs a matching hostname and a return lookup to the same address.

Google Crawling Infrastructure: verify crawler and fetcher requests

Locate an old response without treating every cache as one box

When a reader sees yesterday’s text, first record which response they received. Browser storage, a shared edge cache and an application’s own stored output are different places to investigate. This exercise interprets invented HTTP responses; it does not change a CDN rule or recommend weakening protection.

Work through one response’s age

Suppose a public teaching response has Cache-Control: public, max-age=60, s-maxage=300 and its calculated current age is ninety seconds. Under these simplified inputs, its browser freshness lifetime is sixty seconds and its shared-cache lifetime is three hundred seconds. The same response is stale for the browser but fresh for a compliant shared cache. Actual current-age calculation also accounts for the response dates, transit and residence times; do not substitute your laptop’s stopwatch for that calculation.

  1. Capture the ordinary public request: retain the URL, time, status, relevant cache headers and a harmless body marker. A cache-specific status header may help describe that provider’s response, but is not universal proof of the entire route.
  2. Inspect the browser route: record whether the browser reused a stored response or made a network request. If it revalidates, the shared cache may still serve its fresh representation. A reload is not evidence that the origin was contacted.
  3. Compare an authorised origin observation: use your existing safe origin access, preserving the host and TLS requirements. If the origin marker is version B while the public response is version A, record that mismatch. Do not publish origin addresses, credentials or visitor identifiers.
  4. Recheck the intended public URL after the scoped fix: retain its body marker and headers. A query-string variant may have a different cache key, so its success alone cannot prove that the URL readers use is current.

Fill the case, including what remains unknown

Synthetic record: origin B; public A; calculated age 90 s; browser lifetime 60 s; shared lifetime 300 s. Conclusion: the age fits shared-cache freshness, but the record has not identified every layer or the cause of the old representation. Next observation: the ordinary URL after successful validation or expiry. A 304 response carries validation information rather than a replacement full body; compare the representation the client ultimately uses.

Copy: exact URL ___; body marker ___; request time ___; Date/Age observations ___; directives ___; browser route ___; shared-cache evidence ___; authorised origin marker ___; remaining unknown ___; same-URL result ___. An application cache needs its own evidence. None of these fields identifies human readers or proves that a named bot is hostile.

At a calculated age of ninety seconds, a response with max-age sixty is stale for a browser, while s-maxage three hundred still permits shared-cache freshness. Origin and public body versions require separate observations.
Simplified header exercise; actual age calculation has additional inputs. © Open Systems Fieldbook · CC BY 4.0.

RFC 9111: shared-cache freshness and s-maxage