Log File Analysis Without Enterprise Tools

TL;DR: Server logs are the only record of what search engines actually requested from your site, as opposed to what they chose to report. For most local sites this is a once-a-year check rather than an ongoing practice, and you can do it with a spreadsheet. Three patterns are worth looking for. If none are present, you’ve spent an hour and can stop worrying about crawling.

Table of Contents

• What logs tell you that Search Console won’t
• Getting logs off shared and managed hosting
• Parsing them without buying anything
• Three patterns worth acting on
• What not to bother with
• When it isn’t worth your afternoon

What Logs Tell You That Search Console Won’t

Search Console’s Crawl Stats report is aggregated, sampled, and limited to Google. It tells you roughly how much crawling happened and the response distribution. It does not tell you which specific URLs were requested, in what order, how often, or by which bot.

Server access logs record every single request: timestamp, IP, requested URL, response code, user agent, bytes, and often referrer. Four questions only logs answer:

Which URLs is Googlebot actually fetching? Not which you think are important. Which it requests, and how often. The gap between those two lists is frequently the whole finding.

Is Googlebot reaching your new pages? You published eight city pages three weeks ago. Have they been requested? Search Console will tell you if they got indexed. Logs tell you if they were even fetched.

What are the other bots doing? In 2026 this matters more than it used to. AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others — now make up a meaningful share of traffic on many sites. Logs are the only place you can see their behavior, which robots directives they’re honoring, and how much bandwidth they’re consuming.

What’s erroring for bots specifically? A URL that returns 200 to you in a browser and 403 to a bot behind a firewall rule is invisible in Search Console but obvious in logs.

Getting Logs Off Shared and Managed Hosting

This is the step that stops most people. Options by host type:

cPanel shared hosting. Look for “Raw Access Logs” in the Metrics section. Enable archiving so logs aren’t rotated away, then download the gzipped files. Usually you get the current day plus a short archive.

WP Engine. Available in the User Portal under the environment’s utilities, or by support request. They retain a limited window, so grab them promptly.

Kinsta. Available in MyKinsta under Logs, downloadable, with access logs separate from error logs.

Flywheel. Request from support; not exposed in the dashboard by default.

SiteGround. Site Tools → Statistics → Access Logs.

Cloudflare in front of your site. This is the important case. If Cloudflare or another CDN sits in front, your origin logs may be incomplete because cached requests never reach the server. Cloudflare’s own Logpush is a paid feature on higher plans. For a small site, the practical answer is to check origin logs and accept they undercount, or temporarily use Cloudflare analytics for bot data.

VPS or dedicated. /var/log/nginx/access.log or /var/log/apache2/access.log. You have everything, including history if rotation is configured sensibly.

Ask for a week of logs minimum, ideally a month. A single day is too noisy to read.

If your host flatly won’t provide them, that’s a data point about the host. It’s not worth switching hosts over, but note it.

Parsing Them Without Buying Anything

You don’t need a log analyzer. Three approaches by comfort level.

Spreadsheet (works up to a few hundred thousand lines).
1. Open the log in a text editor and confirm the format. Standard combined format is: IP, timestamp, request, status, bytes, referrer, user agent.
2. Import into Excel or Google Sheets as delimited text using space as the delimiter, quote character as ".
3. Add a column flagging Googlebot rows: search the user agent field for “Googlebot”.
4. Pivot: requested URL by count, filtered to Googlebot. That single pivot answers most of your questions.

Command line (works at any size, faster).
Count Googlebot requests per URL:
grep Googlebot access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -50

Count status codes served to Googlebot:
grep Googlebot access.log | awk '{print $9}' | sort | uniq -c | sort -rn

Count requests by bot:
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20

Three commands, and you have the core of the analysis.

Screaming Frog Log File Analyser. Paid but inexpensive, with a free tier limited to 1,000 lines. If you’ll do this more than twice a year it’s worth it. For an annual check, the command line is fine.

One critical verification step: many user agents claim to be Googlebot and aren’t. Before drawing conclusions about Googlebot behavior, verify a sample of the IPs. Google publishes its crawler IP ranges, and a reverse DNS lookup on a genuine Googlebot IP resolves to a googlebot.com or google.com hostname. Fake Googlebot traffic is common enough that skipping this can invalidate your whole analysis.

Three Patterns Worth Acting On

Pattern 1 — Crawl concentrated on pages that don’t matter.
Sort Googlebot requests by URL count. If the top twenty are attachment pages, tag archives, paginated blog URLs, or filter parameters, your crawl is being consumed by pages that will never earn anything. Fix: the index cleanup work — noindex the archives, canonical the parameters, redirect attachment pages to their parents.

Pattern 2 — Important pages rarely or never fetched.
Filter for your money pages — service pages, city pages — and count requests over the log period. A service page fetched once in a month, or a three-week-old page never fetched at all, means a discovery problem. Fix: internal links from established pages, a sitemap entry, and a URL Inspection submission. This pattern is the single most actionable finding logs produce.

Pattern 3 — Errors served to bots but not to you.
Count status codes for Googlebot. Any 5xx is worth investigating immediately — it means your server failed under crawl, and sustained 5xx is the fastest way to reduce how much Google crawls you. 403s to bots usually mean a security plugin or firewall rule is blocking crawlers; we see this regularly with aggressive WAF configurations, and it can silently suppress an entire site.

A fourth, newer pattern worth a look even though it’s rarely actionable: AI crawler volume. Count requests by user agent and see what share comes from GPTBot, ClaudeBot, PerplexityBot, Bytespider and similar. If AI crawlers are consuming substantial bandwidth, you have a decision to make about robots.txt directives — a business decision about visibility in AI answers versus resource cost, not a technical one.

What Not to Bother With

Crawl budget optimization on a small site. If you have 150 pages, the crawl allocation isn’t your constraint. Logs will confirm this in ten minutes, which is a legitimate reason to look.

Chasing individual 404s from logs. Search Console’s Pages report is a better source, because it shows what Google considers worth reporting. Logs will show you thousands of 404s from vulnerability scanners probing for /wp-admin variants. Ignore those entirely.

Building a monthly log reporting habit. For an enterprise site, sensible. For a local service business, the return doesn’t justify the time. Check annually and after major changes.

Correlating log data with rankings. Tempting and usually spurious at small scale. There isn’t enough data to draw the conclusion you’ll want to draw.

When It Isn’t Worth Your Afternoon

Skip log analysis entirely if all of these are true:

• Under a few hundred URLs
• Search Console shows your important pages indexed
• Crawl Stats shows healthy response times and no significant 5xx
• No recent migration, replatform, or security-plugin change

In that situation logs will confirm everything is fine, which is a fine outcome but a slow way to reach it.

Do look if any of these apply: pages aren’t getting indexed and you don’t know why, you’re behind an aggressive firewall or security plugin, you’ve just migrated, you have hundreds of programmatic pages, or your host reports unexplained resource usage.

Quick Recap

• Logs are the only complete record of what bots actually requested, and the only place you can see AI crawler behavior.
• Get them from your host’s control panel or support; ask for at least a week, ideally a month. CDN caching undercounts origin logs.
• Parse with a spreadsheet pivot or three command-line one-liners. Verify Googlebot IPs before trusting the user agent.
• Three patterns matter: crawl wasted on junk URLs, important pages never fetched, and errors served only to bots.
• Skip it on a small healthy site. Do it after a migration, or when indexing problems have no other explanation.

Frequently Asked Questions

How much log data do I need?
A week minimum for pattern detection, a month for confidence. A single day is too noisy — bot behavior varies substantially day to day.

Can I see logs if I’m on Cloudflare?
Partially. Cached requests never reach your origin, so origin logs undercount. Full log access through Cloudflare requires a paid Logpush plan; Cloudflare’s bot analytics is a reasonable substitute for a small site.

Is Googlebot in my logs always really Googlebot?
No. Spoofing is common. Verify a sample by reverse DNS lookup on the IP — genuine Googlebot resolves to a Google-owned hostname.

Should I block AI crawlers in robots.txt?
A business decision, not a technical one. Blocking reduces resource use and stops your content training models; it may also reduce your presence in AI-generated answers. Most local service businesses currently benefit more from being cited than from blocking.

What’s the single most useful thing logs reveal?
Whether your important pages are being fetched at all. Everything else is refinement; that one finding changes what you do next.