log file analysis seo — illustration

Log File Analysis Without Enterprise Tools

What server logs tell you that Search Console won't, how to get them off managed hosting, how to parse them free, and the three patterns worth acting on.
Developer reviewing server access logs on a dark terminal screen
Server access logs record every single request: timestamp, IP, requested URL, response code, user agent, bytes, and often referrer.

TL;DR: Server logs are the only record of what search engines actually requested from your site, as opposed to what they chose to report. For most local sites this is a once-a-year check rather than an ongoing practice, and you can do it with a spreadsheet. Three patterns are worth looking for. If none are present, you’ve spent an hour and can stop worrying about crawling.

Table of Contents

  • What logs tell you that Search Console won’t
  • Getting logs off shared and managed hosting
  • Parsing them without buying anything
  • Three patterns worth acting on
  • What not to bother with
  • When it isn’t worth your afternoon

What Logs Tell You That Search Console Won’t

Search Console’s Crawl Stats report is aggregated, sampled, and limited to Google. It tells you roughly how much crawling happened and the response distribution. It does not tell you which specific URLs were requested, in what order, how often, or by which bot.

In this video

  1. 0:00 What This Training Covers
  2. 1:14 Jeremy’s Background and PushLeads Origin
  3. 1:33 What Google Search Console Is
  4. 2:14 Search Console Shows Only Your Local Neighborhood
  5. 3:16 Four Levels of Local Search Visibility
  6. 4:27 How to Export Keywords from GSC
  7. 6:28 Reading the Exported Spreadsheet
  8. 8:36 Spot-Checking Rankings with Ad Preview Tool
  9. 10:32 Tracking Keywords in SEMrush
  10. 12:58 Deleting Zero-Traffic Keywords
  11. 14:09 Running a Free Scan at PushLeads.com

Full transcript: How to Find SEO Keywords with Search Console

Server access logs record every single request: timestamp, IP, requested URL, response code, user agent, bytes, and often referrer. Four questions only logs answer:

Which URLs is Googlebot actually fetching? Not which you think are important. Which it requests, and how often. The gap between those two lists is frequently the whole finding.

Is Googlebot reaching your new pages? You published eight city pages three weeks ago. Have they been requested? Search Console will tell you if they got indexed. Logs tell you if they were even fetched.

What are the other bots doing? In 2026 this matters more than it used to. AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others — now make up a meaningful share of traffic on many sites. Logs are the only place you can see their behavior, which robots directives they’re honoring, and how much bandwidth they’re consuming.

What’s erroring for bots specifically? A URL that returns 200 to you in a browser and 403 to a bot behind a firewall rule is invisible in Search Console but obvious in logs.

Want more customers from Google & AI search?

Get a free SEO audit of your site — see exactly what to fix first.

Get My Free Audit Book a Call

Getting Logs Off Shared and Managed Hosting

This is the step that stops most people. Options by host type:

cPanel shared hosting. Look for “Raw Access Logs” in the Metrics section. Enable archiving so logs aren’t rotated away, then download the gzipped files. Usually you get the current day plus a short archive.

WP Engine. Available in the User Portal under the environment’s utilities, or by support request. They retain a limited window, so grab them promptly.

Kinsta. Available in MyKinsta under Logs, downloadable, with access logs separate from error logs.

Flywheel. Request from support; not exposed in the dashboard by default.

SiteGround. Site Tools → Statistics → Access Logs.

Cloudflare in front of your site. This is the important case. If Cloudflare or another CDN sits in front, your origin logs may be incomplete because cached requests never reach the server. Cloudflare’s own Logpush is a paid feature on higher plans. For a small site, the practical answer is to check origin logs and accept they undercount, or temporarily use Cloudflare analytics for bot data.

VPS or dedicated. /var/log/nginx/access.log or /var/log/apache2/access.log. You have everything, including history if rotation is configured sensibly.

Ask for a week of logs minimum, ideally a month. A single day is too noisy to read.

If your host flatly won’t provide them, that’s a data point about the host. It’s not worth switching hosts over, but note it.

Parsing Them Without Buying Anything

Five-step process for parsing server logs without paid tools
Three commands, and you have the core of the analysis.

You don’t need a log analyzer. Three approaches by comfort level.

Spreadsheet (works up to a few hundred thousand lines).
1. Open the log in a text editor and confirm the format. Standard combined format is: IP, timestamp, request, status, bytes, referrer, user agent.
2. Import into Excel or Google Sheets as delimited text using space as the delimiter, quote character as ".
3. Add a column flagging Googlebot rows: search the user agent field for “Googlebot”.
4. Pivot: requested URL by count, filtered to Googlebot. That single pivot answers most of your questions.

Command line (works at any size, faster).
Count Googlebot requests per URL:
grep Googlebot access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -50

Count status codes served to Googlebot:
grep Googlebot access.log | awk '{print $9}' | sort | uniq -c | sort -rn

Count requests by bot:
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20

Three commands, and you have the core of the analysis.

Screaming Frog Log File Analyser. Paid but inexpensive, with a free tier limited to 1,000 lines. If you’ll do this more than twice a year it’s worth it. For an annual check, the command line is fine.

One critical verification step: many user agents claim to be Googlebot and aren’t. Before drawing conclusions about Googlebot behavior, verify a sample of the IPs. Google publishes its crawler IP ranges, and a reverse DNS lookup on a genuine Googlebot IP resolves to a googlebot.com or google.com hostname. Fake Googlebot traffic is common enough that skipping this can invalidate your whole analysis.

Three Patterns Worth Acting On

Pattern 1 — Crawl concentrated on pages that don’t matter.
Sort Googlebot requests by URL count. If the top twenty are attachment pages, tag archives, paginated blog URLs, or filter parameters, your crawl is being consumed by pages that will never earn anything. Fix: the index cleanup work — noindex the archives, canonical the parameters, redirect attachment pages to their parents.

Pattern 2 — Important pages rarely or never fetched.
Filter for your money pages — service pages, city pages — and count requests over the log period. A service page fetched once in a month, or a three-week-old page never fetched at all, means a discovery problem. Fix: internal links from established pages, a sitemap entry, and a URL Inspection submission. This pattern is the single most actionable finding logs produce.

Pattern 3 — Errors served to bots but not to you.
Count status codes for Googlebot. Any 5xx is worth investigating immediately — it means your server failed under crawl, and sustained 5xx is the fastest way to reduce how much Google crawls you. 403s to bots usually mean a security plugin or firewall rule is blocking crawlers; we see this regularly with aggressive WAF configurations, and it can silently suppress an entire site.

A fourth, newer pattern worth a look even though it’s rarely actionable: AI crawler volume. Count requests by user agent and see what share comes from GPTBot, ClaudeBot, PerplexityBot, Bytespider and similar. If AI crawlers are consuming substantial bandwidth, you have a decision to make about robots.txt directives — a business decision about visibility in AI answers versus resource cost, not a technical one.

Checklist of three crawl patterns worth acting on from log data
A URL that returns 200 to you in a browser and 403 to a bot behind a firewall rule is invisible in Search Console but obvious in logs.

What Not to Bother With

Crawl budget optimization on a small site. If you have 150 pages, the crawl allocation isn’t your constraint. Logs will confirm this in ten minutes, which is a legitimate reason to look.

Chasing individual 404s from logs. Search Console’s Pages report is a better source, because it shows what Google considers worth reporting. Logs will show you thousands of 404s from vulnerability scanners probing for /wp-admin variants. Ignore those entirely.

Building a monthly log reporting habit. For an enterprise site, sensible. For a local service business, the return doesn’t justify the time. Check annually and after major changes.

Correlating log data with rankings. Tempting and usually spurious at small scale. There isn’t enough data to draw the conclusion you’ll want to draw.

When It Isn’t Worth Your Afternoon

Skip log analysis entirely if all of these are true:

  • Under a few hundred URLs
  • Search Console shows your important pages indexed
  • Crawl Stats shows healthy response times and no significant 5xx
  • No recent migration, replatform, or security-plugin change

In that situation logs will confirm everything is fine, which is a fine outcome but a slow way to reach it.

Do look if any of these apply: pages aren’t getting indexed and you don’t know why, you’re behind an aggressive firewall or security plugin, you’ve just migrated, you have hundreds of programmatic pages, or your host reports unexplained resource usage.

In this video

  1. 0:00 Ranking Drops Require Diagnosis Before Action
  2. 0:55 Three Conditions That Confirm an Algorithm Hit
  3. 1:58 Technical Causes That Mimic Algorithm Penalties
  4. 3:06 Monitoring Layers to Catch Drops Early
  5. 3:44 Four Data Sets to Pull Simultaneously
  6. 4:17 Matching Update Type to Your Symptoms
  7. 4:56 Ten-Step Audit Sequence in Order
  8. 5:56 Categorize Every Page Keep Improve Remove
  9. 6:41 Rebuilding E-E-A-T and Authority Correctly
  10. 7:58 Manual Action vs Algorithmic Impact
  11. 8:40 Quarterly Prevention Beats Costly Recovery
  12. 9:32 Recovery Timeline and What to Monitor
Show transcript

A ranking drop after a Google update feels like an emergency. Most businesses react immediately — rewriting content, deleting pages, rebuilding links — before they know what happened. That almost always slows recovery.

This video walks you through the complete diagnosis and recovery pipeline: confirm first, rule out technical causes, audit content, categorize every page, and rebuild authority correctly. By the end, you will know exactly what to check, in what order, and what to expect on the timeline.

I’m Jeremy with PushLeads. Helping business owners and marketers track and recover from SEO drops is my specialty, not a side project. I’ve been doing this for over 20 years, long before AI made it fashionable.

In this one, we’re walking through three signals that tell you in minutes whether Google caused your traffic drop or your own site did.

Before changing anything, confirm that an algorithm update actually caused your drop. Three conditions must be true. First, timing must align — your drop date should match a confirmed Google update rollout. Second, the decline must be site-wide, not just two or three pages. Third, you must have ruled out technical causes. A gradual decline with no clear starting point is almost never an algorithm hit. It is far more likely technical debt or a competitor gaining ground.

Here is how you confirm the correlation. Open Google Search Console’s performance report and note the exact date clicks and impressions fell. Match that date to a confirmed rollout on Google’s Search Central blog or Search Engine Land’s update timeline. Filter the performance report by page and sort by clicks. If dozens of pages dropped on the same date, that is an algorithmic pattern. If only a handful dropped, you are likely looking at a targeted issue — a competitor, a crawl problem, or a manual action — not a broad core update.

A botched migration, an accidental noindex tag, or a new disallow rule in your robots.txt file can produce a drop that looks exactly like an algorithm penalty. The key difference appears in Google Search Console. Technical drops create coverage errors — excluded or noindexed URLs visible in the coverage report. Algorithmic drops show clean indexing but lower impressions and lower positions. That distinction tells you which direction to go before spending a single hour on content changes.

Run these technical checks before touching any content. Pull the coverage report and look for a spike in excluded or noindexed URLs. Use the URL inspection tool to confirm Googlebot can render your affected pages. Check your robots.txt for new disallow rules from a recent deploy. Search the source code for noindex strings — one misplaced tag on a shared template can noindex an entire site section. Confirm your XML sitemap is current and submitted. Review server logs for downtime around the drop date. Finish all of this before opening a single page to rewrite.

BrightLocal research shows businesses monitoring rankings weekly catch drops an average of 23 days faster than those reviewing monthly. 23 days is a significant head start. Set up three monitoring layers: Google Analytics automated alerts when sessions drop more than 20% week over week, Google Search Console email notifications for manual actions, and weekly position snapshots for your top 20 most valuable keywords. Recovery from a broad core update typically takes one to three subsequent core update cycles, often two to six months. Catching the drop early shortens total time spent behind.

The moment you suspect a drop, pull four data sets simultaneously. Export Google Search Console performance data for the last 90 days filtered by page. Export Google Analytics organic sessions by landing page for the same window. Pull a position history report for your top 20 keywords. Having all four together lets you match URL-level traffic loss to position loss and indexing errors — the fastest way to determine whether you are facing a technical problem, an algorithmic one, or a competitor gaining ground.

Different updates target different problems. A broad core update targets overall content quality and E-E-A-T signals — experience, expertise, authoritativeness, and trustworthiness. The helpful content system targets pages written to rank rather than to genuinely answer questions. Spam updates go after manipulative link schemes and auto-generated content. Page experience updates focus on Core Web Vitals and intrusive interstitials. Product reviews updates catch thin affiliate content lacking original testing or firsthand experience. Match your symptoms to the update type and you cut audit time significantly.

Here is the full ten-step audit in sequence.

Step one: confirm the drop date and update correlation. Step two: run technical checks — coverage report, robots.txt, noindex tags, sitemap, and server logs. Step three: crawl the full site with Screaming Frog and export all URLs with status codes, title tags, meta descriptions, and word counts. Step four: merge crawl data with analytics and identify the lowest-performing 20% by traffic and engagement. Step five: check for duplicate content and confirm canonical tags. Step six: review your backlink profile for drops in referring domains or spikes in low-quality links, and confirm that author names, credentials, publication dates, and sources appear on your top pages. Skipping to rewrites before finishing these steps is the most common way businesses waste weeks chasing the wrong fix.

Once the audit is done, sort every page into one of three categories: keep, improve, or remove. Pages generating real traffic and serving a clear purpose stay. Pages with thin content get substantially rewritten — not lightly edited, but rebuilt with real depth and original insight. Pages with essentially no traffic, duplicate content, or no clear purpose get consolidated into a stronger page or removed with a proper 301 redirect.

Be more honest than feels comfortable. Keeping low-quality pages out of reluctance to delete is one of the most common reasons recovery stalls. Google evaluates aggregate quality across your entire domain, not just your strongest pages.

Recovery is not just fixing what broke. It is becoming demonstrably better than before the update. Add firsthand experience to content that reads like a summary of other sources. If you cover a product, test it. If you cover a process, show your own results. Add real author bylines with credentials linked to author pages that establish expertise. Cite primary sources — Google’s own documentation, peer-reviewed research, recognized industry organizations. Build backlinks from genuinely relevant sites. One link from a relevant, authoritative source carries more weight than many links from unrelated sites.

One of the most tempting moves during recovery is publishing a wave of new pages. One genuinely useful 1,500-word piece is worth more than five thin 300-word pages. Every new page becomes part of the aggregate quality Google evaluates. Pages that fail to fully satisfy the user’s question compound whatever caused the original drop. Publish carefully, target questions your existing content does not answer, and make every new page demonstrate real experience and depth.

Understanding whether you are dealing with a manual action or an algorithmic impact completely changes your recovery path. A manual action is applied by a human Google reviewer and appears directly in Google Search Console under the Manual Actions report with a specific description of the violation. You fix the cited issue and submit a reconsideration request. An algorithmic impact has no notification in Google Search Console — you see it only as a position decline in the performance report. There is no reconsideration process. Recovery requires genuinely improving site quality and waiting for the next update reassessment. Confusing the two and submitting a reconsideration request for an algorithmic drop wastes months.

Prevention is far less costly than recovery. Run a full content quality audit quarterly. Check every page published in the last 90 days against your E-E-A-T standards before it ages into your domain’s quality average. Run a technical crawl monthly and compare it to the previous month to catch errors introduced by site updates or plugin changes. Set up Google Search Console and Google Analytics alerts so any future decline gets noticed within days. Many businesses discover a ranking drop months after it started, extending the entire recovery timeline. Regular audits catch thin pages before they accumulate into a site-wide quality problem.

Google’s own core update guidance makes this explicit: there may be nothing wrong with your affected pages. Improvement requires making content more useful, more original, and more trustworthy over time. Recovery takes one to three core update cycles, often two to six months, even when you make the right changes quickly.

Use the waiting period productively. Continue improving content, building genuine authority, and monitoring engagement signals. Watch Google Search Console’s average position metric for gradual improvement before traffic bounces back, because position often improves a few weeks before clicks follow. Businesses that use this period well come out of recovery stronger than before the drop.

A ranking drop is data, not a verdict. The businesses that recover fastest diagnose accurately, fix things in the right order, and monitor consistently so the next update cycle works in their favor. Running through this checklist gives you a clear picture of whether your drop was technical, algorithmic, or competitive.

If you want a second set of eyes on the diagnosis, the team at PushLeads reviews sites regularly and will tell you exactly which factors are most likely driving your current position. Get your free site teardown at the link in the description, or call 828-348-7686.

Quick Recap

  • Logs are the only complete record of what bots actually requested, and the only place you can see AI crawler behavior.
  • Get them from your host’s control panel or support; ask for at least a week, ideally a month. CDN caching undercounts origin logs.
  • Parse with a spreadsheet pivot or three command-line one-liners. Verify Googlebot IPs before trusting the user agent.
  • Three patterns matter: crawl wasted on junk URLs, important pages never fetched, and errors served only to bots.
  • Skip it on a small healthy site. Do it after a migration, or when indexing problems have no other explanation.

Frequently Asked Questions

How much log data do I need?
A week minimum for pattern detection, a month for confidence. A single day is too noisy — bot behavior varies substantially day to day.

Can I see logs if I’m on Cloudflare?
Partially. Cached requests never reach your origin, so origin logs undercount. Full log access through Cloudflare requires a paid Logpush plan; Cloudflare’s bot analytics is a reasonable substitute for a small site.

Is Googlebot in my logs always really Googlebot?
No. Spoofing is common. Verify a sample by reverse DNS lookup on the IP — genuine Googlebot resolves to a Google-owned hostname.

Should I block AI crawlers in robots.txt?
A business decision, not a technical one. Blocking reduces resource use and stops your content training models; it may also reduce your presence in AI-generated answers. Most local service businesses currently benefit more from being cited than from blocking.

What’s the single most useful thing logs reveal?
Whether your important pages are being fetched at all. Everything else is refinement; that one finding changes what you do next.

Want more customers from Google & AI search?

Get a free SEO audit of your site — see exactly what to fix first.

Get My Free Audit Book a Call

Up next

How We Fix Text-Only Pages: Images, Videos & Infographics SOP (PushUX UX Improver)

38 min read · 506 of 871 service pages had no images. Here's the exact manual SOP PushLeads uses to add images, videos and infographics to text-only pages…

Want this done for you?

Scan your site. See the gaps. Decide with numbers.

  • Rankings, Google Business Profile and site-speed teardown
  • Report lands in your inbox in about a minute
  • No obligation · 4.9 from 67 Google reviews

Prefer a human? Book 20 minutes with Jeremy

Continue to Part 12

Share this post

log file analysis seo — illustration