Log File Analysis: See What Crawlers Actually Crawled

Log file analysis is the only method that can prove which URLs crawlers actually crawled. Search Console gives you a total crawl volume curve, server logs give the timestamp, source IP, path, status code, and response size of every single request. The gap between these two sets of data is often the real reason new pages迟迟 don’t get indexed.

Filter the last 30 days of access logs for search engine crawler requests, aggregate by URL, and you’ll see three things clearly: which low-value paths are eating crawl budget, which important pages haven’t been visited in weeks, and how many requests are hitting 404s and redirect chains. After fixing these three things, indexing speed usually shows visible changes within 2-4 weeks, and you don’t need to change a single line of frontend code.

First recognize the six truly useful fields in logs

A standard line of Nginx or Apache access log looks messy, but actually only six fields are directly related to SEO. Pick them out, the rest can all be discarded — processing speed will be much faster.

Don’t skip IP reverse lookup. A lot of scraping programs disguise their User-Agent as Googlebot — without reverse DNS verification, up to 30% of your crawl volume statistics could be fake.

Three steps to turn raw logs into readable tables

Most webmasters get stuck at the first step: hundreds of MB .log files can’t be opened with Notepad. Split the process into three steps and it’s not hard — a laptop can process a month’s volume in half an hour.

• Data extraction: download the last 30 days of access logs from server /var/log/nginx/ or hosting panel; sites using CDN need to separately pull edge logs, otherwise you’ll miss requests blocked by cache.
• Cleaning: only keep lines with crawler User-Agent, reverse DNS verify to remove disguised traffic, unify request paths to lowercase, remove tracking parameters.
• Aggregation: make three pivot tables by URL, by directory, and by status code respectively, export CSV. Screaming Frog Log Analyser, GoAccess, or a pandas script all work — as long as the dimensions are right.

Five metrics to see through crawlers’ real preferences

After the tables are done, don’t rush to conclusions — first watch these five numbers. They can translate vague “poor indexing” into specific, fixable problems.

• Crawl frequency distribution: what proportion of requests do the top 20% of URLs eat? A healthy site should have core categories and product pages as the majority — if what ranks at the top is search result pages or parameter pages, it means crawl budget has already been diluted.
• Zero-crawl pages: do a difference set between sitemap URLs and logs — pages not visited even once in 30 days are orphan pages, usually caused by missing internal links.
• Status code proportion: non-200 requests over 10% should trigger an alert, focus on the absolute volume of 404s and 301s — dead link and redirect chain cleanup can often immediately release considerable budget.
• Average response time: crawlers actively slow down for slow sites — when response time goes from 200ms to 1.2s, the crawl volume drop isn’t an algorithm problem, it’s a server problem.
• Mobile vs desktop ratio: under mobile-first indexing, mobile UA crawl volume should be明显 higher than desktop; an abnormal ratio means the mobile version has redirect or rendering obstacles.

Four types of problems logs can catch that crawler tools can’t see

Tools like Screaming Frog simulate “how crawlers should walk,” logs record “how crawlers actually walk.” The following four types of problems can only be exposed by the latter.

• Ghost URLs repeatedly crawled: long-decommissioned old version paths, leaked test environment addresses, wrong URLs pointed to by external links — they’re not in your site structure, yet continuously consume requests.
• robots rules not taking effect: wrote Disallow but crawlers still crawl — usually path case or wildcard written wrong,对照 logs to review robots.txt and sitemap configuration can locate it.
• Crawl and indexing disconnected: certain pages get crawled every day but don’t enter the index — the problem is in content quality or canonical tags, not in the crawl layer, cross-verify with Search Console coverage report can quickly distinguish.
• Post-publish response delay: the interval from a new article going live to first crawl is the most honest metric for measuring site authority, more accurate than any third-party score.

Schedule log analysis as a monthly fixed action

One-time analysis has limited value — trends are what matter. Every month on a fixed day, run the same set of scripts, record the five metrics in the same table, and after three months you can see the respective effects of redesigns, adding internal links, and cleaning dead links.

When sample size is too small, conclusions aren’t reliable. For small sites with under 500 daily crawls, extend the period to 90 days; for e-commerce or news sites with over 10,000 daily crawls, 30 days is enough.

Next step: pick one of the following four and finish it today, don’t wait until you’ve finished discussing log analysis tool selection to start.

• Log into server or hosting panel, download the last 30 days of access logs, confirm fields are complete and not truncated or lost.
• Filter crawler requests and do reverse DNS verification, count real total crawl volume, compare with Search Console numbers.
• Export status code distribution, list the top 20 URLs by 404 and 301 count, process them within this week.
• Subtract URLs that appeared in logs from sitemap, get a zero-crawl list, add 2-3 related internal links to each page.

FAQ

What’s the difference between log analysis and Search Console reports?

Search Console gives total crawl volume curves, logs give timestamp, path, status code, and response size for every request — the gap is often the reason new pages index slowly.

How to process hundreds of MB log files?

Three steps: download 30 days of logs (CDN sites pull edge logs), filter crawler UA and reverse DNS verify, make three pivot tables by URL, directory, and status code.

How to identify scrapers disguised as Googlebot?

Reverse DNS verify the client IP — only those that resolve back to Googlebot domains are real crawlers. Without verification, up to 30% of your crawl volume statistics could be fake.

Which metrics in logs should you watch most?

Five: crawl frequency distribution, zero-crawl pages, non-200 status proportion, average response time, mobile vs desktop ratio — translating “poor indexing” into specific problems.

How much crawl volume counts as sufficient sample?

Small sites with under 500 daily crawls extend to 90 days, e-commerce/news sites with over 10,000 daily crawls 30 days is enough. Too small a sample yields unreliable conclusions.