Log File Analysis and SEO

How to Leverage Log Files for Search Engine Optimization (SEO)

There’s a technical side to search engine optimization (SEO) that often goes unnoticed. You’ll still need to create content and build backlinks, but you’ll also need to analyze certain types of data, one of the most important being log files. Analyzing log files will allow you to optimize your website more effectively so that it succeeds in the search results.

What Are Log Files?

Also known as raw access logs, log files are documents containing structured information about your website’s traffic. When someone visits your website, information about his or her visit is recorded in a log file. Log files contain blocks of information. Each block is associated with a specific visit.

Most log files contain the following information about a given visit:

•   The visitor’s Internet Protocol (IP) address

•   Date and time of visit

•   Hypertext Transfer Protocol (HTTP) request type and status code

 • Requested URL

•   User agent

•   Referring URL

Measure Crawl Frequency through Log File Analysis

You can use log files to measure crawl frequency. Log files don’t just reveal information about human visitors; they reveal information about search engine visitors as well. Search engines will visit your website to crawl it. By analyzing log files, you can measure the frequency at which Google, Bing or other search engines crawl your website.

Search engines are distinguished from human visitors in log files by their user agent. For human visitors, the user agent is a standard web browser. For search engines, the user agent is a crawler, such as Googlebot or Bingbot. You can see how often search engines crawl a given page by analyzing log files.

Identify Status Code Errors

Log files can help you identify status code errors. Status codes are the way in which your website responds to visitors’ requests. When a visitor requests a URL, your website will send him or her a status code. Certain problems, either client or server side, can result in a status code error. Instead of receiving a 3xx status code, for instance, visitors may receive a 4xx or 5xx status code.

Status code errors can damage your website’s SEO in a few ways. If a particular page is throwing a status code error, search engines may delist it from their indexes. The page won’t earn many backlinks with a status code error, either. It will act as dead weight by degrading your website’s search performance

In log files, status codes are displayed alongside the HTTP type. You’ll see the visitor’s HTTP type, such as GET or POST, as well as a status code he or she received from your website. Not all status codes are errors. Only those that begin with “4” or “5” indicate an error.

Optimize JavaScript

You can use log files to optimize your website’s JavaScript. JavaScript poses challenges in regards to SEO. Unlike with text, search engines struggle to crawl it. Google, in fact, didn’t begin crawling JavaScript until 2008. Even today, Google and other search engines often encounter problems when attempting to crawl JavaScript.

If your website has JavaScript, you should consider using log files to determine whether search engines can crawl it. You can analyze the visits for your website’s JavaScript files. If you discover that a JavaScript file is throwing a status code error, search engines may not be able to crawl it. You can then open the JavaScript file to optimize and fix it.

Preserve Link Equity During Redirects

Analyzing log files can help to preserve your website’s link equity when redirecting URLs. It’s not uncommon for websites to change the URLs of their pages. When you change the URL of a page, you’ll typically want to set up a redirect. A redirect will allow visitors to access the page by visiting its old URL. The old URL will redirect visitors to the respective page’s new URL.

Redirecting URLs, however, can result in a loss of link equity. Links pointing to a page’s old URL will become less valuable. Redirects essentially dilute the links’ ranking authority or equity. Fortunately, you can preserve your website’s link equity by using 301 redirects.

Redirects can be either 301 or 302. They work in the same by moving visitors from an old URL to a new URL. The difference is that 301 redirects are designed for instances when which a page or document has permanently moved, whereas 302 redirects are designed for instances when a page or document has been temporarily moved. With that said, 301 redirects pass substantially more link equity than their 302 counterparts. You can use log files to find 302 redirects — they are shown as the referring URL — so that you can change them to 301 redirects.

Tips on Analyzing Log Files

Before you can analyze log files for SEO, you’ll need to retrieve them. Most web hosts will create at least one log file for each hosted website automatically. You can download this log file from the host’s control panel.

Log files contain a lot of data, so they are usually compressed. You’ll need to download the log file from your host’s control panel, followed by extracting and saving it to your hard drive. After retrieving the log file, it’s time to analyze it.

You can open and view log files using any text-editing program, including Notepad. In addition to being compressed, nearly all log files use the Common Log Format. This text-based specification makes them easy to read by eliminating the need for special programs. When you open a server log in a text-editing program, you’ll see information about each of your website’s visits on a separate line.

While it’s possible to analyze server logs in a text-editing program, doing so can be tedious. An easier method is to use an Apache log viewer. Apache log viewers are programs that are designed specifically to analyze server logs.

Analyzing server logs is a form of technical SEO. As visitors and search engines access your website, information about their visits will be recorded in a server log. You can analyze these logs to measure crawl frequency, identify status code errors, optimize JavaScript and preserve link equity during redirects.