Log files: what they are and how to make the most of them

Have you ever heard of log files? If the answer is yes but you still don’t know what it is, you are in the right place.

If you are a nautical enthusiast, the name might already be a clue: in fact, the term derives from an eighteenth-century seafaring expression, log, i.e. a wooden board on which the boat’s speed, weather, wind strength and significant events occurring on board were noted. Three centuries later we find this term still associated with navigation, but in a context much more familiar to all of us, namely the internet.

Today, in computer science, a log file is a digital record in which all the steps, accesses and actions that take place within a computer system, specifically on the server hosting your website, are noted down. It is a fundamental tool because it allows you to monitor the behaviour of web crawlers, i.e. the search engine bots that perform periodic scans of the content present on the web, giving you the possibility of turning this information into opportunities to optimise your site.

Contrary to what you might think, these are not endless, unreadable strings of text, but files in text format, organised to be analysed with specific tools. Their structure may vary depending on the server, but overall they all contain the following information:

• the timestamp, i.e. the time stamp, which indicates the date and/or time at which a given event occurred;
• the log level, which indicates the severity of the event (e.g. info for
• common activities, warning for potential problems and error for errors);
• the log message, which reveals the details of the event such as the name of the service that generated it, the action or the error.

Analysing log files can therefore be fundamental to improving site performance, following a detailed analysis of events, including unwanted ones. For instance, should you experience a slowdown, log file analysis allows you to understand the cause, whether it is a traffic problem, a code error or a computer attack.

Log files are a fundamental tool for SEO practitioners, as they allow us to understand how search engines interact with a site: it is possible to know the type of visitors in order to optimise and improve indexing. By identifying the pages visited by bots and those visited by human users, it will be possible to understand which content is perceived to be of greater value, giving us the possibility of adapting our strategy to this information.

If it is true that log files are like digital diaries, it will therefore be possible to:

• identify the most important pages and identify those that are not crawled frequently;
• determine the crawl budget, i.e. the time and resources the bots spend on crawling a website and indexing pages;
• check for large or slow pages and recognise those crawled most frequently;
• monitor crawler activity, paying attention to drops or increases in activity.

It is crucial for SEO to make sure that the site is crawled efficiently and effectively, especially making sure that key pages, both new and constantly updated ones, are found easily, crawled quickly and with appropriate frequency. Furthermore, knowing how Google crawls your site allows you to improve your organic performance.

#1 Data, data, and more data!
Data analysis is the first crucial step in understanding how to modify and improve your website. In this case, log files are the main source of data collection for network observability.

#2 Problems… and solutions
Thanks to the textual logging of events produced by the system, software and hardware developers are able to act quickly if problems are detected thanks to the log files.

#3 Waiting times are also important
Excessively long waiting times are known to cause frustration in users, so it is necessary to detect slow queries, errors that cause transactions to take too long or bugs that affect the performance of websites or applications, and this is possible by analysing log files.

Have you ever analysed the log files of your website? We are sure you will start doing so today!