Parsing IIS logs

By  Liisa Tallinn and Raido Karro

This article is a step-by-step guide on parsing IIS logs using SpectX.  Once the parser matches the raw data, SpectX makes it quick and easy to run queries on a large number of log files from their current location. This is especially handy if the volumes are large and the time is limited.

IIS logs seem fairly easy to parse - the structure is well standardized and available in the header of the files. At the same time, there is no pattern to rule them all because logging use cases are unique, storage is limited and not all IIS servers have every field and option turned on in their configuration. The situation gets even trickier when looking at multiple servers with multiple configurations. The solution we've drawn up at SpectX is a full IIS log pattern (or schema) to cover all fields potentially available at IIS versions 8.5 and up. Fields not present in the data can be commented out in the pattern.

To get started, download and install SpectX. The Desktop edition is free of charge. If you happen to get stuck anywhere, feel free to join our Slack community to ask for advice.

Locating IIS Log Files

If the IIS server runs on-prem and centralized logging has not been set up, the logs are most likely sitting in the server itself. The default configuration has them collected in the folder

#drive#:\inetpub\logs\LogFiles

The fastest option for reading, parsing and analyzing these files with SpectX is joining the AD security group governing access to this specific folder. This will allow mapping the log folder to the machine running SpectX and running complex queries without first importing or copying the data from the current location.  If the IIS server runs in Azure and the logs sit at an Azure blob, SpectX can also read the files directly from that blob. The third option is centralized logging. SpectX can read files from the central file storage via ssh or much faster if adding the SpectX Source Agent to the server.

The Trouble with IIS Headers

Looking at the raw IIS file, it is quite easy for a human to grasp the data structure since all the fields  are specified in the header starting with #Fields:


The tricky part for a machine when parsing the logs is the fact that the header lines starting with the '#' are present not only at the beginning of the file but also in the middle, repeating after a certain number of events. This metadata needs to be skipped to be able to analyze the actual data. Here's how to do it with SpectX. 

Point to the Storage and Parse the Logs

To access IIS logs with SpectX, create a datastore to specify where they are located. Click on New > Datastore and pick System as the location for the datastore.  In the default installation (for security reasons), you can only access local files by logging into SpectX as an admin and creating a datastore into the System folder (the contents of which are also only accessible for admins).

Select 'file' as the store protocol if the logs are mapped to the machine running SpectX. Select 'wasbs' (Windows Azure Storage Blob Secure) as the store protocol if your data is an Azure blob. Create the datastore and close the small window. Then, navigate to the file by clicking on the Input Data and navigating a file in the datastore you just created. Continue to 'prepare pattern’.

In the upper pattern developer pane, replace the default schema ‘LD EOL’ with this pattern we've pasted to Github. The best part is this particular pattern will parse the data into typified fields where applicable, not just text. The next step is to tweak the pattern to match the fields present in your data.

Customizing the Pattern (Schema)

To match the pattern with your data, take a look at the '# Fields' section in the file header. Compare the list with the fields listed in the SpectX pattern. Comment out fields in the pattern that are missing in the header.


When the pattern matches the data, fields in the parse preview light up in yellow and blue:

Run the First Query

When the pattern matches the data, click on 'Prepare Query' and 'Run'.  The result should give you the first 1000 lines of the IIS log file you selected, parsed and typified.

Tip! If you’d like to look at more than just one file, add an asterisk to the path at the beginning of the query. For example, the scope for

%SystemDrive%\inetpub\logs\u_ex181029.log

can be expanded by

%SystemDrive%\inetpub\logs\u_ex1810*.log

to look at the whole month or with

%SystemDrive%\inetpub\logs\u_ex18*.log

to look at the full year.

If your logs are stored in multiple locations and you would like to analyze them all simultaneously, first create a datastore for each of the sources. Then use the following syntax to list all the datastore URIs as a source for the parser. 

@src = PARSE(pattern:$pattern, src: [‘C:\inetpub\logs\u_ex1901*.log, C:\inetpub\logs\u_ex11902*.log, C:\inetpub\logs\u_ex1903*.log’]);

More Queries

Now that the data is parsed and the scope of the data is as broad as needed, you can start experimenting with queries. Take a look at these 20 questions and copy-pastable queries you can ask from your IIS logs. 

Back to articles