By Liisa Tallinn and Raido Karro
This article is a step-by-step guide on parsing IIS logs using SpectX. Once the parser matches the raw data, SpectX makes it quick and easy to run queries on a large number of log files from their current location. This is especially handy if the volumes are large and the time is limited.
IIS logs seem fairly easy to parse - the structure is well standardized and available in the header of the files. At the same time, there is no pattern to rule them all because logging use cases are unique, storage is limited and not all IIS servers have every field and option turned on in their configuration. The situation gets even trickier when looking at multiple servers with multiple configurations. The solution we've drawn up at SpectX is a full IIS log pattern (or schema) to cover all fields potentially available at IIS versions 8.5 and up. Fields not present in the data can be commented out in the pattern.
To get started, download and install SpectX. The Desktop edition is free of charge. If you happen to get stuck anywhere, feel free to join our Slack community to ask for advice.
#drive#:\inetpub\logs\LogFilesThe fastest option for reading, parsing and analyzing these files with SpectX is joining the AD security group governing access to this specific folder. This will allow mapping the log folder to the machine running SpectX and running complex queries without first importing or copying the data from the current location. If the IIS server runs in Azure and the logs sit at an Azure blob, SpectX can also read the files directly from that blob. The third option is centralized logging. SpectX can read files from the central file storage via ssh or much faster if adding the SpectX Source Agent to the server.
To access IIS logs with SpectX, create a datastore to specify where they are located. Click on New > Datastore and pick System as the location for the datastore. In the default installation (for security reasons), you can only access local files by logging into SpectX as an admin and creating a datastore into the System folder (the contents of which are also only accessible for admins).
Select 'file' as the store protocol if the logs are mapped to the machine running SpectX. Select 'wasbs' (Windows Azure Storage Blob Secure) as the store protocol if your data is an Azure blob. Create the datastore and close the small window. Then, navigate to the file by clicking on the Input Data and navigating a file in the datastore you just created. Continue to 'prepare pattern’.
%SystemDrive%\inetpub\logs\u_ex181029.logcan be expanded by
%SystemDrive%\inetpub\logs\u_ex1810*.logto look at the whole month or with
%SystemDrive%\inetpub\logs\u_ex18*.logto look at the full year.
@src = PARSE(pattern:$pattern, src: [‘C:\inetpub\logs\u_ex1901*.log, C:\inetpub\logs\u_ex11902*.log, C:\inetpub\logs\u_ex1903*.log’]);