Using Logs to Chase Changes in 3rd Party Services

Tarmo Mamers, Network Security R&D Engineer

At RebelRoam we offer Wi-Fi optimization service that helps to improve Wi-Fi quality, increase security and reduce data consumption. Keeping up with changes in third-party technologies and services is a must. SpectX is a tool that helps us analyse logs and detect changes in traffic statistics, approaching the data swiftly and with great flexibility.

A significant part of our cloud-based Wi-Fi optimization solution and one of our primary expertise is content filtering. To keep running and to improve our service, we need to look at traffic statistics, play around with logs, enrich and aggregate data. To keep our customers satisfied, we need to spot changes in the behaviour of internet services that end-users are using. For example, we need to quickly find out the reason behind significant changes in the amount of traffic directed at Facebook or Google. We are proactively following all the public news and release notes but this is not enough, there are many situations where we can only be reactive. When Youtube changes their video codecs or buffering logic - this is their internal know-how and might be their intellectual property - it’s clear that this might not be covered in public, but as a traffic optimiser, we need to deal with such changes immediately. None of the cases looks the same and they all need different approaches to collecting data as well as to analyzing it.

SpectX is an excellent tool for getting any data quickly into a table or onto a chart, no matter if that data is already in csv format or unstructured in logs or even in binary traffic capture files. My first use of SpectX was to query a session database and do AS number and AS name lookups with additional non-trivial decisions, meddling also with IP subnet addressing. The result was a simple bar chart. Solving this case gave me a good impression of the SpectX tool and a feeling that product support is close if need will be.

We’re currently setting up SpectX to monitor our traffic volumes with data coming via logs and via APIs to third-party systems. We’ll be looking at volumes and distribution of data within time periods, nature of traffic between connections, specific behaviour of application traffic, AS distribution and so on.
A lot of our work goes around finding out reasons for pattern changes in the network and in application traffic and having to fiddle ad-hoc with all the above data and get responses quickly. SpectX saves me the hassle of parsing log lines from one separator to another, meddling with string functions and typifying the data in some programming language or bash scripts. Encountering quotation marks is always a challenge when scripting things manually. Such approach is not very helpful to get fast results to an offhand hypothesis. But SpectX helps with both the programming and analyzing part and gets me the results in no time.

Once past the initial learning curve, SpectX offers a quick way of doing all this - parsing and querying data, getting immediate typified results. I also like the fact that you can install SpectX on Windows or Mac desktop - it is therefore also a tool you can use to play around with local data while offline.
Looking beyond our primary use cases, SpectX looks like a flexible tool for investigating security incidents. Situations where something is going on, but it’s not easy to discover what exactly. What you need to do in these situations is correlating a lot of different data. However, log analysis can be a real challenge if it’s not just the network logs but application logs and data produced by integrations with external systems. Centralised application logs are a good starting point for discovering true anomalies and patterns.

About RebelRoam
RebelRoam provides modern onboard Wi-Fi Routers, onboard Wi-Fi Administration Service and pan-European mobile broadband Data Service for passenger transportation and tour operator companies. RebelRoam solutions also include carrier and equipment-agnostic “RebelRocket Cloud” and “RebelRocket Hybrid Cloud” Data Traffic Optimization Services that can be used worldwide with any internet connectivity setup.

Back to case studies