Finding New Ways to Look at Your Data

We are surrounded on all sides by mountains and mountains of data. The amount of personal data generated by the average citizen on a daily basis is staggering, doubly so when you add professional endeavors. Whether it’s personal fitness data, personal spending reports from the bank, or web traffic analytics and build logs, we are now routinely asked to look at larger and larger amounts of data and make important decisions based on that data. Sometimes it’s useful (or just plain fun) to take a step back, remove ourselves from the eye of the information storm and attempt to see things from a different perspective. Here are a few examples of creative tools I’ve found that allow people to look at their data from a different perspective in the hopes of gaining some new insight into the process.

 

---

 

Logstalgia

Logstalgia is a tool that allows you to visualize the logs from your web server. Instead of lines and lines of IP addresses and HTTP status codes, you are instead presented with colored balls traveling across the screen from left to right. Each ball represents a connection to your website. The place where the ball lands on the right side of the screen is the destination page. It might be difficult to visualize, so here’s an actual video of the software in action.

This particular video depicts a user, or group of users using automated software to “scrape” our website for information. Using Logstalgia, it was easy to see that most of our web traffic at the time was coming from a handful of IP address and they were opening multiple simultaneous connections, systematically going through our site in alphabetical order looking for user names and email addresses. Visualizing the traffic gave us a much more intuitive understanding of what was happening to our site and it allowed us to react a bit more quickly to the problem we were seeing and get our site back to normal quickly.

 

---

 

Space Shooter Web Logs

During our most recent hackathon, I remembered the video above and it inspired me to see if there was a more interactive way to glean information from our weblogs. That lead me to create a tiny prototype for a space shooter game based on the data in our web logs.

The full concept was to have a user-controlled spaceship at the bottom of the screen. The alien ships that fall from the top of the screen each represent a connection to your web server. Different categories of pages would have different colors/types of aliens, as would different categories of connections. The IP address of the user would be translated into a country code which could then be represented by a little flag on the body of each alien ship. Looking at the types of ships would give you an overall impression of the types of traffic currently visiting the site.

 

---

 

Doom Process Killer

psDoom is a small project created to demonstrate an interesting concept: What if system administrators were able to view their servers, their systems, and software as artifacts in a first person perspective 3D interface? psDoom allows the player to view all the system processes on the server as “monsters” roaming around a 3D level. Killing one of the monsters in the game actually triggers a process to be killed.

So for example, shooting the monster labeled “webserver” would cause the webserver process to actually be killed. This opens the mind to even more uses of this gaming paradigm. What if you were able to see resource starved processes as looking pale, gaunt, and hungry? What if you were able to see processes contending for shared resources as two quarreling entities in a 3D game? It might give administrators a more natural, intuitive way to evaluate the health of their systems aside from combing through hundreds or thousands of lines of log files.

Those are just a few examples of small projects that allow people to analyze their data to diagnose system problems, but there are other people using novel ways of viewing data to diagnose and solve much higher stakes problems.

 

---

 

Protein Folding

A game called “Fold It!” allows players to solve three-dimensional puzzles based on protein folding research being conducted to help find cures for serious illnesses. By solving these puzzles, the players are using the unique, problem-solving ability of the human brain to identify patterns and solve “puzzles” that sometimes confound traditional computer algorithms. The solutions to these puzzles contribute directly to understanding and subsequently contribute to the creation of new medications and treatments.

While some of these examples might not be immediately relevant to most people’s day to day workflow, I hope it helps to illustrate how taking a novel approach to understanding our data can assist in the development of more robust, intuitive, and easy to manage systems and software. At the very least, it might help to give you a break the monotony of data analysis and troubleshooting, and reinvigorate your imagination!

 

---