I frequently analyze logs from Linux command line. Typically this is the best way to get to the root cause of a problem you are having whether you are a developer, a sysadmin, or even a regular user. In the beginning, reviewing logs can be a daunting task, but there are a few tricks that make the process much more painless. I'll share a few of the things I've picked up with you here. These are all some common Unix tools. Information is readily available on them and the man pages can tell you all about them (just type man COMMAND in the terminal), but I'll simplify it down a bit and show how I use them day to day.
A note on piping and redirection
One thing we will be using a lot is pipes. This is when we place the | character between two commands to redirect output into input. The syntax is command 1 | command 2. The effect of this is to redirect the standard output of command 1 to the standard input of command 2. This allows us to perform consecutive processing operations on log text to generate our final readable output.
There is also redirection which uses the > character to send the standard output of a command to a file. The key difference here is that a pipe sends a program's output to the standard input of another program, whereas redirection sends a program's output to a file.
A note on log locations
Log files are generally located in the /var/log directory unless otherwise specified in a configuration file. Take a look in this directory with the command ls /var/log and see what is available to you. As a desktop user you should at least have an Xorg log and a syslog with lots of juicy information for you to dig into.
cat
If you want to see the contents of a file without opening it in an editor, you can use the cat command. The syntax is cat [filename]. Multiple file names can be specified and they will be displayed one after another.
Now this is useful, but for a large log file this doesn't cut it. Enter less.
less
If we want to scroll through a large file, we can pipe the output of cat to less with cat [filename] | less. Piping with the | character will redirect the standard output of the cat command into the standard input of the less command. Alternatively, we can just call less on the file directly e.g. less [filename]. Less will open a navigation screen. We can scroll up and down in the file using the arrow keys or often our mouse scroll wheel. You can directly go to the beginning and end of the file by entering g or G respectively.
Less also allows you to search through the file by typing / followed by the text you want to search for. You can use n and N to jump to the next and previous instances of the search pattern.
While not necessarily for logs, a nifty trick if you want to edit a file you have open in less is that you can type v and the text file will be opened in the default system text editor. To the uninitiated, beware that this editor will often be vi if not previously changed. If you find yourself unable to exit vi, type :q!. You can edit files in this way and when you exit the editor be returned back into less.To exit less type q.
grep
Grep allows us to search through a file for a pattern. The syntax is grep [pattern] [filename]. This will output all lines of the file that contain the given pattern specified. An alternative syntax to achieve the same thing would be cat [filename] | grep [pattern].
One useful option for grep is -v. It means filter out the given pattern. For example, if I want to show my NGINX access log without showing traffic from 127.0.0.1 I would use grep -v 127.0.0.1 /var/log/nginx/access.log.
Another useful option is -i. It means ignore case. For example, grep -i iNdEx /var/log/nginx/access.log will show me all lines that match index, INDEX, indeX, etc - case is not taken into account. An example of this might be if I want to find my graphical device descriptor from my Xorg log.
tim@localhost:/var/log$ grep -i ">device" Xorg.0.log [ 27.439] (**) | |-->Device "nvidia"
We can also pipe the output of grep into less. If we take our example of filtering out 127.0.0.1, we can use grep -v 127.0.0.1 /var/log/nginx/access.log | less to navigate through our entire access log without 127.0.0.1 entries.
Another useful technique is that we can use regular expressions in our search. We do this using the -E option. An example would be if I was to show all 5XX range errors from my nginx access log I would use grep -E '5[0-9]{2}' /var/log/nginx/access.log. In this example [0-9] will match all numerical digits and {2} indicates that we want to match two digits in a row.
tail
Tail will show the end of the a text file. The syntax is tail [filename]. By default it shows the last 10 lines. This can be changed with -n. For example, tail -n 30 /var/log/syslog will show the last 30 lines of the system log.
One option I use all the time with tail is -f. This stands for follow. This will output the end of the log as it updates in real time. This is especially useful for debugging. Let's say that I have a web application and want to observe 5XX range errors in real time while I interact with my application. I might use the following: tail -f /var/log/nginx/access.log | grep -E '5[0-9]{2}'.
head
Head is like tail by for the beginning of the file. I use it much less than I use tail as it doesn't make much sense to follow the head, but it is still quite useful.
In Conclusion
These are a few of the tricks I use to look through logs. There is no real limit to how many pipes you can use and a lot of the power to this strategy lies in piping the output of these commands into each other. Both grep and searching within less allow for the usage of regular expressions to add variability to your searches.
Often when people are new to troubleshooting and debugging using logs, they can be a bit overwhelming. Hopefully this helps to make them a bit more approachable.
