Hello World!
If you are still reading after the title of this post, I commend you. Writing about logging, and more specifically, a syslog server is difficult to get excited about and I can only assume that it is equally hard to get excited about reading.
I should preface this by saying I am not a “Nix” guy. I’ve dabbled in nearly every flavor of Linux over the last 20 years or so but I’m definitely not a Linux over Windows guy (with the notable exception of purpose-built hardened linux appliances) nor will I ever claim to be. With that said, I like the simplicity of Linux specifically in the utility space. A Linux TFTP/FTP/SFTP server makes far more sense than a Windows box doing the same job, as does a Linux jump box (depending on what you are jumping to). To that end, when I decided to add a syslog server to my lab/development environment; Linux was far and away the best choice.
I am choosing to use Ubuntu as my flavor of choice for this particular Linux project as I generally like the look and feel of it. I am using the server distro and while I am now running 18.0.4 LTS, I did not start out that way. I tried 18.0.4 as a fresh install in my virtual (VMWare ESXi) environment but found that I could not get past configuring the network interface in the setup UI. I would verify the interface (it was correct) and then setup my IP address (static in this case, although I did also try DHCP). This seems simple enough, but once I attempted to save my network settings the UI would reset and I would once again be back at the start of the setup process. I experienced this several times before I gave up and went another way. The “Other Way” in this case was to first install release 16.0.4 and then allow the system to upgrade to 18.0.4. This worked without problem. I just so happened to have 16.0.4 in my software repository but I’m sure I could have upgraded from other releases as well. In doing some research, it seems that Ubuntu introduced a different network configuration manager in 18.0.4 and I’m guessing that it was not playing nice with ESXi.
Now that I have my Linux platform, the installation of the RSyslog server packages is unimpressive. The command to fetch and install the package on Ubuntu is listed below.
apt-get install rsyslog
Once the install is complete the configuration can begin. If all you want is an external place to drop your syslog data this will take no time at all.
In /etc you’ll need to edit the rsyslog.conf file. For the most basic functionality you need to make sure that RSyslog is listening. In my environment UDP for syslog is fine and I also use the standard port (514).
When you initially open the configuration file you’ll notice that UDP (and TCP as well) are commented out. This means that RSyslog is only processing local syslog events (See snippet below).
# provides UDP syslog reception # module(load="imudp") # input(type="imudp" port="514") # provides TCP syslog reception #module(load="imtcp") #input(type="imtcp" port="514")
Since I am using UDP, I want to un-comment the module and input lines associated to that protocol (see snippet below).
# provides UDP syslog reception module(load="imudp") input(type="imudp" port="514")
Once I have done this, I can restart the RSyslog service and my server is now ready to receive. The restart command is below for your reference.
sudo systemctl restart rsyslog
Upon a successful restart (assuming there is a device setup to send syslogs), I should see new messages in my syslog file located at /var/log/syslog (snippet shown below).
Mar 18 06:25:02 linutil01 rsyslogd: [origin software="rsyslogd" swVersion="8.32.0" x-pid="1051" x-info="http://www.rsyslog.com"] rsyslogd was HUPed Mar 18 15:30:01 vmhost02.sprnet.local Rhttpproxy: message repeated 2 times: [ verbose rhttpproxy[FFFC3B70] [Originator@6876 sub=Proxy Req 01543] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x1f08e318] _serverNamespace = /sdk _isRedirect = true _port = 8307] Mar 18 15:30:01 vmhost02.sprnet.local Rhttpproxy: message repeated 2 times: [ verbose rhttpproxy[56E54B70] [Originator@6876 sub=Proxy Req 01543] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x1f08e318] _serverNamespace = /sdk _isRedirect = true _port = 8307]
In the above snippet, there are three entries. The first entry is from my local Linux box (linutil01) and the next two are from one of my VMWare ESXi hosts (vmhost02.sprnet.local).
If you have a small environment and only a handful of devices sending to your syslog server, the above configuration may be all you need. If, like me, you want something that is more organized, I think I can help.
While there are several ways of accomplishing organization in RSyslog, I am choosing to organize my logs by hostname and service name. For the purposes of this example, I have two ESXi hosts that are logging to the server.
To start, I need to go back to the rsyslog.conf file that I wrote about earlier. I need to tell the server what logs I want to be stored and where/how I want to store them. I can do this with a template. I’ve posted the template configuration snippet below.
# Templates $template RemoteHost,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" #Remote Logging $Ruleset remote *.* ?RemoteHost #Bind Ruleset to UDP Listener $InputUDPServerBindRuleset remote $UDPServerRun 514
In the snippet above, I have my template name RemoteHost and I have where I want to store/categorize my logs /var/log/%HOSTNAME%/%PROGRAMNAME%.log. The %HOSTNAME% and %PROGRAMNAME% sections are variables and are filled in by the servers internal syslog facilities. Note that I have logs being stored in the /var/log/ directory. If you want to store your logs here you will have to change ownership properties of the directory (snippet below) so that syslog can create and write there. I also have a Ruleset that I am binding my template to. In my configuration that ruleset is called remote. I bind my template to the remote ruleset which includes every message coming into the server on UDP port 514.
cd /var && sudo chown syslog:syslog log
Once I complete the configuration above and restart my RSyslog service, I should start seeing directory entries like the ones shown below in my /var/log/ directory.
drwxr-xr-x 2 syslog syslog 4096 Mar 16 13:56 vmhost01.sprnet.local drwxr-xr-x 2 syslog syslog 4096 Mar 16 13:55 vmhost02.sprnet.local
If I go into one of these directories, I should see several different .log file entries (snippet shown below).
-rw-r----- 1 syslog adm 0 Mar 16 13:55 bootstop.log -rw-r----- 1 syslog adm 10263 Mar 16 11:15 cimslp.log -rw-r----- 1 syslog adm 97516 Mar 18 07:01 crond.log -rw-r----- 1 syslog adm 10474 Mar 18 07:01 heartbeat.log -rw-r----- 1 syslog adm 777730 Mar 18 07:00 hostd-probe.log -rw-r----- 1 syslog adm 5345 Mar 18 06:01 ImageConfigManager.log -rw-r----- 1 syslog adm 0 Mar 16 13:55 init.log -rw-r----- 1 syslog adm 3033018 Mar 18 07:01 Rhttpproxy.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 root.log -rw-r----- 1 syslog adm 0 Mar 16 13:55 sfcbd.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 sfcbd-watchdog.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 sfcb-ProviderManager.log -rw-r----- 1 syslog adm 6384 Mar 18 06:36 sfcb-vmware_raw.log -rw-r----- 1 syslog adm 16384 Mar 16 13:56 slpd.log -rw-r----- 1 syslog adm 171223 Mar 18 06:35 smartd.log -rw-r----- 1 syslog adm 47693 Mar 18 07:00 syslog.log -rw-r----- 1 syslog adm 128 Mar 15 16:05 tmpwatch.log -rw-r----- 1 syslog adm 4096 Mar 16 13:55 vmauthd.log -rw-r----- 1 syslog adm 6189399 Mar 18 07:01 vmkernel.log -rw-r----- 1 syslog adm 827731 Mar 18 07:00 vmkwarning.log -rw-r----- 1 syslog adm 0 Mar 16 13:55 VMware.log -rw-r----- 1 syslog adm 3633994 Mar 18 07:01 vobd.log -rw-r----- 1 syslog adm 69577951 Mar 18 07:01 Vpxa.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-cdp.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-dcbd.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-net-lacp.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-net-lbt.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-nscd.log -rw-r----- 1 syslog adm 0 Mar 16 13:55 watchdog-openwsmand.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-sdrsInjector.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-smartd.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-vobd.log -rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-vpxa.log
I now have an organized version of the chaos that syslog directories often hold. I can look at the systems and services that I need to without sorting through all of the other notifications that often get in the way.
I hope this has been helpful to someone, there are a lot of great resources online and mine is simply a collection of what worked for me. Thank you for reading, please ask any questions in the comments section and I’ll do my best to respond in a timely manner.
-Justin
Justin On VOIP (and more) 