Tag: virtualization

My Linux Logging Adventure: RSyslog

Hello World!

If you are still reading after the title of this post, I commend you. Writing about logging, and more specifically, a syslog server is difficult to get excited about and I can only assume that it is equally hard to get excited about reading.

I should preface this by saying I am not a “Nix” guy. I’ve dabbled in nearly every flavor of Linux over the last 20 years or so but I’m definitely not a Linux over Windows guy (with the notable exception of purpose-built hardened linux appliances) nor will I ever claim to be. With that said, I like the simplicity of Linux specifically in the utility space. A Linux TFTP/FTP/SFTP server makes far more sense than a Windows box doing the same job, as does a Linux jump box (depending on what you are jumping to). To that end, when I decided to add a syslog server to my lab/development environment; Linux was far and away the best choice.

I am choosing to use Ubuntu as my flavor of choice for this particular Linux project as I generally like the look and feel of it. I am using the server distro and while I am now running 18.0.4 LTS, I did not start out that way. I tried 18.0.4 as a fresh install in my virtual (VMWare ESXi) environment but found that I could not get past configuring the network interface in the setup UI. I would verify the interface (it was correct) and then setup my IP address (static in this case, although I did also try DHCP). This seems simple enough, but once I attempted to save my network settings the UI would reset and I would once again be back at the start of the setup process. I experienced this several times before I gave up and went another way. The “Other Way” in this case was to first install release 16.0.4 and then allow the system to upgrade to 18.0.4. This worked without problem.  I just so happened to have 16.0.4 in my software repository but I’m sure I could have upgraded from other releases as well. In doing some research, it seems that Ubuntu introduced a different network configuration manager in 18.0.4 and I’m guessing that it was not playing nice with ESXi.

Now that I have my Linux platform, the installation of the RSyslog server packages is unimpressive. The command to fetch and install the package on Ubuntu is listed below.

apt-get install rsyslog

Once the install is complete the configuration can begin. If all you want is an external place to drop your syslog data this will take no time at all.

In /etc you’ll need to edit the rsyslog.conf file. For the most basic functionality you need to make sure that RSyslog is listening. In my environment UDP for syslog is fine and I also use the standard port (514).

When you initially open the configuration file you’ll notice that UDP (and TCP as well) are commented out. This means that RSyslog is only processing local syslog events (See snippet below).

# provides UDP syslog reception
# module(load="imudp")
# input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

Since I am using UDP, I want to un-comment the module and input lines associated to that protocol (see snippet below).

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

Once I have done this, I can restart the RSyslog service and my server is now ready to receive. The restart command is below for your reference.

sudo systemctl restart rsyslog

Upon a successful restart (assuming there is a device setup to send syslogs), I should see new messages in my syslog file located at /var/log/syslog (snippet shown below).

Mar 18 06:25:02 linutil01 rsyslogd: [origin software="rsyslogd" swVersion="8.32.0" x-pid="1051" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Mar 18 15:30:01 vmhost02.sprnet.local Rhttpproxy: message repeated 2 times: [ verbose rhttpproxy[FFFC3B70] [Originator@6876 sub=Proxy Req 01543] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x1f08e318] _serverNamespace = /sdk _isRedirect = true _port = 8307]
Mar 18 15:30:01 vmhost02.sprnet.local Rhttpproxy: message repeated 2 times: [ verbose rhttpproxy[56E54B70] [Originator@6876 sub=Proxy Req 01543] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x1f08e318] _serverNamespace = /sdk _isRedirect = true _port = 8307]

In the above snippet, there are three entries. The first entry is from my local Linux box (linutil01) and the next two are from one of my VMWare ESXi hosts (vmhost02.sprnet.local).

If you have a small environment and only a handful of devices sending to your syslog server, the above configuration may be all you need. If, like me, you want something that is more organized, I think I can help.

While there are several ways of accomplishing organization in RSyslog, I am choosing to organize my logs by hostname and service name. For the purposes of this example, I have two ESXi  hosts that are logging to the server.

To start, I need to go back to the rsyslog.conf file that I wrote about earlier. I need to tell the server what logs I want to be stored and where/how I want to store them.  I can do this with a template. I’ve posted the template configuration snippet below.

# Templates
$template RemoteHost,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log"
#Remote Logging
$Ruleset remote
*.* ?RemoteHost
#Bind Ruleset to UDP Listener
$InputUDPServerBindRuleset remote
$UDPServerRun 514

In the snippet above, I have my template name RemoteHost and I have where I want to store/categorize my logs /var/log/%HOSTNAME%/%PROGRAMNAME%.log. The %HOSTNAME% and %PROGRAMNAME% sections are variables and are filled in by the servers internal syslog facilities. Note that I have logs being stored in the /var/log/ directory. If you want to store your logs here you will have to change ownership properties of the directory (snippet below) so that syslog can create and write there. I also have a Ruleset that I am binding my template to. In my configuration that ruleset is called remote. I bind my template to the remote ruleset which includes every message coming into the server on UDP port 514.

cd /var && sudo chown syslog:syslog log

Once I complete the configuration above and restart my RSyslog service, I should start seeing directory entries like the ones shown below in my /var/log/ directory.

drwxr-xr-x 2 syslog syslog 4096 Mar 16 13:56 vmhost01.sprnet.local
drwxr-xr-x 2 syslog syslog 4096 Mar 16 13:55 vmhost02.sprnet.local

If I go into one of these directories, I should see several different .log file entries (snippet shown below).

-rw-r----- 1 syslog adm 0 Mar 16 13:55 bootstop.log
-rw-r----- 1 syslog adm 10263 Mar 16 11:15 cimslp.log
-rw-r----- 1 syslog adm 97516 Mar 18 07:01 crond.log
-rw-r----- 1 syslog adm 10474 Mar 18 07:01 heartbeat.log
-rw-r----- 1 syslog adm 777730 Mar 18 07:00 hostd-probe.log
-rw-r----- 1 syslog adm 5345 Mar 18 06:01 ImageConfigManager.log
-rw-r----- 1 syslog adm 0 Mar 16 13:55 init.log
-rw-r----- 1 syslog adm 3033018 Mar 18 07:01 Rhttpproxy.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 root.log
-rw-r----- 1 syslog adm 0 Mar 16 13:55 sfcbd.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 sfcbd-watchdog.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 sfcb-ProviderManager.log
-rw-r----- 1 syslog adm 6384 Mar 18 06:36 sfcb-vmware_raw.log
-rw-r----- 1 syslog adm 16384 Mar 16 13:56 slpd.log
-rw-r----- 1 syslog adm 171223 Mar 18 06:35 smartd.log
-rw-r----- 1 syslog adm 47693 Mar 18 07:00 syslog.log
-rw-r----- 1 syslog adm 128 Mar 15 16:05 tmpwatch.log
-rw-r----- 1 syslog adm 4096 Mar 16 13:55 vmauthd.log
-rw-r----- 1 syslog adm 6189399 Mar 18 07:01 vmkernel.log
-rw-r----- 1 syslog adm 827731 Mar 18 07:00 vmkwarning.log
-rw-r----- 1 syslog adm 0 Mar 16 13:55 VMware.log
-rw-r----- 1 syslog adm 3633994 Mar 18 07:01 vobd.log
-rw-r----- 1 syslog adm 69577951 Mar 18 07:01 Vpxa.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-cdp.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-dcbd.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-net-lacp.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-net-lbt.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-nscd.log
-rw-r----- 1 syslog adm 0 Mar 16 13:55 watchdog-openwsmand.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-sdrsInjector.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-smartd.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-vobd.log
-rw-r----- 1 syslog adm 0 Mar 16 13:56 watchdog-vpxa.log

I now have an organized version of the chaos that syslog directories often hold. I can look at the systems and services that I need to without sorting through all of the other notifications that often get in the way.

I hope this has been helpful to someone, there are a lot of great resources online and mine is simply a collection of what worked for me. Thank you for reading, please ask any questions in the comments section and I’ll do my best to respond in a timely manner.

-Justin

 

 

 

 

 

 

 

 

Advertisement

The Upgrade Follies: Communications Manager 9.x to 11.x

If you’ve been in the Cisco voice game for more than a second, you’ve probably done a Call Manager upgrade or two. In my case, I lost count around version 4.1 to 4.3. My record for the longest upgrade that I was a part of is 72 hours…straight! It was painful but such was life back in those days.

With the advent of virtualized Cisco voice and all of its associated parts the upgrades have definitely improved, but that does not mean that the gotchas aren’t still lurking.

For one of my current customer projects I am upgrading a virtualized 9.x environment to 11.x. Unity Connection was upgraded first using CLI. CUC luckily doesn’t have a whole lot of gotchas assuming the engineer that built it originally used an OVA template and followed the correct steps. All you need to do is apply the new “keys” COP file (ciscocm.version3-keys.cop.sgn) and you are good to go. Maybe it isn’t quite that easy, but you get the picture. Communications Manager (Call Manager) is a far different animal. For the sake of automation, I’m doing the CM upgrade using Prime Collaboration Deployment (PCD). PCD is a separate application server that comes with your upgrade order on Cisco’s Product Upgrade Tool (PUT). PCD basically allows you to take a CM cluster (including IM&Presence) and script the upgrade so that you can basically click once and wait. If only it were that easy…

There are a couple gotchas that I’ve learned the hard way. I’ll list them below, hopefully they can help you out during your next upgrade.

  1. Licensing. If you are going to do an upgrade, do your homework. I could write an entire post on licensing but for a 9.x to 11.x upgrade it really just involves doing clean up. If you are getting that ugly little licensing warning from Cisco when you log into your system, clean it up before attempting to upgrade, you’ll save yourself from a lot of pain later.
  2. Disk Space. I get it, hard drive space is cheap, but OVAs are not future proof. The original mid-size build OVA for CM 9.x specified an 80 gig virtual drive. The 80 gig drive model is not not supported by fresh 11.x installs. What bites engineers in the ass is a pesky little storage location known as the common partition. When the 11.x upgrade script first verifies that an upgrade is possible, it checks the amount space available in that common partition, if less than 25 gigs space is available the upgrade will fail. There is an excellent Cisco Support Forums post about this failure here.  So what do you do if the above scenario is true?

There are 3 options…

  1. You can go into your server’s TFTP directory and manually remove old crap that you don’t need. If you happen to remove crap that you do need, bad things WILL happen, so keep that in mind….
  2. You can apply the following COP file ciscocm.free_common_space_v1.3.k3.cop.sgn. This little beauty will remove any software tied to the inactive partition of the system which may include software tied to the common partition, thus giving you your required space. This is a really cool idea/theory, but I’ve had mixed results.
  3. This is the scariest sounding one, but its actually not that bad. There is a second COP file that you can run. The ciscocm.vmware-disk-size-reallocation-1.0.cop.sgn; This file will allow you to resize your virtual disk in VMWare ESXi and make CM ok with it (I tend to expand 80 gig disks to 160 as long as the physical systems allow it). **Changing the size of the disk without the COP file basically guarantees you a rebuild from scratch and possibly a resume` generating event.** As I said, its not that bad, but there is a catch. In a couple of different cases, your results may vary.

1. If your CM system is running on a snapshot within ESXi, your virtual disk size adjustment option will be grayed out, this will inevitably cause you to panic and wonder if a PCD migration is your only option, its not. You can actually delete the snap shot and once you do that you should be able to change your disk space. Once you do this, restart your VM and let it go through its process. It will reboot 2 times during the start-up process as it expands the disk in the “BIOS” and then in the initial CM boot process If you’ve done it correctly you’ll see your partitions aligned and your new disk size in both the CLI and web console.

2. In some cases the allocation of your virtual disk may be as large as the blocks with in the disk controller will allow it to be. If this is the case, you have two options. Option 1 (see option 1 above). Options 2, migrate instead of upgrade using PCD. It will take more time but you and/or your customer’s data should be safe and back where it belongs.

Whew… that was a long post. I hope it helps someone. For those of you new to upgrades, you’re lucky, they keep getting easier. If you have questions, leave a comment and we can have a discussion. Thanks for reading!

 

-Justin