Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Tag: rhel

  • Packstack Installer Failure: “Error: Could not start Service[rabbitmq-server]: Execution of ‘/usr/bin/systemctl start rabbitmq-server’ returned 1”

    openstack

    Sitting in my hotel room today, I kept running into this error while trying to install OpenStack on a RHEL 7.1 VM running on my laptop. Digging through logs was not helping me one bit, and neither was trying to run “puppet apply” on the failing puppet manifests to see if I could get more info with which to troubleshoot.

    Below is the specific error that I was running into. Note that my RHEL VM’s IP address is 192.168.122.75. This IP address is pre-pended to the puppet module names. Your output, will obviously, vary. Note that this output is truncated.

    Applying 192.168.122.75_amqp.pp
    Applying 192.168.122.75_mariadb.pp
    192.168.122.75_amqp.pp: [ ERROR ]
    Applying Puppet manifests [ ERROR ]

    ERROR : Error appeared during Puppet run: 192.168.122.75_amqp.pp
    Error: Could not start Service[rabbitmq-server]: Execution of ‘/usr/bin/systemctl start rabbitmq-server’ returned 1: Job for rabbitmq-server.service failed. See ‘systemctl status rabbitmq-server.service’ and ‘journalctl -xn’ for details.
    You will find full trace in log /var/tmp/packstack/20150415-183003-mn6Kfx/manifests/192.168.122.75_amqp.pp.log
    Please check log file /var/tmp/packstack/20150415-183003-mn6Kfx/openstack-setup.log for more information
    Additional information:

    Each and every time, the failure occurred when the installer was trying to install/start and rabbitmq-server via the puppet module amqp.pp. Attempting to start rabbitmq manually yielded the same result.

    In this instance, I was trying to be fancy and I had given my VM the hostname packstack01.local (instead of sticking with localhost).

    [root@packstack01 20150415-183254-Kv8u6k]# hostnamectl
    Static hostname: packstack01.local
    Icon name: computer
    Chassis: n/a
    Machine ID: ca64b7fb0c9d4459a4d313dd17b19d76
    Boot ID: fc3397657ed040fca72f3d229d014b74
    Virtualization: kvm
    Kernel: Linux 3.10.0-229.1.2.el7.x86_64
    Architecture: x86_64

    Fresh out of any good ideas, I noticed that a simple nslookup on my made up hostname actually returned results. Results that I would not have expected to be valid.

    [root@packstack01 20150415-183254-Kv8u6k]# nslookup packstack01.local
    Server: 192.168.1.1
    Address: 192.168.1.1#53

    Name: packstack01.local.local
    Address: 198.105.244.104
    Name: packstack01.local.local
    Address: 198.105.254.104

    Despite never referencing my made up hostname in my answer file (by default, the answer file is generated with IP addresses only)  the Rabbitmq service was attempting to connect to itself via hostname, which obviously failed as this is a valid ip and since I was working in a hotel room without proper dns, my server was trying to connect to a machine on the opposite side of the country.

    A quick bit of tinkering in the /etc/hosts file resolved this issue, and I was able to complete my install.

    Note that there are probably many other reasons why one might run into this error during an OpenStack install via Packstack, however this is the one that I ran into, and thankfully it was easy to fix.

    Note to self – always use localhost when working without a valid DNS entry.

  • Redhat Satellite 5: How to Clone Security Errata to a Software Channel

    space_dogFirst check to see if the errata is available to your local satellite server. To accomplish this log into your organizations satellite server and click on the “Errata” tab. Then on the left side of the page click on “Advanced Search”.

    In the search box enter the RHSA number (Redhat Security Advisory Number) for the errata that you want to clone/update. In this example I am searching for RHSA-2014:1924, which is a Thunderbird security update.

    If your search does not return any results, you will need to manually sync your local Satellite Server with Redhat.To accomplish this you need to ssh into your local satellite server and run the command shown below. Note that this does not update any packages/errata. This does update the list of availbile packages/errata.

    /usr/bin/satellite-sync
    [root@myserver ~]# satellite-sync –email
    10:08:09 Red Hat Satellite – live synchronization
    10:08:09 url: https://satellite.rhn.redhat.com
    10:08:09 debug/output level: 1
    ….truncated….

    Once you are able to locate the specific fix in via “Erratum Search” you may proceed to the next step. In this example, as I stated above, I am searching for RHSA-2014:1924.

    clone_erratta

    Now that our local Satellite server is aware of our specific errata, click on “Clone Errata” on the left side of the page. Search the page “Errata Management” for the specific fix that you want to apply. Note that the “Errata Management” page does have built in search functionality — don’t ask me why — so you must search using your browser’s own page search function.

    clone_thunderbird

    Once you have located the correct Security Advisory, put a check in the box and spend about 5 minutes scrolling down to the bottom of the page. Stop when your arm is tired, or once you locate the “Clone Errata” button. Obviously you want to click this.

    Note that your newly added and updated errata/package may not become immediatley availible to install. You nay need to run the following commands to refresh/reload your repos.

    #yum clean all

    Then check for updates with the command below.

    #yum check-update

  • Enabling Remote Command Execution in Redhat Satellite

    9i4b57pkTAs many of you know, a very useful feature of Redhat Satellite is the ability to execute remote commands on a set of servers. You pick a set of servers and plug in your command and schedule the job. Most of the time these remote commands run just fine, however they can error out if the server is not configured to allow remote command execution. Let’s discuss how to fix this.

    If remote command execution is not allowed, you will probably see an error similar to those shown below.

    Local permission not set for action type script.run

    or

    Invalid function call attempted

    Note that these two error messages usually mean the same thing. You need to enable remote command execution on the remote server.

    First off you need to make sure that the command rhn-actions-control is installed. If it is not then we first figure out which rpm provides the command. You can do so with the command below

    # yum whatprovides */rhn-actions-control

    Now that we know what to install, lets install it.

    # yum -y install rhncfg-actions

    Now we enable remote command execution

    # rhn-actions-control --enable-all

    Now you should be able to reschedule your job against the server via the webUI.

  • RHEL6: Cool PAM Tricks – Logging Terminal Keystokes

    Pam-original-cooking-spray-72450This is a neat and very useful trick that I learned today. Lets say that you want to be able to monitor and log all keystrokes that are typed as root. This is particularly useful as normally you can only log when a user uses sudo to run a command. If the user has the abilty to become root however,  then they have effectively eluded yourattempts to track their activity. Like Thomas Magnum shaking a tail, they are free to scoot around your island with the top down.

    So how do you stop this from occuring? How to you log all activity and keystrokes made by root without implementing a bloated 3rd party software that will probably cost and arm and a leg? You use PAM you dingbat.
     

    The secret sauce in this security burrito is the pam_tty_audit.so module. Here is how to use it,

    Below is my stock /etc/pam.d/system-auth file

    #%PAM-1.0
    # This file is auto-generated.
    # User changes will be destroyed the next time authconfig is run.
    auth        required      pam_env.so
    auth        sufficient    pam_fprintd.so
    auth        sufficient    pam_unix.so nullok try_first_pass
    auth        requisite     pam_succeed_if.so uid >= 500 quiet
    auth        required      pam_deny.so

    account     required      pam_unix.so
    account     sufficient    pam_localuser.so
    account     sufficient    pam_succeed_if.so uid < 500 quiet
    account     required      pam_permit.so

    password    requisite     pam_cracklib.so try_first_pass retry=3 type=
    password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
    password    required      pam_deny.so

    session     optional      pam_keyinit.so revoke
    session     required      pam_limits.so
    session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
    session     required      pam_unix.so

    Now look above and then look below at my modified system-auth file. Note the additonal session entry for pam_tty_audit.so.

    [root@ip-172-31-21-28 pam.d]# cat system-auth
    #%PAM-1.0
    # This file is auto-generated.
    # User changes will be destroyed the next time authconfig is run.
    auth        required      pam_env.so
    auth        sufficient    pam_fprintd.so
    auth        sufficient    pam_unix.so nullok try_first_pass
    auth        requisite     pam_succeed_if.so uid >= 500 quiet
    auth        required      pam_deny.so

    account     required      pam_unix.so
    account     sufficient    pam_localuser.so
    account     sufficient    pam_succeed_if.so uid < 500 quiet
    account     required      pam_permit.so

    password    requisite     pam_cracklib.so try_first_pass retry=3 type=
    password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
    password    required      pam_deny.so

    session     optional      pam_keyinit.so revoke
    session     required      pam_tty_audit.so enable=root
    session     required      pam_limits.so
    session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
    session     required      pam_unix.so

    Note that you will need to add the pam_tty_audit.so call to your /etc/pam.d/password-auth as well to ensure that you capture all of root's keystrokes, no matter how they log in.

     

    Related articles

    Much Todo About Linux/RHEL Passwords
    RHEL6 – How to Manually Logout of an Iscsi Disk
    Fun With PAM: Working with pam_cracklib and pam_tally2

  • Mastering Auditd: A Guide to Linux Auditing

    Mastering Auditd: A Guide to Linux Auditing

    Auditd is the userland piece of the RHEL audit tool suite. When its up and running, audit messages sent by the kernel will be send to log files that you have configured. By default, only a small and limited number of messages will be picked up by Auditd; these are mostly messages related to authentication and authorization.

    Auditd has three main config files as shown below

    • /etc/sysconfig/auditd – basic configuration options
    • /etc/audit/auditd.conf – main config file
    • /etc/audit/audit.rules – auditing rules

    Auditd and Syslog:

    Its possible to send audit messages to a syslog. By setting active=yes in /etc/audisp/plugins.d/syslog.conf you can send all your audit messages to syslog. If your system is setup to log to a remote syslog server, then your audit messages will go along for the ride as well. Note that you can also send audit messages to a remote logging server via native audit protocol over TCP. I am not going to go into this option, but I want to make sure that we are aware that it exists.

     

    Looking for Audit Events in All the Wrong Places:

    Auditd includes a handy-dandy tool for searching audit logs. Ausearch. You can check out all your current audit log messages using the command below.

    [root@ip-172-31-21-28 ~]# ausearch -l

    Viewing audit logs in their raw format can be accomplished with the command below

    [root@ip-172-31-21-28 ~]# ausearch --raw

    The -a option allows you to search by audit event ids

    [root@ip-172-31-21-28 ~]# ausearch -a 282

    Auditd also includes ausearch, which allows you to get a quick summary of audit events, rather than trying to view massive audit logs. Usage and output shown below.

    root@ip-172-31-21-28 ~]# aureport
Summary Report
======================
Range of time in logs: 07/17/2014 10:21:36.438 - 07/17/2014 19:52:49.556
Selected time for report: 07/17/2014 10:21:36 - 07/17/2014 19:52:49.556
Number of changes in configuration: 4
Number of changes to accounts, groups, or roles: 24
Number of logins: 20
Number of failed logins: 4
Number of authentications: 75
Number of failed authentications: 3
Number of users: 3
Number of terminals: 18
Number of host names: 19
Number of executables: 14
Number of files: 0
Number of AVC's: 10
Number of MAC events: 20
Number of failed syscalls: 10
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 244
Number of keys: 0
Number of process IDs: 203
Number of events: 1132

    You can also use aureport and ausearch together. Simliar to the powerfull partnership between Batman and Robin, these two tools complement each other in ways that you can only imagine. Check out my sexy bits below.

    [root@ip-172-31-21-28 ~]# ausearch --start today --raw | aureport
Summary Report
======================
Range of time in logs: 07/17/2014 10:21:36.438 - 07/17/2014 20:01:01.911
Selected time for report: 07/17/2014 10:21:36 - 07/17/2014 20:01:01.911
Number of changes in configuration: 4
Number of changes to accounts, groups, or roles: 24
Number of logins: 20
Number of failed logins: 4
Number of authentications: 75
Number of failed authentications: 3
Number of users: 3
Number of terminals: 18
Number of host names: 19
Number of executables: 14
Number of files: 0
Number of AVC's: 10
Number of MAC events: 20
Number of failed syscalls: 10
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 244
Number of keys: 0
Number of process IDs: 205
Number of events: 1144

    Want to know another cool tool that is part of auditd? I know, its a lot to take in at one time, but I am sure that you can handle it. Using autrace you can trace and investigate system calls made by a process. 

    Want to see everything that nslookup is doing? Then run the command below.

    [root@ip-172-31-21-28 ~]# autrace /usr/bin/nslookup google.com

    This will output a pid for you to trace with ausearch.

    Trace complete. You can locate the records with 'ausearch -i -p 3359'

     
    Related articles

    RHEL6- Getting Up Close and Personal With Rsyslog
    Linux audit files to see who made changes to a file

  • RHEL6: Using Advanced Log File Filtering in Rsyslog

    100-8591So by default when you forward logs to a syslog/rsyslog server all the logs end up in the same file (ususally configured to go to the messages file). Sometimes one may prefer to forward logs from a particular server to a separate logfile. I know for a fact that my sometimes friends in our info-sec group prefers this setup.

    While managing such a setup for more than a handful of hosts would probably be a nightmare, its not actaually hard to setup on the most basic level.

    First create a custom filters.conf (or whatever you want to name it) in your /etc/rsyslog.d directory. Below is the file that I created.

    [root@ip-172-31-21-28 rsyslog.d]# cat filters.conf
    :fromhost, isequal, "ip-172-31-25-104.ec2.internal" /var/log/ip-172-31-25-104.ec2.internal/messages
    :fromhost, isequal, "ip-172-31-25-104.ec2.internal" ~

    Note that you will need to include two lines in your file for each host, one with the specific file/location that you want to send your filtered log messages, and a second line that directs rsyslog to discard any messages from the specified hosts once they have been logged to the specified locations. This keeps these messages from ending up in the messages file as well as the file defined in your filter file.

     
    Related articles

    RHEL6 – Introduction to IPtables, Part II
    HomeLab: Basic Syslog Configuration on Cisco Catalyst Devices
    RHEL6- Getting Up Close and Personal With Rsyslog

  • Disabling Ctrl+Alt+Delete in RHEL 6

    Ctrl-Alt-Del-Cup-Set

    Let me start off by saying that I am not a fan of disabling ctrl+alt+delete, especially if you do not have physical access to a server.

    Sometimes the old three finger salute is the best and quickest method to reboot a locked and unresponsive Operating System.

    Regardless of this fact, some Info Sec folks think that disabling this functionality is a good idea for security.

    So for their sake, I will show you how to do it.

    First ssh into your server and become root and change directory to what is shown below.

    [root@ip-172-31-22-45 ~]# cd /etc/init

    Now copy the file below exactly as I have illustrated. You do not want to try to modify the original file as it could be overwritten by Upstart.

    [root@ip-172-31-22-45 init]# cp control-alt-delete.conf control-alt-delete.override

    Now modify the line below and change it to run a command that will exit with an exit code of 0.

    exec /sbin/shutdown -r now "Control-Alt-Delete pressed"

    Something like /bin/true will work, although I am sure that you can come up with more creative ideas.

    exec /bin/true

    Now try ctrl+atl+delete, nothing should happen.