Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Tag: redhat

  • Mastering Auditd: A Guide to Linux Auditing

    Mastering Auditd: A Guide to Linux Auditing

    Auditd is the userland piece of the RHEL audit tool suite. When its up and running, audit messages sent by the kernel will be send to log files that you have configured. By default, only a small and limited number of messages will be picked up by Auditd; these are mostly messages related to authentication and authorization.

    Auditd has three main config files as shown below

    • /etc/sysconfig/auditd – basic configuration options
    • /etc/audit/auditd.conf – main config file
    • /etc/audit/audit.rules – auditing rules

    Auditd and Syslog:

    Its possible to send audit messages to a syslog. By setting active=yes in /etc/audisp/plugins.d/syslog.conf you can send all your audit messages to syslog. If your system is setup to log to a remote syslog server, then your audit messages will go along for the ride as well. Note that you can also send audit messages to a remote logging server via native audit protocol over TCP. I am not going to go into this option, but I want to make sure that we are aware that it exists.

     

    Looking for Audit Events in All the Wrong Places:

    Auditd includes a handy-dandy tool for searching audit logs. Ausearch. You can check out all your current audit log messages using the command below.

    [root@ip-172-31-21-28 ~]# ausearch -l

    Viewing audit logs in their raw format can be accomplished with the command below

    [root@ip-172-31-21-28 ~]# ausearch --raw

    The -a option allows you to search by audit event ids

    [root@ip-172-31-21-28 ~]# ausearch -a 282

    Auditd also includes ausearch, which allows you to get a quick summary of audit events, rather than trying to view massive audit logs. Usage and output shown below.

    root@ip-172-31-21-28 ~]# aureport
Summary Report
======================
Range of time in logs: 07/17/2014 10:21:36.438 - 07/17/2014 19:52:49.556
Selected time for report: 07/17/2014 10:21:36 - 07/17/2014 19:52:49.556
Number of changes in configuration: 4
Number of changes to accounts, groups, or roles: 24
Number of logins: 20
Number of failed logins: 4
Number of authentications: 75
Number of failed authentications: 3
Number of users: 3
Number of terminals: 18
Number of host names: 19
Number of executables: 14
Number of files: 0
Number of AVC's: 10
Number of MAC events: 20
Number of failed syscalls: 10
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 244
Number of keys: 0
Number of process IDs: 203
Number of events: 1132

    You can also use aureport and ausearch together. Simliar to the powerfull partnership between Batman and Robin, these two tools complement each other in ways that you can only imagine. Check out my sexy bits below.

    [root@ip-172-31-21-28 ~]# ausearch --start today --raw | aureport
Summary Report
======================
Range of time in logs: 07/17/2014 10:21:36.438 - 07/17/2014 20:01:01.911
Selected time for report: 07/17/2014 10:21:36 - 07/17/2014 20:01:01.911
Number of changes in configuration: 4
Number of changes to accounts, groups, or roles: 24
Number of logins: 20
Number of failed logins: 4
Number of authentications: 75
Number of failed authentications: 3
Number of users: 3
Number of terminals: 18
Number of host names: 19
Number of executables: 14
Number of files: 0
Number of AVC's: 10
Number of MAC events: 20
Number of failed syscalls: 10
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 244
Number of keys: 0
Number of process IDs: 205
Number of events: 1144

    Want to know another cool tool that is part of auditd? I know, its a lot to take in at one time, but I am sure that you can handle it. Using autrace you can trace and investigate system calls made by a process. 

    Want to see everything that nslookup is doing? Then run the command below.

    [root@ip-172-31-21-28 ~]# autrace /usr/bin/nslookup google.com

    This will output a pid for you to trace with ausearch.

    Trace complete. You can locate the records with 'ausearch -i -p 3359'

     
    Related articles

    RHEL6- Getting Up Close and Personal With Rsyslog
    Linux audit files to see who made changes to a file

  • RHEL6: Using Advanced Log File Filtering in Rsyslog

    100-8591So by default when you forward logs to a syslog/rsyslog server all the logs end up in the same file (ususally configured to go to the messages file). Sometimes one may prefer to forward logs from a particular server to a separate logfile. I know for a fact that my sometimes friends in our info-sec group prefers this setup.

    While managing such a setup for more than a handful of hosts would probably be a nightmare, its not actaually hard to setup on the most basic level.

    First create a custom filters.conf (or whatever you want to name it) in your /etc/rsyslog.d directory. Below is the file that I created.

    [root@ip-172-31-21-28 rsyslog.d]# cat filters.conf
    :fromhost, isequal, "ip-172-31-25-104.ec2.internal" /var/log/ip-172-31-25-104.ec2.internal/messages
    :fromhost, isequal, "ip-172-31-25-104.ec2.internal" ~

    Note that you will need to include two lines in your file for each host, one with the specific file/location that you want to send your filtered log messages, and a second line that directs rsyslog to discard any messages from the specified hosts once they have been logged to the specified locations. This keeps these messages from ending up in the messages file as well as the file defined in your filter file.

     
    Related articles

    RHEL6 – Introduction to IPtables, Part II
    HomeLab: Basic Syslog Configuration on Cisco Catalyst Devices
    RHEL6- Getting Up Close and Personal With Rsyslog

  • Password Protecting Grub in RHEL 6

    19552780-illustration-of-cartoon-wormGrub, is the standard boot loader used by each and every Linux type operating system that I can think of. RHEL 6 uses what I guess we are now calling grub 1.o, since grub 2.0 has been released and in use by Fedora for the last few releases. You will also find that grub 2.0 has replaced grub 1.0 in RHEL 7. At some point I plan to explore grub 2 at lenght, but today is not that day (unless something strange happens before I go to bed tonight — you never know).

     

    Anyway – I digress. Below is an excerpt from the current grub.conf that is in use by my EC2 RHEL 6.5 instance. I've made it really tiny to save space.

    # grub.conf generated by anaconda
    #
    # Note that you do not have to rerun grub after making changes to this file
    # NOTICE:  You do not have a /boot partition.  This means that
    #          all kernel and initrd paths are relative to /, eg.
    #          root (hd0)
    #          kernel /boot/vmlinuz-version console=ttyS0 ro root=/dev/vda1
    #          initrd /boot/initrd-[generic-]version.img
    #boot=/dev/vda
    default=0
    timeout=1
    splashimage=(hd0)/boot/grub/splash.xpm.gz
    hiddenmenu
    title Red Hat Enterprise Linux Server (2.6.32-431.17.1.el6.x86_64)
            root (hd0)
            kernel /boot/vmlinuz-2.6.32-431.17.1.el6.x86_64 console=ttyS0 ro root=UUID=05e7b919-2577-40a7-91fb-1ccdede87fc4 rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 xen_blkfront.sda_is_xvda=1 console=ttyS0,115200n8 console=tty0 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet
            initrd /boot/initramfs-2.6.32-431.17.1.el6.x86_64.img

    To configure a password to use for grub you need to use the grub-crypt command. as you can see in the example below, grub-crypt prompts you for your password, although you can also pass the password to grub-crypt in the inital command. Grub-crypt spits out your hashed and salted password.

    [root@ip-172-31-22-45 ~]# grub-crypt
    Password:
    Retype password:
    $6$V0qkcD4b/CouCY/L$.7vrHd.X6MZWzzDxvXdQn3avPwv2rUX80XKuS.Hvh1YqR11ZyINZszvJ/e0qH6VcsFMAVWAPWXtugHHYTzYj90

    Now stick this password in your grub.conf using the format below.

    password –encrypted $6$V0qkcD4b/CouCY/L$.7vrHd.X6MZWzzDxvXdQn3avPwv2rUX80XKuS.Hvh1YqR11ZyINZszvJ/e0qH6VcsFMAVWAPWXtugHHYTzYj90

    You have a couple of options when inserting the password into the grub.conf. You can either stick this entry near the top of the file, which protects each and every boot image that you might be configured to use, or you can stick the entry directly into a particular stanza to protect a specific boot image. For example, if you are building your servers with the option to pxe-boot from grub, or wipe the disk from grub, you might want to protect these options with a password.

    Also note that adding a password to grub does not keep an unauthorized user from being able to boot or reboot a server, rather what it does is protect the grub menu from being edited manually durring a reboot. This keeps nefarious users from being able to break the boot sequence and boot into single-user mode where they can compromise your system.

     
    Related articles

    RHEL6 – Restore Grub on MBR
    Grub
    RHEL6 – Mount: Could Not Find Any Free Loop Device

  • Fun With PAM: Working with pam_cracklib and pam_tally2

    Liberty-bell_13850_smPlugable Authentication Modules, or PAM, is the standard mechanism that most Unix and Linux Operatng Systems use for user credential authentication. By design, PAM is broken out into a number of files, each with a specific purpose. Before you can get started with PAM you need to understand a bit about how PAM configuration files are formatted.  So lets get into that first before we try to bite off anything more.

     

    PAM Config File Standards:

    PAM config files follow a standard format as shown below.

    Rule Types        Control        Module [module arguments]

     

    There are 4 Rule Types

    • auth – used to authenticate users/passwords
    • account – set properties for a user's account
    • password – controls password changes
    • session – sets and controls environmental variables

    And there are 5 Control Types

    • required – a module that a user is required to pass
    • sufficient – a user is not required to pass a sufficient module
    • optional – these modules do not have to be passed sucessfully
    • include – these rules reference other PAM config files
    • requisite – similar to required, but if failed, no further rules are checked

     

    Configuring pam_cracklib:

    Pam_cracklib is used to define password complexity. It has several module arguments that can be used to define password complexity and lenght. Its most common arguments are show below

    • ucredit – when used in the following format (ucredit=-n) requires the defined number of uppercase characters in a password
    • dcredit – when used in the following format (dcredit=-n) requires the defined number of digits in a password
    • ocredit – when used in the following format (ocredit=-n) requires the defined number of other (think symbols) type charaters in a password
    • lcredit – when used in the following format (lcredit=-n) requires the defined number of lower case letters  in a password
    • minclass – defines the minimum number of different character classes that must be present in a password.
    • minlen – defines the minimum required lenght of a password.

    Here's a usage example of the cracklib module from a /etc/pam.d/system-auth file. In this example try_first_pass tells pam to try to use any cached credentials, while retry allows a user to try their password 3 times before the fail this module.

    password    requisite     pam_cracklib.so minlen=8 ocredit=-1 ucredit=-1 try_first_pass retry=3

     

    Configuring pam_tally2:

    Pam_tally2 can be used to lock users after a defined number of failed login attempts. The example below, taken from the system-auth file will lock a user after 3 failed login attempts, will automatically unlock the user after 300 seconds, and will do so quietly, without any notification to the user.

    auth        required        pam_tally2.so deny=3 unlock_time=300 quiet

    The command pam_tally2 can be used to list users with failed logins and can also be used to reset a user's failed login count. See reset example below

    pam_tally2 –reset –u testuser

    Note that pam_tally2 deprecates the faillock module.

     

    Supplementary PAM Configuration Options:

    Want to limit a user to a particular number of concurrent ssh sessions? You can set this up in /etc/security/limits.conf if you are calling the pam_limits.so in your pam configs. Limits.conf provides the example below. Just copy the format and you are off to the races. Remember to remove the #.

    #@student        -       maxlogins       4

     

     
    Related articles

    How to enforce password complexity on Linux
    Secure Linux Servers
    You are not allowed to access to (crontab) because of pam configuration.

  • Introduction to AIDE – Advanced Intrusion Detection Environment

    Wally_Gator_PhotoEver heard of AIDE, neither had I. Apparently its a simple intrusion detection application that can be used to monitor file changes.  It can be confired to monitor permission, ownership, timestamp, or content changes.

    Lets install it. Its in the stock Redhat repos, so its a piece of cake to install via yum.

     

    [root@localhost ~]# yum -y install aide

    Once installed, you can tweak the config file (/etc/aide.conf) to your liking. The stock config is pretty robust, so I am going to trim it down a bit and just monitor /etc for permission changes, and /bin for what are defined as normal changes. Normal looks at file hashes to see if the files have been modified.

    /bin    NORMAL
    /etc    PERMS
     

    Now lets start aide

    [root@localhost ~]# aide –init

    AIDE, version 0.15.1

    ### AIDE database at /var/lib/aide/aide.db.new.gz initialized.

     

    Now this part is silly, we need to rename the database created above to the name that aide is configured to use.

    [root@localhost ~]# cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

     

    Now lets check for changes.

    [root@localhost ~]# aide –check

    AIDE, version 0.15.1

    ### All files match AIDE database. Looks okay!

    Hey no changes. Now lets monkey with something and see if aide catches it. In this example we are creating a new file in /etc. As seen below aide catches the new file and reports on it.

     

    [root@localhost ~]# touch /etc/aide.test.change
    [root@localhost ~]# aide –check
    AIDE 0.15.1 found differences between database and filesystem!!
    Start timestamp: 2014-07-15 19:51:14

    Summary:
      Total number of files:        5054
      Added files:                  1
      Removed files:                0
      Changed files:                0

    —————————————————
    Added files:
    —————————————————

    added: /etc/aide.test.change

     

    So now lets re-initialize the database, which is pretty much a snapshot.

    [root@localhost ~]# aide –init

    AIDE, version 0.15.1

    ### AIDE database at /var/lib/aide/aide.db.new.gz initialized.

     

    Don't forget to overwrite the old database.

    [root@localhost ~]# cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
    cp: overwrite ‘/var/lib/aide/aide.db.gz’? yes

    Now lets change the permissions on our test file and see if aide catches the change.  I'll spare you the suspense and let you know that aide did its job. See below.

     

    [root@localhost ~]# chmod 777 /etc/aide.test.change
    [root@localhost ~]# aide –check                    
    AIDE 0.15.1 found differences between database and filesystem!!
    Start timestamp: 2014-07-15 19:54:09

    Summary:
      Total number of files:        5054
      Added files:                  0
      Removed files:                0
      Changed files:                2

    —————————————————
    Changed files:
    —————————————————

    changed: /etc/aide.test.change
    changed: /root/.mozilla/firefox/8u03e3hs.default/sessionstore.js

    —————————————————
    Detailed information about changes:
    —————————————————

    File: /etc/aide.test.change
     Perm     : -rw-r–r–                       , -rwxrwxrwx
     ACL      : old = A:
    —-
    user::rw-
    group::r–
    other::r–
    —-
                      D: <NONE>
                new = A:
    —-
    user::rwx
    group::rwx
    other::rwx
    —-
                      D: <NONE>

    Now aide on its own is just a simple tool, but run via cron with a bit of tuning and a bit more logic behind it and I can see it being a very useful tool. Looking forward to playing with it more.
    Related articles

    How To Install Aide on a DigitalOcean VPS
    RHEL6 – How to Setup an Anonymous Download Only FTP Server

  • Finding Files with Special Permissions in Linux

    Special-k-diet-1Ok, before you even attempt to read this post, I am assuming that you not only understand standard UNIX file permissions, but that you also understand special file permissions. What are special file permissions you ask. Well you know them as setuid, setgid, and the stickbit. If you don’t know what these things are then I will give you a very brief introduction.

    Setuid – when set on a file, runs a command as root, a good example of this is the password command as shown below.

    [root@localhost ~]# ls -l /usr/bin/passwd
    -rwsr-xr-x. 1 root root 27156 Aug  3  2013 /usr/bin/passwd

    Setgid – when set on a file, elevates permissions to the group ownership of a file. When set on a directory, causes all files created in that directory to be inherit group ownership from the directory.

    Sticky Bit – when set on a directory, blocks a user from renaming or removing a file that they do not own.

    Now lets use the find command to find files on our linux box with special permissions.

    To match a special permission with a specific octal number specify the octal number as shown below. Note that this command does not return anything at all as there are no files on my system with the perms 4000.

    [root@localhost ~]# find / -perm 4000

    To match the permissions on a file using a specific prefix use the ‘-‘ switch as shown below. Its basically a wildcard search, which searches on the fields that you specified.

    [root@ip-172-31-22-45 ~]# find /bin -perm -4000 | head -n 5
    /bin/umount
    /bin/mount
    /bin/ping
    /bin/su
    /bin/ping6

    As shown below, this command returns us all files in the /bin directory with the setuid bit set. In octal 4 is setuid.

    [root@ip-172-31-22-45 ~]# ls -l /bin/ping
    -rwsr-xr-x. 1 root root 40760 Sep 17  2013 /bin/ping

    The next example returns us all files in /bin with the setgid bit set. Note that the octal value of the setgid bit is 2.

    [root@ip-172-31-22-45 ~]# find /bin -perm /2000 | head -n 3
    /bin/cgexec
    /bin/cgclassify

    See I told you so…

    [root@ip-172-31-22-45 ~]# ls -l /bin/cgexec
    -rwxr-sr-x. 1 root cgred 16384 Dec  2  2013 /bin/cgexec

    Now lets get silly and try to match two octal values at once. This time we are going to search for files where both the setuid and setgid bits are set. As we know 4 + 2 = 6, so our command is as shown below

    [root@ip-172-31-22-45 ~]# find /bin -perm -6000 | head -n 3

    Note that this command does not return anything at all. However if we modify it a bit so that it searches for files that either have the setuid or the setgid bit set. To do this we use the ‘/’ modifier as shown below in my awesome example.

    [root@ip-172-31-22-45 ~]# find /bin -perm /6000 | xargs ls -l
    -rwxr-sr-x. 1 root cgred 16352 Dec  2  2013 /bin/cgclassify
    -rwxr-sr-x. 1 root cgred 16384 Dec  2  2013 /bin/cgexec
    -rwsr-xr-x. 1 root root  77336 Apr 10 04:50 /bin/mount
    -rwsr-xr-x. 1 root root  40760 Sep 17  2013 /bin/ping
    -rwsr-xr-x. 1 root root  36488 Sep 17  2013 /bin/ping6
    -rwsr-xr-x. 1 root root  34904 Mar  5 07:21 /bin/su
    -rwsr-xr-x. 1 root root  53472 Apr 10 04:50 /bin/umount

    In this example below I am searching for files that either have the setuid (4), the setgid (2), or the stickybit (1) set.

    [root@ip-172-31-22-45 ~]# find /bin -perm /7000 | xargs ls -l
    -rwxr-sr-x. 1 root cgred 16352 Dec  2  2013 /bin/cgclassify
    -rwxr-sr-x. 1 root cgred 16384 Dec  2  2013 /bin/cgexec
    -rwsr-xr-x. 1 root root  77336 Apr 10 04:50 /bin/mount
    -rwsr-xr-x. 1 root root  40760 Sep 17  2013 /bin/ping
    -rwsr-xr-x. 1 root root  36488 Sep 17  2013 /bin/ping6
    -rwsr-xr-x. 1 root root  34904 Mar  5 07:21 /bin/su
    -rwsr-xr-x. 1 root root  53472 Apr 10 04:50 /bin/umount

    To review. when using find if you do not specify any switches or options when searching for files by permissions, then you are attempting to do a specific match for files with those exact permissions. When you use the ‘-‘ option you are searching for files that match on a specific field. When you use the ‘/’ switch you are basically doing a wild card logical OR search. Use this option when you are searching for more than one type of special permission.

    Related articles

    CVE-2014-0907 – SetUID/SetGID Programs Allow Privilege Escalation Via Insecure RPATH In IBM DB2
    setuid/setgid binaries in Debian’s Wheezy release?
    Learn octal file permissions easily with stat

  • Working With Extended Filesystem Attributes in Linux

    GarfatIn my previous post I went over standard filesystem attibutes in Linux, and how to set and view those attibutes with lsattr and chattr. You can view that post here if you are interested.

    In this post we are going to go over extended filesystem attibutes. Now there is not much to this as you are probably not going to ever have to use these settings. That being said, its not a bad thing to be aware of.

    Attribute names are strings that can be set and configured at will using the setfatr command. They can be viewed with the getfattr command. There are 4 namespaces of attibutes, security, system, trusted, and user.

    When using getfattr ( which I pronounce as getfatter) the -d option dumps only user namespace attibutes. The rest of the namespace attributes can be viewed by using the -m option along with the namespace name. In the example below you can see that there are no user namespace attibutes set on my anaconda-ks.cfg file, however there are attibutes set in the security namespace.

    [root@localhost ~]# getfattr -d anaconda-ks.cfg
    [root@localhost ~]# getfattr -d -m security anaconda-ks.cfg
    # file: anaconda-ks.cfg
    security.selinux="unconfined_u:object_r:admin_home_t:s0"

     

    Using setfattr you can define and set custom attributes. See the useless example below.

    [root@localhost ~]# setfattr -n user.example -v example anaconda-ks.cfg
    [root@localhost ~]# getfattr -d anaconda-ks.cfg
    # file: anaconda-ks.cfg
    user.example="example"

    However this could be cool if you got an md5 sum on a file and dumped it into a file attibute. You could, in theory, use this process to see if someone has messed with one of your config files.

     

    [root@localhost ~]# md5sum anaconda-ks.cfg
    fda1aa550d3cf82423d1b1ad1ae53a13  anaconda-ks.cfg
    [root@localhost ~]# setfattr -n user.md5sum -v fda1aa550d3cf82423d1b1ad1ae53a13 anaconda-ks.cfg
    [root@localhost ~]# getfattr -d anaconda-ks.cfg
    # file: anaconda-ks.cfg
    user.example="example"
    user.md5sum="fda1aa550d3cf82423d1b1ad1ae53a13"

     
    Related articles

    Why Extended Attributes are Coming to HDFS
    Advanced Filesystem Attributes in Linux
    How to create a file that root user also can't delete?