Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Tag: rhel

  • RHCE Exam Objectives

    TCBND00ZWant to become an Red Hat Certified Engineer and ride around in trains all the live-long day. Well, you best get to studying and the exam is notoriously hard with a very agressive time limit. Bottom line, you walk in the door and sit down at the desk you better have a good idea of whats on the test, and your better be prepared to work fast.

    Anyway, below is the link to the exam outline, courtesy of Redhat.

    http://www.redhat.com/training/courses/ex300/examobjective

    One item of note, if you want cannot become a RHCE until you are a RHCSA, however you can pass the RHCE and then take the RHCSA. You do not have to pass them in order.

    Link below to the RHCSA exam outline, also from Redhat.

    http://www.redhat.com/training/courses/ex200/examobjective

    The links above will open in a new page, and are guaranteed to bring you pain and suffering.

     

  • Broadcom (bnx2) Network Adapters Dropping Recieved Packets Under Linux

    VampSo a few weeks ago some of our Centos 5.4 and OEL 5.5 servers started exibiting strange connectivity problems. Monitoring started alerting that hosts were down when they weren't; some boxes could ping target hosts and some couldn't; some boxes became unresponsive when interfaces were failed over, and the strangest of all is that some of the boxes would magically "repair" themselves. Like I said, strange.

    Over the next week or so we ran into the issue a few more times and were able to see a pattern emerge. All the affected servers were running Centos 5.4 or Oracle Linux 5.5 and had broadcom (bnx2) adapters that were on the recieving end of some pretty decent traffic. Most importantly, all had a good number of dropped recieved packets that was continuously, albeit slowly, increasing.

    A bit of google research led us to this bugzilla, which suggested changing the adapter's coalescense settings…. so a bit on coalescense.

    Coalescense

    In your network adapter, coalescence is all about interupts. Traditionally interupt coalescense (or IC) is used to reduce the number of interupts generated by the system by delaying the generating of an interrupt by a very short period of time…think less then a milisecond. In turn more traffic will be recieved by the host and the next interupt generated will be larger in size. You can find out more than you would ever want to know about coalesence here

    The Fix

    So apparently the Broadcom IC settings were not aggressive enough. Packets would come in, fill up the receive queue, and get dropped before they could be sent off for processing via an interrupt. This takes us back to the bugzilla above and the suggested settings below which you set with the ethtool command

     ethtool -C ethX rx-usecs 8 rx-usecs-irq 8 rx-frames 0 rx-frames-irq 0

    Note that this was not an issue on any in Centos 5.6, any server with Intel adapters, or any server with 10g adapters. As a matter of fact, those servers had IC settings even more agressive then those above. See the Intel 82599EB 10-gigabit settings below

    rx-usecs: 1
    rx-frames: 0
    rx-usecs-irq: 0
    rx-frames-irq: 0

    Final Configuration

    So now that we know the fix we need to make it permanent, which is not as easy as editing a config file for the device as the coalescence config is set at boot and it part of the installed driver for the device. Rather than muck around with trying to modify the driver itself, we decided to set and configure our devices at boottime with a rc script that checks the checks the each network interface on the box and modifys their IC settings if they are using the bnx2 (Broadcom) driver.  We dropped the script below into /etc/rc.d and created a symbolic link to it in /etc/rc3.d.

    #!/bin/bash

    case "$1" in
    start)

    IFACE=$(ls /etc/sysconfig/network-scripts/ifcfg-eth*| grep -v bak | cut -d – -f 3)

    for ETH in $IFACE
            do
                    if ( ethtool -i $ETH | grep -qw bnx2 )
                    then
                            echo "$Changing Settings for $ETH"
                            ethtool -C $ETH rx-usecs 8 rx-usecs-irq 8 rx-frames 0 rx-frames-irq 0
                    else
                            echo "$ETH is not a broadcom"

                    fi
            done
    exit 1
    ;;

    stop)

    echo " hammer time"
    ;;

    *)
        echo "usage: $0 (start|stop)"
    ;;
    esac

  • Poor Man’s eSATA Drive Hot Swap without AHCI or Hotplug Support Under Linux

     

     

    ServerRoomFireWeb

    Unfortunately, hot-swapping an eSATA drive is a bit more complicated than hot swapping a USB drive.

    First off,  your BIOS needs to support AHCI (click here for more info on AHCI), and your SATA controller also needs to support it as well. Secondly your OS, needs to specifically support hot plug, and in the case of Windows 7, it wont boot if you change to AHCI after the OS has been installed.

    So, In my case I need to update firmware on lots of SATA SSDs and want to do so without rebooting, and without worrying about changing bios settings. So in order to keep things simple, I followed the procedure below.

    First, you need to detect your drive. So watch dmesg to see what drive letter is assigned to your new disk upon initial connection.

    #dmesg

    [86527.985994]  sdd: unknown partition table
    [86528.012820] sd 8:0:0:0: [sdd] Assuming drive cache: write through
    [86528.012823] sd 8:0:0:0: [sdd] Attached SCSI disk
    [86528.456281] device label btrfs devid 1 transid 11 /dev/sdd

    Then, when its time to remove the disk device to the following. Subsitute your disk device letters.

    # echo 1 > /sys/block/sdd/device/delete

    Now you are free to swap your disk. No reboot, no bios changes, required.

  • RHEL6 – Simple Iptables How To

    Firewall supportYour mother and I were talking last night about how important it is to properly configure Iptables, and how despite that fact, many just choose to disable it. So today we are going to discuss iptables.

    Overview:

    By far the easiest way to setup a simple firewall using Iptables is to use system-config-firewall, or system-config-firewall-tui. I prefer this method as iptables can be a bit confusing on the command line and in its config file (/etc/sysconfig/iptables) is not exactly user friendly. At the very least you can create a basic set of rules and then customize by hand. Lets take a look at the file in its default form on my RHEL 6 box.

    But before we do that, lets review a couple of terms that we need to know.

    1. INPUT – are inbound packets
    2. OUTPUT are outbound packets
    3. FORWARD – packets from another machine that the firewall should forward (like to a vm on the host).
    4. ACCEPT – the packet is accepted
    5. DROP – the packet is dropped as if it never existed
    6. REJECT – the packed is rejected and and error message is returned to sender
    7. RULE – the basic building block — tells the firewall what to do with a packet
    8. CHAIN – a list of all rules which will be checked in order from first to last
    9. POLICY – the default action, like accept, drip, reject, forward

    Now that you have memorized the list above, here is my /etc/sysconfig/iptables.

    # Firewall configuration written by system-config-firewall
    # Manual customization of this file is not recommended.
    *filter
    :INPUT ACCEPT [0:0]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
    -A INPUT -p icmp -j ACCEPT
    -A INPUT -i lo -j ACCEPT
    -A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j ACCEPT
    -A INPUT -j REJECT –reject-with icmp-host-prohibited
    -A FORWARD -j REJECT –reject-with icmp-host-prohibited.

    Now lets run system-config-firewall tui and enable apache and ftp, plus we want to configure our box to respond to ICMP ping requests. This process is pretty self explanitory once you start.

    Once that is done lets view /etc/sysconfig/iptables again.

    # Firewall configuration written by system-config-firewall
    # Manual customization of this file is not recommended.
    *filter
    :INPUT ACCEPT [0:0]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
    -A INPUT -p icmp -m icmp –icmp-type echo-request -j REJECT –reject-with icmp-host-prohibited
    -A INPUT -p icmp -m icmp –icmp-type echo-reply -j REJECT –reject-with icmp-host-prohibited
    -A INPUT -p icmp -m icmp –icmp-type destination-unreachable -j REJECT –reject-with icmp-host-prohibited
    -A INPUT -p icmp -j ACCEPT
    -A INPUT -i lo -j ACCEPT
    -A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j ACCEPT
    -A INPUT -m state –state NEW -m tcp -p tcp –dport 80 -j ACCEPT
    -A INPUT -m state –state NEW -m tcp -p tcp –dport 21 -j ACCEPT
    -A INPUT -m state –state NEW -m tcp -p tcp –dport 443 -j ACCEPT
    -A INPUT -j REJECT –reject-with icmp-host-prohibited
    -A FORWARD -j REJECT –reject-with icmp-host-prohibited
    COMMIT

    Iptables Command:

    The iptables command can be used in several different ways.

    List the current rules in use, similar to viewing the /etc/sysconfig/iptables file

    #iptables -L

    To set a default policy use iptables -P, in the example below we are setting the default INPUT policy to DROP.

    #iptables -P INPUT DROP

    Now lets say we want to delete all our existing rules, note that i did not say policy

    #iptables -F

    To add a rule use iptables -a, for example lets say you have a default policy of INPUT DROP but we want to accept all established and related packets. Note that -m must be used when adding rules to a chain as it forces modprobe to load any necessary modules.

    #iptables -A INPUT -m state –state ESTABLISHED, RELATED

    Now lets say that we want to reject all packets from 192.168.10.10. Note -j specifies the action that the rule is to take — in the case below, REJECT

    #iptables -A INPUT -s 192.168.10.10 -j REJECT

    Now lets say we want to ACCEPT all ICMP traffic from our local subnet. The -p is protocol

    #iptables -A INPUT -p ICMP -s 192.168.1.0/24 -j ACCEPT

    Please note that under RHEL you can use following commands to save firewall rules.Make sure that you do this before you restart iptables.

    #service iptables save

  • RHEL6 — Troubleshooting SELinux Violations

    Sad_face1Dear Reader: Welcome to my third and not final installment on SELinux. The first two can be read here and here. They are exciting reads and are sure to have you on the edge of your seat.

    Anyway, the best way to implement SELinux sucessfully is to know how to troubleshoot when things aren’t going your way. If you panic at the first sign of trouble, you are just going to end up turning off SELinux and not reap the rich rewards that it will bring you in life. Now that I have convinced you to run SELinux lets get started.

    First install the package setroubleshoot, which will send SELinux messages to our messages file.

    yum -y install setroubleshoot-server.x86_64

    Now you can search the messages file for SELinux Violations. Use sealert -l UUID to find information on a specific incident, or sealert -a  /var/log/audit.log to search an entire log file for violations.

    In this specfic example, I created a test file and dropped it in /var/www/html, however I did not set the context to httpd_sys_content_t, then i attempted to view the file in a browser. Obviously access was denied. The output of sealert shows me the error and then tells me how to fix it.

    Summary:

    SELinux is preventing /usr/sbin/httpd “getattr” access to /var/www/html/file3.

    Detailed Description:

    SELinux denied access requested by httpd. /var/www/html/file3 may be a
    mislabeled. /var/www/html/file3 default SELinux type is httpd_sys_content_t, but
    its current type is admin_home_t. Changing this file back to the default type,
    may fix your problem.

    …TRUNCATED…

    Allowing Access:

    You can restore the default system context to this file by executing the
    restorecon command. restorecon ‘/var/www/html/file3’, if this file is a
    directory, you can recursively restore using restorecon -R
    ‘/var/www/html/file3’.

    Fix Command:

    /sbin/restorecon ‘/var/www/html/file3’

    Boom goes the dynomite! Problem solved.

  • RHEL6 – SELinux Modes and Contexts

    Anime,art,cartoon,chibi,darth,vader,darthvader-26d94e794e11142d417a277b8fbdc0eb_hSELinux, or Security-Enhanced Linux as its known by the guy who invented it, is a Linux feature that provides an additional level of security by setting rules for which processes can access which files, directories, ports, etc.

    Display and Modify SELinux Modes:

    You can use /etc/sysconfig/selinux to change the default SELinux mode at boot, and the setenforce command can be used to change the default level on the fly. Getenforce can be used to determine the current SELinux mode. 

    # getenforce
    Enforcing

    Display SELinux Contexts:

    Under SELinux, every file, process, directory, or port is assigned a special security label called a context.

    To view the contexts assigned to a file or directory use the '-Z' option. Coupled with and 'ls" or a 'ps" this is a formidable command.

    To view a list of all possible assigned contexts use semanage.

    # semanage fcontext -l

    Modify SELinux Contexts:

    For example, lets create two test files in /tmp called testfile1 and testfile2, and then lets check their contexts.

    # ls -lZ test*
    -rw-r–r–. root root unconfined_u:object_r:user_tmp_t:s0 testfile1
    -rw-r–r–. root root unconfined_u:object_r:user_tmp_t:s0 testfile2

    Now compare this to the default context assigned to apache content

    # ls -ldZ /var/www/html
    drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 /var/www/html

    So lets say that I want to move testfile1 to /var/www/html and make it accessable via a webbrowser; in order to do this I must assign the correct context to the file using the restorecon command.

    # restorecon -Rv /var/www/html

    The command above restores or even better, allows the testfiles to inherit the contexts assigned to the parent directory, which in this case is /var/www/html/

    Add SELinux Contexts:

    Now lets say that you need to add a directory and apply a context directly to that directory, instead of allowing a context to be inheritted. For example, lets say that I need to setup an apache virtual server under /virtual_server2, so lets first create the directory and a simple index.html

    # mkdir /virtual_server2
    # vi /virtual_server2/index.html
    # ls -Zd /virtual_server2
    drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 /virtual_server2

    # ls -Zd /virtual_server2/index.html
    -rw-r–r–. root root unconfined_u:object_r:default_t:s0 /virtual_server2/index.html

    Now we need to set and apply the correct http content context to /virtual_server2 and its contents (say that 5 times fast)

    # semanage fcontext -a -f "" -t httpd_sys_content_t '/virtual_server2(/.*)?'

    # restorecon -RFvvv /virtual_server2/

    restorecon reset /virtual_server2 context unconfined_u:object_r:default_t:s0->system_u:object_r:httpd_sys_content_t:s0
    restorecon reset /virtual_server2/index.html context unconfined_u:object_r:default_t:s0->system_u:object_r:httpd_sys_content_t:s0

    Now that our context is correct apache should have no issue serving up our new content (once configured on the apache side)

    # ls -Zd /virtual_server2/
    drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 /virtual_server2/

     

     

     

     

     

     

  • RHEL6 – Managing Swap Space

    54af9-6a00e551c39e1c8834017ee46a8c54970d-piSwap space on a Linux box is an area on disk that is used to hold inactive memory pages. This occurs when the system needs more memory then is currently available, so it swaps these inactive memory pages to disk.

    To create additional swap space on the fly you are either going to need a spare disk or free partition on a disk that you can use.

    First, using fdisk,  you will need to make sure that the partition type for the disk (or partition) is set to 82.

    Then setup the swap area using mkswap. In this example I am using /dev/sdb2, but your setup is bound to be different.

    >mkswap /dev/sdb2

    Then determine the UUID of the new swap space.

    >blkid /dev/sdb2

    Then add an entry to the /etc/fstab, so that the swap space is mounted at boottime. The show in the example below is the output of the blkid command above.

    >UUID=7b05f0a9-18d5-42e5-b259-78ba3a8cc1b7 swap                    swap    defaults        0 0

    Then activate your new swap space

    >swapon -a

    Then check to make sure everything worked by checkign for your new swap partition in the output of the command below.

    >swapon -s

    /dev/sdb2                             partition       4193276 7480    0