Tag: centos

  • Broadcom (bnx2) Network Adapters Dropping Recieved Packets Under Linux

    VampSo a few weeks ago some of our Centos 5.4 and OEL 5.5 servers started exibiting strange connectivity problems. Monitoring started alerting that hosts were down when they weren't; some boxes could ping target hosts and some couldn't; some boxes became unresponsive when interfaces were failed over, and the strangest of all is that some of the boxes would magically "repair" themselves. Like I said, strange.

    Over the next week or so we ran into the issue a few more times and were able to see a pattern emerge. All the affected servers were running Centos 5.4 or Oracle Linux 5.5 and had broadcom (bnx2) adapters that were on the recieving end of some pretty decent traffic. Most importantly, all had a good number of dropped recieved packets that was continuously, albeit slowly, increasing.

    A bit of google research led us to this bugzilla, which suggested changing the adapter's coalescense settings…. so a bit on coalescense.

    Coalescense

    In your network adapter, coalescence is all about interupts. Traditionally interupt coalescense (or IC) is used to reduce the number of interupts generated by the system by delaying the generating of an interrupt by a very short period of time…think less then a milisecond. In turn more traffic will be recieved by the host and the next interupt generated will be larger in size. You can find out more than you would ever want to know about coalesence here

    The Fix

    So apparently the Broadcom IC settings were not aggressive enough. Packets would come in, fill up the receive queue, and get dropped before they could be sent off for processing via an interrupt. This takes us back to the bugzilla above and the suggested settings below which you set with the ethtool command

     ethtool -C ethX rx-usecs 8 rx-usecs-irq 8 rx-frames 0 rx-frames-irq 0

    Note that this was not an issue on any in Centos 5.6, any server with Intel adapters, or any server with 10g adapters. As a matter of fact, those servers had IC settings even more agressive then those above. See the Intel 82599EB 10-gigabit settings below

    rx-usecs: 1
    rx-frames: 0
    rx-usecs-irq: 0
    rx-frames-irq: 0

    Final Configuration

    So now that we know the fix we need to make it permanent, which is not as easy as editing a config file for the device as the coalescence config is set at boot and it part of the installed driver for the device. Rather than muck around with trying to modify the driver itself, we decided to set and configure our devices at boottime with a rc script that checks the checks the each network interface on the box and modifys their IC settings if they are using the bnx2 (Broadcom) driver.  We dropped the script below into /etc/rc.d and created a symbolic link to it in /etc/rc3.d.

    #!/bin/bash

    case "$1" in
    start)

    IFACE=$(ls /etc/sysconfig/network-scripts/ifcfg-eth*| grep -v bak | cut -d – -f 3)

    for ETH in $IFACE
            do
                    if ( ethtool -i $ETH | grep -qw bnx2 )
                    then
                            echo "$Changing Settings for $ETH"
                            ethtool -C $ETH rx-usecs 8 rx-usecs-irq 8 rx-frames 0 rx-frames-irq 0
                    else
                            echo "$ETH is not a broadcom"

                    fi
            done
    exit 1
    ;;

    stop)

    echo " hammer time"
    ;;

    *)
        echo "usage: $0 (start|stop)"
    ;;
    esac

  • Poor Man’s eSATA Drive Hot Swap without AHCI or Hotplug Support Under Linux

     

     

    ServerRoomFireWeb

    Unfortunately, hot-swapping an eSATA drive is a bit more complicated than hot swapping a USB drive.

    First off,  your BIOS needs to support AHCI (click here for more info on AHCI), and your SATA controller also needs to support it as well. Secondly your OS, needs to specifically support hot plug, and in the case of Windows 7, it wont boot if you change to AHCI after the OS has been installed.

    So, In my case I need to update firmware on lots of SATA SSDs and want to do so without rebooting, and without worrying about changing bios settings. So in order to keep things simple, I followed the procedure below.

    First, you need to detect your drive. So watch dmesg to see what drive letter is assigned to your new disk upon initial connection.

    #dmesg

    [86527.985994]  sdd: unknown partition table
    [86528.012820] sd 8:0:0:0: [sdd] Assuming drive cache: write through
    [86528.012823] sd 8:0:0:0: [sdd] Attached SCSI disk
    [86528.456281] device label btrfs devid 1 transid 11 /dev/sdd

    Then, when its time to remove the disk device to the following. Subsitute your disk device letters.

    # echo 1 > /sys/block/sdd/device/delete

    Now you are free to swap your disk. No reboot, no bios changes, required.

  • RHEL6 – How to Setup an Anonymous FTP Server


    tow-truck-driver-cartoon-character-final-coghillToday on the fatmin we are going to setup an ftp server on RHEL6 that accepts anonymous uploads. We are going to do so with SELinux support and will be making modifications to iptables as well. Sounds fun, right?

    Installation:

    First and formost we need to install vsftpd

    # yum -y install vsftpd && service vsftpd start && chkconfig vsftpd on

    Our anonymous upload directory will be /var/ftp/anon, and we need to change group ownership to the ftp group and then change permissions so that the members of that group can write to it. Note that no one other than root can read or execute anything under /var/ftp/anon.

    # chgrp ftp /var/ftp/anon
    # chmod 730 /var/ftp/anon
    # ls -ld /var/ftp/anon
    drwx-wx—. 3 root ftp 4096 Oct 19 13:34 /v1

    SELinux Support:

    Next we need to configure SELinux support and assign the correct context to the /v1 directory and its future contents. Note -a is add -t is type.

    # semanage fcontext -a -t public_content_rw_t ‘/var/ftp/anon(/.*)’

    Now lets go ahead and apply the new context. Note -vv is verbose, -F force and R is recursive

    # restorecon -vvFR /var/ftp/anon

    Now we need to get and set the allow_ftpd_anon_write boolean

    # setsebool -P allow_ftpd_anon_write=1

    Now lets check to make sure the setting “stuck”.

    # getsebool -a | grep allow_ftpd_anon_write
    allow_ftpd_anon_write –> on

    Configure Vsftpd:

    Now vi /etc/vsftpd/vsftpd.conf and ensure that the following configuration values are set and un-commented. Note that I had to add the last line to my config file.

    anonymous_enable=YES

    anon_upload_enable=YES

    chown_uploads=YES

    chown_username=daemon

    anon_umask=077

    Configure iptables:

    Add the following to /etc/sysconfig/iptables-config. In my case I only needed to add the ip_nat_ftp part to the line

    IPTABLES_MODULES=”nf_conntrack_ftp ip_nat_ftp”

    Now you are going to want to make sure that these two lines exist in /etc/sysconfig/iptables.

    -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
    -A INPUT -p tcp –dport 21 -j ACCEPT

    Now restart iptables

    Addendum:

    Note that I ran into issues with the semanage command below.

    # semanage fcontext -a -t public_content_rw_t ‘/var/ftp/anon(/.*)’

    It seems that the context assigned to the /var/ftp/anon directory was not changing correctly from public_content_t to public_content_rw_t.

    # ls -Zd /var/ftp
    drwxrwxrwx. root root system_u:object_r:public_content_t:s0 /var/ftp

    However when I checked the file_contexts file all looked correct.

    # cat /etc/selinux/targeted/contexts/files/file_contexts.local/var/ftp/anon(/.*)    system_u:object_r:public_content_rw_t:s0

    So I ran the chon command seen below and did not run the restorecon command. This worked as afterwards the context on the directory /var/ftp/anon was correct.

    # chcon -R -t public_content_rw_t /var/ftp/anon

     

  • RHEL6 – How to Setup a Caching-Only DNS Server


    26868_1First off lets get this straight, all DNS Servers cache. However, some DNS Servers intended to only provide the caching function. Which is what we are going to configure today.

    A Caching-only DNS server does not contain zone information or a zone database. Its cache only contains information based on the results of queries that it has already performed. In this case, the cache takes the place of the zone database file for the lookups that you are already doing.

    Here’s how its done

    First step you need to install bind via yum.

    # yum install bind && chkconfig bind

    Now configure named to start at boot and start it up

    # chkconfig named on && service named start

    Then modify /etc/named.conf and change these two lines

    listen-on port 53 { 127.0.0.1; };
    listen-on-v6 port 53 { ::1; };

    to this

    listen-on port 53 { any; };
    listen-on-v6 port 53 { any; };

    then change this the dnssec-validation line to no as seen below.

    dnssec-validation no;

    Then configure named to accept lookups from its local network by changing the line below

    allow-query     { localhost; };

    To what you see below, note that 10.1.224.0 is my local network

    allow-query     { localhost; 10.1.224.0/24; };

    Now don’t forget to insert a forwarders entry to forward requests to your local dns server. Look in your resolv.conf for this info.

    forwarders { 10.100.4.16; };

    Then modify your /etc/resolv.conf so that your machine uses itself for name lookups, you do this by adding the line below as the first nameserver.

    nameserver localhost

    Now start named and do an nslookup, the server and address should point back to local host.

    # nslookup fatmin.com
    Server:         127.0.0.1
    Address:        127.0.0.1#53

    Non-authoritative answer:
    Name:   fatmin.com
    Address: 64.202.189.170

  • RHEL6 – SELinux Modes and Contexts

    Anime,art,cartoon,chibi,darth,vader,darthvader-26d94e794e11142d417a277b8fbdc0eb_hSELinux, or Security-Enhanced Linux as its known by the guy who invented it, is a Linux feature that provides an additional level of security by setting rules for which processes can access which files, directories, ports, etc.

    Display and Modify SELinux Modes:

    You can use /etc/sysconfig/selinux to change the default SELinux mode at boot, and the setenforce command can be used to change the default level on the fly. Getenforce can be used to determine the current SELinux mode. 

    # getenforce
    Enforcing

    Display SELinux Contexts:

    Under SELinux, every file, process, directory, or port is assigned a special security label called a context.

    To view the contexts assigned to a file or directory use the '-Z' option. Coupled with and 'ls" or a 'ps" this is a formidable command.

    To view a list of all possible assigned contexts use semanage.

    # semanage fcontext -l

    Modify SELinux Contexts:

    For example, lets create two test files in /tmp called testfile1 and testfile2, and then lets check their contexts.

    # ls -lZ test*
    -rw-r–r–. root root unconfined_u:object_r:user_tmp_t:s0 testfile1
    -rw-r–r–. root root unconfined_u:object_r:user_tmp_t:s0 testfile2

    Now compare this to the default context assigned to apache content

    # ls -ldZ /var/www/html
    drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 /var/www/html

    So lets say that I want to move testfile1 to /var/www/html and make it accessable via a webbrowser; in order to do this I must assign the correct context to the file using the restorecon command.

    # restorecon -Rv /var/www/html

    The command above restores or even better, allows the testfiles to inherit the contexts assigned to the parent directory, which in this case is /var/www/html/

    Add SELinux Contexts:

    Now lets say that you need to add a directory and apply a context directly to that directory, instead of allowing a context to be inheritted. For example, lets say that I need to setup an apache virtual server under /virtual_server2, so lets first create the directory and a simple index.html

    # mkdir /virtual_server2
    # vi /virtual_server2/index.html
    # ls -Zd /virtual_server2
    drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 /virtual_server2

    # ls -Zd /virtual_server2/index.html
    -rw-r–r–. root root unconfined_u:object_r:default_t:s0 /virtual_server2/index.html

    Now we need to set and apply the correct http content context to /virtual_server2 and its contents (say that 5 times fast)

    # semanage fcontext -a -f "" -t httpd_sys_content_t '/virtual_server2(/.*)?'

    # restorecon -RFvvv /virtual_server2/

    restorecon reset /virtual_server2 context unconfined_u:object_r:default_t:s0->system_u:object_r:httpd_sys_content_t:s0
    restorecon reset /virtual_server2/index.html context unconfined_u:object_r:default_t:s0->system_u:object_r:httpd_sys_content_t:s0

    Now that our context is correct apache should have no issue serving up our new content (once configured on the apache side)

    # ls -Zd /virtual_server2/
    drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 /virtual_server2/

     

     

     

     

     

     

  • RHEL6 – Accessing ISCSI Disks with Iscsiadm

    Pdp-8ISCSI is scsi (Small Computer System Interface) over IP. Where SCSI is the protocol and IP is the transport mechanism. This is going to be a quick and dirty overview of how to login to and mount an iscsi disk. This will not be an overview of how to share out a disk via iscsi. I may get around to writing that post, but it will not be today.

    Before we get started we need to understand a few simple terms.

    • ISCSI Initiator – the client that needs access to the ISCSI disk.
    • ISCSI Target – the remote disk.
    • ISCSI Target Portal – the server that is providing the Target to the Initiator
    • IQN – The “ISCSI Qualified Name” or more simply a unique name for the Initiator and the Target

    Now lets get down to business.

    First on the Target you must install the ISCSI utils.

    >yum -y install iscsi-initiator-utils.x86_64

    Then discover the any ISCSI targets provided by the Target Portal. In this case the IP address of my target portal will be 192.168.0.55.

    >iscsiadm -m discovery -t st -p 192.168.0.55.

    In the example above -m puts iscsiadm is discovery mode, -t st tells iscsiadm to look for a SCSI target, and -p is the target portal. The output of the above command will be the IQN of the target disk.

    For this example we will say that the IQN is

    iqn.2001-05.com.equallogic:6-8a0900-5e1812101-1cf000245514a8da-disk1

    Then log into the ISCSI target

    >iscsiadm -m node -T iqn.2001-05.com.equallogic:6-8a0900-5e1812101-1cf000245514a8da-disk1 -p 192.168.0.55 -l

    In the example above we are setting the mode to node, -T prefaces the ISCSI target, -p the portal, and -l means login.

    Now you need to identify which device is the iscsi target by looking at the output of dmesg or tail the messages file.

  • RHEL6 – Managing Swap Space

    54af9-6a00e551c39e1c8834017ee46a8c54970d-piSwap space on a Linux box is an area on disk that is used to hold inactive memory pages. This occurs when the system needs more memory then is currently available, so it swaps these inactive memory pages to disk.

    To create additional swap space on the fly you are either going to need a spare disk or free partition on a disk that you can use.

    First, using fdisk,  you will need to make sure that the partition type for the disk (or partition) is set to 82.

    Then setup the swap area using mkswap. In this example I am using /dev/sdb2, but your setup is bound to be different.

    >mkswap /dev/sdb2

    Then determine the UUID of the new swap space.

    >blkid /dev/sdb2

    Then add an entry to the /etc/fstab, so that the swap space is mounted at boottime. The show in the example below is the output of the blkid command above.

    >UUID=7b05f0a9-18d5-42e5-b259-78ba3a8cc1b7 swap                    swap    defaults        0 0

    Then activate your new swap space

    >swapon -a

    Then check to make sure everything worked by checkign for your new swap partition in the output of the command below.

    >swapon -s

    /dev/sdb2                             partition       4193276 7480    0