• Iostat Primer

    Harddisk
    iostat
    (input/output statistics) is a computer operating system monitoring tool used to collect and show storage input and output statistics. ( I stole this from Wiki)

    It is often used to identify performance issues with local disks or networked file systems such as NFS.

    For local disks and SAN disks I usually run it like so.

    iostat -zxnM 5

    this returns stats in the format below:

                        extended device statistics
        r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
        1.3    2.9    0.0    0.0  0.0  0.0    2.1   10.1   1   2 d0
        0.0    0.0    0.0    0.0  0.0  0.0    0.4    8.8   0   0 d1
        0.4    0.2    0.0    0.0  0.0  0.0    2.6    8.6   0   0 d4
        0.7    2.9    0.0    0.0  0.0  0.0    0.0   10.2   0   2 d10
        0.0    0.0    0.0    0.0  0.0  0.0    0.0    8.1   0   0 d11
        0.2    0.2    0.0    0.0  0.0  0.0    0.0    9.3   0   0 d14
        0.7    2.9    0.0    0.0  0.0  0.0    0.0    7.2   0   2 d20
        0.0    0.0    0.0    0.0  0.0  0.0    0.0    7.7   0   0 d21
        0.2    0.2    0.0    0.0  0.0  0.0    0.0    7.8   0   0 d24
        0.0    0.0    0.0    0.0  0.0  0.0    0.0   62.3   0   0 c0t0d0
        1.0    4.5    0.0    0.0  0.0  0.1    0.0   11.0   0   3 c1t0d0
        1.0    3.1    0.0    0.0  0.0  0.0    0.0    6.9   0   2 c1t1d0
        0.2    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t2d0
        0.2    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t3d0
        2.4    1.0    0.3    0.0  0.0  0.0    0.1    4.2   0   1 c7t600A0B800029A7A000000E4247DAAF20d0
        0.2    0.2    0.0    0.0  0.0  0.0    0.0    3.4   0   0 c7t600A0B800029A7A000000E8447DFA700d0
        0.3    0.3    0.0    0.0  0.0  0.0    0.0    3.7   0   0 c7t600A0B800029A76800000C444820020Dd0
        0.9    0.8    0.1    0.0  0.0  0.0    0.0    2.9   0   0 c7t600A0B800029A76800000C4548200237d0

    Use the key below to figure out what each heading means.

     The fields have the following meanings:

    disk name of the disk
    r/s reads per second
    w/s writes per second
    Kr/s kilobytes read per second
    Kw/s kilobytes written per second
    wait average number of transactions waiting for service (Q length)
     actv average number of transactions actively 
     being serviced (removed from the
        queue but not yet
    completed)
    %w percent of time there are transactions waiting
    for service (queue non-empty)
    %b percent of time the disk is busy (transactions
    in progress)

    Most of the time I am troubleshooting hot application mount points, which are usually SAN disks controlled by Veritas. So at this point you need to correspond the OS's long ass disk name with Veritas' short and clean name. Use vxdisk list

    vxdisk -e list

    <

    p style=”font-size:9px;font-family:Trebuchet MS”>SUN6140-0_0  auto      disk disgroup online     c7t600A0B800029A7A000000E8447DFA700d0s2
    SUN6140-0_2  auto      disk disgroup online     c7t600A0B800029A76800000C4548200237d0s2
    SUN6140-0_3  auto      disk disgroup online     c7t600A0B800029A76800000C444820020Dd0s2

    At this point know Solaris' name for each disk and can look for hotspots on these disks.

  • SCS4805 System Configuration

    Lantronix
    I have had several opportunities as of late work with these Lantronix terminal servers. Usually I am connecting older Sun boxes… those without an SC. In this instance I am connecting to a LOM port on a SUN v120 using a straight through Ethernet cable.

    First of log into the SCS480 using the sysadmin user id and its password. In this instance I am connected to port 23.

    Typing help will show you a list of commands.

    Typing direct and then the port number will connect you directly to the port, <ESC>A will disconnect you.

     Entering Direct mode…Server = DEVICE_23
    (enter '<ESC>A' to exit, enter '<ESC>B' to generate a break.)

    Type listdev to see all the ports, and their names. You should set the name so it corresponds with the server.

    Type devices 23 to see how port 23 is configured.

    sysadmin-DEVICE_23>devices 23
    Enter accepts present value
    D23: Enter device name                      | NAME: DEVICE_23>
    D23: 0=2400 1=4800, 2=9600, 3=19200,
         4=38400, 5=57600, 6=115200             | BAUD RATE: 9600>
    D23: 1=1, 2=2                               | STOP BITS: 1>
    D23: 0=None, 1=Odd, 2=Even, 3=Mark, 4=Space | PARITY: NONE>
    D23: 6=6, 7=7, 8=8                          | DATA BITS: 8>
    D23: 0=OFF, 1=DCE, 2=DTE                    | EQUIP PORT TYPE: DTE>
    D23: 0=XON/OFF, 1=RTS/CTS                   | FLOW CONTROL: XON/XOFF>
    D23: 0=No, 1=Yes                            | INHIBIT BUFFERING IN DIRECT: No>

    To configure the port I need to change the device name and change the EQUIP PORT TYPE to DCE. Click here for more info on DCE/DTE.

    Using the devices command you can configure the port. See syntax below

    devices –name=devname n
      Set name of device n to devname
     devices –baud=2400|4800|9600|19200|38400|57600|115200 n
      Set baudrate of device n to 2400, 4800, 9600, 19200, 38400, 57600,
                  or 115200
     devices –baud=0|1|2|3|4|5|6 n
      Set baudrate of device n to 0=2400 1=4800, 2=9600, 3=19200,
                  4=38400, 5=57600, or 6=115200
     devices –stop=1|2 n
      Set stop bits of device n to 1=1, 2=2
     devices –parity=None|Odd|Even|Mark|Space n
      Set parity of device n to None, Odd, Even, Mark, or Space
     devices –parity=0|1|2|3|4 n
      Set parity of device n to 0=None, 1=Odd, 2=Even, 3=Mark, or 4=Space
     devices –data=6|7|8 n
      Set number of data bits of device n to 6, 7, or 8
     devices –type=OFF|DCE|DTE n
      Set port type of device n to OFF, DCE, or DTE
     devices –type=0|1|2 n
      Set port type of device n to 0=OFF, 1=DCE, or 2=DTE
     devices –flow=XON|RTS n
      Set flow control of device n to XON/OFF, or RTS/CTS
     devices –flow=0|1 n
      Set flow control of device n to 0=XON/OFF, or 1=RTS/CTS
     devices –inhibitbuffer=No|Yes n
      Set inhibit buffering of device n to No, or Yes
     devices –inhibitbuffer=0|1 n
      Set inhibit buffering of device n to 0=No, or 1=Yes

    Using this info I can now configure my port

    <

    p style=”font-size:11px;font-family:Trebuchet MS”>sysadmin-DEVICE_23>devices –name=myserver 24
    alaudpnc004
    Config files have been updated.
    Updating hardware, please wait….  port: 24
    Hardware has been updated.
    sysadmin-DEVICE_23>devices –type=DCE 24
    Config files have been updated.
    Updating hardware, please wait….  port: 24
    Hardware has been updated.

  • Rsync How to

    Rsync is a very very useful command with a whole bunch of options. However most of the time if you run rsync on the command line you only need to know a few.

    #rsync -avz foo:src/bar /data/tmp

    This would recursively transfer all files from the directory src/bar
    on the machine foo into the /data/tmp/bar directory on the local machine.

    The files are transferred in “archive” mode, which ensures that symbolic links,
    devices, attributes, permissions, ownerships etc are preserved in the transfer.
    Additionally, compression will be used to reduce the size of data portions of the transfer.

    #rsync -avz foo:src/bar/ /data/tmp

    A trailing slash on the source changes this behavior to transfer all files
    from the directory src/bar on the machine foo into the /data/tmp/.

    A trailing / on a source name means “copy the contents of this directory”.
    Without a trailing slash it means “copy the directory”.

    This difference becomes particularly important when using the –delete option.

    Below are a few of the options that I find useful.

    What to Copy:
    -r recursive

    How to Copy:
    -n dry run
    -g preserve group

    Destination Options:
    -a archive
    -b backup

    Syntax Examples:
    rsync [OPTION]… SRC [SRC]… [USER@]HOST:DEST
    rsync [OPTION]… [USER@]HOST:SRC DEST
    rsync [OPTION]… SRC [SRC]… DEST
    rsync [OPTION]… [USER@]HOST::SRC [DEST]
    rsync [OPTION]… SRC [SRC]… [USER@]HOST::DEST
    rsync [OPTION]… rsync://[USER@]HOST[:PORT]/SRC [DEST]
    rsync [OPTION]… SRC [SRC]… rsync://[USER@]HOST[:PORT]/DEST

  • Changing a Solaris Server’s IP Address

    The following steps may be used to change the IP address of a
    Solaris system.

    1. Change the host’s IP in /etc/hosts for the change to take effect
    after reboot. If you are using Solaris 10, you must also change the host’s IP in /etc/inet/ipnodes for the change to take effect after reboot.

    2. Run ifconfig interface ip_address netmask broadcast_address for the
    IP address change to take effect immediately. The netmask and broadcast_address should be specified if you are using variable length subnet masks (VLSM), but may be omitted otherwise.


    3. If you are using variable length subnet masks (VLSM), add the host’s network number and subnet mask to /etc/netmasks.

    <

    p style=”font-family:Arial”>If the new IP address places the system on a different network:

    4. Place the host’s new default gateway in /etc/defaultrouter, place new netmask in /etc/netmasks

    5. Run route add default new_gateway for the new default gateway to take effect immediately.

    6. Run route delete default old_gateway to delete the old default gateway from the routing table.

  • Using Filesystem Labels in Linux

    Ist2_3533724_yellow_sticky_note When rebooting a RHEL box with a whole bunch of SAN luns I ran into an issue where the system was failing when trying to mount a device ( /dev/sdm). When I logged into the box in single user mode this device did not even exist. However, it was brought to my attention that a few luns where removed from this box recently, so apparently when I rebooted the box generated a new device list, and my renamed /devd/sdm to something different.

    Apparently this is common.As the simple act such as adding or removing a
    LUN can cause Linux to remap your /dev/sd entries, causing what used to
    be /dev/sde1 to become /dev/sdd1. This problem can be solved by using filesystem labels, which are entries in the ext3 filesystem header.

    have found 2 ways of doing this.

    First is with e2label, Usage: e2label device [newlabel]

    You can also show your label as below. In this instance my label is 'storage'

    root@kubuntu:~/downloads# e2label /dev/sdb1
    storage

    Or you can use tune2fs

    L argument to set, l to show a whole bunch of stuff

    tune2fs -L storage /dev/sdb1

    root@kubuntu:~/downloads# tune2fs -l /dev/sdb1
    tune2fs 1.40.8 (13-Mar-2008)
    Filesystem volume name:   storage
    Last mounted on:          <not available>
    Filesystem UUID:          1cc16930-8891-4307-97fb-e267117dead1
    Filesystem magic number:  0xEF53
    Filesystem revision #:    1 (dynamic)
    Filesystem features:      ext_attr resize_inode dir_index filetype sparse_super large_file
    Filesystem flags:         signed_directory_hash
    Default mount options:    (none)
    [truncated]

    Then make your changes to the fstab, both entries below are same, just one uses the newly assigned label.

    #/dev/sdb1      /mnt            auto    rw,user,noauto,exec     0       3
    LABEL=storage   /mnt            auto    rw,user,noauto,exec     0       3

  • Verify Solaris 10 Multipathing/Configure SAN Disk

    Fibre

    I was attempting to troubleshoot issues as a user was complaining about slow performance on a SAN disk. First thing that I did was check to ensure that there were not any performance issues on any disks that might have been causing this users issues

    A quick iostat verified that everything was looking fine
    iostat -cxzn 1

     

    This box is running Veritas so lets check out the disks. Vxdisk list shows one Sun6140 disk.

    # vxdisk list
    DEVICE       TYPE            DISK         GROUP        STATUS
    Disk_0       auto:none       –            –            online invalid
    Disk_1       auto:none       –            –            online invalid
    SUN6140_0_1  auto:cdsdisk    diskname_dg02  diskname_dg online nohotuse

    Luxadm is an utility, which discovers FC devices (luxadm probe), shut
    downs devives (luxadm shutown_device …) runs a firmware upgrade
    (luxadm download_firmware …) and many other things. In this instance I use luxadm to get the true device name for my disk


    # luxadm probe
    No Network Array enclosures found in /dev/es

    Found Fibre Channel device(s):
    Node WWN:200600a0b829a7a0  Device Type:Disk device
    Logical Path:/dev/rdsk/c4t600A0B800029A7A000000DC747A8168Ad0s2

    I then run a luxadm on the device. Below you can see that I do indeed have two paths to the device.
    1 controller = one path, 2 controllers = 2 paths

    # luxadm display /dev/rdsk/c4t600A0B800029A7A000000DC747A8168Ad0s2
    DEVICE PROPERTIES for disk: /dev/rdsk/c4t600A0B800029A7A000000DC747A8168Ad0s2
    Vendor:               SUN
    Product ID:           CSM200_R
    Revision:             0619
    Serial Num:           SG71009283
    Unformatted capacity: 12288.000 MBytes
    Write Cache:          Enabled
    Read Cache:           Enabled
    Minimum prefetch:   0x1
    Maximum prefetch:   0x1
    Device Type:          Disk device
    Path(s):

    /dev/rdsk/c4t600A0B800029A7A000000DC747A8168Ad0s2
    /devices/scsi_vhci/ssd@g600a0b800029a7a000000dc747a8168a:c,raw
    Controller           /devices/pci@1f,4000/SUNW,qlc@5,1/fp@0,0
    Device Address              203700a0b829a7a0,1
    Host controller port WWN    210100e08bb370ab
    Class                       secondary
    State                       STANDBY
    Controller           /devices/pci@1f,4000/SUNW,qlc@5/fp@0,0
    Device Address              203600a0b829a7a0,1
    Host controller port WWN    210000e08b9370ab
    Class                       primary
    State                       ONLINE

    Had I only had one path I would have run cfgadm. I would have seen that one of the fc-fabric devices would have been unconfigured. I then could have used cfgadm to configure it and enable my mulitpathing

    # cfgadm
    Ap_Id                          Type         Receptacle   Occupant     Condition
    c0                             scsi-bus     connected    configured   unknown
    c1                             scsi-bus     connected    unconfigured unknown
    c2                             fc-fabric    connected    configured   unknown
    c3                             fc-fabric    connected    configured   unknown

    MPXIO Primer

    Solaris I/O multipathing gives you the ability to set up multiple
    redundant paths to a storage system and gives you the benefits of load
    balancing and failover.

    Need to enable MPXIO

    Solaris 10 is the easier, because the mpxio capability is
    built-in. You just need to turn it on!

    To enable it, edit the file /kernel/drv/fp.conf
    file. At the end it should say:

    mpxio-disable="yes";

    Just change yes to no and it will be enabled:

    mpxio-disable="no";

    Before multipathing, you should see two copies of each disk in
    format. Afterwards, you’ll just see the one copy.

    It assigns the next available controller ID, and makes up some
    horrendously long target number. For example:

    Filesystem kbytes used avail capacity Mounted on
    /dev/dsk/c6t600C0FF000000000086AB238B2AF0600d0s5
    697942398 20825341 670137634 4% /test

  • How to Change Root’s Ilom Password

    ILOM is the system Service Processor Sun includes with Sun x64 systems.ILOM, can also be accessed via an integrated HTTPS server, meaning that a user friendly graphical user interface is available. Currently the server models below use ILOM.
    Sun Fire[TM]  X4100

    Sun Fire[TM]  X4200

    Sun Fire[TM]  X4500

    Sun Fire[TM]  X4600

    Sun Fire[TM]  X4100 M2

    Sun Fire[TM]  X4200 M2

    Sun Fire[TM]  X4600 M2

    Sun Blade[TM] 6000

    Sun Blade[TM] 8000

    Sun Netra[TM] X4200 M2

    Sun SPARC[TM] Enterprise T5120

    Sun SPARC[TM] Enterprise T5220

    The default root password for the ILOM is changeme. This is a Sun default and you should make sure that you modify this when you are setting up your new server.

    Below I have outlined the procedure that you need to follow

    1. Connect and Login to your Serial Management port.
    2. cd /SP/users
    3. type “set /SP/users/root password”, and voila

    Changing password for user /SP/users/root…
    Enter new password: ********
    Enter new password again: ********
    New password was successfully set for user /SP/users/root