Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Category: Vmware

  • Enabling SNMP in ESXi 4.1 using the Remote CLI

    cdc63-6a00e551c39e1c88340168ea399c7e970c-pi

    Based on the fact that ESX 4.1 is the last major release of ESX, I decided that I would make myself familiar with managing ESXi hosts.  Since I monitor all my hosts via Zenoss, I figured that I needed to get snmp up and running first.

    So I first when out and installed the remote cli for ESX on my ubuntu desktop.  The rcli can be downloaded here. The remote cli allows you to run command administrative commands against ESX/ESXi systems. Its availbile for Windows or Linux.

    Configuring on ESXi 4.1, Licensed

    First configure your community string, target, and port:

    vicfg-snmp –server <ESXi_ip> -c <communityname> -p 161 -t <destination_host>@161/<community name>

    Then enable it using the command below:

    vicfg-snmp –server <ESXi_ip> -E

    Next verify your settings:

    vicfg-snmp –server <ESXi_ip> -s

    Now test your settings:

    vicfg-snmp –server <ESXi_ip> -T

    These settings are written out to /etc/vmware/snmp.xml. Sample file below.

    /etc/vmware # cat snmp.xml
    <config>
    <snmpSettings>
    <communities>pubic</communities>
    <enable>true</enable>
    <port>161</port>
    <targets>10.1.xx.xx@161 public</targets>
    </snmpSettings>

    Configuring on ESXi 4.0, Free/Foundation

    I had a couple of ESXi 4.0 free hosts to configure, but my attempts to configure them using the cli failed as the snmp settings via cli were read only. So the first thing that you need to do is enable the unsupported console. Instructions can be found here.

    Once you are able to ssh to the ESXi box, you need to edit the following file by hand, /etc/vmware/snmp.xml. Use the sample file above as a template and modify your ip, port, and string as needed. I use vi to edit mine.

    Then run the command below

    services.sh restart

    You can then verify your settings using the remote cli by running the command below against your esxi box.

    vicfg-snmp –server <ESXi_ip> -T

  • Getting Started with the VMware Management Assistant

    Vmware_computer_logo_100x100I've seen a lot of blog posts as of late regarding the death of the Service Console (COS). The rumors are that ESX is going to be replace completely with ESXi. While most Widows/Vmware Admins probably don't care, us Linux/Vmware admins are less than happy as we want nothing to do with the Windows PowerShell.

    So I did some poking around and ran into the Vmware Management Assistant, which is a tiny VM. Here is the official spiel from Vmware

    "The vSphere Management Assistant (vMA) allows administrators and
    developers to run scripts and agents to manage
    ESX/ESXi and vCenter Server systems. vMA is a virtual machine that
    includes prepackaged software, a logging
    component, and an authentication component that supports non-interactive
    login."

    You can download and install it here

    Documentation is here.
    Basically the VMA give you a remote server that you can use to run commands against your ESX and ESXi boxes.

    You can add a server to your list of managed servers with the command below. You will be prompted for your login credentials.Complete this step for all ESX and ESXi boxes.

    sudo vifp addserver <servername>

    To view your list type

    vifp listservers

    Then you can run a command against the boxes in your list, see below

    vicfg-nics –server –<servername> –list 

    You can also choose an execution server to run multiple commands against using the following

    vifpinit <servername>

    There is a lot more to the VMA, but this tip of the iceberg will get you started.

  • Resolving SCSI Reservation Conflicts/Locks in Vsphere 4.0

    Blue_lock-main1A few days ago we got hit with a ton of alerts which indicated that a handful of VMs were down, then up, and down again. This cycle continued several times.

    At first, after a bit of digging through logs, we thought that the issue was related to scsi reservation errors, but we were already compliant with the best practices for 3PAR mentioned here. So we dug deeper and found that we were in fact suffering from SCSI locks. Go here for more information.

     According to VMware…

    "The second
    category involves acquisition of locks. These are locks related to VMFS
    specific meta-data (called cluster locks) and locks related to files
    (including directories). Operations in the s    econd category occur much more frequently than operations in the first category. The following are examples of VMFS operations that require locking metadata:

    • Creating a VMFS datastore
    • Expanding a VMFS datastore onto additional extents
    • Powering on a virtual machine
    • Acquiring a lock on a file
    • Creating or deleting a file
    • Creating a template
    • Deploying a virtual machine from a template
    • Creating a new virtual machine
    • Migrating a virtual machine with VMotion
    • Growing a file, for example, a Snapshot file or a thin provisioned Virtual Disk

    To resolve a SCSI Lock, log into each of your ESX boxes and run the following command. 

    # esxcfg-info | egrep -B5 "s Reserved|Pending

    Look for the output below, as the host that has "Pending Reservation" value greater than one is causing the lock.

    |—-Pending Reservations……………. 1

    Now reset the lun.

    vmkfstools –lock lunreset /vmfs/devices/disks/vml.02000000006001c230d8abfe000ff76c198ddbc13e50455243


  • ESX Post Install – Enable NTP and SNMP

    FirewallThis post is the first in what I suspect will be a semi-long list of post-install hints and tips as I go through and start rebuilding my cluster as Vsphere 4. Hopefully I will learn a lot along the way… like for example the fact the ntp and snmp traffic is not allowed by default by the ESX Firewall.

    But before we go there we first need to make sure that our services are starting at boot.

    chkconfig ntp on, … do the same for snmp

    Then lets fix the firewall. First lets fix ntp.

    esxcfg-firewall -e ntpClient

    Then lets verify that all is well with…

    esxcfg-firewall -q ntpClient

    This command returns…

    Service ntpClient is enabled

    Ok now lets fix snmp using the same commands above, but specific for snmp.

    esxcfg-firewall -e snmp and esxcfg-firewall -q snmpd.

    While you are at it add the following to you snmp.conf

    dlmod SNMPESX /usr/lib/vmware/snmp/libSNMPESX.so

    Then restart snmp and ntp and you should be good.

  • How to Configure NTP in a RHEL/CentOS Vmware Guest

    HourglassSuccessful time keeping in a Virtual Machine can be a bit confusing. At times I have been told to use Vmware Tools to sync time between the Guest and the Host, and at time I have been advised to avoid this functionality and use NTP. The following information is direct from a VMware KB article (updated, 4/16/2010) so I am going to follow their lead on this and use NTP exclusively.

    First off VMware advises using the NTP service to keep time in sync, but it suggests
    using an additional kernel parameter that you add to your grub.conf. See the KB Article for more info on how to do this.

    • notsc for RHEL/Centos 4.6 64bit
    • notsc divider=10 for  RHEL/Centos 5.3 64bit

    Note that there are no additional params needed for 5.4

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427

    Also, inside the ntp.conf the following line should be added to the top of the file.

    tinker panic 0

    This configuration directive instructs NTP not to give up
    if it sees a large jump in time. This is important for coping with large
    time drifts and also resuming virtual machines from their suspended state.

    It is also important not to use the local clock as a time source, often
    referred to as the Undisciplined Local Clock. NTP has a tendency to fall
    back to this in preference to the remote servers when there is a large
    amount of time drift.

    An example of such a configuration is below. You should comment out both
    lines.

    server 127.127.1.0

    fudge 127.127.1.0 stratum 10

    Also, if you are using ntp, you want to make sure that you disable
    Vmware tools time sync. You can do so with the following command via the
    guest OS.

    vmware-guestd –cmd “vmx.set_option synctime 1 0”

  • Troubleshooting Performance Issues on Multiple Vcpu Virtual Machines

    I’ve been doing a bit of research on performance issues with virtual machines that have multiple Vcpus. In my case there are several 4 Vcpu machines that often seem to have performance issues. Specifically these are Windows 2003 Virtual Machines running Cold Fusion.

    Found the information below in the Vmware Community Forum.

    http://communities.vmware.com/thread/129970?tstart=0

    “An ESX server has to provide ALL processors at the the time a VM
    requests it, it can’t give 1 CPU then the other, they both have to be
    available. As your ESX server becomes busier, it has less time to
    allocate to VM’s that need more CPU’s, thus you get less and less time
    slice, which causes your VM to wait for CPU cycles, thus causing the
    programs inside the VM to miss a few cycles, and then it causes delays
    which causes slower performance. So it does get slower, depending on the number of multi-CPU’s you have
    and the CPU of the physical cores on the ESX host. Also Adding More
    CPU’s isn’t necessary unless you have applications that can use it.
    Just because a VM shows both CPU in use, don’t assume that to mean that
    meaningful work is being performed.”

    Also found this information on how to test weather or not your multiple Vcpu machines are shooting themselves in the foot.

    To determine if multiple vCPUs assigned to your virtual machine is causing poor performance:
    1. Open a console prompt on the ESX host or initiate an SSH connection to it. For more information, see Opening a command or shell prompt (1003892).
    2. Type esxtop and press Enter.
    3. On the CPU screen, check the %CSTP
      value. If this number is higher than 100, the performance issues may be
      caused by the vCPU count. Try lowering the vCPU count of the virtual
      machine by 1.      Note: The %CSTP value
      represents the amount of time a virtual machine with multiple virtual
      CPUs is waiting to be scheduled on multiple cores on the physical
      host. The higher the value, the longer it waits and the worse its
      performance.  Lowering the number of vCPUs reduces the scheduling wait time.

    See the KB article below for more info…

    http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1005362&sliceId=1&docTypeID=DT_KB_1_1&dialogID=75028447&stateId=0%200%2080939873

    Note that my issues are occuring in ESX 3.5. Apparently in ESX4 many improvements have been made to the CPU scheduler. I have yet had a chance to see this for myself.

    “In ESX 4, many improvements have been introduced in CPU scheduler. This includes further relaxed co-scheduling, lower lock-contention, and multi-core aware load balancing. Co-scheduling overhead has been further reduced by the accurate measurement of the co-scheduling skew, and by allowing more scheduling choices. Lower lock-contention is achieved by replacing scheduler cell-lock with finer-grained locks. By eliminating the scheduler-cell, a virtual machine can get higher aggregated cache capacity and memory bandwidth. Lastly, multi-core aware load balancing achieves high CPU utilization while minimizing the cost of migrations.”

    See this article for more information: http://www.vmware.com/files/pdf/perf-vsphere-cpu_scheduler.pdf

  • Troubleshooting Vswif0 Via the Command Line (Vsphere 4.0)

    Vm Troubleshooting network connectivity issues in ESX can be a real pain in the rear if the only method that you have to connect to your ESX host is via the command line over the ILO. In this instance a simple reboot turned into a host down with only ILO connectivity remaining. When this occurs you need to get the Service Console back on the network, and sort out the details of what and why later.


    First figure out what vmnics are connected to the vSwitch connected to your Service Console.

    > esxcfg-vswitch -L

    In my case the Service Console is connected to Vswitch0 via vmnic0 and vmnic1.Pick a vmnic and remove it from the vswitch. Below I am removing vmnic1.

    > esxcfg-vswitch -U vmnic1 vSwitch0

    Try to ping the Service Console's gateway. In my case it did not respond, so I added vmnic1 back and removed vmnic0.

    > esxcfg-vswitch -L vmnic1 vSwitch0

    > esxcfg-vswith -U vmnic0 vSwitch0

    Now try to ping your gateway. If it works then get the CDP information for vmnic0 using the command below, send this information along with a firm scolding to your local network admin.

    > vmware-vim-cmd hostsvc/net/query_networkhint