Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

  • OpenStack director Overcloud Image Build Using the RHN Portal

    a17f5-6a00e551c39e1c8834015436149671970c-pi

    Most of the documentation that I see on the subject of building your own Overcloud images use repos that are not available to most users. Today, I am going to document two additional scenarios. One, using a local Satellite server, and the other using the Red Hat (RHN) Portal.

    RHEL Portal Registration

    Here are the steps to use if you want your overcloud machines to register to the RHEL portal.

    First export a bunch of ENV variables.

    export NODE_DIST=rhel7
    export DIB_LOCAL_IMAGE=rhel7-guest.qcow2
    export USE_DELOREAN_TRUNK=0
    export REG_METHOD=portal
    export REG_USER=”RHN-USERID”
    export REG_PASSWORD='<RHN-PASSWORD!’
    # Find this with `sudo subscription-manager list –available`
    export REG_POOL_ID=”[pool id]”
    export REG_REPOS=”rhel-7-server-rpms \
    rhel-7-server-extras-rpms \
    rhel-ha-for-rhel-7-server-rpms \
    rhel-7-server-optional-rpms \
    rhel-7-server-openstack-7.0-rpms”

    Now start your build. Should take about 30 minutes, so be patient.

    #time openstack overcloud image build –all 2>&1 | tee openstack_image_build.log

     

    RHEL Satellite Registration

    When registering to a local satellite server, use the format below. Note that you must use an activation key, as using id and password authentication is not supported for security reasons.

    export REG_METHOD=satellite
    # REG_SAT_URL should be in the format of:
    # http://<satellite-hostname
    export REG_SAT_URL=”[satellite url]”
    export REG_ORG=”[satellite org]”
    # Activation key must enable these repos:
    # rhel-7-server-rpms
    # rhel-7-server-optional-rpms
    # rhel-7-server-extras-rpms
    # rhel-7-server-openstack-7.0-rpms
    export REG_ACTIVATION_KEY=”[activation key]”

    Now start your build. Remember to be patient.

    time openstack overcloud image build –all 2>&1 | tee openstack_image_build.log

  • OpenStack Ironic – CLI Cheat Sheet

    home-iron

    Ironic is the baremetal provisioning component of OpenStack. Ironic interacts with hardware through a set of plug-ins. Technologies leveraged are PXE and IPMI as well as a few vendor specific plug-ins.

    Here are a few OpenStack Ironic commands that I have found useful when troubleshooting.

    List nodes registered with Ironic.

    $ ironic node-list

     

    Show details of a specific node using the node UUID – your node UUID will differ.

    $ ironic node-show 5492ee96-218f-4e0e-b5b3-4931ba2c9ede

     

    Disable maintenance mode for a specific node.

    $ ironic node-set-maintenance 5492ee96-218f-4e0e-b5b3-4931ba2c9ede off

     

    The command below is useful for watching logs. Note…

    • -u is used to specify the unit
    • -f is used to tail logs

    $sudo journalctl -u openstack-ironic-conductor -u openstack-ironic-api -f

     

    Make a node available for scheduling by nova. Below is an example using a UUID in my environment.

    $ ironic node-set-provision-state 5492ee96-218f-4e0e-b5b3-4931ba2c9ede provide

     

    Show Mac address for a particular ironic node – uses UUID.

    $ ironic node-port-list 5492ee96-218f-4e0e-b5b3-4931ba2c9ede

    More examples to come.

    Delete a node – using UUID

    $ironic node-set-provision-state f7f58a13-1ac8-4d3a-b1b8-e7a02a07a323 deleted

    Force Delete a Node from the Database – Use this option when your node is stuck in the “deleting” Provisioning State.

    Log into mysql database

    [stack@undercloud] # sudo mysql -u root

    Switch to the ironic database

    MariaDB [(none)]> use ironic;
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -A

    Force state change using UUID of Ironic Node.

    MariaDB [ironic]> UPDATE nodes SET provision_state=”available”, target_provision_state=NULL WHERE uuid=”f7f58a13-1ac8-4d3a-b1b8-e7a02a07a323″;
    Query OK, 1 row affected (0.01 sec)
    Rows matched: 1 Changed: 1 Warnings: 0

    MariaDB [ironic]> exit
    Bye

     

  • Numa CPU Pinning with KVM/Virsh

    Intel_CPU_Pentium_4_640_Prescott_bottom

    According to Wikipedia, Numa is — “a computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor. Under NUMA, a processor can access its own local memory faster than non-local memory (memory local to another processor or memory shared between processors). The benefits of NUMA are limited to particular workloads, notably on servers where the data are often associated strongly with certain tasks or users.“

    So what does this mean for a Virtual Machine optimization under KVM/Libvirt? It means that for best performance, you want to configure your multi-vcpu VMs to use only cores from the same physical CPU (or numa node).

    So how do we do this? See the example below from one of my homelab servers. This machine has two hyperthreaded quad core Xeons (x5550) — for a total of 16 cores.

    First we use the “lspcu” command to determine which cpu cores are tied to which CPU. This is in bold below.

    # lscpu
    Architecture: x86_64
    CPU op-mode(s): 32-bit, 64-bit
    Byte Order: Little Endian
    CPU(s): 16
    On-line CPU(s) list: 0-15
    Thread(s) per core: 2
    Core(s) per socket: 4
    Socket(s): 2
    NUMA node(s): 2
    Vendor ID: GenuineIntel
    CPU family: 6
    Model: 26
    Model name: Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
    Stepping: 5
    CPU MHz: 2668.000
    BogoMIPS: 5319.11
    Virtualization: VT-x
    L1d cache: 32K
    L1i cache: 32K
    L2 cache: 256K
    L3 cache: 8192K
    NUMA node0 CPU(s): 0-3,8-11 
    NUMA node1 CPU(s): 4-7,12-15

    Using the virsh command, we can inspect the CPU pinning for my test VM called “mytestvm“.

    (more…)

  • OpenStack Ironic : JSON Validator

    12340anvil

    This handy tool, developed by an engineer at Red Hat, checks your Ironic JSON file for errors such as missing passwords, or non-unique MAC addresses. The script also tests connections to your bare metal (or virtual nodes in my test lab) to ensure that they are accessible.

    Learn more via the link below:

    https://github.com/rthallisey/clapper

    Or download the script directly via this link.

    Usage, at least in my case, is shown below. Here, I am running the script against my instackenv.json file.

    [code language=”css”]
    # python instackenv-validator.py -f instackenv.json
    INFO:__main__:Checking node 192.168.122.1
    DEBUG:__main__:Identified virtual node
    INFO:__main__:Checking node 192.168.122.1
    DEBUG:__main__:Identified virtual node
    INFO:__main__:Checking node 192.168.122.1
    DEBUG:__main__:Identified virtual node
    INFO:__main__:Checking node 192.168.122.1
    DEBUG:__main__:Identified virtual node
    INFO:__main__:Checking node 192.168.122.1
    DEBUG:__main__:Identified virtual node
    DEBUG:__main__:Baremetal IPs are all unique.
    DEBUG:__main__:MAC addresses are all unique.

    ——————–
    SUCCESS: instackenv validator found 0 errors
    [/code]

  • Getting Started with Red Hat OpenStack director

    OpenStack-logo

    Previously,  I wrote a short blog post on OpenStack director, the new Red Hat installer for Openstack 7 (kilo). That post can be found here, and includes a short video introduction and demo of the new installer – RHEL OSP director.

    OpenStack director introduces the concepts of an undercloud and an overcloud. These concepts derived from TripleO, AKA, OpenStack on OpenStack. It’s a bit tricky at first viewing, but once you have worked through an install or two, it makes much more sense.

    That being said… In this post, I am going to provide a few helpful links that can be used to help you get started with installing OpenStack director and your OpenStack overcloud.

    First, below is the link to the official installation documentation. I have been following along with this guide, in addition to my internal notes, and have found the official documentation to be very useful.

    Red Hat OpenStack Director Installation Guide

    In addition to the link above, I am including the link below which directs you to the download page of the images used to build your overcloud. You are going to need these to get started, as the above guide will tell you. I figured I would provide that link as well to help move the process along. Download all three tar files.

     

    Overcloud Image Download

    Enjoy !!

    PS. Note that there are some users experiencing issues with the 7.2 images. See link below. 

    https://bugzilla.redhat.com/show_bug.cgi?id=1296330

    As of today 7.3 (2/18/2016) 7.3 images are available via the link download link above.

     

  • RHEL 7 – Setup Software Raid Via mdadm

    smarta

    In this post I will walk though the steps that I used to setup software raid using mdadm on my RHEL 7.2 server.

    The disks in my server are shown below.

    • /dev/sda – 64gb SSD – OS disk
    • /dev/sdb – 4TB – Unused
    • /dev/sdc – 4TB – Unused 

    My plan is to create a RAID 1 mirror from the two 4TB drives (sdb and sdc) and mount the mirrored device to “/var/lib/libvirt/images” for use via KVM.

    First we need to partition the disks. Note that we cannot use fdisk as does not support partition sizes over 4Tb in size.  See below.

    [root@titan]# fdisk /dev/sdb

    WARNING: The size of this disk is 4.0 TB (4000000000000 bytes).
    DOS partition table format can not be used on drives for volumes
    larger than (2199023255040 bytes) for 512-byte sectors. Use parted(1) and GUID partition table format (GPT).

     

    Instead we will use parted. See steps below. Note that each step is repeated as we have two disks.

    [root@titan]# parted --script /dev/sdb "mklabel gpt"
    [root@titan]# parted --script /dev/sdc "mklabel gpt"
    [root@titan]# parted --script /dev/sdb "mkpart primary 0% 100%"
    [root@titan]# parted --script /dev/sdc "mkpart primary 0% 100%"
    [root@titan]# parted --script /dev/sdb "set 1 raid on"
    [root@titan]# parted --script /dev/sdc "set 1 raid on"

    Now we will create the mirrored device using mdadm.

    [root@titan]# mdadm --create /dev/md0 --level=raid1 --raid-devices=2 /dev/sdb1 /dev/sdc1

    The new device needs to sync – we can watch its progress using mdstat.

    [root@titan]# cat /proc/mdstat
    Personalities : [raid1]
    md0 : active raid1 sdc1[1] sdb1[0]
    3906117632 blocks super 1.2 [2/2] [UU]
    [>....................] resync = 0.0% (1893888/3906117632) finish=515.3min speed=126259K/sec
    bitmap: 30/30 pages [120KB], 65536KB chunk

    Note that I pretty much followed this guide line for line. Its an older article, but it checks out.

    Using fdisk we can see our new mirror device.

    [root@titan]# fdisk -l /dev/md0

    Disk /dev/md0: 3999.9 GB, 3999864455168 bytes, 7812235264 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes

    Let’s put a bird on it. No, let’s create a partition on it

    [root@titan]# parted --script /dev/md0 "mklabel gpt"
    [root@titan]# parted --script /dev/md0 "mkpart primary 0% 100%"

    Great work. Now lets take a peak at our new partition.

    [root@titan]# fdisk -l /dev/md0
    Disk /dev/md0: 3999.9 GB, 3999864455168 bytes, 7812235264 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk label type: dos
    Disk identifier: 0x00000000
    Device Boot Start End Blocks Id System
    /dev/md0p1 1 4294967295 2147483647+ ee GPT

    Do you want to create a file system? I know I do. See below. Note the device name that we are using.

    [root@titan]# mkfs -t ext4 /dev/md0p1

    Now lets mount it up.

    [root@titan]# mount /dev/md0p1 /var/lib/libvirt/images/

    Don’t forget to add an entry to “/etc/fstab” so that our filesystem mounts at boot time.

  • RHEV/oVirt – 503 Service Temporarily Unavailable

    OVirt-logo-highres

    This post will walk you through fixing the RHEV/oVirt error shown below. Note that this error was received when trying to load the RHEV-M/oVirt Engine web interface.

    ovirt-rhev

    Note that these instructions worked for me in a known working environment, where the web interface was previously working but had failed after an unscheduled restart of the host.

    First lets restart the ovirt-engine.

    # service ovirt-engine restart
    Stopping oVirt Engine: [FAILED]
    Starting oVirt Engine: [ OK ]

    Next we restart the ovirt-websocket-proxy

    # service ovirt-websocket-proxy restart
    Stopping oVirt Engine websockets proxy: [ OK ]
    Starting oVirt Engine websockets proxy: [ OK ]

    Finally, we restart Apache.

    # service httpd restart
    Stopping httpd: [ OK ]
    Starting httpd: [ OK ]