Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Tag: openstack.

  • OpenStack Ironic Troubleshooting – Neutron Port in Use

    keyboard-key-board-melt-broken-computer_p

    Ran into issues with a deploy via RHEL OSP director, which caused our heat stack scale compute scale-out to fail.

    We corrected the issue, and then attempted our deploy again. This time around, we were able to scale out by several compute nodes. However, several nodes failed to deploy properly.

    After trudging through the logs for a bit we were able to find the error below in /var/log/nova-conductor.log

    2016-07-25 18:34:39.453 27374 ERROR nova.scheduler.utils [req-1caa3ca0-9e13-4340-b8b2-80b1cf2f8a7f 8408462bd5a8445c9742ea4dfbc20d70 cc44dc9e68064e64899697ac610c8f06 – – -] [instance: e44e3a04-a47e-4a8e-9eb5-0037c1175e4d] Error from last host: fatmin.lab.localdomain (node c5177fbf-ae0f-49db-94f4-087537b3dd53): [u’Traceback (most recent call last):\n’, u’ File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 1905, in _do_build_and_run_instance\n filter_properties)\n’, u’ File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 2058, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n’, u’RescheduledException: Build of instance e44e3a04-a47e-4a8e-9eb5-0037c1175e4d was re-scheduled: Port 14:18:77:3e:1a:bf is still in use.\n’]

    To resolve we ran the following command, using the MAC address shown above to narrow down the search.

    #neutron port-list | grep “14:18:77:3e:1a:bf”
    | 25a366df-1e6c-4eb8-853c-5b7db82637f0 | | 14:18:77:3e:1a:bf | {“subnet_id”: “0310f210-63ad-4616-9338-d59ac13cc0be”, “ip_address”: “10.20.0.134”} |

    The command “neutron port show“, showed us that this port was down and was not responding to ping.

    We then deleted the port via neutron

    #neutron port-delete 25a366df-1e6c-4eb8-853c-5b7db82637f0

    We then re-ran our deploy and were able to scale without issue.

     

  • NUMA Node to PCI Slot Mapping in Red Hat Enterpise Linux

    NUMA Node to PCI Slot Mapping in Red Hat Enterpise Linux

     

    understanding-dpdk-31-638
    Sandybridge I/O Controller to PCI-E Mapping

     

    Using a few simple commands you can easily map a PCI slot back to its directly connected NUMA node. This information comes in very handy when implementing NFV leveraged technologies such as CPU Pinning and SRIOV.

     

    First, you will need to install hwloc and hwloc-gui, if it is not already installed on your system. hwloc-gui provides the lstopo command, so you will need to install the gui package even if you are going to run the command on a headless system.

    # yum -y install hwloc.x86_64 hwloc-gui.x86_64

    Now you can run lstopo. Below is the output from one of my dual socket, quad core Xeon systems.

    # lstopo
    Machine (40GB)
    NUMANode L#0 (P#0 16GB) + Socket L#0 + L3 L#0 (8192KB)
    L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
    PU L#0 (P#0)
    PU L#1 (P#8)
    L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
    PU L#2 (P#1)
    PU L#3 (P#9)
    L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
    PU L#4 (P#2)
    PU L#5 (P#10)
    L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
    PU L#6 (P#3)
    PU L#7 (P#11)
    NUMANode L#1 (P#1 24GB) + Socket L#1 + L3 L#1 (8192KB)
    L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
    PU L#8 (P#4)
    PU L#9 (P#12)
    L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
    PU L#10 (P#5)
    PU L#11 (P#13)
    L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
    PU L#12 (P#6)
    PU L#13 (P#14)
    L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
    PU L#14 (P#7)
    PU L#15 (P#15)
    HostBridge L#0
    PCIBridge
    PCI 8086:10c9
    Net L#0 "enp8s0f0"
    PCI 8086:10c9
    Net L#1 "enp8s0f1"
    PCIBridge
    PCIBridge
    PCIBridge
    PCI 8086:10e8
    Net L#2 "enp5s0f0"
    PCI 8086:10e8
    Net L#3 "enp5s0f1"
    PCIBridge
    PCI 8086:10e8
    Net L#4 "enp4s0f0"
    PCI 8086:10e8
    Net L#5 "enp4s0f1"
    PCIBridge
    PCI 102b:0532
    GPU L#6 "card0"
    GPU L#7 "controlD64"
    PCI 8086:3a22
    Block L#8 "sr0"
    Block L#9 "sda"
    Block L#10 "sdb"
    Block L#11 "sdc"

    The first 27 lines of output tell you which cores are in each socket.

    Lines starting with “HostBridge L#0” list the PCI devices attached to socket 0. On more modern dual socket systems (think Sandybridge) you would have a “HostBridge L#8” section as well.

     

    “The PCI host bridge provides an interconnect between the processor and peripheral components. Through the PCI host bridge, the processor can directly access main memory independent of other PCI bus masters. For example, while the CPU is fetching data from the cache controller in the host bridge, other PCI devices can also access the system memory through the host bridge. The advantage of this architecture lies in its separation of the I/O bus from the processor’s host bus.”

    Unfortunately, my lab systems are Nehalem based machines which implement what is called QPI to share a host bridge between CPU sockets.  See image below.

     

    019_QPI_1IOH
    Nehalem QPI Architecture

    Nonetheless, we are able to determine which CPU socket is associated with a specific PCI device. For this example, we will focus on the devices below since they are both directly attached to the PCI Host Bridge and not the PCI Bus.

     

    HostBridge L#0
    PCIBridge
    PCI 8086:10c9
    Net L#0 "enp8s0f0"
    PCI 8086:10c9
    Net L#1 "enp8s0f1"

    Now using the lspci command I can find the exact devices per NUMA node.

    lspci -nn | grep 8086:10c9
    08:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
    08:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)

     

     

     

     

  • Mapping Libvirt VM Names with OpenStack Instance Names

    openstack

    Within OpenStack, each virtual machine instance running on a Compute nodes also a virtual machine running on a libvirt node.

    If you ssh to a Compute node and run the command below you can get the names of each VM running or registered on this Compute node.

    # virsh list –all
    Id Name State
    —————————————————-
    2 instance-000000f0 running
    – instance-00000024 shut off
    – instance-00000039 shut off
    – instance-000000ea shut off

    So there is only one VM currently running on this Compute node, but which VM is it?

    Well, we can figure that out pretty easily. See below.

    # virsh dumpxml instance-000000f0 | grep uuid | grep name

    <entry name=’uuid’>3103d38c-447d-40af-9607-56b26473ee72</entry>

    Now we just have to map this UUID back to an OpenStack instance name

    Here we have a nasty little awk grep to get the UUID and name of each OpenStack instance running in our cluster.

    # nova list | grep -v “+” | grep -v ID | awk ‘{print $2 $3 $4}’

    a402716b-73d0-4303-9331-202bc2386ab8|storage-perf-a
    bd73f092-88c9-4af1-b569-c1176290841c|storage-perf-b
    0ab8decb-6623-4e68-b8e4-b9cd522f6ea9|storage-perf-c
    493c8afe-836e-4846-bdd0-029bb6e7f70e|storage-perf-d

    Note that you can also get the instance name using a UUID,  see below

    #nova show db79f6a2-455e-4f17-88d0-b3018d279c7c | grep instanceOS-EXT-SRV-ATTR:instance_name | instance-0000002a

    OS-EXT-SRV-ATTR:instance_name | instance-0000002a

     

    Update:

    Thanks to TB for showing me this one. You can accomplish all of this via nova cli using the FQDN of the host.

    (overcloud) [stack@undercloud ~]$ nova list --all --host comp39.localdomain --fields=name,host,instance_name,status
+--------------------------------------+----------------------+-------------------------------+-------------------+--------+
| ID | Name | Host | Instance Name | Status |
+--------------------------------------+----------------------+-------------------------------+-------------------+--------+
| 30fccd83-e517-4e56-a1df-5d5d454818ff | vm-name1 | comp39.localdomain | instance-0023a423 | ACTIVE |
| bc6bb4da-1cac-4603-ad52-2078f13cf6fd | vm-name-2 | comp39.localdomain | instance-0023a348 | ACTIVE |
+--------------------------------------+----------------------+-------------------------------+-------------------+--------+

     

     

  • Red Hat OpenStack Technical Preview Features

    redhat-logo

     

    According to this page,  a Technology Preview is a feature that is  currently unsupported, may not have complete functionality, and are not suitable for deployment in production. However, Red Hat provides these features and makes them available to the customer as a courtesy with the primary goal of exposing the feature to a wider audience.

    I am quite often asked by my clients, which features are in tech preview for each release.  Instead of spending time looking these things up each time I figured I would document them here.

     

  • How to Determine Installed OpenStack director Version

    OpenStack-logo

    Trying to determine that version of Red Hat OpenStack director that is installed on an Undercloud Controller is not as straight forward as one would hope.

    I have created the simple table below that you can use to determine your installed director version via the installed version of the openstack-tripleo-heat-templates rpm.

    Minimum RPM Version Release Date director Version
    openstack-tripleo-heat-templates-0.8.6-123.el7ost UNK 7.3.1
    openstack-tripleo-heat-templates-0.8.6-94.el7ost.noarch.rpm 2015-12-16 7.2
    openstack-tripleo-heat-templates-0.8.6-71.el7ost.noarch.rpm 2015-10-05 7.1
    openstack-tripleo-heat-templates-0.8.6-45.el7ost.noarch.rpm 2015-07-30 7.0
    openstack-tripleo-heat-templates-0.8.14-11.el7ost.noarch 2016-04-20 8.0
    oopenstack-tripleo-heat-templates-2.0.0-41.el7ost.noarch ?? 9.0
    openstack-tripleo-heat-templates-5.2.0-15.el7ost.noarch ?? 10.0
    openstack-tripleo-heat-templates-6.0.0-10.el7ost.noarch ?? 11.0

    Hopefully, in the near future, we will see an rpm similar to redhat-release-server (the rpm that create /etc/redhat-release) included in future OSP director updates. Until then, this table should be of some assistance.

    Related Bugzillas

    https://bugzilla.redhat.com/show_bug.cgi?id=1295512

    https://bugzilla.redhat.com/show_bug.cgi?id=1085625

    Secondary Method – Updated 9/20/2016

    You can also run the following command on your overcloud

    [root@tpacpuictrl0 nova]# nova-manage –version
    2015.1.3

    Now cross reference this info with this webpage.

    https://wiki.openstack.org/wiki/Releases

  • OpenStack: Viewing a Child Heat Resource

    empty birds nest

    In this post we are going to walk through a process you can use to troubleshoot a failed heat stack deploy by viewing information of nested resources.

    In the command below we are looking for any failed resource. Our grep allows us to grab the name of Parent id.

    [stack@undercloud] # heat resource-list -n5 overcloud | grep -A5 -B5 -i Failed

    What is below its very hard to read as I am unable to get the copy/paste to format correctly. What we are looking for is the parent resource for the failed child resource. I have bolded this line below to make is easier to find.

    | StoragePort | ed706420-e4e2-4e46-b3d3-8dfd47855a3a | OS::Neutron::Port | CREATE_COMPLETE | 2016-03-03T18:48:42Z | StorageVirtualIP |
    | VipPort | 2833b468-5f0f-4162-a0bb-e928126b3767 | OS::Neutron::Port | CREATE_COMPLETE | 2016-03-03T18:48:42Z | RedisVirtualIP |
    | 0 | 6c8b39f8-4ed2-47e4-957b-f74b8520d386 | OS::TripleO::Compute | CREATE_IN_PROGRESS | 2016-03-03T18:48:47Z | Compute |
    | 0 | 829a6b13-7a92-483a-b9d3-52a99dc655f8 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z | Controller |
    | 1 | 556da671-baa6-4fc8-be83-f3acfb324f46 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z | Controller |
    | 2 | c28d5942-a206-4734-bfd1-c6d263fafc52 | OS::TripleO::Controller | CREATE_FAILED | 2016-03-03T18:48:49Z | Controller |
    | InternalApiPort | 59800552-0b66-4437-bf43-833df8131673 | OS::TripleO::Compute::Ports::InternalApiPort | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
    | NetIpMap | 2d455861-3cbe-40cd-8443-f0e63b0c24b7 | OS::TripleO::Network::Ports::NetIpMap | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
    | NetworkConfig | 062eb0ed-7914-435b-8f1e-61395fed6548 | OS::TripleO::Compute::Net::SoftwareConfig | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
    | NetworkDeployment | 5a11128b-c62a-4720-a8b2-6f27b0d83d74 | OS::TripleO::SoftwareDeployment | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z | 0 |
    | NodeUserData | 94deb049-9015-490a-9a2b-925ff17c5c9f | OS::TripleO::NodeUserData | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |

     

    Our parent resource is “Controller” so let’s take a look at that first. We want to grab the “physical_resource_id“. Again, hard to see so I have bolded that line below.

    [stack@undercloud] # heat resource-show overcloud Controller

    +————————+————————————-+
    | Property | Value |
    +————————+—————————–+
    | attributes | { |
    | | “attributes”: null, |
    | | “refs”: null |
    | | } |
    | description | |
    | links | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud/

    054fde22-87f2-44cb-8318-784d1fa46323/resources/Controller (self) |
    | | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud/

    054fde22-87f2-44cb-8318-784d1fa46323 (stack) |
    | | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/

    overcloud-Controller-swdfj36bsf6e/6faa959c-3f1b-4f60-aeee-c01e34f399e7 (nested) |
    | logical_resource_id | Controller |
    | physical_resource_id | 6faa959c-3f1b-4f60-aeee-c01e34f399e7 |
    | required_by | ControllerNodesPostDeployment |
    | | VipDeployment |
    | | ControllerIpListMap |
    | | ControllerBootstrapNodeDeployment |
    | | ControllerClusterDeployment |
    | | CephClusterConfig |
    | | ControllerSwiftDeployment |
    | | SwiftDevicesAndProxyConfig |
    | | ControllerBootstrapNodeConfig |
    | | ControllerCephDeployment |
    | | allNodesConfig |
    | | ControllerAllNodesDeployment |
    | | ControllerClusterConfig |
    | resource_name | Controller |
    | resource_status | CREATE_IN_PROGRESS |
    | resource_status_reason | state changed |
    | resource_type | OS::Heat::ResourceGroup |
    | updated_time | 2016-03-03T18:48:34Z |
    +————————+————————————–+

    Now that we know the physical_resource_id we can drill down into that resource.  Note resource “2”  is the one we want to focus on.

    [stack@undercloud] # heat resource-list 6faa959c-3f1b-4f60-aeee-c01e34f399e7
    +—————+—————————+———————-+
    | resource_name | physical_resource_id | resource_type | resource_status | updated_time |
    +—————+————————+———————-+
    | 0 | 829a6b13-7a92-483a-b9d3-52a99dc655f8 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z |
    | 1 | 556da671-baa6-4fc8-be83-f3acfb324f46 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z |
    | 2 | c28d5942-a206-4734-bfd1-c6d263fafc52 | OS::TripleO::Controller | CREATE_FAILED | 2016-03-03T18:48:49Z |
    +—————+———————————————–+

    Now we run “heat resource-show” using the physical resource id plus the resource_name (which is 2).

    [stack@undercloud] # heat resource-show 6faa959c-3f1b-4f60-aeee-c01e34f399e7 2
    +————————+——————————————–+
    | Property | Value |
    +————————+——————————————–+
    | attributes | { |
    | | “storage_mgmt_ip_address”: null, |
    | | “hostname”: “overcloud-controller-2”, |
    | | “config_identifier”: “,”, |
    | | “nova_server_resource”: “52872fac-9db2-4913-8e02-281fc01f44eb”, |
    | | “tenant_ip_address”: null, |
    | | “external_ip_address”: null, |
    | | “swift_device”: “r1z1-:%PORT%/d1”, |
    | | “corosync_node”: { |
    | | “ip”: “172.16.0.136”, |
    | | “name”: “overcloud-controller-2” |
    | | }, |
    | | “hosts_entry”: ” overcloud-controller-2.localdomain overcloud-controller-2 overcloud”, |
    | | “swift_proxy_memcache”: “:11211”, |
    | | “storage_ip_address”: null, |
    | | “internal_api_ip_address”: null, |
    | | “ip_address”: “172.16.0.136” |
    | | } |
    | description | |
    | links | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/

    overcloud-Controller-swdfj36bsf6e/6faa959c-3f1b-4f60-aeee-c01e34f399e7/resources/2 (self) |
    | | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud-Controller-swdfj36bsf6e/6faa959c-3f1b-4f60-aeee-c01e34f399e7 (stack) |
    | | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud-Controller-swdfj36bsf6e-2-j5zi5fahlwf3/c28d5942-a206-4734-bfd1-c6d263fafc52 (nested) |
    | logical_resource_id | 2 |
    | parent_resource | Controller |
    | physical_resource_id | c28d5942-a206-4734-bfd1-c6d263fafc52 |
    | required_by | |
    | resource_name | 2 |
    | resource_status | CREATE_FAILED |
    | resource_status_reason | CREATE aborted |
    | resource_type | OS::TripleO::Controller |
    | updated_time | 2016-03-03T18:48:49Z |
    +————————+——————————————————————————————————————–+

  • Red Hat Openstack & Big Cloud Fabric Introduction

    BSN-Logo-color

     

    Big Cloud Fabric, is a SDN solution from Big Switch Networks, designed to integrate into OpenStack (or VMware).

    Via OpenStack, BCF (Big Cloud Fabric) integrates directly into OpenStack Neutron by way of a plugin. BCF supports L2/L3 networking and L4-7 insertion. BCF runs on whitebox, or brightbox hardware.

    Below are a couple of videos that will give you a high-level view of the solution and let you see it in action.

    Red Hat & Big Switch: Integrated OpenStack Solution for Simplified Cloud Deployment:

    Webinar – Unified P+V Networking for OpenStack Clouds:

     

    You can also access the Big Switch Labs and play around with the techology via the link below. Note that you will have to provide an email address.

    http://labs.bigswitch.com/home