AI, Virtualization, Containers, Infrastructure, Linux
Now that we have set up the guest virtual machine instances so that they will only be allowed to run on cores 2, 3, 6, and, 7 we must also set up the host processes so that they will not run on these cores – restricting themselves instead to cores 0, 1, 4, and 5. To do this we must set the isolcpus kernel argument – adding this requires editing the system’s boot configuration.
Source: Driving in the Fast Lane – CPU Pinning and NUMA Topology Awareness in OpenStack Compute
In this post we are going to walk through a process you can use to troubleshoot a failed heat stack deploy by viewing information of nested resources.
In the command below we are looking for any failed resource. Our grep allows us to grab the name of Parent id.
[stack@undercloud] # heat resource-list -n5 overcloud | grep -A5 -B5 -i Failed
What is below its very hard to read as I am unable to get the copy/paste to format correctly. What we are looking for is the parent resource for the failed child resource. I have bolded this line below to make is easier to find.
| StoragePort | ed706420-e4e2-4e46-b3d3-8dfd47855a3a | OS::Neutron::Port | CREATE_COMPLETE | 2016-03-03T18:48:42Z | StorageVirtualIP |
| VipPort | 2833b468-5f0f-4162-a0bb-e928126b3767 | OS::Neutron::Port | CREATE_COMPLETE | 2016-03-03T18:48:42Z | RedisVirtualIP |
| 0 | 6c8b39f8-4ed2-47e4-957b-f74b8520d386 | OS::TripleO::Compute | CREATE_IN_PROGRESS | 2016-03-03T18:48:47Z | Compute |
| 0 | 829a6b13-7a92-483a-b9d3-52a99dc655f8 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z | Controller |
| 1 | 556da671-baa6-4fc8-be83-f3acfb324f46 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z | Controller |
| 2 | c28d5942-a206-4734-bfd1-c6d263fafc52 | OS::TripleO::Controller | CREATE_FAILED | 2016-03-03T18:48:49Z | Controller |
| InternalApiPort | 59800552-0b66-4437-bf43-833df8131673 | OS::TripleO::Compute::Ports::InternalApiPort | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
| NetIpMap | 2d455861-3cbe-40cd-8443-f0e63b0c24b7 | OS::TripleO::Network::Ports::NetIpMap | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
| NetworkConfig | 062eb0ed-7914-435b-8f1e-61395fed6548 | OS::TripleO::Compute::Net::SoftwareConfig | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
| NetworkDeployment | 5a11128b-c62a-4720-a8b2-6f27b0d83d74 | OS::TripleO::SoftwareDeployment | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z | 0 |
| NodeUserData | 94deb049-9015-490a-9a2b-925ff17c5c9f | OS::TripleO::NodeUserData | CREATE_COMPLETE | 2016-03-03T18:48:49Z | 0 |
Our parent resource is “Controller” so let’s take a look at that first. We want to grab the “physical_resource_id“. Again, hard to see so I have bolded that line below.
[stack@undercloud] # heat resource-show overcloud Controller
+————————+————————————-+
| Property | Value |
+————————+—————————–+
| attributes | { |
| | “attributes”: null, |
| | “refs”: null |
| | } |
| description | |
| links | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud/054fde22-87f2-44cb-8318-784d1fa46323/resources/Controller (self) |
| | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud/054fde22-87f2-44cb-8318-784d1fa46323 (stack) |
| | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud-Controller-swdfj36bsf6e/6faa959c-3f1b-4f60-aeee-c01e34f399e7 (nested) |
| logical_resource_id | Controller |
| physical_resource_id | 6faa959c-3f1b-4f60-aeee-c01e34f399e7 |
| required_by | ControllerNodesPostDeployment |
| | VipDeployment |
| | ControllerIpListMap |
| | ControllerBootstrapNodeDeployment |
| | ControllerClusterDeployment |
| | CephClusterConfig |
| | ControllerSwiftDeployment |
| | SwiftDevicesAndProxyConfig |
| | ControllerBootstrapNodeConfig |
| | ControllerCephDeployment |
| | allNodesConfig |
| | ControllerAllNodesDeployment |
| | ControllerClusterConfig |
| resource_name | Controller |
| resource_status | CREATE_IN_PROGRESS |
| resource_status_reason | state changed |
| resource_type | OS::Heat::ResourceGroup |
| updated_time | 2016-03-03T18:48:34Z |
+————————+————————————–+
Now that we know the physical_resource_id we can drill down into that resource. Note resource “2” is the one we want to focus on.
[stack@undercloud] # heat resource-list 6faa959c-3f1b-4f60-aeee-c01e34f399e7
+—————+—————————+———————-+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time |
+—————+————————+———————-+
| 0 | 829a6b13-7a92-483a-b9d3-52a99dc655f8 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z |
| 1 | 556da671-baa6-4fc8-be83-f3acfb324f46 | OS::TripleO::Controller | CREATE_IN_PROGRESS | 2016-03-03T18:48:49Z |
| 2 | c28d5942-a206-4734-bfd1-c6d263fafc52 | OS::TripleO::Controller | CREATE_FAILED | 2016-03-03T18:48:49Z |
+—————+———————————————–+
Now we run “heat resource-show” using the physical resource id plus the resource_name (which is 2).
[stack@undercloud] # heat resource-show 6faa959c-3f1b-4f60-aeee-c01e34f399e7 2
+————————+——————————————–+
| Property | Value |
+————————+——————————————–+
| attributes | { |
| | “storage_mgmt_ip_address”: null, |
| | “hostname”: “overcloud-controller-2”, |
| | “config_identifier”: “,”, |
| | “nova_server_resource”: “52872fac-9db2-4913-8e02-281fc01f44eb”, |
| | “tenant_ip_address”: null, |
| | “external_ip_address”: null, |
| | “swift_device”: “r1z1-:%PORT%/d1”, |
| | “corosync_node”: { |
| | “ip”: “172.16.0.136”, |
| | “name”: “overcloud-controller-2” |
| | }, |
| | “hosts_entry”: ” overcloud-controller-2.localdomain overcloud-controller-2 overcloud”, |
| | “swift_proxy_memcache”: “:11211”, |
| | “storage_ip_address”: null, |
| | “internal_api_ip_address”: null, |
| | “ip_address”: “172.16.0.136” |
| | } |
| description | |
| links | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud-Controller-swdfj36bsf6e/6faa959c-3f1b-4f60-aeee-c01e34f399e7/resources/2 (self) |
| | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud-Controller-swdfj36bsf6e/6faa959c-3f1b-4f60-aeee-c01e34f399e7 (stack) |
| | http://172.16.0.10:8004/v1/94508024f96c426abac45b7e1acdfe39/stacks/overcloud-Controller-swdfj36bsf6e-2-j5zi5fahlwf3/c28d5942-a206-4734-bfd1-c6d263fafc52 (nested) |
| logical_resource_id | 2 |
| parent_resource | Controller |
| physical_resource_id | c28d5942-a206-4734-bfd1-c6d263fafc52 |
| required_by | |
| resource_name | 2 |
| resource_status | CREATE_FAILED |
| resource_status_reason | CREATE aborted |
| resource_type | OS::TripleO::Controller |
| updated_time | 2016-03-03T18:48:49Z |
+————————+——————————————————————————————————————–+
Note: The RT-AC66U and the RT-AC66R are identical other than their product number. The RT-AC66R is the product sold through Best Buy and RT-AC66U is the product sold directly from ASUS.
IPKG is a cli utility used for package management. It is required to have ipkg installed if you want to configure your router for SNMP, install BIND, or view I/O to a usb disk as you will need to install software packages that are not included in the stock firmware.
Note: I am running the ASUSWRT-Merlin firmware, you can find it here. The stock firmware may be a bit different, so you might not see the exact screens as shown below.
To install ipkg you will need to install Download Master. To Install Download Master you will need to have a USB drive plugged into the router and mounted. In the image below you can see my USB disk, labeled as “SMI USB DISK”
I had a bit of an issue here as I was trying to mount a drive that was formatted as EXT4. Apparently this was not supported. See error below.
EXT3-fs: sda1: couldn’t mount because of unsupported optional features (240).
EXT2-fs: sda1: couldn’t mount because of unsupported optional features (240).
I pulled the drive, reformatted as EXT3 and was off and running.
usb 1-1.1: USB disconnect, address 4
usb 1-1.2: new high speed USB device using ehci_hcd and address 5
usb 1-1.2: configuration #1 chosen from 1 choice
scsi2 : SCSI emulation for USB Mass Storage devices
scsi 2:0:0:0: Direct-Access SMI USB DISK 1100 PQ: 0 ANSI: 4
sd 2:0:0:0: [sda] 8196096 512-byte hardware sectors (4196 MB)
sd 2:0:0:0: [sda] Write Protect is off
sd 2:0:0:0: [sda] Mode Sense: 43 00 00 00
sd 2:0:0:0: [sda] Assuming drive cache: write through
sd 2:0:0:0: [sda] Assuming drive cache: write through
sda: sda1
sd 2:0:0:0: [sda] Attached SCSI removable disk
sd 2:0:0:0: Attached scsi generic sg0 type 0
You can find the list of supported formats and partition sizes here. I would suggest EXT3. NTFS and FAT32 are supported, but I have heard that you might run into issues.
Now that you have your USB disk mounted you can install Download Master.
Select USB Applications on the left pane. Then click on Download Master. In the example below, you can see that I have already performed the install.
You should see the screen below once Download Master is installed.
Now you can either telnet or SSH to your router using its LAN IP address. Note that telnet is the default protocol, however you can enable SSH by clicking on “Administration” in the lower left pane, and then clicking the “System” tab. Under “SSH Daemon” select “Enable SSH”.
Ipkg installs to the path shown below.
# which ipkg
/opt/bin/ipkg
A guide to using ipkg can be found here.
Open Vswitch is an Open Source software switch designed specifically to be used in virtualized environments such as OpenStack or RHEV-H. OVS (Open Vswitch) was designed to make it easier to manage, configure, and monitor virtual traffic within a virtualized environment.
Below, is Part 1 in what I suspect will be a multiple part series on configuring, viewing, and managing your virtualized network via OVS.
To view OVS Bridges configured on a system, use the command ovs-vsctl as shown below. On my test system, we have three configured bridges; br-ex, br-int, and br-tun.
# ovs-vsctl show
b5aa3cf4-d962-4cb2-b3b6-20f0b4858f59
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port “eth0”
Interface “eth0”
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}Bridge br-int
fail_mode: secure
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internalBridge br-tun
fail_mode: secure
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
ovs_version: “2.4.0”
The output from the command above is rather verbose. If you prefer a more terse output you best try the command below. In this instance the only output is the name of our bridges.
Big Cloud Fabric, is a SDN solution from Big Switch Networks, designed to integrate into OpenStack (or VMware).
Via OpenStack, BCF (Big Cloud Fabric) integrates directly into OpenStack Neutron by way of a plugin. BCF supports L2/L3 networking and L4-7 insertion. BCF runs on whitebox, or brightbox hardware.
Below are a couple of videos that will give you a high-level view of the solution and let you see it in action.
You can also access the Big Switch Labs and play around with the techology via the link below. Note that you will have to provide an email address.
http://labs.bigswitch.com/home
A while ago, I needed to tear down one of the nodes in my RHEV cluster, as I wanted to re-purpose it as a RHEL7 host. A few hours before a planned to shut it down, I logged into my RHEV-M console and put the node in “Maintenance” status.
Later, when I came home, I powered off the node and rebuilt it, not once checking to ensure that the VMs running on the host had migrated properly. They had not, and this was a problem. I unknowingly fubared my cluster.
When I logged back into RHEV-M, I found the several VMs with a status of “?” or “Unknown State“, and a physical host that I was unable to remove from the cluster, despite the fact that it had been rebuilt.
So now what….
There were several VMs that were running on the now retired RHEV-H host when the server was powered off. I needed to fix these before I could remove the host. I tried powering them off, on, or migrating them from the WebUI, but this did not work. So I had to delete them manually from the RHEV-M database. Note that its possible that I could have figured out how to save the VMs instead of deleting them, but this was not a priority since this is my homelab environment.
First lets SSH to the RHEV-M server and log into the database.
# source /etc/ovirt-engine/engine.conf.d/10-setup-database.conf
# export PGPASSWORD=$ENGINE_DB_PASSWORD
#psql -h localhost -U engine engine
Now we figure out the vm_guids for each VM. Below I am starting with my logging server (log.lab.localdomain).
Below we are telling RHEV to mark each VM as Powered Off. Nothing here will delete your VM.
engine=> select vm_guid from vm_static where vm_name = 'log.lab.localdomain';
vm_guid
--------------------------------------
f2c43e33-fd02-4b74-b86c-9e9ff9b8c51b
(1 row)engine=> update vm_dynamic set status = 0 where vm_guid = 'f2c43e33-fd02-4b74-b86c-9e9ff9b8c51b';
UPDATE 1I needed to run through this process a total of four times, once for each VM that was orphaned, scared, and alone.
First open up another ssh session to your RHEV-M host, and stop Jboss
# service jbossas stop
Now back in your other window, which should still be connected to the engine database….
My node is named titan.lab.localdomain – we need to get ids vds_id.
engine=# select vds_id, storage_pool_name from vds where vds_name = 'titan.lab.localdomain' ;
vds_id | storage_pool_name
--------------------------------------+-------------------
e59198b0-fc75-4c5d-b31e-1ab639a1f708 | Auburn_Datacenter
(1 row)
Now lets put it in maintenance mode.
engine=# update vds_dynamic set status = 4 where vds_id = 'e59198b0-fc75-4c5d-b31e-1ab639a1f708';
UPDATE 1
Now lets start jboss in our other window.
# service jbossas start
At this point I was able to log back into the WebUI and remove the host.
Some of the OpenStack director guides that I have seen and read suggest using ACH Tools to perform role matching, which matches Ironic nodes to the most appropriate roles (flavors) bases on their CPU, Memory, and Disk configurations.
However if you prefer to manually assign a role to a node follow the steps outlined below.
Before we can manually assign a role to a flavor, we must first create the flavor. In this example we are going to create 4 flavors.
First lets create the flavor that will be used for our OpenStack Overcloud Controllers.
# openstack flavor create –id auto –ram 8192 –disk 26 –vcpus 4 control
+—————————-+————————————–+
| Field | Value |
+—————————-+————————————–+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 26 |
| id | 4007f8a4-6ea9-400e-9142-6926f0538b29 |
| name | control |
| os-flavor-access:is_public | True |
| ram | 8192 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 4 |
+—————————-+————————————–+
Now lets create the flavor that will be used for our OpenStack Overcloud Compute Nodes.
# openstack flavor create –id auto –ram 6144 –disk 18 –vcpus 2 compute
+—————————-+————————————–+
| Field | Value |
+—————————-+————————————–+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 18 |
| id | 60d05417-d2ba-4f62-a37c-1bafbfaec30a |
| name | compute |
| os-flavor-access:is_public | True |
| ram | 6144 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 2 |
+—————————-+————————————–+
Since our POC deployment will also include CEPH nodes, we need to also create a CEPH role.
# openstack flavor create –id auto –ram 4096 –disk 18 –vcpus 2 ceph
+—————————-+————————————–+
| Field | Value |
+—————————-+————————————–+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 18 |
| id | 5480e8e6-f895-4870-b4dc-e5fff5a32911 |
| name | ceph |
| os-flavor-access:is_public | True |
| ram | 4096 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 2 |
+—————————-+————————————–+
Important Note: You need to have a baremetal role created otherwise your deployment will fail, use the command below to create it.
# openstack flavor create –id auto –ram 8192 –disk 58 –vcpus 4 baremetal
Now let’s check our work with the command below. You should see your 4 roles listed (Baremetal, Ceph, Control, and Compute).
#openstack flavor list