The downstream version of RDO – Manager, which is the current installer that is shipped with the current release of RDO, will be called RHEL OSP director when it is released as part of RHEL OSP 7 (Openstack Kilo).
OSP- director (note the lowercase d in director) will not only install RHEL OSP 7, it will also handle updating and upgrading as well as lifecycle management.
These three additional functionalities are not currently available in RHEL OSP 6 – Foreman/Staypuft installer.
Below is a nice demo of the upstream version of the installer
This process can be used whenever your databases are out of sync. For example when someone without thinking reboots all nodes in a cluster without shutting down the databases first.
Resolution
Place all cluster hosts into standby mode and cleanup any failed resources.
I sugguest making a back-up of Mariadb on each controller nodes – just in case.
Then on each Controller Node – verify that mariadb (mysql) has stopped. If any instance has not stopped properly via PCS as shown below, please stop them manually.
# systemctl status mariadb.service
mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled)
Active: inactive (dead)
Find the latest (i.e – largest) version of the DB (seqno), or choose a node if all have the same version. In my instance, controller3 had the largest/highest version number, so we will be doing most of our recovery work on the instance below.
Stop puppet if it is running and edit the /etc/my.cnf.d/galera.cnf file. If you are not running puppet, or if puppet is not managing your Galera cluster you can skip this step and go ahead and edit the file.
root@controller3 # systemctl stop puppet
Set wsrep_cluster_address manually in /etc/my.cnf.d/galera.cnf making a note of the original value and restart mariadb manually.
For example, your default configuration should look something like what is shown below. Each IP address listed is the IP address of a mariadb instance in your cluster.
# Group communication system handle
wsrep_cluster_address="gcomm://172.17.9.23,172.17.9.24,172.17.9.22"
Modify to this by commenting out the default string and adding the string shown in the example below. Note that in this example we are working on controller3.
# Group communication system handle
#wsrep_cluster_address="gcomm://172.17.9.23,172.17.9.24,172.17.9.22"
wsrep_cluster_address="gcomm://"
Now restart Mariadb manually on the controller3.
root@controller3 # systemctl start mariadb
Now start mariadb on one of the remaining controllers
root@controller2 # systemctl start mariadb
Below you can see the newly started Maridb instance as requested to sync
Jun 15 10:08:02 controller2 mysqld_safe[24999]: 150615 10:08:02 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Jun 15 10:08:02 controller2 mysqld_safe[24999]: 150615 10:08:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Jun 15 10:08:02 controller2 mysqld_safe[24999]: 150615 10:08:02 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsr...ver.pid'
Jun 15 10:08:04 controller2 mysqld_safe[24999]: 150615 10:08:04 mysqld_safe WSREP: Recovered position 2fb3bbe0-eed6-11e4-ac79-4b77a337d598:12175047
Jun 15 10:08:06 controller2 systemd[1]: Started MariaDB database server.
Once running you can un-standby the nodes and monitor the status as the remaining resources are loaded.
root@rcontroller3 # pcs cluster unstandby --all
root@node # pcs status
Finally, correct the value of wsrep_cluster_address to it’s original value in /etc/my.cnf.d/galera.cnf and restart the service, monitoring that the resource remains active.
root@controller3 # vi /etc/my.cnf.d/galera.cnf
root@controller3 # systemctl restart mariadb
root@controller3 # pcs status
Now check your databases to make sure that they are in sync. The following file should be the same on each controller.
oslo.messaging._drivers.impl_rabbit PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - cannot redeclare exchange 'openstack' in vhost '/' with different type, durable, internal or autodelete value
EXAMPLE 2:
2015-06-08 09:52:17.367 8437 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to declare consumer for topic 'cinder-scheduler': Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'cinder-scheduler' in vhost '/' not equivalent
Warning
Note that you need to make sure that you are only working with OpenStack specific queues.
Cause
This is caused by a mismatch in the durable queues setting in the configuration files on the controller nodes. You cannot just change this setting to correct the issue, you must manually delete the affected queues.
The ccommand above will give you a list of all the queues, so now you can search for the affected queue and delete it as shown. In this example I am deleting the queue “notifications.info”.
To access a novnc console in OpenStack without logging into the Horizon WebUi, follow the steps below.
First and foremost you need to find your virtual instance. In this example I am looking for one of my test instances.
# nova list | grep test
Note that if you are logged in as root and the instance you are looking for was created under a guest tenant, then the command is a bit different. See below.
# nova list –all-tenants | grep test
The nova list command above can spew out a lot of information if you have many test instances. However this is the one that I am looking for specifically.
On this standalone host we have just added an additional disk, which has been enumerated as “/dev/vdc”. First lets go ahead and create a partition on this disk and label it with the type “lvm”.
Now lets view the partition that we created. Note that our new disk is approx 21gb in size. Since we are working in our lab, we are not working with large disks.
Now lets make sure that cinder services are up and running.
[root@packstack cinder(keystone_admin)]# systemctl | grep cinder | awk '{print $1}'| xargs -tn1 systemctl status | grep Active systemctl status openstack-cinder-api.service Active: active (running) since Tue 2015-04-28 17:02:29 EDT; 1min 21s ago systemctl status openstack-cinder-backup.service Active: active (running) since Tue 2015-04-28 17:02:29 EDT; 1min 21s ago systemctl status openstack-cinder-scheduler.service Active: active (running) since Tue 2015-04-28 17:02:29 EDT; 1min 21s ago systemctl status openstack-cinder-volume.service Active: active (running) since Tue 2015-04-28 17:02:30 EDT; 1min 20s ago
Now we must declare the new volume as block storage using the commands below.
[root@packstack cinder(keystone_admin)]# cinder type-create lvm1 +--------------------------------------+------+ | ID | Name | +--------------------------------------+------+ | 082fc208-9d38-4ad4-ab48-110d056a2856 | lvm1 | +--------------------------------------+------+
Now lets link the backend volume to the newly created type-key.
[root@packstack cinder(keystone_admin)]# cinder type-key lvm1 set volume_backend_name=cinder-volumes-1
Below you can see our new cinder backend volume, and its backend name.
Looking to get started with the nova command, well look no further. Here are a few simple commands that you can use to get started. Nothing fancy here, rather these commands serve as a basic introduction to nova command and the OpenStack CLI.
Note that the following commands are all run on a standalone OpenStack install deployed via the PackStack installer. Not that this matters all that much, but it will explain the hostname of my controller/compute node.
List all uploaded images. Here you can see that I only have one active image available via Glance.
[root@packstack ~(keystone_admin)]# nova image-list
+————————————–+————–+——–+——–+
| ID | Name | Status | Server |
+————————————–+————–+——–+——–+
| b60448e9-48b8-4727-9cb8-415662ec3bf8 | Cirros-0.3.3 | ACTIVE | |
+————————————–+————–+——–+——–+
Show detailed information pertaining to a specific image.
In this instance I am using the ID from the entry above.
[root@packstack ~(keystone_admin)]# nova image-show b60448e9-48b8-4727-9cb8-415662ec3bf8
+———————-+————————————–+
| Property | Value |
+———————-+————————————–+
| OS-EXT-IMG-SIZE:size | 13200896 |
| created | 2015-04-14T22:02:17Z |
| id | b60448e9-48b8-4727-9cb8-415662ec3bf8 |
| minDisk | 0 |
| minRam | 0 |
| name | Cirros-0.3.3 |
| progress | 100 |
| status | ACTIVE |
| updated | 2015-04-14T22:02:18Z |
+———————-+————————————–+
List all hypervisors in an OpenStack Cluster…
[root@packstack ~(keystone_admin)]# nova hypervisor-list
+—-+———————+
| ID | Hypervisor hostname |
+—-+———————+
| 1 | packstack.local |
+—-+———————+
Here you can see both the private ip address, and the public, floating ip address of each instance.
[root@packstack ~(keystone_admin)]# nova list –all_tenants
+————————————–+———–+——–+————+————-+————————————-+
| ID | Name | Status | Task State | Power State | Networks |
+————————————–+———–+——–+————+————-+————————————-+
| 473170c0-c7b5-4da7-909a-682b14bba1ae | instance1 | ACTIVE | – | Running | Private=172.16.0.2, 192.168.122.201 |
| 3c25fa8a-0476-40fd-9b27-f4c0c7082dc1 | instance2 | ACTIVE | – | Running | Private=172.16.0.4, 192.168.122.202 |
+————————————–+———–+——–+————+————-+————————————-+
Get detailed information on running instances.
Note that this output is a bit ugly below. It will look better on your console.
[root@packstack ~(keystone_admin)]# nova-manage vm list
instance node type state launched image kernel ramdisk project user zone index
instance1 packstack.local m1.tiny active 2015-04-14 22:08:24 b60448e9-48b8-4727-9cb8-415662ec3bf8 0254b6c215de43e697256fa27bcd3477 a2be9c15f7cd49a3afbfae3963cf0553 nova 0
instance2 packstack.local m1.tiny active 2015-04-27 22:33:50 b60448e9-48b8-4727-9cb8-415662ec3bf8 0254b6c215de43e697256fa27bcd3477 a2be9c15f7cd49a3afbfae3963cf0553 nova 0
Sitting in my hotel room today, I kept running into this error while trying to install OpenStack on a RHEL 7.1 VM running on my laptop. Digging through logs was not helping me one bit, and neither was trying to run “puppet apply” on the failing puppet manifests to see if I could get more info with which to troubleshoot.
Below is the specific error that I was running into. Note that my RHEL VM’s IP address is 192.168.122.75. This IP address is pre-pended to the puppet module names. Your output, will obviously, vary. Note that this output is truncated.
ERROR : Error appeared during Puppet run: 192.168.122.75_amqp.pp
Error: Could not start Service[rabbitmq-server]: Execution of ‘/usr/bin/systemctl start rabbitmq-server’ returned 1: Job for rabbitmq-server.service failed. See ‘systemctl status rabbitmq-server.service’ and ‘journalctl -xn’ for details.
You will find full trace in log /var/tmp/packstack/20150415-183003-mn6Kfx/manifests/192.168.122.75_amqp.pp.log
Please check log file /var/tmp/packstack/20150415-183003-mn6Kfx/openstack-setup.log for more information
Additional information:
Each and every time, the failure occurred when the installer was trying to install/start and rabbitmq-server via the puppet module amqp.pp. Attempting to start rabbitmq manually yielded the same result.
In this instance, I was trying to be fancy and I had given my VM the hostname packstack01.local (instead of sticking with localhost).
Fresh out of any good ideas, I noticed that a simple nslookup on my made up hostname actually returned results. Results that I would not have expected to be valid.
Despite never referencing my made up hostname in my answer file (by default, the answer file is generated with IP addresses only) the Rabbitmq service was attempting to connect to itself via hostname, which obviously failed as this is a valid ip and since I was working in a hotel room without proper dns, my server was trying to connect to a machine on the opposite side of the country.
A quick bit of tinkering in the /etc/hosts file resolved this issue, and I was able to complete my install.
Note that there are probably many other reasons why one might run into this error during an OpenStack install via Packstack, however this is the one that I ran into, and thankfully it was easy to fix.
Note to self – always use localhost when working without a valid DNS entry.