os-collect-config is a tool that starts up via systemd when a system boots. It initially runs at boot time, but continues to run looking for changes in heat metadata. In a nutshell, os-collect-config is responsible for monitoring and downloading metadata from the Heat API.
When data changes, os-collect-config makes a call to os-refresh-config. This data provides the node with all of the information it needs to make configuration changes to the host – this data will be node specific.
Os-collect-config polls for data from sources (nova-metadata and heat) and stores them in /var/lib/os-collect-config/.
os-refresh-config is called by os-collect-config once it recognizes that metadata has changed within the Heat API for that specific node.
The important steps that os-refresh-config take are shown below
1) Apply systemctl configurables
2) Run os-apply-config (see below)
3) Configure the networking for the host
4) Download and set the hieradata files for puppet parameters
5) Configure /etc/hosts
6) Deploy software configuration with Heat
os-apply-config
os-apply-config is called by os-refresh-config to sets up configuration files on specific nodes. It is called via ‘/usr/libexec/os-refresh-config/configure.d/20-os-apply-config‘.
As is does with the undercloud deploy, os-refresh-config executes scripts under /usr/libexec/os-refresh-config/ in a specific order based on numbering.
First scripts within thje pre-configure.d/ directory are run, then configure.d/ scripts are applied, and finally scripts in post-configure.d/.
It is within these scripts that the metadata downloaded by os-collect-config will be acted upon.
Any call to os-apply-config uses the files in /var/lib/os-collect-config as its configuration source.
The appropriate script files for doing so are as follows:
overcloud$ ll /usr/libexec/os-refresh-config/configure.d/
total 32
-rwxr-xr-x. 1 root root 396 Aug 5 07:31 10-sysctl-apply-config
-rwxr-xr-x. 1 root root 42 Aug 5 07:31 20-os-apply-config
-rwxr-xr-x. 1 root root 189 Aug 5 07:31 20-os-net-config
-rwxr-xr-x. 1 root root 629 Aug 5 07:31 25-set-network-gateway
-rwxr-xr-x. 1 root root 2265 Aug 5 07:31 40-hiera-datafiles
-rwxr-xr-x. 1 root root 1387 Aug 5 07:31 51-hosts
-rwxr-xr-x. 1 root root 5784 Aug 5 07:31 55-heat-config
If we run os-apply-config manually, we can see that it does the following:
The directory /os/net-config/ holds the config.json file that is used to modify the networking configuration on each host. The config found in this file is derived from the os-collect-config data in /var/lib/os-collect-config/.
Again you can use this file to review your networking configuration and compare and contrast to your templates. Formatting is off in the file below, but you get the point.
Heat is the main orchestration engine for OpenStack, and is used my OpenStack director to install an OpenStack Overcloud environment.
When we run the “openstack deploy overcloud” command, we are specifically telling RHEL OSP director that we want it to use the pre-defined Heat templates from /usr/share/openstack-tripleo-heat-templates/. OSP director will manage the deployment of a new overcloud heat stack, using files from this directory. When RHEL OSP director calls the Heat stack, it needs the following data…
A top-level Heat template to use that describes the overall environment and the resources required.
An environment/resource registry to tell Heat where to find resource definitions for non-standard Heat elements, e.g. TripleO components.
A set of parameters to declare the deployment-specific options (via -e)
The most important files for us to focus on are in our deployment directory, these are the default files that get called by OSP director.
The top-level Heat template that OSP director uses for deployment is /usr/share/openstack-tripleo-heat-templates/overcloud-without-mergepy.yaml
The resource registry, which tells Heat where to find the templates for deployment resources is /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet. yaml
Creating a Heat Stack
To create the stack we run the command below. This command instructs heat to use the templates in ~/my_templates/, as well as the override templates specified with the ‘-e’ option.
This is just an example of what I am using in my lab environment, your deploy command will be much different. Also note that I have copied the templates from /usr/share/openstack-tripleo-heat-templates to ~/my_templates/.
Unfortunately our deploy failed with the following errors.
Exception: Heat Stack create failed. DEBUG: openstackclient.shell clean_up DeployOvercloud DEBUG: openstackclient.shell got an error: Heat Stack create failed. ERROR: openstackclient.shell Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 176, in run return super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 230, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 864, in take_action self._deploy_tripleo_heat_templates(stack, parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 535, in _deploy_tripleo_heat_templates parsed_args.timeout) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 478, in _heat_deploy raise Exception("Heat Stack create failed.") Exception: Heat Stack create failed.
We can verify that the deploy failed with the command below.
The failed resources are named “Compute” and “Controller“. Lets take a closer look at those using the “resource-show” argument.
#heat resource-show overcloud Compute
| resource_status_reason | ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: StackValidationFailed: Property error : OsNetConfigImpl: config The Parameter (BondInterfaceOvsOptions) was not provided."" |
Let’s now do the same for Controller.
#heat resource-show overcloud Controller
| resource_status_reason | ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: StackValidationFailed: Property error : OsNetConfigImpl: config The Parameter (BondInterfaceOvsOptions) was not provided."" |
Apparently I have some issues with my OVS bonding options, so I need to get those straight before I can continue.
Deleting a Failed Heat Stack
Since our last deploy failed, we need to delete the failed stack before we can kick off another stack deploy. Below is an example of that command – note we are using the UUID of the stack.
Ok, let’s take a look at /var/log/heat/heat/heat-engine.log for more details. I also suggest opening another ssh session and tailing the log while the delete is attempting to do its thing.
If the output is too verbose to follow, I suggest attempting to thin out the output using the command below
Apparently SELinux is blocking the reads for the certificates. There are two ways to work around this issue. You can run “restorecon -v /path/to/certs/“, or you can work around by disabling selinux by running “setenforce 0” or by editing the /etc/selinux/config file and setting ‘SELINUX=DISABLED’. You may need to rerun the delete, in my case it was stuck in “DELETE_IN_PROGRESS”. I restarted all heat releated services to force the delete to error.
I decided to local host a small Minecraft server on a RHEL 7 vm. There are tons of guides on how to install Minecraft on Linux, but I did not find much of anything that details how to get the software to start at boot-time via systemd.
Below are the steps that I took. These instructions should work on RHEL 6/7, Centos 6/7, or any other Linux that utilizes systemd for starting services.
Change directory to the systemd directory.
# cd /etc/systemd/system
Create a systemd service file using your favorite text editor. Mine is called minecraft.service.
Next, create a startup script. Again use your favorite text editor. In the example above, systemd is configured to run the script “/root/start_minecraft_server.bash”. We now need to create that script. Its contents are below.
So this one is pretty simple. However, I found a lot of misinformation along the way, so I figured that I would jot the proper (and most simple) process here.
Symptoms: a RHEL (or variant) VM that takes a very long time to boot. On the VM console, you can see the following output while the VM boot process is stalled and waiting for a timeout. Note that the message below has nothing to do with cloud init, but its the output that I have most often seen on the console while waiting for a VM to boot.
[106.325574} random: crng init done
Note that I have run into this issue in both OpenStack (when booting from external provider networks) and in KVM.
Upon initial boot of the VM, run the command below.
touch /etc/cloud/cloud-init.disabled
Seriously, that’s it. No need to disable or remove cloud-init services. See reference.
TestPMD is a lightweight application running in user space, utilizing ovs-dpdk, that can be used for testing DPDK in packet forwarding mode.
In this example we want to setup TestPMD on a RHEL VM running in our SR-IOV capable Red Hat OpenStack 10 overcloud. Our passthrough adapters are Intel X520s. Our plan here is to run performance tests via an external load generator.
Before we can get started we need to build a test VM.
VM Details
RHEL 7.x
Two VFs
eth0 – ssh access via admin network
eth1 – load generator private network
4 vCPUs
4096 MB Mem
150 GB Disk
Deploy RHEL VM
Your first step is to deploy your RHEL VM, configure your primary network interface (eth0 for ssh) via VM console. Eth1 needs to be up and configured to start at boot, but do not assign it an IP address. Next, register your VM with your local satellite server or with RH CDN.
Download DKDP
Use the link below to download the “Latest Major” version of DPDK. Place the tarfile in /root on the VM and untar.
Install Prerequisites
Before we can compile DPDK, we need to install a few prereqs.
In this test we are using the Intel x520 NIC, which is directly accessible to our VM via SR-IOV passthrough. If you are passing through a different NIC, your process will differ.