When deploying OpenStack via Red Hat OSP director you configure the hostname of your baremetal (ironic) nodes at time of import. This is done via json file, by default named instack-env.json (but often re-named, nodes.json). Below is an excerpt from that file.
In the sample instance above, I am importing a node named, “fatmin-ctrl01”. This will be the server name as it appears in Ironic. When heat deploys the overcloud, this node will by default be renamed overcloud-controller0, and any controller nodes will iterate by 1. Same situation for compute nodes.
What is preferable is to configure what is referred to as “Predictable Hostnames”. Using “Predictable Hostnames” we can do one of two things.
Specify the hostname format to use and allow nova to iterate through nodes on its own.
Specify the exact hostname for nova to use for each baremetal node
Nova Scheduler Hints
Before we can use either of the two options above, we must first update each baremetal node with a nova scheduler hint. In the examples below we are tagging one node to build as controller-0 (overcloud-controller0) and one node to build as (overcloud-compute-0).
Using the method above the first compute node will be names fatmin-controller-01, and the first compute node will be names fatmin-compute-01. Additional nodes will iterate the index.
While this is nice, as it allows us to set a customized hostname format for each type of node, it does not allow us to specify the exact hostname to be used for a specific ironic node. We can do that will the HostnameMap.
HostnameMap
Now you may want to take this a bit further. You may want to use a custom nova name for each node compute/controller node. You can accomplish this using a HostnameMap as shown below.
Note, when specifying the flavor profiles in the deploy command for preassigned nodes, they should be specified as ‘baremetal‘ instead of ‘control‘ and ‘compute‘. This means that you will not have to assign a profile to each host. You will let the nova scheduler hints handle the decision
So at this point – we will be able to allign the compute or controller index in ironic, with the index in Ironic. For example you can now map your ironic-node name (for example) fatmin-ctrl0 to fatmin-controller0.
Special Notes for Special People
I do not suggest setting the nova name to the exactly the same name that you defined for the ironic name. While the indexes should match, the name formats should vary enough that you can easily tell if you are looking at a nova name or an ironic name.
The use of HostnameMap will easily facilitate the replacement of a failed node so that you can provision the new node with the same nova name that was used by the original node before its premature death. Otherwise, nova will blacklist the nova name of the failed node. For example if controller0 dies and you need to replace and redeploy it, it will end up being named controller4 since this is the next number in the index.
Heat is the main orchestration engine for OpenStack, and is used my OpenStack director to install an OpenStack Overcloud environment.
When we run the “openstack deploy overcloud” command, we are specifically telling RHEL OSP director that we want it to use the pre-defined Heat templates from /usr/share/openstack-tripleo-heat-templates/. OSP director will manage the deployment of a new overcloud heat stack, using files from this directory. When RHEL OSP director calls the Heat stack, it needs the following data…
A top-level Heat template to use that describes the overall environment and the resources required.
An environment/resource registry to tell Heat where to find resource definitions for non-standard Heat elements, e.g. TripleO components.
A set of parameters to declare the deployment-specific options (via -e)
The most important files for us to focus on are in our deployment directory, these are the default files that get called by OSP director.
The top-level Heat template that OSP director uses for deployment is /usr/share/openstack-tripleo-heat-templates/overcloud-without-mergepy.yaml
The resource registry, which tells Heat where to find the templates for deployment resources is /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet. yaml
Creating a Heat Stack
To create the stack we run the command below. This command instructs heat to use the templates in ~/my_templates/, as well as the override templates specified with the ‘-e’ option.
This is just an example of what I am using in my lab environment, your deploy command will be much different. Also note that I have copied the templates from /usr/share/openstack-tripleo-heat-templates to ~/my_templates/.
Unfortunately our deploy failed with the following errors.
Exception: Heat Stack create failed. DEBUG: openstackclient.shell clean_up DeployOvercloud DEBUG: openstackclient.shell got an error: Heat Stack create failed. ERROR: openstackclient.shell Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 176, in run return super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 230, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 864, in take_action self._deploy_tripleo_heat_templates(stack, parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 535, in _deploy_tripleo_heat_templates parsed_args.timeout) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 478, in _heat_deploy raise Exception("Heat Stack create failed.") Exception: Heat Stack create failed.
We can verify that the deploy failed with the command below.
The failed resources are named “Compute” and “Controller“. Lets take a closer look at those using the “resource-show” argument.
#heat resource-show overcloud Compute
| resource_status_reason | ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: StackValidationFailed: Property error : OsNetConfigImpl: config The Parameter (BondInterfaceOvsOptions) was not provided."" |
Let’s now do the same for Controller.
#heat resource-show overcloud Controller
| resource_status_reason | ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: StackValidationFailed: Property error : OsNetConfigImpl: config The Parameter (BondInterfaceOvsOptions) was not provided."" |
Apparently I have some issues with my OVS bonding options, so I need to get those straight before I can continue.
Deleting a Failed Heat Stack
Since our last deploy failed, we need to delete the failed stack before we can kick off another stack deploy. Below is an example of that command – note we are using the UUID of the stack.
Ok, let’s take a look at /var/log/heat/heat/heat-engine.log for more details. I also suggest opening another ssh session and tailing the log while the delete is attempting to do its thing.
If the output is too verbose to follow, I suggest attempting to thin out the output using the command below
Apparently SELinux is blocking the reads for the certificates. There are two ways to work around this issue. You can run “restorecon -v /path/to/certs/“, or you can work around by disabling selinux by running “setenforce 0” or by editing the /etc/selinux/config file and setting ‘SELINUX=DISABLED’. You may need to rerun the delete, in my case it was stuck in “DELETE_IN_PROGRESS”. I restarted all heat releated services to force the delete to error.
What is below its very hard to read as I am unable to get the copy/paste to format correctly. What we are looking for is the parent resource for the failed child resource. I have bolded this line below to make is easier to find.
Our parent resource is “Controller” so let’s take a look at that first. We want to grab the “physical_resource_id“. Again, hard to see so I have bolded that line below.
