Chris Paquin

AI, Virtualization, Containers, Infrastructure, Linux

Category: neutron

  • RHEL OSP 10/11 – OVS+DPDK Tunables

    93905-6a00e551c39e1c883401a511e32c88970c-pi

    Tunables for Dell R630s for use when deploying OVS+DPDK

    [terminal]

    OSP 10/11 DPDK Tunables

    #

    R630 NUMA locality – CPUs

    node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22

    24 26 28 30 32 34 36 38 40 42 44 46

    #

    node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23

    25 27 29 31 33 35 37 39 41 43 45 47

    #
    #

    R630 NUMA locality – NIC

    node 0 dpdk interface – p3p1

    node 1 dpdk interface – p1p1

    #
    #
    #

    NovaVcpuPinSet (OSP 10+)

    These are the cores that Nova will use for scheduling instances. Pair sibling threads together.

    Using cores from NUMA node 0 only to prevent crossing NUMA boundaries

    NovaVcpuPinSet: “‘4,6,8,10,12,14,16,18,20,22,28,30,32,34,36,38,40,42,44,46′”
    #

    NeutronDpdkCoreList (OSP 10/11) OvsPmdCoreList (OSP 12+)

    This parameter configures a list of CPU cores to be used by the OVS-DPDK Poll Mode Drivers

    The first core from a CPU, should be reserved for host processes, and should be excluded from this list.

    NeutronDpdkCoreList: “‘2,26,3,27′”
    #

    HostIsolatedCoreList (OSP 10/11) IsolCpusList (OSP 12+)

    A set list or range of cores (and their sibling threads) to be appended to the tuned cpu-partitioning profile and isolated from the host.

    These cores will be isolated from any host processes

    Assuming you want to isolate nova cores from all system processes, NovaVcpuPinSet + NeutronDpdkCoreList = HostIsolatedCoreList

    HostIsolatedCoreList: “‘2,3,4,6,8,10,12,14,16,18,20,22,26,27,28,30,32,34,36,38,40,42,44,46′”
    #

    HostCpusList (OSP 10/11) & OvsDpdkCoreList (OSP 12+)

    A list of logical cores used by OVS-DPDK processes for dpdk-lcore-mask for non-datapath operations

    These cores must be mutually exclusive from the list of cores in NeutronDpdkCoreList/OvsPmdCoreList and NovaVcpuPinSet.

    Allocate the first physical core (and sibling thread) from each NUMA node irrespective of DPDK interface NUMA locality.

    HostCpusList: “‘0,24,1,25′”
    #

    Provide the number of memory channels in the format – [allowed_pattern: “[0-9]+”]:

    NeutronDpdkMemoryChannels: “4”
    #

    Set the memory allocated for each socket:

    NeutronDpdkSocketMemory: “‘2048,2048′”
    #

    An array of filters used by Nova to filter a node.These filters will be applied in the order they are listed,

    so place your most restrictive filters first to make the filtering process more efficient.

    NovaSchedulerDefaultFilters: “RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,NUMATopologyFilter”
    #

    Kernel arguments for Compute node

    ComputeKernelArgs: “default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on”
    [/terminal]

     

  • OpenStack: Configuring SR-IOV in RHEL OSP 8

    openstack

    Introduction

    This article documents the configuration used to configure SR-IOV in OSP 8/Liberty on Dell hardware

    Compute Node Configuration

    This section will outline the changes needed to configure SR-IOV on each Compute Node.

    Bios Configuration on Dell Compute Nodes

    First, you will need to ssh to the drac of each Compute Node. Then, type the command below to enter racadm command line.

    #racadm

    Type the command below to enable SRIOV.

    #racadm set BIOS.IntegratedDevices.SriovGlobalEnable Enabled
    [Key=BIOS.Setup.1-1#IntegratedDevices]
    RAC1017: Successfully modified the object value and the change is in
    pending state.
    To apply modified value, create a configuration job and reboot
    the system. To create the commit and reboot jobs, use “jobqueue”
    command. For more information about the “jobqueue” command, see RACADM
    help.

    Type the command below to verify your configuration.

    #racadm>>get BIOS.IntegratedDevices.SriovGlobalEnable

    racadm get BIOS.IntegratedDevices.SriovGlobalEnable
    [Key=BIOS.Setup.1-1#IntegratedDevices]
    SriovGlobalEnable=Enabled

    If the server already has an OS, reboot it to make these settings stick. If it doesn’t, use these racadm commands to power cycle.

    #racadm serveraction powerdown
    #racadm serveraction powerup

    Grub Configuration on Compute Nodes

    Add “intel_iommu=on” to the GRUB_CMDLINE_LINUX line as shown below.

    # cat /etc/default/grub
    GRUB_TIMEOUT=5
    GRUB_DISTRIBUTOR=”$(sed ‘s, release .*$,,g’ /etc/system-release)”
    GRUB_DEFAULT=saved
    GRUB_DISABLE_SUBMENU=true
    GRUB_TERMINAL_OUTPUT=”console”
    GRUB_CMDLINE_LINUX=”console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on”
    GRUB_DISABLE_RECOVERY=”true”
    audit=1

    First, make a backup of /etc/default/grub.

    # cp -p /etc/default/grub /etc/default/grub/.$(date +%F_%R)

    Edit the line below.

    “GRUB_CMDLINE_LINUX=\”console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet\”

    Change it to this.

    “GRUB_CMDLINE_LINUX=\”console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on\”

    Now, rebuild GRUB config as shown below.

    # grub2-mkconfig -o /boot/grub2/grub.cfg

    Specify the number of VFs to Create in rc.local

    Add the following line to /etc/rc.d/rc.local, adjusting for the device name and #VFs. In this instance, our physical adapters each support 32 VFs.

    echo 32 > /sys/class/net/<device>/device/sriov_numvfs

    For example:

    # echo 32 > /sys/class/net/p1p1/device/sriov_numvfs
    # echo 32 > /sys/class/net/p3p1/device/sriov_numvfs

    Also,ensure the correct SELinux context is restored.

    # restorecon -R -v /etc/rc.d/rc.local

    Ensure that /etc/rc.d/rc.local is executable.

    #chmod +x /etc/rc.d/rc.local

    Whitelist PCI devices nova-compute (Compute)

    Tell nova-compute which pci devices are allowed to be passed through. Edit the file /etc/nova/nova.conf:

    [default]
    pci_passthrough_whitelist = [{“vendor_id”:”8086″,”product_id”:”154d”}][{“devname”:”p1p1″,”physical_network”:”sriov_net1″},{“devname”:”p3p1″,”physical_network”:”sriov_net2″}]

    This tells nova that all VFs belonging to the physical interface, “p1p1“, are allowed to be passed through to VMs and belong to the neutron provider network “sriov_net1” and all VFs belonging to the physical interface, “p3p1“, are allowed to be passed through for the network “sriov_net2“.

    Restart nova-compute on each compute node with the command shown below.

    #systemctl restart openstack-nova-compute

    Install and Enable Neutron Sriov-Agent (Compute)

    Note that the sriov-agent is not required, however, we are going to install and configure it anyway.

    Install the following rpm.

    # yum -y install openstack-neutron-sriov-nic-agent

    Now, on each compute node edit the file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini:

    [securitygroup]
    firewall_driver = neutron.agent.firewall.NoopFirewallDriver

    [sriov_nic]
    physical_device_mappings = sriov_net1:p1p1,sriov_net2:p3p1

    Now enable and start the nic agent.

    # systemctl enable neutron-sriov-nic-agent.service && systemctl start neutron-sriov-nic-agent.service

    Controller Node Configuration

    Perform the following steps on each Controller Node. Note we will modify Nova and Neutron config files.

    Neutron-Server changes in /etc/neutron/plugins/ml2/ml2_conf.ini(Controller)

    The following changes take place in the file /etc/neutron/plugins/ml2/ml2_conf.ini.

    Add sriovnicswitch as mechanism driver.

    mechanism_drivers =openvswitch,bsn_ml2,sriovnicswitch

    Set type_drivers to vlan as shown below.

    type_drivers = vlan

    Set tenant_network_types to vlan.

    tenant_network_types = vlan

    Set flat_networks as shown below where “sriov_net1” and “sriov_net2” are the networks we are going to create.

    flat_networks =datacentre,sriov_net1,sriov_net2

    Add VLAN ranges for the SRIOV networks to the network_vlan_ranges line as shown below.

    network_vlan_ranges =datacentre:10:100,datacentre:101:122,sriov_net1:200:300,sriov_net1:200:300

    Neutron-Server changes in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini(Controller)

    The change below needs to be made in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on each controller

    Update the /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on each controller.
    In our case,the vendor_id is 8086 and the product_id is 10ed.

    supported_pci_vendor_devs = 8086:10ed

    Modify Nuetron-Server Startup

    Edit /usr/lib/systemd/system/neutron-server.service. Here we add –config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini to the ExecStart line. See example below.

    # cat neutron-server.service
    [Unit]
    Description=OpenStack Neutron Server
    After=syslog.target network.target

    [Service]
    Type=notify
    User=neutron
    ExecStart=/usr/bin/neutron-server –config-file /usr/share/neutron/neutron-dist.conf –config-dir /usr/share/neutron/server –config-file /etc/neutron/neutron.conf –config-file /etc/neutron/plugin.ini –config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini –config-dir /etc/neutron/conf.d/common –config-dir /etc/neutron/conf.d/neutron-server –log-file /var/log/neutron/server.log
    PrivateTmp=true
    NotifyAccess=all
    KillMode=process

    [Install]
    WantedBy=multi-user.target

    Restart neutron on the controllers via pacemaker. See command below.

    #pcs resource restart neutron-server-clone

    Configure nova-scheduler (Controller)

    On every controller node running nova-scheduler add PCIDeviceScheduler to the scheduler_default_filters parameter.

    Also  add a new line for scheduler_available_filters parameter under the [default] section in /etc/nova/nova.conf.

    [DEFAULT]
    scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter
    scheduler_available_filters = nova.scheduler.filters.all_filters
    scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter

    Now restart nova-scheduler via Pacemaker as shown below.

    # pcs resource restart openstack-nova-scheduler-clone

     

    Reference

    http://docs.openstack.org/liberty/networking-guide/adv-config-sriov.html

  • Mapping Virtual Networks with plotnetcfg

    plot-sheet-large

    Plotnetcfg is a Linux utility that you can use to scan the networking configuration on a server and output the configuration hierarchy to a file. Plotnetcfg is most useful when troubleshooting complex virtual networks with all sorts of bonds and bridges, the likes of which you will find on KVM nodes, or OpenStack Controller nodes.

    You can install plot on RHEL/Centos as shown below.

    # yum -y plotnetcfg.x86_64

    You will also want to install the “dot” command which is installed with graphiz. See below.

    # yum -y install graphviz.x86_64

    Now that the bits and pieces are installed we can run the command below which outputs to PDF file named file.pd

    # plotnetcfg | dot -Tpdf > file.pd

    If you want to, you can also use “convert” to convert the PDF to a jpg. For example, I exported to jpg to embed below.

    file

    Super clean, and super easy to read and understand

     

     

  • OpenStack Ironic Troubleshooting – Neutron Port in Use

    keyboard-key-board-melt-broken-computer_p

    Ran into issues with a deploy via RHEL OSP director, which caused our heat stack scale compute scale-out to fail.

    We corrected the issue, and then attempted our deploy again. This time around, we were able to scale out by several compute nodes. However, several nodes failed to deploy properly.

    After trudging through the logs for a bit we were able to find the error below in /var/log/nova-conductor.log

    2016-07-25 18:34:39.453 27374 ERROR nova.scheduler.utils [req-1caa3ca0-9e13-4340-b8b2-80b1cf2f8a7f 8408462bd5a8445c9742ea4dfbc20d70 cc44dc9e68064e64899697ac610c8f06 – – -] [instance: e44e3a04-a47e-4a8e-9eb5-0037c1175e4d] Error from last host: fatmin.lab.localdomain (node c5177fbf-ae0f-49db-94f4-087537b3dd53): [u’Traceback (most recent call last):\n’, u’ File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 1905, in _do_build_and_run_instance\n filter_properties)\n’, u’ File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 2058, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n’, u’RescheduledException: Build of instance e44e3a04-a47e-4a8e-9eb5-0037c1175e4d was re-scheduled: Port 14:18:77:3e:1a:bf is still in use.\n’]

    To resolve we ran the following command, using the MAC address shown above to narrow down the search.

    #neutron port-list | grep “14:18:77:3e:1a:bf”
    | 25a366df-1e6c-4eb8-853c-5b7db82637f0 | | 14:18:77:3e:1a:bf | {“subnet_id”: “0310f210-63ad-4616-9338-d59ac13cc0be”, “ip_address”: “10.20.0.134”} |

    The command “neutron port show“, showed us that this port was down and was not responding to ping.

    We then deleted the port via neutron

    #neutron port-delete 25a366df-1e6c-4eb8-853c-5b7db82637f0

    We then re-ran our deploy and were able to scale without issue.

     

  • Red Hat Openstack & Big Cloud Fabric Introduction

    BSN-Logo-color

     

    Big Cloud Fabric, is a SDN solution from Big Switch Networks, designed to integrate into OpenStack (or VMware).

    Via OpenStack, BCF (Big Cloud Fabric) integrates directly into OpenStack Neutron by way of a plugin. BCF supports L2/L3 networking and L4-7 insertion. BCF runs on whitebox, or brightbox hardware.

    Below are a couple of videos that will give you a high-level view of the solution and let you see it in action.

    Red Hat & Big Switch: Integrated OpenStack Solution for Simplified Cloud Deployment:

    Webinar – Unified P+V Networking for OpenStack Clouds:

     

    You can also access the Big Switch Labs and play around with the techology via the link below. Note that you will have to provide an email address.

    http://labs.bigswitch.com/home

     

  • OpenStack Post Install Validation Checklist

    smarta

    Below is a “Post Install Validation Checklist” that I am working on for validating an OpenStack Deployment. This is a work in progress and is sure to be missing some items. I plan to attempt to treat this post as a live document an update as I find new items to add to the checklist

    <

    table border=”1″>

    Component Task
    All On each node (compute, controller) apply post-install configurations. Ensure ssh keys for root and nova exist on all hosts
    All If deploying with Staypuft installer, ensure that puppet is stopped and disabled so that it does not overwrite post-install configurations
    Glance Ensure that you can upload and configure OS Images
    Nova Ensure that you can create flavors
    Nova Ensure that you can successfully provision test instances. Test with each flavor and OS Image
    Cinder Provision and assign a block device from Cinder. Also ensure that you can detach and delete a block device
    Neutron Create and test tenant networks
    Neutron Provision and assign a floating IP from an external network for each tenant network. Ensure that connectivity is successful
    Nova Define and assign a security group to an instance, enabling at least SSH and ICMP access
    All Reboot of all nodes to ensure configuration persistence, document any configuration changes. Push any fixes out to nodes when necessary
    Nova Generate and test keypairs to be used for access to instances
    Nova Ensure you are able to create and delete instance snapshots
    Swift If using swift, verify that you can create object stor and upload or create files
    Nova Test Live Migration by migrating instances across each compute node