Tag: galera

  • How to Recover an Out of Sync Mariadb Galera OpenStack Database cluster

    MariaDB-Seal-with-Text

    Introduction

    This process can be used whenever your databases are out of sync. For example when someone without thinking reboots all nodes in a cluster without shutting down the databases first.

    Resolution

    Place all cluster hosts into standby mode and cleanup any failed resources.

    I sugguest making a back-up of Mariadb on each controller nodes – just in case.

    root@controler3#  mysqldump --all-databases > mariadb_dump_06152015
    root@controller3 # pcs cluster standby --all
    root@controller3 # pcs status
    root@controller3 # pcs resource cleanup ${resource}

    Then on each Controller Node – verify that mariadb (mysql) has stopped. If any instance has not stopped properly via PCS as shown below, please stop them manually.

    # systemctl status mariadb.service
    mariadb.service - MariaDB database server
       Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled)
       Active: inactive (dead)

    Find the latest (i.e – largest) version of the DB (seqno), or choose a node if all have the same version. In my instance, controller3 had the largest/highest version number, so we will be doing most of our recovery work on the instance below.

    root@controller3 # cat /var/lib/mysql/grastate.dat
    # GALERA saved state
    version: 2.1
    uuid:    2fb3bbe0-eed6-11e4-ac79-4b77a337d598
    seqno:   12175047
    cert_index:
    
    

    Stop puppet if it is running and edit the /etc/my.cnf.d/galera.cnf file. If you are not running puppet, or if puppet is not managing your Galera cluster you can skip this step and go ahead and edit the file.

    root@controller3 # systemctl stop puppet

    Set wsrep_cluster_address manually in /etc/my.cnf.d/galera.cnf making a note of the original value and restart mariadb manually.

    For example, your default configuration should look something like what is shown below. Each IP address listed is the IP address of a mariadb instance in your cluster.

    # Group communication system handle
    wsrep_cluster_address="gcomm://172.17.9.23,172.17.9.24,172.17.9.22"

    Modify to this by commenting out the default string and adding the string shown in the example below. Note that in this example we are working on controller3.

    # Group communication system handle
    #wsrep_cluster_address="gcomm://172.17.9.23,172.17.9.24,172.17.9.22"
    wsrep_cluster_address="gcomm://"

    Now restart Mariadb manually on the controller3.

    root@controller3 # systemctl start mariadb

    Now start mariadb on one of the remaining controllers

    root@controller2 # systemctl start mariadb

    Below you can see the newly started Maridb instance as requested to sync

    Jun 15 10:08:02 controller2 mysqld_safe[24999]: 150615 10:08:02 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
    Jun 15 10:08:02 controller2 mysqld_safe[24999]: 150615 10:08:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    Jun 15 10:08:02 controller2 mysqld_safe[24999]: 150615 10:08:02 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsr...ver.pid'
    Jun 15 10:08:04 controller2 mysqld_safe[24999]: 150615 10:08:04 mysqld_safe WSREP: Recovered position 2fb3bbe0-eed6-11e4-ac79-4b77a337d598:12175047
    Jun 15 10:08:06 controller2 systemd[1]: Started MariaDB database server.
    
    
    

    Once running you can un-standby the nodes and monitor the status as the remaining resources are loaded.

    root@rcontroller3 # pcs cluster unstandby --all
    root@node # pcs status

    Finally, correct the value of wsrep_cluster_address to it’s original value in /etc/my.cnf.d/galera.cnf and restart the service, monitoring that the resource remains active.

    root@controller3 # vi /etc/my.cnf.d/galera.cnf
    root@controller3 # systemctl restart mariadb
    root@controller3 # pcs status

    Now check your databases to make sure that they are in sync. The following file should be the same on each controller.

    [root@lppcldiuctl01 ~(openstack_admin)]# cat /var/lib/mysql/grastate.dat
    # GALERA saved state
    version: 2.1
    uuid:    2fb3bbe0-eed6-11e4-ac79-4b77a337d598
    seqno:   -1
    cert_index:

    Reference:
    https://access.redhat.com/solutions/1283463