Additional languages
« 15.7 Management of MySQL Cluster
15.7.2 Commands in the MySQL Cluster Management Client »
Section Navigation      [Toggle]

15.7.1. MySQL Cluster Startup Phases

This section describes the steps involved when the cluster is started.

There are several different startup types and modes, as shown here:

  • Initial Start: The cluster starts with a clean filesystem on all data nodes. This occurs either when the cluster started for the very first time, or when it is restarted using the --initial option.

  • System Restart: The cluster starts and reads data stored in the data nodes. This occurs when the cluster has been shut down after having been in use, when it is desired for the cluster to resume operations from the point where it left off.

  • Node Restart: This is the online restart of a cluster node while the cluster itself is running.

  • Initial Node Restart: This is the same as a node restart, except that the node is reinitialized and started with a clean filesystem.

Prior to startup, each data node (ndbd process) must be initialized. Initialization consists of the following steps:

  1. Obtain a Node ID.

  2. Fetch configuration data.

  3. Allocate ports to be used for inter-node communications.

  4. Allocate memory according to settings obtained from the configuration file.

When a data node or SQL node first connects to the management node, it reserves a cluster node ID. To make sure that no other node allocates the same node ID, this ID is retained until the node has managed to connect to the cluster and at least one ndbd reports that this node is connected. This retention of the node ID is guarded by the connection between the node in question and ndb_mgmd.

Normally, in the event of a problem with the node, the node disconnects from the management server, the socket used for the connection is closed, and the reserved node ID is freed. However, if a node is disconnected abruptly — for example, due to a hardware failure in one of the cluster hosts, or because of network issues — the normal closing of the socket by the operating system may not take place. In this case, the node ID continues to be reserved and not released until a TCP timeout occurs 10 or so minutes later.

To take care of this problem, you can use PURGE STALE SESSIONS. Running this statement forces all reserved node IDs to be checked; any that are not being used by nodes actually connected to the cluster are then freed.

After each data node has been initialized, the cluster startup process can proceed. The stages which the cluster goes through during this process are listed here:

  • Stage 0

    Clear the cluster filesystem. This stage occurs only if the cluster was started with the --initial option.

  • Stage 1

    This stage sets up Cluster connections, establishes inter-node communications, and starts Cluster heartbeats.

    Note: When one or more nodes hang in Phase 1 while the remaining node or nodes hang in Phase 2, this often indicates network problems. One possible cause of such issues is one or more cluster hosts having multiple network interfaces. Another common source of problems causing this condition is the blocking of TCP/IP ports needed for communications between cluster nodes. In the latter case, this is often due to a misconfigured firewall.

  • Stage 2

    The arbitrator node is elected. If this is a system restart, the cluster determines the latest restorable global checkpoint.

  • Stage 3

    This stage initializes a number of internal cluster variables.

  • Stage 4

    For an initial start or initial node restart, the redo log files are created. The number of these files is equal to NoOfFragmentLogFiles.

    For a system restart:

    • Read schema or schemas.

    • Read data from the local checkpoint and undo logs.

    • Apply all redo information until the latest restorable global checkpoint has been reached.

    For a node restart, find the tail of the redo log.

  • Stage 5

    If this is an initial start, create the SYSTAB_0 and NDB$EVENTS internal system tables.

    For a node restart or an initial node restart:

    1. The node is included in transaction handling operations.

    2. The node schema is compared with that of the master and synchronized with it.

    3. Synchronize data received in the form of INSERT from the other data nodes in this node's node group.

    4. In all cases, wait for complete local checkpoint as determined by the arbitrator.

  • Stage 6

    Update internal variables.

  • Stage 7

    Update internal variables.

  • Stage 8

    In a system restart, rebuild all indexes.

  • Stage 9

    Update internal variables.

  • Stage 100

    At this point in a node restart or initial node restart, APIs may connect to the node and begin to receive events.

  • Stage 101

    At this point in a node restart or initial node restart, event delivery is handed over to the node joining the cluster. The newly-joined node takes over responsibility for delivering its primary data to subscribers.

After this process is completed for an initial start or system restart, transaction handling is enabled. For a node restart or initial node restart, completion of the startup process means that the node may now act as a transaction coordinator.