Backing up and restoring nodes

Being prepared for catastrophic system failures

Disaster recovery strategies for blockchains are more flexible than those of regular databases, because the content of the chain is automatically replicated to (and can be recovered from) other nodes in the network. Nonetheless, there are two good reasons to back up individual nodes.

First and most importantly, if the private keys for a node’s addresses are not being deliberately stored outside its wallet, then it is vital to back up the wallet and its keys. If the private key for an address is permanently lost, no more transactions can be performed on behalf of that address.

Second, if a node needs to recover quickly from a system failure, without the delay caused by retrieving and reindexing the blockchain’s contents, then it may make sense to back up the node’s entire state instead.

Both of these methods are described below, along with instructions on how to restore each type of backup. It should also be noted that clustering can serve as an alternative to backups, and as a workaround for that fact that nodes must be stopped when backing up their full state.

Performing a wallet backup

A node’s wallet, which includes both private keys and watch-only addresses, is stored in the wallet.dat file in that node’s blockchain directory. By default, this directory is ~/.multichain/[chain-name]/ on Linux and Mac and %APPDATA%\MultiChain\[chain-name]\ on Windows.

In general, the wallet.dat file can be backed up directly. However, if a node is running, it is best to use the backupwallet API command to create a copy of the file under an exclusive lock.

Note that while backing up a node’s wallet, it may also be helpful to back up two more things:

  • The list of assets and streams to which that node is subscribed. Use the listassets and liststreams API commands, and note the name of any entity for which subscribed is true.
  • The username and password for accessing the node’s API along with any other fixed runtime parameters. These are stored in the multichain.conf file in the blockchain directory.

Restoring a wallet backup

The process of recreating a node from a wallet-only backup is as follows:

  1. Obtain the nodeaddress of another node in the chain using the getinfo command.
  2. On the server to be recovered, make sure that the blockchain directory no longer exists.
  3. Start connecting to the blockchain in the usual way, passing the previously obtained node address as a parameter to multichaind.
  4. If multichaind does not stop with a message about needing connect permissions, stop it after a few seconds using multichain-cli [chain-name] stop. (This can happen in a blockchain with parameter anyone-can-connect=true.)
  5. Replace the wallet.dat file in the node’s new blockchain directory with the version that had been backed up.
  6. If a backup was kept of the node’s multichain.conf file, restore that file as well.
  7. Restart the node in the usual way: multichaind [chain-name] -daemon. In the event that the first address in the wallet no longer has connect permissions, use the handshakelocal runtime parameter to specify which address to use for handshaking.
  8. Watch the blocks value in getinfo to see when the node has caught up with the rest of the network. For large chains, this may take a considerable amount of time.
  9. If a record was kept of the subscribed assets and streams, re-subscribe using the subscribe command without rescanning, e.g. subscribe '["stream1","asset1"]' false. Then stop the node, restart it with the -rescan option, and wait for rescanning to complete.

That’s it! Your restored node should be up to date with the blockchain and ready to be used again.

Full state backup and restore

As an alternative to backing up and restoring the node’s wallet and some other small pieces of information, it is possible to make a straightforwards full backup of all the files in the blockchain directory, e.g. ~/.multichain/[chain-name]/ on Linux or Mac, or %APPDATA%\MultiChain\[chain-name]\ on Windows. Restoring from a full backup is simply a matter of putting these files back in the right place, restarting the node, and waiting for it to catch up with any blocks created after the backup was made.

There are two key advantages to backing up the full state instead of the wallet only:

  • The process of backup and recovery is much simpler, based on copying an entire directory of files.
  • A functioning node can be instantly recovered without needing to retrieve and reindex the entire blockchain.

However full state backups also have two key disadvantages:

  • The node must be stopped before the backup is performed, to ensure consistency in the backed up files. Clustering can help here – while one node in a cluster is stopped for backup, another can continue to serve the application.
  • The total amount of data backed up will be drastically larger. However it should be noted that standard compression techniques such as gzip or zip can reduce this size by 50-75%, depending on the pattern of activity within the chain.