Any application that runs in production should be backed up regularly. Even if the application runs inside a Kubernetes or Openshift cluster. To back up an application in a Kubernetes cluster, a user should back up all the resources and definitions that the application uses. It could be pretty easy in case when the application is, for example, a deployment with attached volume. But what if the application has a lot of objects to back up? In such case the task becomes more complicated and requires an understanding on how the components of the application work and interact with each other. Or… just back up the whole cluster, however, such approach has a lot of overhead.
To address this problem in
Eclipse Che, the backup
and restore feature was implemented. With it, an admin doesn’t
have to be aware of Eclipse Che internals in order to create a
backup or do recovery of Che. Eclipse Che (Eclipse Che operator to
be more precise) can create backups and restore the installation
even if a Che installation was completely deleted! This works only
if Che has been installed using the operator (
Let me show you how easy the process of backing up and restoring Che is now. But first, let’s talk about backup servers a bit.
Internal vs external backup server
When all data for backup is gathered into a snapshot, then it is encrypted and sent to a backup server. The backup server should be set up beforehand and be accessible from within the cluster. This step requires choosing the backup server type and manual configuration of it.
To make life a bit easier, Eclipse Che can automatically set up and configure a backup server in the same cluster. Such approach requires no additional configuration as everything is automated, but the main downside of it is that backups are stored in the same cluster and even the same namespace as Eclipse Che.
Note, for production use, it is recommended to set up a backup server outside of the cluster.
How to back up and restore Che using chectl
To create a backup of Eclipse Che with chectl one should run:
$ chectl server:backup
The command above will create a backup snapshot and send it to the configured backup server. But if no backup server is configured, Che operator will deploy internal backup server and configure itself to use the server by default.
To use an external backup server (or switch to another one), its URL and backups repository password should be provided, for example:
$ checlt server:backup -r rest:my-backups.domain.net:1234/che-backups -p encryption-password
After execution of the command above, a new backup will be created
and sent to the specified backup server. Also, it will configure
Che to use that backup server by default, so for the next backups
will be enough.
Note, instead of using
it is possible to set
environment variable. Note, losing repository password means
losing all the data stored in it as the password is used to
decrypt the repository content.
Supported types of backup servers
Eclipse Che uses an external tool called
restic to manage backup
stores backup snapshots in a backup repository, where each
snapshot is identified by a hash. It also can connect to different
kinds of servers that provide data storage capabilities.
As of now, Eclipse Che supports the following types of backup servers:
AWS S3and API compatible
is a dedicated server that’s specially designed to be used with
It supports optional authentication by username and password:
$ export REST_SERVER_USERNAME=user
$ export REST_SERVER_PASSWORD=password
$ chectl server:backup -r rest:http://backups.my-domain.net:1234/che -p encryption-password
Internal backup server is of type
storage and all API compatible implementations can be used as a
backup server. Requires setting
environment variables. Example:
$ export AWS_ACCESS_KEY_ID=BZK8W5****
$ export AWS_SECRET_ACCESS_KEY=JKTa9TKoL*****dH6U+kP
$ chectl server:backup -r s3:s3.amazonaws.com/che-bucket -p password
storage. It requires providing SSH key for passwordless login.
That could be done by providing the path to the file with the SSH
key or the key itself (choose one):
$ export SSH_KEY_FILE=/home/user/.ssh/sftp.key
$ # export SSH_KEY=-----BEGIN RSA PRIVATE KEY-----*****
$ chectl server:backup sftp:email@example.com:1234//srv/static/che-backups
Restoring Che installation form a backup
To restore Eclipse Che installation, simply run:
$ chectl server:restore
It will download the latest backup snapshot from the configured backup server and restore all Eclipse Che data from it. If needed, it will even deploy a new Che cluster and apply data from the backup snapshot.
But what if we created a dozen of backups and want to restore not
from the latest backup available on the backup server, but an
older one? It is possible! Just add backup snapshot ID with
to the restore command:
$ chectl server:restore -s f801da5c
Where to get snapshot IDs? There are two ways. Snapshot ID is printed when a backup command executed:
Backup snapshot ID: f801da5c
Another way is to use
$ restic -r rest:my-backups.domain.net:1234/che-backups snapshots
Also, it is possible to use differnet backup server to restore from. Just provide a backup server URL and repository password with needed credentials. For example:
$ chectl server:restore -r sftp:firstname.lastname@example.org:/srv/data/che-backups/ -p encryption-password --ssh-key-file=~/.ssh/che-sftp.key
Note, that the command above will change default backup server, so the next backup will be sent there unless another configuration provided.
How to back up and restore Che via custom resources objects
If someone doesn’t want to use
or want to have more control over the backup and restore process,
it is possible to control backup and restore processes by directly
managing backup related custom resources (CRs). There are 3 types
CheBackupServerConfigurationthat holds information about a backup server and references to the secrets with credentials.
CheClusterBackuprequests a new backup and also points to an instance of
CheBackupServerConfigurationto where the backup snapshot should be sent.
CheClusterRestorerequests a new restore and holds reference to
CheBackupServerConfigurationfrom where the backup snapshot should be downloaded.
Please note, that only creation of
instances triggers backup and restore processes correspondingly.
Any editing of these resources has no effect.
Under the hood,
deals with the described CRs in order to create a backup or
trigger a restore process.
Configuring a backup server
Before backing up or restoring Che installation, at least one backup server configuration should be created. Also, all secrets that are referenced from the CR must exist. Then, the configuration might be referenced from backup and/or restore CR.
Example backup server configuration for AWS S3 storage:
keys must exist.
As it was described above, under
allowed. CR definitions have self-explanatory fields and it will
be easy to create a backup server configuration. But note, that
each subsection mutually excludes the others. However, it is
allowed to create as many backup server configurations as needed.
To create a new backup, a new CR of
type should be created:
Right after the CR creation a new backup process will be started.
To monitor backup process state, one should look at
section of the created CR:
$ kubectl get CheClusterBackup eclipse-che-backup -n eclipse-che -o yaml | grep -A 5 ^status
The output of the command above looks like:
message: 'Backup is in progress. Start time: <timestamp>'
stage: Collecting Che installation data
messageshows overall human readable status or an error message.
stagedisplays human readable current phase of backup process
stateindicates the overall state of the backup. Only
When the process finishes successfully, the
section will contain
field that could be used when restoring. The CR might be deleted
after backup is finished.
If one need to request internal backup server and create a backup,
property set to
should be created:
Note, it will create an instance of
and corresponding secrets automatically.
To restore from a backup snapshot, a new CR of
type should be created:
By default the latest snapshot is taken. However, it is possible
to restore from a specific snapshot by adding
To monitor the restore state, one may read status of the corresponding CR:
$ kubectl get CheClusterRestore eclipse-che-restore -n eclipse-che -o yaml | grep -A 5 ^status
Once the restore finishes, the CR can be deleted.
As of now, there are two major limitations with backup and restore:
- Backing up of user’s projects inside workspaces hasn’t been implemented yet. So, all not committed changes will not be restored.
- Backup snapshots are bind to the specific cluster, so it is not possible to restore snapshot on another cluster in general case. This is because Che binds to some cluster ID’s.
Other than that, back up and restore is a user friendly and straightforward process now.