Q100740: Best practices for upgrading large Flix deployments

SUMMARY

Upgrading from one Flix version to another often involves upgrading the Flix database schema. There are a few common issues with this database schema upgrade and this article aims to explain how you can avoid them.

  1. Large database schemas can take a very long time, causing the upgrade process to time out. This will lead to a failed Flix update unless you increase the MySQL timeouts in Flix before the upgrade. Please revert the change once the upgrade process is completed.
  2. Large database schemas can use up all free space on the MySQL server, causing the upgrade to fail. To avoid this, you need to make sure that you have at least twice as much free space in your /tmp directory as the size of your Flix MySQL schema.
  3. Flix database schemas created in MySQL 5.4 use the default MySQL 5.4 charset (utf8mb3/latin1). Flix versions 8.1 and newer assume the database schema is using the default MySQL 8 charset (utf8mb4). This discrepancy causes the Flix update to fail unless you disable the foreign key check and convert the schema to use the utf8mb4 charset. The foreign key check should be re-enabled once the upgrade process is completed.
  4. Flix database backup from the upgrade process can take a while. It is faster to manually create a database backup and skip the one provided by Flix.

 

MORE INFORMATION

Part of the Flix server upgrade process is updating the MySQL Flix schema to match the schema of the new Flix version. Often, there is no difference between the current MySQL Flix schema and the version required by the new Flix version. You can find details on how to upgrade Flix and the required database versions at the Managing Flix Deployment page.

 

Large Flix database schemas can time out

Over time, Flix generates an increasing amount of data in the MySQL schema. If you have used Flix for a long time, your MySQL schema size can become very large, and the upgrade process will take a very long time. This can lead to MySQL connections timing out.

To prevent this, we recommend raising the MySQL timeouts in Flix for the upgrade process. Here are the suggested settings for your Flix config.yml file: 

## The maximum amount of time to wait when attempting to connect to a database.
mysql_dial_timeout: 2000s
## The maximum amount of time to wait when reading incoming data.
mysql_read_timeout: 2000m
## The maximum amount of time to wait when writing outgoing data.
mysql_write_timeout: 2000m
## The maximum amount of time to wait before aborting a stalled database transaction
mysql_transaction_timeout: 2000m

Please make sure to revert those values after the upgrade process is complete. Raising them over the default values can cause unexpected slowness in Flix.


Please make sure the timeout settings on the MySQL side are at least as high as the corresponding settings in Flix.

To do this, update the following settings in the MySQL config files and restart the MySQL server. The MySQL configuration file is located on:

Linux - /etc/my.cnf.d/mysql-server.cnf
Windows - C:\ProgramData\MySQL\MySQL Server X.Y\my.ini
macOS - /usr/local/mysql/etc/my.cnf

You will need to add the following settings in the [mysqld] section:

wait_timeout = 10000
interactive_timeout = 10000
Connect_timeout = 1000

 

Large Flix database schemas can use up all available disk space

During the upgrade process, Flix changes the MySQL schema it uses. During that process, MySQL creates a temporary folder inside /tmp. With very large Flix databases, the upgrade can eat up all available disk space. This will cause the update to fail and the database can get corrupted. To ensure you don’t run into this problem, please make sure that you have at least twice as much free space in the /tmp directory as the size of your Flix MySQL schema. You can check the size of your MySQL schema by running the following MySQL query:

SELECT table_schema "Database_Name"
     SUM(data_length + index_length) / (1024 * 1024) AS "Database Size in MB"
FROM information_schema.TABLES
GROUP BY table_schema;

Alternatively, you can check the size of all MySQL schemas on disk by running the following command in a terminal:

du -sh /var/lib/mysql/*

 

Flix database schema created in MySQL 5.4

Flix database schemas created in MySQL 5.4 use the default MySQL 5.4 charset utf8mb3/latin1. If a MySQL server was upgraded from 5.4 to 8, the charset for the Flix database schema will remain unchanged.

Flix 8 assumes shows were created using the utf8mb4 charset (the default charset in MySQL 8), which is different from the MySQL 5.4 default charset.

From Flix 8.1 onward, this discrepancy will cause a database upgrade to fail with the following error:

ERRO schema.(*MigrationRunner).runMigration: error during migration, attempting to reverse steps error="could not run step: could not run actions for step 104-2 (schema.createTableStep): could not run action 3: Error 3780 (HY000): Referencing column 'access_key_id' and referenced column 'key_id' in foreign key constraint 'fk_api_key_access_key' are incompatible."

To fix this problem, you will need to convert the schema to use the utf8mb4 charset. 

The sequence_revision and job_chain tables share a foreign key, so the conversion will fail unless you disable the foreign key check.

The MySQL query will look like this:

SET FOREIGN_KEY_CHECKS = 0; 
ALTER DATABASE flix_db CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci; 
SET FOREIGN_KEY_CHECKS = 1; 

Where flix_db is the name of the Flix schema you are using. 

Please make sure to enable foreign_key_checks after the process is complete, as keeping it disabled can cause issues in the future.

 

Flix database schema backup can take a while

During the Flix upgrade process, you will be asked to perform a database backup. We recommend skipping this option and backing up the MySQL database manually, as the manual backup can take less time. The manual backup process is described in this knowledge base article - Q100615: Backing up Flix.

Do not skip the MySQL schema backup offered by the upgrade process, unless you have created a manual backup.

 

 

FURTHER READING

Before upgrading to a new Flix version, we strongly recommend backing up your database as described in Q100615 Backing up Flix.

If something goes wrong during the upgrade process, you can restore your database to its previous state as described in the following article: Q100567: How to restore a Flix database backup.

 

    We're sorry to hear that

    Please tell us why