Disk was 100% full, noticed following folder was taking up most of the space, with lot of files like 'binlog.000024'.
Quick fix was purge those files via MySQL see end of this doc.
/var/lib/mysql
Show variables like 'binlog%';
mysql> show variables like 'binlog%';
+------------------------------------------------+--------------+
| Variable_name | Value |
+------------------------------------------------+--------------+
| binlog_cache_size | 32768 |
| binlog_checksum | CRC32 |
| binlog_direct_non_transactional_updates | OFF |
| binlog_encryption | OFF |
| binlog_error_action | ABORT_SERVER |
| binlog_expire_logs_seconds | 2592000 |
| binlog_format | ROW |
| binlog_group_commit_sync_delay | 0 |
| binlog_group_commit_sync_no_delay_count | 0 |
| binlog_gtid_simple_recovery | ON |
| binlog_max_flush_queue_time | 0 |
| binlog_order_commits | ON |
| binlog_rotate_encryption_master_key_at_startup | OFF |
| binlog_row_event_max_size | 8192 |
| binlog_row_image | FULL |
| binlog_row_metadata | MINIMAL |
| binlog_row_value_options | |
| binlog_rows_query_log_events | OFF |
| binlog_stmt_cache_size | 32768 |
| binlog_transaction_compression | OFF |
| binlog_transaction_compression_level_zstd | 3 |
| binlog_transaction_dependency_history_size | 25000 |
| binlog_transaction_dependency_tracking | COMMIT_ORDER |
+------------------------------------------------+--------------+
23 rows in set (0.01 sec)
Note : 2592000 (seconds) / 60 / 60 / 24 is 30 days.
Now added to config file
# vi /etc/mysql/my.cnf
[mysqld]
binlog_expire_logs_seconds=172800
Restart MySQL
# service mysql stop
# service mysql start
mysql> show variables like 'binlog%';
+------------------------------------------------+--------------+
| Variable_name | Value |
+------------------------------------------------+--------------+
| binlog_cache_size | 32768 |
| binlog_checksum | CRC32 |
| binlog_direct_non_transactional_updates | OFF |
| binlog_encryption | OFF |
| binlog_error_action | ABORT_SERVER |
| binlog_expire_logs_seconds | 172800 |
| binlog_format | ROW |
| binlog_group_commit_sync_delay | 0 |
| binlog_group_commit_sync_no_delay_count | 0 |
| binlog_gtid_simple_recovery | ON |
| binlog_max_flush_queue_time | 0 |
| binlog_order_commits | ON |
| binlog_rotate_encryption_master_key_at_startup | OFF |
| binlog_row_event_max_size | 8192 |
| binlog_row_image | FULL |
| binlog_row_metadata | MINIMAL |
| binlog_row_value_options | |
| binlog_rows_query_log_events | OFF |
| binlog_stmt_cache_size | 32768 |
| binlog_transaction_compression | OFF |
| binlog_transaction_compression_level_zstd | 3 |
| binlog_transaction_dependency_history_size | 25000 |
| binlog_transaction_dependency_tracking | COMMIT_ORDER |
+------------------------------------------------+--------------+
23 rows in set (0.01 sec)
Manullay purge log from MySql, files are deleted from filesystem (worked fine on 16 Jan 2021)
PURGE BINARY LOGS BEFORE '2020-12-28 22:46:26';
