Appendix B. HyperSQL Database Files and Recovery
Keywords: HSQLDB, Data, Files

| Appendix B. HyperSQL Database Files and Recovery |
Appendix A. Lists of Keywords |  | Appendix C. Building HSQLDB Jars

# HyperSQL Database Files and Recovery

$Revision: 5925 $

2022-10-20

## Database Files

Database catalogs opened with the file: protocol are stored as a set of files. This document describes the contents of these files and how they are stored.

A database named 'test' is used in this description. The database files will be as follows.

Database Files

test.properties | Contains the entry 'modified'. If the entry 'modified' is set to 'yes' then the database is either running or was not closed correctly. When the database is properly shutdown, 'modified' is set to 'no'.
test.script | This file contains the SQL statements that makes up the database up to the last checkpoint - it is in sync with the contents of `test.backup`.
test.data | This file contains the binary data records for CACHED tables only.
test.backup | Depending on the backup mode ( SET FILES BACKUP INCREMENT {TRUE | FALSE} ), this file contains either a backup of the parts of the `test.data` that have been modified since the last checkpoint (the default setting, TRUE) or the complete compressed backup of the `test.data` file at the time of last checkpoint (when FALSE).
test.log | This file contains the extra SQL statements that have modified the database since the last checkpoint. It is used as a redo log.
test.lobs | This file contains the lobs. If a database has no BLOB or CLOB object, this file will not be present. This file contains all the lobs that are currently in the database, as well as those that belong to rows that have been deleted since the last checkpoint. The space for deleted lobs is always reused after a CHECKPOINT.

A CHECKPOINT is an operations that saves all the changed data and removes the `test.log` followed by the creation of an empty log. A SHUTDOWN is equivalent to a CHECKPOINT followed by closing the database.

## States

Database is closed correctly

State after running the `SHUTDOWN` statement

- The `test.data` file is fully updated.

- When BACKUP INCREMENT TRUE is used, there is no `test.backup` at all. Otherwise the `test.backup` contains the full compressed `test.data` file.

- The `test.script` contains all the metadata and CREATE TABLE and other DDL statements. It also contains the data for MEMORY tables.

- The `test.properties` contains the entry 'modified' set to 'no'.

- There is no `test.log` file.

Database is closed correctly with SHUTDOWN SCRIPT

State after running the `SHUTDOWN SCRIPT` statement

- The `test.data` file does not exist; all CACHED table data is in the `test.script` file

- The `test.backup` does not exist.

- The `test.script` contains all the metadata and DDL statements, followed by the data for MEMORY, CACHED and TEXT tables.

- The `test.properties` contains the entry 'modified' set to 'no'.

- There is no `test.log` file.

Database is aborted

If the database process was terminated with a SHUTDOWN, or the SHUTDOWN IMMEDIATELY was used, the database is in aborted state.

Aborted database state

- The `test.properties` contains 'modified=yes'.

- The `test.script` contains a snapshot of the database at the last checkpoint.

- The `test.data` file is not necessarily consistent.

- The `test.backup` file contains just sections of the original `test.data` file, or a full snapshot of `test.data` that corresponds to `test.script` at the time of the last checkpoint.

- The `test.log` file contain all data change statements executed since the checkpoint. As a result of abnormal termination, the end of file may be incomplete.

## Procedures

The database engine performs the following procedures internally in different circumstances.

### Clean Shutdown

Procedure B.1. Clean HyperSQL database shutdown

1. The `test.data` file is written completely (all the modified cached table rows are written out) and closed.

2. If backup mode is not INCREMENT, the `test.backup.new` is created which contains the compressed `test.data` file.

3. The file `test.script.new` is created using the current state of the database.

4. The entry 'modified' in the properties file is set to 'yes-new-files' (Note: after this step, the `test.data.new` and `test.script.new` files constitute the database)

5. The file `test.log` is deleted

6. The file `test.script` is deleted

7. The file `test.script.new` is renamed to `test.script`

8. The file `test.backup` is deleted

9. If the file `test.backup.new` exists, it is renamed to `test.backup`

10. The entry 'modified' in the properties file is set to 'no'

### Startup

Procedure B.2. Opening the Database

1. Check if the database files are in use by checking a special `test.lck` file.

2. See if the `test.properties` file exists, otherwise create it.

3. If the `test.script` did not exist, then this is a new database.

4. If it is an existing database, check in the `test.properties` file if 'modified=yes'. In this case the RESTORE operation is performed before the database is opened normally.

5. Otherwise, if in the `test.properties` file 'modified=yes-new-files', then the (old) `test.backup` and `test.script` files are deleted and the new `test.script.new` file is renamed to `test.script`.

6. Open the `test.script` file and create the database objects.

7. Create the empty `test.log` to append any data change statements.

### Restore

The current `test.data` file is not necessarily consistent. The database engine takes these steps:

Procedure B.3. Restore a Database

1. Restore the old `test.data` file from the backup. Depending on the backup mode, decompress the `test.backup` and overwrite `test.data`, or copy the original sections from the `test.backup` file.

2. Execute all the statements in the `test.script` file.

3. Execute all statements in the `test.log` file. If due to incomplete statements in this file an exception is thrown, the rest of the lines in the `test.log` file are ignored. This can be overridden with the database connection property `hsqldb.full_log_replay=true` which results in the startup process to fail and allows the user to examine and edit the `test.log` file.

4. Close the database files, before opening the restored database.

$Revision: 6621 $

Appendix A. Lists of Keywords |  | Appendix C. Building HSQLDB Jars