I have a large fts5 virtual table (currently about 90GB).
Because the sqlite ODBC driver, which I use (version 3.43.2) as my main sqlite scripting client, doesn't support fts5, I populate and use fts5 either via sqlitespy (1.9.16) or via python (3.11.2)'s shipped sqlite3 support, which includes fts5.
To achieve a database backup I would normally use the ODBC driver to .clone a database and/or .dump the tables. With fts5 present I am reduced to either doing a file zip or doing a line by line clone of the fts5 virtual table via python:
import sqlite3
con1 = sqlite3.connect('source.db3')
con2 = sqlite3.connect('target.db3')
con1.text_factory = lambda b: b.decode(errors = 'ignore')
counter = 0
for line in con1.iterdump():
try:
if 'INSERT INTO "ftstable"' in line or 'CREATE VIRTUAL TABLE' in line:
counter = counter + 1
if counter % 1000000 == 0:
print(str(counter))
con2.execute(line)
except Exception as e:
print(line)
print(str(e))
continue
con2.commit()
con1.close()
con2.close()
I used an answer here: how do i dump a single sqlite3 table in python?
The 'text_factory' line was required because there is some corrupt data in my database which was causing the script to crash with a decode error. I used an answer here: sqlite3.OperationalError: Could not decode to UTF-8 column
Note that iterdump() picks up fts5's 'underlying' ordinary tables twice: once when dumping the virtual table, which is fine, and then again as ordinary CREATE TABLEs + INSERTs, which will fail if you try to run them after successfully populating the the virtual table.
I can't find any references anywhere on the web to fts5 backups.
Does anyone have a better fts5 backup solution?