0

I have a large fts5 virtual table (currently about 90GB).

Because the sqlite ODBC driver, which I use (version 3.43.2) as my main sqlite scripting client, doesn't support fts5, I populate and use fts5 either via sqlitespy (1.9.16) or via python (3.11.2)'s shipped sqlite3 support, which includes fts5.

To achieve a database backup I would normally use the ODBC driver to .clone a database and/or .dump the tables. With fts5 present I am reduced to either doing a file zip or doing a line by line clone of the fts5 virtual table via python:

import sqlite3

con1 = sqlite3.connect('source.db3')
con2 = sqlite3.connect('target.db3')
con1.text_factory = lambda b: b.decode(errors = 'ignore')
counter = 0
for line in con1.iterdump():
    try:
        if 'INSERT INTO "ftstable"' in line or 'CREATE VIRTUAL TABLE' in line:
            counter = counter + 1
            if counter % 1000000 == 0:
                print(str(counter))
            con2.execute(line)
    except Exception as e:
        print(line)
        print(str(e))
        continue
con2.commit()
con1.close()
con2.close()

I used an answer here: how do i dump a single sqlite3 table in python?

The 'text_factory' line was required because there is some corrupt data in my database which was causing the script to crash with a decode error. I used an answer here: sqlite3.OperationalError: Could not decode to UTF-8 column

Note that iterdump() picks up fts5's 'underlying' ordinary tables twice: once when dumping the virtual table, which is fine, and then again as ordinary CREATE TABLEs + INSERTs, which will fail if you try to run them after successfully populating the the virtual table.

I can't find any references anywhere on the web to fts5 backups.

Does anyone have a better fts5 backup solution?

0

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.