Write-ahead Log (WAL) Pruning

Chroma Write-Ahead Log is unbounded by default and grows indefinitely. This can lead to high disk usage and slow performance. To prevent this, it is recommended to prune/cleanup the WAL periodically. Below we offer a couple of tools, including an official and recommended CLI tool, to help you prune your WAL.

Tooling

There are two ways to prune your WAL:

  • Chroma CLI - this is the official tooling provided by Chroma and is the recommended way to prune your WAL. This functionality is available either from main branch or Chroma release >0.5.5.
  • chroma-ops

Chroma CLI

To prune your WAL you need to install Chroma CLI (it comes as part of the core Chroma package):

1. pip install chromadb

3. chroma utils vacuum --path /path/to/persist_dir

Auto-pruning

Running the above command will enable auto WAL pruning. This means that Chroma will periodically prune the WAL during its normal operations.

Chroma Ops

To prune your WAL you can run the following command:

1. pip install chroma-ops
2. chops cleanup-wal /path/to/persist_dir

⚠️ IMPORTANT: It is always a good thing to backup your data before you prune the WAL.

Manual

Steps:

Stop Chroma

It is vitally important that you stop Chroma before you prune the WAL. If you don’t stop Chroma you risk corrupting

  • ⚠️ Stop Chroma
  • 💾 Create a backup of your chroma.sqlite3 file in your persistent dir
  • 👀 Check your current chroma.sqlite3 size (e.g. ls -lh /path/to/persist/dir/chroma.sqlite3)
  • 🖥️ Run the script below
  • 🔭 Check your current chroma.sqlite3 size again to verify that the WAL has been pruned
  • 🚀 Start Chroma

Script (store it in a file like compact-wal.sql)

wal_clean.py

1. #!/usr/bin/env python3
2. # Call the script: python wal_clean.py ./chroma-test-compact
3. import os
4. import sqlite3
5. from typing import cast, Optional, Dict
6. import argparse
7. import pickle

10. class PersistentData:
11. """Stores the data and metadata needed for a PersistentLocalHnswSegment"""

13. dimensionality: Optional[int]
14. total_elements_added: int
15. max_seq_id: int

17. id_to_label: Dict[str, int]
18. label_to_id: Dict[int, str]
19. id_to_seq_id: Dict[str, int]

22. def load_from_file(filename: str) -> "PersistentData":
23. """Load persistent data from a file"""
24. with open(filename, "rb") as f:
25. ret = cast(PersistentData, pickle.load(f))
26. return ret

29. def clean_wal(chroma_persist_dir: str):
30. if not os.path.exists(chroma_persist_dir):
31. raise Exception(f"Persist {chroma_persist_dir} dir does not exist")
32. if not os.path.exists(f'{chroma_persist_dir}/chroma.sqlite3'):
33. raise Exception(
34. f"SQL file not found int persist dir {chroma_persist_dir}/chroma.sqlite3")
35. # Connect to SQLite database
36. conn = sqlite3.connect(f'{chroma_persist_dir}/chroma.sqlite3')

38. # Create a cursor object
39. cursor = conn.cursor()

41. # SQL query
42. query = "SELECT id,topic FROM segments where scope='VECTOR'"  # Replace with your query

44. # Execute the query
45. cursor.execute(query)

47. # Fetch the results (if needed)
48. results = cursor.fetchall()
49. wal_cleanup_queries = []
50. for row in results:
51. # print(row)
52. metadata = load_from_file(
53. f'{chroma_persist_dir}/{row[0]}/index_metadata.pickle')
54. wal_cleanup_queries.append(
55. f"DELETE FROM embeddings_queue WHERE seq_id < {metadata.max_seq_id} AND topic='{row[1]}';")

57. cursor.executescript('\n'.join(wal_cleanup_queries))
58. # Close the cursor and connection
59. cursor.close()
60. conn.close()

63. if __name__ == "__main__":
64. parser = argparse.ArgumentParser()
65. parser.add_argument('persist_dir', type=str)
66. arg = parser.parse_args()
67. print(arg.persist_dir)
68. clean_wal(arg.persist_dir)

Run the script

1. # Let's create a backup
2. tar -czvf /path/to/persist/dir/chroma.sqlite3.backup.tar.gz /path/to/persist/dir/chroma.sqlite3
3. lsof /path/to/persist/dir/chroma.sqlite3 # make sure that no process is using the file
4. python wal_clean.py /path/to/persist/dir/
5. # start chroma

July 30, 2024