Automate file integrity checks with Python (SHA-256)

Automate file integrity checks with Python (SHA-256)

by

Files sitting on a backup drive or NAS are rarely opened, so silent corruption (bit rot) or an accidental delete can go unnoticed for months. Record a SHA-256 hash for every file and compare regularly, and you catch those changes right away. The script below uses only the standard library and runs on Python 3.8+ with nothing to install.

1. How it works

  • First run: stores size, modification time and SHA-256 for every file in manifest.json (your baseline)
  • Later runs: recalculates, compares with the baseline and reports four categories
CategoryRuleMeaning
missingIn the baseline, not on diskDeleted or moved
addedOn disk, not in the baselineNew file
changedHash and mtime both changedProbably a normal edit
corruptedHash changed but mtime did notPossible corruption, check now

2. The full script

"""Folder integrity check: the first run creates a baseline manifest; later runs report changed, missing and new files."""
import hashlib, json, sys, time
from pathlib import Path

CHUNK = 1024 * 1024  # read 1 MB at a time so huge files use little memory

def sha256(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as f:
        while block := f.read(CHUNK):
            h.update(block)
    return h.hexdigest()

def scan(root: Path) -> dict:
    result = {}
    for p in root.rglob("*"):
        if p.is_file():
            st = p.stat()
            result[str(p.relative_to(root))] = {"size": st.st_size, "mtime": int(st.st_mtime), "sha256": sha256(p)}
    return result

def main(root: str, manifest: str = "manifest.json"):
    root_p, man_p = Path(root), Path(manifest)
    now = scan(root_p)
    if not man_p.exists():
        man_p.write_text(json.dumps(now, ensure_ascii=False, indent=1), encoding="utf-8")
        print(f"Baseline created: {len(now)} files -> {man_p}")
        return 0
    base = json.loads(man_p.read_text(encoding="utf-8"))
    missing = sorted(set(base) - set(now))
    added = sorted(set(now) - set(base))
    changed, corrupted = [], []
    for k in set(base) & set(now):
        if base[k]["sha256"] != now[k]["sha256"]:
            # content changed but mtime did not: possible corruption (bit rot)
            (corrupted if base[k]["mtime"] == now[k]["mtime"] else changed).append(k)
    report = {"time": time.strftime("%Y-%m-%d %H:%M:%S"), "missing": missing, "added": added,
              "changed": sorted(changed), "corrupted": sorted(corrupted)}
    print(json.dumps(report, ensure_ascii=False, indent=1))
    return 1 if (missing or corrupted) else 0

if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:]))

Files are read in 1 MB chunks, so even multi-gigabyte video files use almost no memory. The script exits with code 1 when anything is missing or possibly corrupted, so schedulers and wrapper scripts can treat it as a failure.

3. Running it

# First run: create the baseline (keep the manifest outside the folder you check)
python integrity_check.py "D:\Backup\Photos" "C:\check\photos_manifest.json"

# Every run after that prints the comparison
python integrity_check.py "D:\Backup\Photos" "C:\check\photos_manifest.json"

If the manifest lives inside the checked folder, it shows up as “changed” every time, so keep it outside. After legitimate additions or edits, delete the manifest and run again to create a fresh baseline.

4. Run it daily

Windows Task Scheduler: register it from an elevated Command Prompt and append results to a log file.

schtasks /Create /TN "IntegrityCheck" /SC DAILY /ST 03:00 /TR "cmd /c python C:\check\integrity_check.py D:\Backup\Photos C:\check\photos_manifest.json >> C:\check\integrity.log 2>&1"

Linux or NAS (cron): add a line with crontab -e.

0 3 * * * /usr/bin/python3 /opt/check/integrity_check.py /volume1/photos /opt/check/photos_manifest.json >> /opt/check/integrity.log 2>&1

5. Tips

  • Large folders: hashing runs at roughly 100–200 MB/s on a hard drive, so several TB takes hours. Split folders and check a different one each day.
  • Alerts: wrap the script so it sends an email or chat webhook only when the exit code is 1.
  • Verify backups: build the manifest from the source folder and use it to check the backup copy. You will know whether the backup is a faithful copy.

Leave a Comment