How to remove a (large) file from your Git history

Share & showcase
1

@SkepticMystic had the problem that he auto-committed the plugins folder after the installation of the Dual plugin before the warning to add one of the files to the .gitignore was added to the plugin’s README. The Dual plugin folder contains a file too large for GitHub (at least after using it/installing everything). He didn’t want to lose his Git history but needed to remove the file. As a result, here is the guide for everyone who has the same problem to remove this large file. You can of course also use it, should you want to remove any other file from your Git history of your Obsidian notes.

What to do if you can’t push to GitHub after installing the Dual plugin.

The plugin uses a very large file which GitHub doesn’t accept. The recommended file size to push to your repository (repo) is less than 50MB.
The overly-sized file can be found from your vault root at .obsidian/plugins/Dual/essence/pytorch_model.bin, which we refer to as pytorch_model.bin.

This tutorial can also be used if you committed other files which are too large for GitHub, as said above.

  1. Download JDK and install it.
    • This is necessary for executing the BFG tool used later.
  2. Make a backup of your entire vault. Preferably on an external drive.
  3. Add the file/folder which is too big to .gitignore
  4. Untrack the pytorch_model.bin file
git rm --cached ".obsidian/plugins/Dual/essence/pytorch_model.bin"
  • This must be run inside your original vault root.
  • This does not delete the file, it only stops it being tracked by git.
  • If you want to untrack the entire Dual folder instead of just the pytorch_model.bin file, use this command:
    • git rm -r --cached ".obsidian/plugins/Dual/"
    • (Needs recursive flag -r for folder)
  1. Commit these changes git commit -m "message"
  2. Create a new folder outside your vault (This can be anywhere besides your vault) and copy the contents of your entire vault into it.
  3. Copy bfg-.<Version Number>.jar into the new vault folder
  4. Run BFG on the .git folder in your vault folder. This assumes that the terminal is opened in the vault folder
java -jar bfg-.<Version Number>.jar --delete-files "pytorch_model.bin" ".git"
  • Use the file name, not the path to the file.
    • Because a bare repo (the .git folder) doesn’t have paths which a normal repo would have
    • Saturday/pytorch.py - wrong :x:
    • pytorch.py - right :white_check_mark:
  1. Source

The BFG will update your commits and all branches and tags so they are clean, but it doesn’t physically delete the unwanted stuff. Examine the repo to make sure your history has been updated, and then use the standard git gc command to strip out the unwanted dirty data, which Git will now recognise as surplus to requirements:

cd .git
git reflog expire --expire=now --all && git gc --prune=now --aggressive
  1. Now you should be able to git push.

This assumed that you did not have the changes committed to GitHub already since you couldn’t push them. If pushing worked, but you wanted to change your local repo nonetheless and then update the remote GitHub repo, you will find yourself unable to push because the git commit IDs have changed. To push it anyway and force the local history onto the remote repo, use

git push origin main --force

main is the name of my origin branch, you have to change it if yours is named differently.

Should you find any errors, please leave a comment here. As said above, only operate on a copy of your files. Make sure to have backups!

2 Likes