How-to Remove Sensitive Info from a GIT \ Azure DevOps respository

Owned by Walter Hoban
Last updated: Mar 18, 2019 by John Sewchand (Unlicensed)
3 min read

If you accidentally commit sensitive data, such as a password into your Git repository, you can remove it from the history using the git filter-branch command.     

https://help.github.com/en/articles/removing-sensitive-data-from-a-repository

The link gives two options using a 3rd party tool called BFG Repo-Cleaner  (Java JAR file) or the git filter-branch, In my case I caught this right away and used the git filter-branch to delete it.   

Some key information from the above link:

 "The git filter-branch command and the BFG Repo-Cleaner rewrite your repository's history, which changes the SHAs for existing commits that you alter and any dependent commits. Changed commit SHAs may affect open pull requests in your repository. We recommend merging or closing all open pull requests before removing files from your repository."

Warning

Warning: Once you have pushed a commit to GitHub, you should consider any data it contains to be compromised. If you committed a password, change it! If you committed a key, generate a new one.

This article tells you how to make commits with sensitive data unreachable from any branches or tags in your GitHub repository. However, it's important to note that those commits may still be accessible in any clones or forks of your repository, directly via their SHA-1 hashes in cached views on GitHub, and through any pull requests that reference them. You can't do anything about existing clones or forks of your repository, but you can permanently remove cached views and references to the sensitive data in pull requests on GitHub by contacting GitHub Support or GitHub Premium Support.

Instructions

These are the steps I followed to remove the fie, I found my commit change on the initial load so did not have to deal with any of the warnings in the Git resource link above. recommend you do read the link to review if you have any other steps to do.  

  1. Make sure you have local copy of the repository on your computer, if you do not then clone the repository

  2. Navigate to the repositories working directory the commands will only work form the top level of the working tree

  3. Run the following command  replacing the path-to-the-file with the path and name of your file,  paths use the / character  (example: src/app.config)  Note: the double quotes used here, the article used a single quote and those do not work in windows. Tis command will force git to process but not check out the entire history of every branch and tag and remove the specified file as well as any empty commits   

    • git filter-branch --force --index-filter "git rm --cached --ignore-unmatch path-to-the-file" --prune-empty --tag-name-filter cat -- --all  

    • Note: If the file used any other paths, because it was moved or renamed,  you must run this command on those as well.  

  4. Double check that you removed the files and after the file has been removed you can add the file to the .gitignore to ensure don't accidentally commit again. For .net config files you will want the file back in the repository just without the sensitive data.  Use an external configuration file  (<connectionStrings configSource="connections.config"/>   or <appSettings configSource="appsetting.config"/>) and this file will be added to the .gitignore.

  5. When your ok with the changes to the repository. issue a fore-push to overwrite the Git repository.   You will need the Git force-push permission on your account in order to do this, this is something that needs to be done on the master branch of your repository. See instructions below.   If you don't have the permission the commend will let you know and you can correct and run it again once fixed.  

    • git push origin --force --all

  6. See the article above for Step 10  on doing a dereference and garbage collection 

    After some time has passed and you're confident that git filter-branch had no unintended side effects, you can force all objects in your local repository to be dereferenced and garbage collected with the following commands (using Git 1.8.5 or newer):

    $ git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
$ git reflog expire --expire=now --all
$ git gc --prune=now
> Counting objects: 2437, done.
> Delta compression using up to 4 threads.
> Compressing objects: 100% (1378/1378), done.
> Writing objects: 100% (2437/2437), done.
> Total 2437 (delta 1461), reused 1802 (delta 1048)

Adding force-push to Branch Security

In order to update the repository you will need the Git force-push permission the following article will document how to set this up for your master branch.  For my project I set the Force Push to Allow for the Project Administrators Group where I was a member of.   

https://docs.microsoft.com/en-us/azure/devops/repos/git/branch-permissions?view=azure-devops