{"id":8001,"date":"2021-06-23T12:37:39","date_gmt":"2021-06-23T17:37:39","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=8001"},"modified":"2021-06-23T12:51:31","modified_gmt":"2021-06-23T17:51:31","slug":"git-removing-confidential-info-from-history","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=8001","title":{"rendered":"Git – Removing Confidential Info From History"},"content":{"rendered":"

The first cut of code may contain … not best practice code. Sometimes this is just hard coding something you’ll want to compute \/ look up in the future. Hard coding user input isn’t a problem if my first cut is always searching for ABC123. Hard coding the system creds? Not good. You sort that before you actually deploy<\/em> the code. But some old iteration of the file has MyP2s5w0rD sitting right there in plain text. That’s bad in a system that maintains file history! The quick\/easy way to clean up passwords stashed within the history is to download the BFG JAR<\/a> file.<\/p>\n

For this test, I created a new repository in .\\source then created three clones of the repo (.\\clone1, .\\clone2, and .\\clone3). In each cloneX copy, I created a tX folder that has a file named ldapAuthTest.py — a file that contains a statically assigned password as<\/p>\n

strSystemAccountPass = \"MyP2s5w0rD\"<\/pre>\n
The first thing I did was to redact the password in the files — this means anyone looking at HEAD won’t<\/em> see the password. Source, clone1, and clone2 are all current. The clone3 copy has pulled all changes but has a local change committed but not merged.<\/p>\n
To clean the password from the git history, first create a backup of your repo (just in case!). Then mirror the repo to work on it<\/p>\n
mkdir mirror\r\ncd mirror\r\ngit clone --mirror d:\\git\\testFilterBranch\\source<\/pre>\n
 <\/p>\n
Create file .\\replacements.txt with the string to be redacted — in this case:<\/p>\n
strSystemAccountPass = \"MyP2s5w0rD\"<\/pre>\n
Formatting notes for replacements.txt<\/p>\n
MyP2s5w0rD # Replaces string with default ***REMOVED***\r\nMyP2s4w0rD==>REDACTED # Replaces string using custom string REDACTED\r\nMyP2s3w0rD==> # Replaces string with null -- i.e. removes the string\r\nregex:strSystemAccountPass\\s?=\\s?\"MyP2s2w0rD\"\"==>REDACTED # Uses a regex match -- in this case we may or may not have a space around the equal sign<\/pre>\n
So, in my mirror folder, I have the replacement.txt file which defines which strings are replaced. I have a folder that contains the mirror of my repo.<\/p>\n
lisa@FLEX3 \/cygdrive\/d\/git\/testFilterBranch\/mirror\r\n$ ls\r\nreplacements.txt source.git<\/pre>\n
To replace my “stuff”, run bfg using the –replace-text option. Because I only<\/em> want to replace the text in files named ldapAuthTest.py, I also added the -fi option<\/p>\n
java -jar ..\/bfg-1.14.0.jar --replace-text ..\\replacements.txt -fi ldapAuthTest.py source.git<\/pre>\n
 <\/p>\n
lisa@FLEX3 \/cygdrive\/d\/git\/testFilterBranch\/mirror\r\n$ java -jar ..\/bfg-1.14.0.jar --replace-text replacements.txt -fi ldapAuthTest.py source.git\r\n\r\nUsing repo : D:\\git\\testFilterBranch\\mirror\\source.git\r\n\r\nFound 3 objects to protect\r\nFound 2 commit-pointing refs : HEAD, refs\/heads\/master\r\n\r\nProtected commits\r\n-----------------\r\nThese are your protected commits, and so their contents will NOT be altered:\r\n* commit 87f1b398 (protected by 'HEAD')\r\n\r\nCleaning\r\n--------\r\nFound 5 commits\r\nCleaning commits: 100% (5\/5)\r\nCleaning commits completed in 613 ms.\r\n\r\nUpdating 1 Ref\r\n--------------\r\n\r\nRef Before After\r\n---------------------------------------\r\nrefs\/heads\/master | 87f1b398 | 919c8f0f\r\n\r\nUpdating references: 100% (1\/1)\r\n...Ref update completed in 151 ms.\r\n\r\nCommit Tree-Dirt History\r\n------------------------\r\n\r\nEarliest Latest\r\n| |\r\n. D D D m\r\n\r\nD = dirty commits (file tree fixed)\r\nm = modified commits (commit message or parents changed)\r\n. = clean commits (no changes to file tree)\r\n\r\nBefore After\r\n-------------------------------------------\r\nFirst modified commit | dc2cd935 | 8764f6f1\r\nLast dirty commit | 9665c4e0 | ccdf0359\r\n\r\nChanged files\r\n-------------\r\n\r\nFilename Before & After\r\n-------------------------------------\r\nldapAuthTest.py | 25e79fa6 ? 4d12fdad\r\n\r\nIn total, 8 object ids were changed. Full details are logged here:\r\nD:\\git\\testFilterBranch\\mirror\\source.git.bfg-report\\2021-06-23\\12-50-00\r\n\r\nBFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive<\/pre>\n
Check to make sure nothing looks abjectly wrong. Assuming the repo is sound, we’re ready to clean up and push these changes.<\/p>\n
cd source.git\r\n\r\ngit reflog expire --expire=now --all && git gc --prune=now --aggressive\r\ngit push<\/pre>\n
 <\/p>\n
Pulling the update from my source repo, I have merge conflicts<\/p>\n
\"\"<\/a><\/p>\n
These are readily resolved and the source repo can be merged into my local copy.<\/p>\n
\"\"<\/a><\/p>\n
And the change I had committed but not pushed is still there.<\/p>\n
\"\"<\/a><\/p>\n
Pushing that change produces no errors<\/p>\n
\"\"<\/a><\/p>\n
Now … pushing the bfg changes may not work. In my case, the real repo has a bunch of branchs and I am getting “non fast-forward merges”. To get the history changed, I need to do a force<\/em> push. Not so good for the other developers! In that case, everyone should get their changes committed and pushed. The servers should be checked to ensure they are up to date. Then the force push can be done and everyone can pull the new “good” data (which, really, shouldn’t differ from the old data … it’s just the history that is being tweaked).<\/p>\n","protected":false},"excerpt":{"rendered":"
The first cut of code may contain … not best practice code. Sometimes this is just hard coding something you’ll want to compute \/ look up in the future. Hard coding user input isn’t a problem if my first cut is always searching for ABC123. Hard coding the system creds? Not good. You sort that …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[33],"tags":[233,69],"class_list":["post-8001","post","type-post","status-publish","format-standard","hentry","category-coding","tag-git","tag-security"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/8001","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8001"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/8001\/revisions"}],"predecessor-version":[{"id":8006,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/8001\/revisions\/8006"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8001"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8001"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8001"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}