Git commits are duplicated in the same branch after doing a rebase

I think you skipped an important detail when describing your steps. More specifically, your last step, git push on dev, would have actually given you an error, as you can not normally push non-fastforward changes.

So you did git pull before the last push, which resulted in a merge commit with C6 and C6' as parents, which is why both will remain listed in log. A prettier log format might have made it more obvious they are merged branches of duplicated commits.

Or you made a git pull --rebase (or without explicit --rebase if it is implied by your config) instead, which pulled the original C5 and C6 back in your local dev (and further re-rebased the following ones to new hashes, C7' C5'' C6'').

One way out of this could have been git push -f to force the push when it gave the error and wipe C5 C6 from origin, but if anyone else also had them pulled before you wiped them, you'd be in for a whole lot more trouble... basically everyone that has C5 C6 would need to do special steps to get rid of them. Which is exactly why they say you should never rebase anything that's already published. It's still doable if said "publishing" is within a small team, though.

Short answer

You omitted the fact that you ran git push, got the following error, and then proceeded to run git pull:

To [email protected]:username/test1.git
 ! [rejected]        dev -> dev (non-fast-forward)
error: failed to push some refs to '[email protected]:username/test1.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Despite Git trying to be helpful, its 'git pull' advice is most likely not what you want to do.

If you are:

• Working on a "feature branch" or "developer branch" alone, then you can run git push --force to update the remote with your post-rebase commits (as per user4405677's answer).
• Working on a branch with multiple developers at the same time, then you probably should not be using git rebase in the first place. To update dev with changes from master, you should, instead of running git rebase master dev, run git merge master whilst on dev (as per Justin's answer).

A slightly longer explanation

Each commit hash in Git is based on a number of factors, one of which is the hash of the commit that comes before it.

If you reorder commits you will change commit hashes; rebasing (when it does something) will change commit hashes. With that, the result of running git rebase master dev, where dev is out of sync with master, will create new commits (and thus hashes) with the same content as those on dev but with the commits on master inserted before them.

You can end up in a situation like this in multiple ways. Two ways I can think of:

• You could have commits on master that you want to base your dev work on
• You could have commits on dev that have already been pushed to a remote, which you then proceed to change (reword commit messages, reorder commits, squash commits, etc.)

Let's better understand what happened—here is an example:

You have a repository:

2a2e220 (HEAD, master) C5
ab1bda4 C4
3cb46a9 C3
85f59ab C2
4516164 C1
0e783a3 C0

You then proceed to change commits.

git rebase --interactive HEAD~3 # Three commits before where HEAD is pointing

(This is where you'll have to take my word for it: there are a number of ways to change commits in Git. In this example I changed the time of C3, but you be inserting new commits, changing commit messages, reordering commits, squashing commits together, etc.)

ba7688a (HEAD, master) C5
44085d5 C4
961390d C3
85f59ab C2
4516164 C1
0e783a3 C0

This is where it is important to notice that the commit hashes are different. This is expected behaviour since you have changed something (anything) about them. This is okay, BUT:

Trying to push will show you an error (and hint that you should run git pull).

$ git push origin master
To [email protected]:username/test1.git
 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to '[email protected]:username/test1.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

If we run git pull, we see this log:

7df65f2 (HEAD, master) Merge branch 'master' of bitbucket.org:username/test1
ba7688a C5
44085d5 C4
961390d C3
2a2e220 (origin/master) C5
85f59ab C2
ab1bda4 C4
4516164 C1
3cb46a9 C3
0e783a3 C0

Or, shown another way:

And now we have duplicate commits locally. If we were to run git push we would send them up to the server.

To avoid getting to this stage, we could have run git push --force (where we instead ran git pull). This would have sent our commits with the new hashes to the server without issue. To fix the issue at this stage, we can reset back to before we ran git pull:

Look at the reflog (git reflog) to see what the commit hash was before we ran git pull.

070e71d HEAD@{1}: pull: Merge made by the 'recursive' strategy.
ba7688a HEAD@{2}: rebase -i (finish): returning to refs/heads/master
ba7688a HEAD@{3}: rebase -i (pick): C5
44085d5 HEAD@{4}: rebase -i (pick): C4
961390d HEAD@{5}: commit (amend): C3
3cb46a9 HEAD@{6}: cherry-pick: fast-forward
85f59ab HEAD@{7}: rebase -i (start): checkout HEAD~~~
2a2e220 HEAD@{8}: rebase -i (finish): returning to refs/heads/master
2a2e220 HEAD@{9}: rebase -i (start): checkout refs/remotes/origin/master
2a2e220 HEAD@{10}: commit: C5
ab1bda4 HEAD@{11}: commit: C4
3cb46a9 HEAD@{12}: commit: C3
85f59ab HEAD@{13}: commit: C2
4516164 HEAD@{14}: commit: C1
0e783a3 HEAD@{15}: commit (initial): C0

Above we see that ba7688a was the commit we were at before running git pull. With that commit hash in hand we can reset back to that (git reset --hard ba7688a) and then run git push --force.

And we're done.

But wait, I continued to base work off of the duplicated commits

If you somehow didn't notice that the commits were duplicated and proceeded to continue working atop of duplicate commits, you've really made a mess for yourself. The size of the mess is proportional to the number of commits you have atop of the duplicates.

What this looks like:

3b959b4 (HEAD, master) C10
8f84379 C9
0110e93 C8
6c4a525 C7
630e7b4 C6
070e71d (origin/master) Merge branch 'master' of bitbucket.org:username/test1
ba7688a C5
44085d5 C4
961390d C3
2a2e220 C5
85f59ab C2
ab1bda4 C4
4516164 C1
3cb46a9 C3
0e783a3 C0

Or, shown another way:

In this scenario we want to remove the duplicate commits, but keep the commits that we have based on them—we want to keep C6 through C10. As with most things, there are a number of ways to go about this:

Either:

• Create a new branch at the last duplicated commit¹, cherry-pick each commit (C6 through C10 inclusive) onto that new branch, and treat that new branch as canonical.
• Or run git rebase --interactive $commit, where $commit is the commit prior to both the duplicated commits². Here we can outright delete the lines for the duplicates.

¹ It doesn't matter which of the two you choose, either ba7688a or 2a2e220 work fine.

² In the example it would be 85f59ab.

TL;DR

Set advice.pushNonFastForward to false:

git config --global advice.pushNonFastForward false

You should not be using rebase here, a simple merge will suffice. The Pro Git book that you linked basically explains this exact situation. The inner workings might be slightly different, but here's how I visualize it:

• C5 and C6 are temporarily pulled out of dev
• C7 is applied to dev
• C5 and C6 are played back on top of C7, creating new diffs and therefore new commits

So, in your dev branch, C5 and C6 effectively no longer exist: they are now C5' and C6'. When you push to origin/dev, git sees C5' and C6' as new commits and tacks them on to the end of the history. Indeed, if you look at the differences between C5 and C5' in origin/dev, you'll notice that though the content is the same, the line numbers are probably different -- which makes the hash of the commit different.

I'll restate the Pro Git rule: never rebase commits that have ever existed anywhere but your local repository. Use merge instead.