How to use GIT on multi dev environment

There are a couple issues with how you're using git, let me talk about each.

The problem

First, you're treating your Salesforce server as the truth, instead of git. By doing a Refresh from Server each time before you're starting to code, you are updating your working directory with whatever has changed in Salesforce. This will be reflected as a change pending commit on your local branch.

Second, you're having developers share a Salesforce instance. Imagine that you're each working on separate features on separate branches in git but on the same instance. If you do something like refreshing from server, all the stuff from your colleague's branch is going to show up in your index, ready to be committed. Worse, what if you're editing the same metadata files, you're constantly overwriting each other.

The solution

Each developer needs his/her own Salesforce instance. Yes, keeping those instances up-to-date with a stable build (master branch) poses its own challenges but there are a ton of articles out there on continuous integration with Salesforce. This allows for feature development and accurate testing (since Salesforce controls the unit test environment) within your own branch and instance.

Also, you should be treating git as the truth. Right now, you're trusting your instance but why wouldn't you look to your stable build in git first. Step 1 in a VCS system should be a Pull, not a refresh from server. In this scenario, who cares what is on the Salesforce server, if it's not in git it does not exist. And would you want it any other way, git is really designed to hold the history of your changes and Salesforce is not.