users are asked for password while using gitolite

In your config file, I see:

User dilawar

That is wrong. ssh communication to a gitolite server are always done with the same account (here gitolite).
What changes is the private key used, which will help gitolite determine your identity.

What you ~/.ssh/config file should look like is:

Host admin
    HostName 10.107.105.13 
    User gitolite
    IdentityFile ~/.ssh/gitolite

Host dilawar
    HostName 10.107.105.13 
    User gitolite
    IdentityFile ~/.ssh/id_rsa

For cloning gitolite-admin, you would use:

git clone admin:gitolite-admin

For cloning a repo dilawar has access to:

git clone dilawar:aRepo

See more at "Gitolite: adding user not working, and DENIED by fallthru when cloning as root?".
See also "how gitolite uses ssh"

Adding your public key to the server's ~git/.ssh/authorized_keys file is how ssh uses pubkeys to authenticate users.
Let's say [email protected] is trying to log in as git@server.
What you have to do is take the ~sita/.ssh/id_rsa.pub file for user sita on work-station and append its contents (remember it's only one line) to ~git/.ssh/authorized_keys for user git on server.

The authorized_keys file can have multiple public keys (from many different people) added to it so any of them can log in to git@server.

I have got it working by cloning the repository using the gitolite username.

git clone gitolite@server:repo 

If keys are added successfully then further pull and push will go smoothly.

I am accepting VomC answer as a better answer.