How to install TortoiseGit without admin rights?

It is not possible. See Issue 726 on TortoiseGit homepage.

Not yet: bug 726 is still opened.
This installation with reduced features has been done for mercurial (portable TortoiseHg), but not for TortoiseGit yet.

The only workaround, mentioned in this comment is to install it once (with elevated privileges), zip the installation directory, and use that archive on other machine.
Obviously the all menu integration in the Windows Explorer wouldn't be present.

That requires a Microsoft Visual C++ Redistributable version though (also mentioned here):

If you run a application and got the error message

The program can’t start because MSVCR110.dll is missing from your computer.

then the Microsoft Visual C++ Redistributable is missing on the current computer.

This error appears when you wish to run a software which require the Microsoft Visual C++ Redistributable 2012.
The redistributable can easily be downloaded on the Microsoft website as x86 or x64 edition.

Depending on the software you wish to install you need to install either the 32 bit or the 64 bit version.
If you do not know which version you need to install, start to install the 32 bit version and if you still got the error install then the 64 bit version.
On a 32 bit OS you can install only the 32 bit redistributable.

You need to install the correct Microsoft Visual C++ Redistributable. If the software you wish to run require the 2012 DLLs it will not start with the 2010 ones.

You can install both versions on the same computer at the same time (if this is a 64bit OS).

---

Mark Ch adds in the comment:

Once done, you can run it as:

c:\[full path]\TortoiseGitProc.exe /command:log 

from your workspace folder using a command prompt or a shortcut.
This brings up the log window, from which it is possible to get to virtually every other function.