How does Mathematica understand branchcuts of the complex logarithm?

The standard built-in logarithm function is defined for complex variables as follows:

Log[z] = Log[Abs[z]] + I Arg[z]

The location of the branch cut is simply caused by the convention that polar angles of z are assumed to be in the range $-\pi$ to $\pi$. This same branch cut is also part of the definition of the built-in Arg function.

Here is a different logarithm function called simply (lower-case) log to which you can supply a polar angle $\sigma$ at which the branch cut is going to be:

Clear[arg, log];
arg[z_, σ_: - Pi] := Arg[z Exp[-I (σ + Pi)]] + σ + Pi;
log[z_, σ_: - Pi] := Log[Abs[z]] + I arg[z, σ]

The first argument is the same as in the usual Log function. The second argument is optional, and if it's omitted then the new function agrees with its built-in counterpart. The default branch cut of the phase angle arg is $\sigma = -\pi$.

To illustrate these branch cut locations, I defined a "convenience function" that plots a function of the complex variable z in a square of side length 6 around the origin:

plot[f_] := Module[
  {fn = f /. z -> x + I y},
  Show[
   ContourPlot[
    Im[fn],
    {x, -3, 3}, {y, -3, 3},
    ContourShading -> Automatic,
    ExclusionsStyle -> Red
    ],
   ContourPlot[
    Re[fn],
    {x, -3, 3}, {y, -3, 3},
    ContourShading -> False,
    ContourStyle -> Blue,
    ExclusionsStyle -> Red
    ],
   FrameLabel -> {"Re(z)", "Im(z)"},
   Background -> Lighter[Orange],
   PlotRangePadding -> 0,
   PlotLabel -> 
    Framed[Grid[{{Style["-", Bold, Blue], 
        "Real part"}, {Style["-", Bold, Gray], "Imaginary part"}}, 
      Alignment -> Left], FrameStyle -> None, Background -> White, 
     RoundingRadius -> 5],
   ImageSize -> 300
   ]
  ]

For example, see the location of the branch cuts (red) rotate from the default to a non-standard direction:

GraphicsRow[{plot[Log[z]],plot[log[z,π/4]]}]

But now look at your function, with a = 1 because that constant is irrelevant:

plot[z Tanh[π z] Log[z^2 + 1]]

Here, the default choice of branch cut of the Log function leads to vertical branch cuts in the complex plane. This means the upper half plane is not a safe place to be integrating in...

Depending on your needs, you could try to change the branch cut location to something else by using the log function with an appropriate choice for $\sigma$.

Here is where the freedom of choice in the function log comes in. Try this:

plot[z Tanh[π z] log[z^2 + 1, 0]]

And as you can see, the branch cuts are now in a different position, no longer cutting through the middle of your integration domain.

Edit: consequences for integration

The question states that an integral along a semicircle in the upper half plane is to be performed, and the integrand now could be one of these two functions, corresponding to the default branch cut and the rotated branch cut shown in the previous image. Here I define them as functions:

f[z_] := z Tanh[π z] Log[z^2 + 1];
g[z_] := z Tanh[π z] log[z^2 + 1, 0]

The question further asks whether the result of the integration is "trustworthy." The answer is yes, but you have to know what exactly you're asking Mathematica to do. Therefore, I'll add some different plots just to clarify what I already said above. Pick a radius for your integration contour:

r = 6.2;

This is chosen to be not too large so as to obscure things, but also to avoid hitting one of the singularities of the Tanh functions which appear at regular intervals along the imaginary axis.

Now plot the two functions versus the polar angle in the interval corresponding to the upper half plane, and look at the numerical integrals of the two:

Plot[Evaluate[{Re[#], Im[#]} &@f[r Exp[I ϕ]]], {ϕ, 
  0, Pi}]

NIntegrate[f[r Exp[I ϕ]], {ϕ, 0, Pi}]

(* ==> 20.271 - 3.55271*10^-15 I *)

This integral is perfectly allowable, but since it goes over the branch cut, you probably get something you didn't expect: in this case, a vanishing real part due to the cancelation of the area under the magneta, discontinuous curve. Mathematica doesn't produce a warning because isolated discontinuities are integrable. So is this result "trustworthy"? Sure. Do you want it? That's up to you.

Compare this to the function I suggested earlier:

Plot[Evaluate[{Re[#],Im[#]}&@g[r Exp[I ϕ]]],{ϕ,0,π}]

NIntegrate[g[r Exp[I ϕ]], {ϕ, 0, Pi}]

(* ==> 58.2644 + 37.6991 I *)

Now you're integrating over a continuous function, and that's true for any given r as long as it avoids the isolated singularities.

This result is also "trustworthy." Unfortunately, I can't tell from the question if this is what's needed. In any case, since the asymptotic behavior of both functions is divergent at large r (which answers the other question in the post), the integral along a semicircle would have to be done at finite radius.

From the comment I am guessing that the goal is somehow to apply Cauchy's theorem, which is perfectly OK, although in this problem it doesn't yield any benefit. Nevertheless, if this is the goal then you definitely have to avoid crossing branch cuts with the integration contour. So which of the cuts is a better choice in this situation? The answer is: g[z], because even though the branch cut now extends along the real axis, you can certainly draw a closed contour that completes the semicircle by hugging the cuts from above.

When doing this, you also end up having to calculate the integral over an infinitesimal circle around the branch point at $z = i$. By doing a series expansion of the factors around $z = i$ you can see that the integrand goes to zero and hence the integral is also zero. This answers the last question (which is not related to the choice of branch cuts).

Edit 2

Just for fun, here is how the branch cuts of the different logarithm functions evolve into each other as I change the parameter $\sigma$:

frames = Table[
   plot[z Tanh[Pi z] log[z^2 + 1, σ]], {σ, 
    0, Pi, Pi/16}];

ListAnimate[Join[Reverse[frames], frames], AnimationRepetitions -> 1]

The GIF was actually created with:

Export["cuts.gif",Join[Reverse[frames],frames],
  ImageResolution>72,AnimationRepetitions->Infinity,"DisplayDurations"->.1]

An alternative to @Jens' solution is to add the phase to the argument of the Log instead, and then remove the associated rotation in the output of the Log:

logbranch[σ_] := With[{phase = I (σ + Pi)},
    Function[Log[# Exp[-phase]] + phase]
]

I used a slightly different definition so that you just replace Log with logbranch[ϕ] instead of replacing Log[z] with log[z, ϕ]

Here is a side-by-side ComplexPlot of the two versions for a branch cut along the positive real axis:

log0 = logbranch[0];
GraphicsRow[{
    ComplexPlot[log[z,0], {z, -2-2I, 2+2I}],
    ComplexPlot[log0[z], {z, -2-2I, 2+2I}]
}]