How to calculate the mirror point along a line?
Suppose the equation of the line is ax + by + c = 0. Now imagine a line perpendicular to it, which can be represented by -bx + ay + d = 0 (product of slopes of two perpendicular lines is -1). Now the problem is to find d. Put the co-ordinate of the point on the second line, and you'll get the value of d easily.
The second part is, to find a point on the second line which is equidistant as the first point from the first line. For that, you can find the intersection of the two lines. Calculate the differences in x and y of the given point and the intersection point. Now add those to the x and y value of the intersecting point. That gives the point you need (you may need to negate the differences - that's up to the order of subtraction you use).
When things like that are done in computer programs, one of the issues you might have to deal with is to perform these calculations using integer arithmetic only (or as much as possible), assuming the input is in integers. Doing this in integers as much as possible is a separate issue that I will not cover here.
The following is a "mathematical" solution, which if implemented literally will require floating-point calculations. I don't know whether this is acceptable in your case. You can optimize it to your taste yourself.
(1) Represent your line L by
A * x + B * y + C = 0
equation. Note that vector (A, B) is the normal vector of this line.
For example, if the line is defined by two points X1(x1, y1) and X2(x2, y2), then
A = y2 - y1
B = -(x2 - x1)
C = -A * x1 - B * y1
(2) Normalize the equation by dividing all coefficients by the length of vector (A, B). I.e. calculate the length
M = sqrt(A * A + B * B)
and then calculate the values
A' = A / M
B' = B / M
C' = C / M
The equation
A' * x + B' * y + C' = 0
is still an equivalent equation of your line L except that now the normal vector (A', B') is a unit vector.
(3) Take your point P(px, py) and calculate the value
D = A' * px + B' * py + C'
This will give you the signed distance D from your point P to your line L. In other words, this is the distance from P to the closest point on L (we don't really care about the closest point itself, we just need the distance).
The sign says which side of the line L the point P lies on. If P lies on the same side the vector (A', B') is pointing to ("positive" side), the distance is positive. If P lies on the other side ("negative" side), the distance is negative.
(4) In order to find your mirror point P'(px', py') you need to move your point P by the absolute distance |2 * D| across the line L to the other side.
"Across the line" really means that if point P was lying on the "positive" side of L, then we have to move it against the direction of vector (A', B') to the "negative" side. And vice versa, if point P was lying on the "negative" side of L, then we have to move it in the direction of vector (A', B') to the "positive" side.
This can simply be expressed as moving the point the distance of -2 * D (note the minus) in the direction of vector (A', B').
That means that
px' = px - 2 * A' * D
py' = py - 2 * B' * D
gives you your mirror point P'(px', py').
Alternatively, you can use an approach based on finding the actual closest point N on line L and then reflecting your point P with relation to N. This is already suggested in other answers, I'll just describe how I'd do it.
(1) Build an equation
A*x + B*y + C = 0
for your line L exactly as described in step 1 above. There's no need to normalize this equation.
(2) Build an equation for the perpendicular line that passes through P. Let's say that perpendicular line is represented by
D*x + E*y + F = 0
The D and E coefficients are known right away
D = B
E = -A
while F can be calculated by substituting point P into the equation
F = -D*px - E*py
(3) Find the intersection of these two lines by solving the system of two linear equations
A*x + B*y = -C
D*x + E*y = -F
Cramer's rule works very well in this case. The formula given in Line intersection article in Wikipedia are nothing else than an application of Cramer's rule to this system.
The solution gives you the nearest point N(nx, ny) we were looking for.
(4) Now just calculate
px' = nx + (nx - px)
py' = ny + (ny - py)
to find your point P'(px', py').
Note that this approach can be almost entirely implemented in integers. The only step that might lose precision is division inside the Cramer's rule at step 3. Of course, as usual, the price you'll have to pay for "almost integral" solution is the need for large-number arithmetic. Even coefficients C and F can overflow, not even mentioning computations inside Cramer's rule formulae.