Computing parameters for QGIS Affine transformation?

First you need to project your geographic coordinates to a cartesian 2D coordinate system, since affine transformations don't apply to geographic coordinate systems.

You can apply an affine transformation from control points or from transformation parameters. The QGIS plugin asks you transformation parameters, but it is much more common for a user to have control points.

From control points you can calculate transformation parameters. For an affine transformation there are 6 transformation parameters, so you need at least 3 control points (each control point implies 4 coordinates: Xsource, Ysource, Xtarget, Ytarget), but more control points are recommended to have redundancy and thus be able to apply Least Squares, which would give you an estimation of the transformation's quality. Remember that affine transformations can rotate, shift, scale (even applying different factors on each axis) and skew geometries.

Control points must have the form:

X SOURCE: Xs
Y SOURCE: Ys
X TARGET: Xt
Y TARGET: Yt

Parameters are:

a: Scale X 
e: Scale Y
d: Rotation X
b: Rotation Y
c: Translation X
f: Translation Y

And we know:

Xt = X*a + Y*b + c
Yt = X*d + Y*e + f

So, you need to solve this system of equations (for 3 control points):

¦ Xs1 Ys1 1 0 0 0 ¦ | a ¦   ¦ Xt1 ¦
¦ Xs2 Ys2 1 0 0 0 ¦ ¦ b ¦   ¦ Xt2 ¦ 
¦ Xs3 Ys3 1 0 0 0 ¦ ¦ c ¦ = ¦ Xt3 ¦ 
¦ 0 0 0 Xs1 Ys1 1 ¦ ¦ d ¦   ¦ Yt1 ¦ 
¦ 0 0 0 Xs2 Ys2 1 ¦ ¦ e ¦   ¦ Yt2 ¦ 
¦ 0 0 0 Xs3 Ys3 1 ¦ ¦ f ¦   ¦ Yt3 ¦

Where parameters a, b, c, d, e, and f are unknown.

Once you calculate parameters a, b, c, d, e, and f, (for example with this online equation solver) place them into the QGIS plugin interface like this:

X' = a*x + b*y + c
Y' = d*x + e*y + f

or:

I think this solves your two questions.

Using OpenJump to calculate the affine transformation parameters yields:

Scale x: 0.02550720529745378
Scale y: 0.025669710194697357
Rotation: -88.6538203230914
Dx: 357101.9 (Translation x)
Dy: 4512814.6 (Translation y)

These parameters when properly applied should transform your shapefile from the local CRS that it uses to WGS84/UTM Zone 17 North (EPSG:32617). In QGIS the transformed data could then be exported ("saved as") to any required CRS; Google Mercator or lat/lon, for example.

The bad news is that I have never once been able to use successfully the QGIS affine plug-in for transformations that include rotation. By some accounts it works for other people but not for me, so it must be something I'm doing wrong. If anyone could tell me what this is I'd be grateful.

There was a thread here on affining vectors that is well-worth reading. The method I use for affining vectors is explained here at the bottom of the first page.

If you get really stuck you could post a link to the shapefile and we could all have a go at getting it transformed.

An afterthought. If you decide to try the OpenJump method you'll need the WGS84/UTM Zone 17 North (EPSG:32617) coordinates for your lat/lon points:

EPSG:32617: 361154.4 4513930.1
lat/long: 40.764622,-82.645056
shapefile coor: 947.3,1242.8

EPSG:32617: 361152.2 4513850.3
lat/long: 40.763903,-82.645064
shapefile coor: 4050.3,1035.7

EPSG:32617: 361246.5 4513886.7
lat/long: 40.764247,-82.643956
shapefile coor: 2710.5,4765.2