What coordinate system should be used for projecting a non-geographical scan?

Use a Cartesian coordinate system you can set 0,0 arbitrarily (ie: Center Of Gravity or a corner. The units can be anything and any math you want to apply will require elementry level geometry skills.

You will want to georectify the *.tiff here is an example of how to do it there are many other tutorials if you google georectify tiff this is an esri article Fundamentals of georeferencing a raster dataset .

If you do not have real world coordintates you will need to start from 0,0.

Knowing the resolution helps. If you have access to Python and GDAL, you can do something like (not run):

from osgeo import gdal

# Read the original file
fn = "c:/fullpath/myfile.tif"
ds = gdal.Open(fn)

# Write out a copy, changing the GeoTransform
driver = gdal.GetDriverByName("GTIFF")
ds_out = driver.CreateCopy("c:/fullpath/out.tif", ds, 0)
ds_out.SetGeoTransform([0, ds.RasterXSize / 2400, 0, 0, ds.RasterYSize / 2400])
ds_out = None
ds = None

The key part is the GeoTransform, it is [xorigin, xpixelsize, 0, yorigin, 0, ypixelsize].

Having the proper origin (0,0) in the upper-left, and a physical pixel size (in inches), you will be able to make relatively accurate measurements. You don't need to set a projection when opening it ArcGIS. It will show as unknown units, but you know that they are in inches. Having a ruler or other known measure in the picture would help confirm that everything is as it should be.

The simpler alternative is to do any tracing / analysis in 1x1 unit pixels, as it is in the raw image, then convert distances and areas appropriately using the known resolution (1 pixel = 1/2400 inches, for distances, in your case).

If you need to assign a projection, something like this "+proj=eqc +ellps=sphere +units=int-in" might work:

PROJ.4 : '+proj=eqc +lat_ts=0 +lat_0=0 +lon_0=0 +x_0=0 +y_0=0 +a=6370997 +b=6370997 +units=m +no_defs '

OGC WKT :
PROJCS["unnamed",
    GEOGCS["Normal Sphere (r=6370997)",
        DATUM["unknown",
            SPHEROID["sphere",6370997,0]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433]],
    PROJECTION["Equirectangular"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",0],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["int-in",1]]

I don't think it really matters what you pick as long as the units and pixel size are correct, because you are dealing with such a small area, geographically speaking.