Could Cat5e be used for RS485
Short answer: yes. That should work just fine. It is quite common to use CAT5 for RS485.
More detailed:
RS485 requires a common ground reference for all devices. Yes, you can use spare pairs in the cable as ground reference. Proper shielded cables provide better noise immunity. But it is not by any means necessary for RS485 communication to work. Depends on your environment. Lots of high power electrical motors and welders around? Then you may want to use shielded cables.
Depending on your exact cable length requirements, you may want to consider thicker gauge CAT5. Up to ~100m/300ft any CAT5 should do the job. Beyond that, the resistance of the copper conductor can start playing a role in attenuating your signal. Thinner conductor=higher attenuation. RS485 cables designed for 1000m/3000ft have really thick copper wires.
Definitely yes, the CAT5/5E or CAT6 cable is perfect for industrial use of RS485 network.
Pay attention of the shield connection and for long distance networks is recommended to use ISO tranciever like ADM2483 (Datasheet here).
If the shield is not connected (that is a common mistake) an isolated tranciever will almost solve communication issues (a traditional MAX485 or non isolated tranciever will probably not work and it will potentially be damaged due to compensation currents to earth)
In addition to that, the termination resistor on the farthest device can help to balance the entire bus.
Without termination resistor and the ADM2483 ISO tranciever i've reached 1350 meters with a CAT6 cable (shield unconnected), only A and B communication cables at 57600 baud.
Yes it could work, but you have mentioned something about the ground and shield. The RS485 forlong distance is better to be isolated one. The ground goes nowhere, since both devices are connected to the local earth (usually not directly, rather capacitive coupling from environment, the transceiver is floating).
The shield goes connected to the earth in a single point or it is divided into sections.
This situation may be different if you are using non-isolated RS485 transceivers, a third wire is needed - GND. If devices have GND shared with local earth, then large compensation currents will flow over this tinny wire.