Should I ground the heatsink?

By the way, transistors(npn&pnp) are used in a class B audio amplifier.

This is an important detail. The transistor tab is usually connected to the Collector pin, and your standard insulator will add about 15pF capacitance between the tab and the heat sink, so this is about knowing whether this capacitance can cause problems or not.

If the heat sink is grounded, then the collector will have 15pF to ground capacitance in series with the connection inductance. Then the heat sink will also capacitively couple to whatever traces or signals are around.

If your amp uses an emitter-follower output stage, then both power transistors collectors are connected to power supplies (or ground if it is a single supply amp). Since power supplies are decoupled to ground, there is already a low impedance HF path between collectors and ground, therefore grounding the heat sink should make no difference. However, if the heat sink is external, or part of the enclosure, and the enclosure is already grounded at another point, grounding the heat sink on the amp pcb will introduce a ground loop.

If your amp uses a common emitter output stage, then the collectors are connected to the output, and the capacitance will be between output and heat sink. Some amps omit the insulator for better thermal resistance and have the heat sink connected to the output voltage, but this requires it to be inside an enclosure and not accessible. Anyway, if you ground it, then you have to consider if the extra 15pF to ground could make your amp unstable. For an audio amp, this is unlikely.

when I inspect the signals in the circuit via oscilloscope, I see there is some noise on waveforms, while trying to understand where it comes from, I have touched the heatsink (...) It becomes much smooter.

If the output voltage appears fuzzy on the scope, your amp could have stability issues. You can try single shot mode and see if you observe oscillations or just noise. If the amp does have stability issues, they are probably not related to case-to-heatsink capacitance, the problem would come from somewhere else, like layout or decoupling.

The electrical insulator such as Mica or Thermoplastic becomes a capacitor (100pF ??) but if you press your finger onto it your fingerpad, it could be 10x bigger (1nF) and shunt the crosstalk.

If the insulator is needed for safety, ground it. If just for DC and not personal safety but to protect the IC, it doesn't matter. You know what to do.

For HV and highspeed FETS, IGBT's its a different game and load capacitance should be minimized during high dV/dt.

AS long as you have an RC snubber on the speaker output to suppress parasitic oscillations... 10R+0.1uF

Your ungrounded Heatsink is actually a not too shabby capacitively coupled Antenna which radiates into sensitive low level circuits .Ground your isolated heatsink and the problems will go away .The time not to ground is say a SMPS that generates lots of noise at the drain of the switching transistor that you do not want to be injected into the ground causing an EMC fail.