Find the value fluctuating in lists

fluctuatingPos1 = Pick[Range @ Length @ #, 
   MatchQ[#, {___, a_, ___, b_, ___, c_, ___} /; a < b > c || a > b < c]& /@ #] & @
    Transpose[#]&; 
fluctuatingPos2 =  Flatten@Position[#, {___, a_, ___, b_, ___, c_, ___} /; 
        a < b > c || a > b < c, 1] & @ Transpose[#] &;
fluctuatingPos3 = Flatten @ Position[Unitize[Length[DeleteDuplicates[#]] -  1 & /@ 
  (Sign[Differences[#]] & /@ Transpose[#])], 1, 1] &;
fluctuatingPos4 = Flatten @ Position[(Boole @ Nor[OrderedQ[#], OrderedQ[Reverse@#]] &) /@ 
 Transpose[#], 1] &;

fluctuatingPos1[{X1, X2, X3, X4}]

{4, 7}

Equal[fluctuatingPos1@#, fluctuatingPos2@#, fluctuatingPos3@#, 
  fluctuatingPos4@#] &@{X1, X2, X3, X4}

True

SeedRandom[1]
Y = RandomInteger[10, {1000, 1000}];
{r1 = fluctuatingPos1[Y]; // AbsoluteTiming // First,
 r2 = fluctuatingPos2[Y]; // AbsoluteTiming // First,
 r3 = fluctuatingPos3[Y]; // AbsoluteTiming // First,
 r4 = fluctuatingPos4[Y]; // AbsoluteTiming // First}

{0.747986, 0.755511, 0.031081, 0.009527}

r1 == r2 == r3 == r4

True

It sound like you want to fit a linear growth model and pick out those data sets that have high error variance. This can be done with LinearModelFit.

Here, I merge you data into a single matrix and add also a list tlist for the time points of your measurements.

X = Transpose[{X1, X2, X3, X4}];
tlist = {1, 2, 3, 4};

This computes fitted models for you data.

fit = LinearModelFit[Transpose[{tlist, #}], {t, 1}, t] &;
data = fit /@ X;

And this returns the estimated error variances. High values mean high fluctuation:

variances = #["EstimatedVariance"] & /@ data

{0.1, 0.4, 0.75, 4.35, 3.75, 3.6, 4.5, 7.35, 4.5, 3.15544*10^-29}

So given a threshold, you can find the position of the data sets with higher variance as threshold with

threshold = 0.5
Flatten[Position[variances, _?(# > threshold &)]]

You can use a high-pass filter. It lets quick variations in the data pass through. This is what the four datasets look like when high-pass filtered:

ListLinePlot[HighpassFilter[#, 2] & /@ {X1, X2, X3, X4}]

I have used a cutoff frequency of 2 here, you may want to adjust that according to your data.

To get some quantification, you could sum over the absolute values, for example:

N@Total[Abs@HighpassFilter[#, 2]] & /@ {X1, X2, X3, X4}
(* {0.477118, 19.1493, 37.1699, 49.6903} *)