Generating Tuples with restrictions

Select[
 IntegerPartitions[24, {8}, Range[5]],
 #.# == 86 &
]

{{5, 5, 4, 2, 2, 2, 2, 2}, {5, 5, 3, 3, 3, 2, 2, 1}, 
{5, 4, 4, 4, 2, 2, 2, 1}, {5, 4, 4, 3, 3, 3, 1, 1}, {4, 4, 4, 4, 4, 2, 1, 1}}

---

Slightly more general approach (in case where IntegerPartitions is not what we need):

ClearAll[ar, a];
ar = Array[a, 8]

ar /. Solve[Flatten@{
     Tr[ar] == 24,
     ar.ar == 86,
     Thread[1 <= ar <= 5],
     LessEqual @@ ar (* this part takes care about
                        droping equivalent permutations*)
     },
     ar,
     Integers
] 

{{1, 1, 2, 4, 4, 4, 4, 4}, {1, 1, 3, 3, 3, 4, 4, 5}, 
 {1, 2, 2, 2, 4, 4, 4, 5}, {1, 2, 2, 3, 3, 3, 5, 5}, {2, 2, 2, 2, 2, 4, 5, 5}}

A slightly different approach with Reduce (or Solve)

We define:

matC = Range[5]; (* the list of the integers from which we build a list *)

Let us use a vector to indicate the multiplicities of any integer:

matX = Array[ x, 5 ]; (* @Kuba: that's more concise, indeed *)

So x[1] will tell us how many times $1$ appears a possible solution.

We can then get all possible solutions by

sol= Reduce[
   {

      matC.matX == 24, (* the numbers add up to 24 *)

      Total[ matC^2 matX ] == 86, (* the sum of their squares is 86 *)

      Total[ matX ] == 8, (* there are exactly 8 numbers in the solution *)

      And @@ Thread[ matX >= 0 ]  (* non-negativity of course *)

   }
   ,
   matX
   ,
   Integers
]

      (x[1] == 0 && x[2] == 5 && x[3] == 0 && x[4] == 1 && x[5] == 2) 
   || (x[1] == 1 && x[2] == 2 && x[3] == 3 && x[4] == 0 && x[5] == 2) 
   || (x[1] == 1 && x[2] == 3 && x[3] == 0 && x[4] == 3 && x[5] == 1) 
   || (x[1] == 2 && x[2] == 0 && x[3] == 3 && x[4] == 2 && x[5] == 1) 
   || (x[1] == 2 && x[2] == 1 && x[3] == 0 && x[4] == 5 && x[5] == 0)

With this we can get back to the octlets as follows:

rules = List @* ToRules @ sol;

Function[
    rule,
    Flatten[
       Table[ #1, {#2} ]& @@@ Transpose[ { matC, matX /. rule } ]
    ]
] /@ rules

{{2, 2, 2, 2, 2, 4, 5, 5}, {1, 2, 2, 3, 3, 3, 5, 5}, {1, 2, 2, 2, 4, 4, 4, 5}, {1, 1, 3, 3, 3, 4, 4, 5}, {1, 1, 2, 4, 4, 4, 4, 4}}

This is not really very efficient but here goes. We can create a rational function that is effectively a generating function in three variables, one to force 8 factors, one to force a sum e1ual to 24, one to force a sum of squares equal to 86. The other parameters just keep track of what factors get used in a coefficient.

vals = Range[5];
aa = Array[a, 5];
ratfunc = Apply[Times, 1/(1 - t*aa*x^vals*y^(vals^2))]

(* Out[253]= 1/((1 - t x y a[1]) (1 - t x^2 y^4 a[2]) (1 - 
     t x^3 y^9 a[3]) (1 - t x^4 y^16 a[4]) (1 - t x^5 y^25 a[5])) *)

listOfLists = 
 Apply[List, 
  Apply[List, 
   SeriesCoefficient[ratfunc, {t, 0, 8}, {x, 0, 24}, {y, 0, 86}]], 1]

(* Out[263]= {{a[1]^2, a[2], a[4]^5}, {a[1]^2, a[3]^3, a[4]^2, 
  a[5]}, {a[1], a[2]^3, a[4]^3, a[5]}, {a[1], a[2]^2, a[3]^3, 
  a[5]^2}, {a[2]^5, a[4], a[5]^2}} *)

listOfLists /. 
 a[j_]^e_. :> Apply[Sequence, ConstantArray[j, e]]

(* Out[264]= {{1, 1, 2, 4, 4, 4, 4, 4}, {1, 1, 3, 3, 3, 4, 4, 5}, {1, 2, 
  2, 2, 4, 4, 4, 5}, {1, 2, 2, 3, 3, 3, 5, 5}, {2, 2, 2, 2, 2, 4, 5, 
  5}} *)