A question about Schwartz-type functions used in analytic number theory

The answer to your question is yes, and it is a pretty well-understood topic.

First of all $X$ is more or less irrelevant for the bounds in 4) so let us take $X = Y$, say, for convenience.

Second, we can always scale everything down by $Y$. So without loss of generality $Y = 1$.

Put $g = f'$. Let us also for simplicity assume that $f$ is symmetric so it is enough to study $g$ on $[1, 2]$. The problem therefore is reduced to the following: for which sequences $t_j$ we can find a function $g$ such that $g = 0, x\notin [1, 2]$, $\int g = 1$ and $|g^{j}(x)|\le C t_j$ for some constant $C$.

The answer to this question in more or less full generality is given by the Denjoy-Carleman Theorem: if the sequence $M_j = \frac{t_j}{j!}$ is logarithmically convex (i.e. $\frac{M_{j+1}}{M_j}$ is increasing in $j$) then such a function exists if and only if $\sum_j \frac{1}{jM_j^{1/j}} < \infty$. For example there exists a function $f$ such that \begin{equation}\label{bound} |f^{(j)}(x)| \le CY^{-j}j^{(1+\varepsilon)j} \end{equation} for any fixed $\varepsilon > 0$ (this is related to the so-called Gevrey classes).

Actually, since you mentioned Fourier transform, let me write about a different result which is more directly applicable to this type of problems: Beurling-Malliavin multiplier theorem. It reads as follows:

Let $w:\mathbb{R}\to \mathbb{R}$ be a nonnegative Lipshitz function (this is a small technical condition). Then there exists a nonzero compactly supported function $f$ with $|\hat{f}(\xi)| \le e^{-w(x)}$ if and only if integral

$$\int_\mathbb{R} \frac{w(x)}{x^2 + 1}dx$$

is convergent. Moreover, support of the function can be arbitrary small.

Lastly, if you want an explicit function $g$ (and therefore $f$), satisfying the above bound, you can take

$$g(x) = e^{-(1-x)^{-m}}e^{-(x+1)^{-m}}\chi_{(-1, 1)},$$

see e.g. this paper, section 3.1.

Not sure what $X$ and $Y$ are. Positive constants? Also, the meaning of the symbol $\ll_j$ should be explained. I may have misunderstood condition 4) but it might be too strong and imply analyticity of $f$ which contradicts 2) and 3). It seems what the OP needs is some explicit bumpfunctionology. Start with $f(x)=0$ for $x\le 0$ and $f(x)=\exp\left(-\frac{1}{x}\right)$ for $x>0$. Then take shifts, reflections, and products. Integrate and also shift, reflect, and take products of the antiderivative. One typically obtains in this way functions that are Gevrey with explicit bounds on derivatives as a function of $j$. These bounds can be proved via Cauchy's Theorem with complex analysis or combinatorially (my favorite approach). For a fully worked out example of how to do the bounds combinatorially, see my answer to this MO question:

Gevrey estimate of derivatives