Mathematical foundations of Quantum Field Theory
Yes, of course, there is much research on mathematical rigor in quantum field theory. Of course, I don't know what "reasonable", "essentially different", and "realistic" mean to you, but I would say that there are "reasonable" approaches and that some of them do address "realistic" field theories. As an aside, "rigor" for its own sake is far from the primary goal of mathematical physics, as has been discussed many times here and on our sister sites. See for example https://physics.stackexchange.com/questions/27665/the-role-of-rigor.
But, in any case, you mention QED, and more generally "realistic" quantum field theory almost certainly means Yang–Mills Theory + matter, as this is what appears in the Standard Model. Here there are deep open questions, like those related to the mass gap. But some parts are by now understood. One part in particular is the perturbative path-integral approach to Lagrangian field theory, where the deepest part of the story is that of renormalization theory. Kevin Costello's book does a good job, I think, of explaining to mathematicians what renormalization theory is, setting it within a language of homological algebra.
But you bring up Wightman's axioms and AQFT, suggesting that you are less interested in making rigorous physics-as-it-is-practiced, and more interested in axiomatizing its general structure. There is, of course, no consensus yet as to the correct axiomatization — almost all proposals have no nontrivial examples — but many structures seem to be common. There is a more flexible version of AQFT, called factorization algebra and detailed in Costello's book with Owen Gwilliam that I think goes a long way towards providing a basic framework. Certainly all physical theories will have more common structure than just being a factorization algebra; but it is very common that we write down an axiom system and then the examples that appear "in nature" are quite special.
I have yet to be convinced that the "Schrodinger" picture is a fundamentally correct one. This is the picture that underscores, for example, geometric quantization, and also the Atiyah–Kontsevich–Segal–etc picture of QFT (originally TQFT, but by now more general). Certainly this version of QFT is an interesting mathematical structure to study, but it arises from the "Heisenberg" picture underpinning factorization algebras only because sometimes certain algebraic objects have unique, or at least canonical, irreducible projective representations.
Finally, I want to make one more comment concerning the success of perturbative and some nonperturbative QFT. Namely, almost every QFT that has been written down has been described in "path integral" formalism in some sense or another. Certainly this is true of the perturbative field theories in Kevin's book. But probably "having a path-integral description" is not fundamental to the notion of QFT. In a related way, "having a classical limit" is not fundamental. I highly recommend Dijkgraaf's Les Houches notes; this point is made in Section 2.2.
Edit: The OP has clarified the question to ask about how nonlinear quantum field theory can lead to the linear algebra of Hilbert Spaces. I think this is part of a standard confusion with the picture as developed of quantization, and what are "quantum spaces". The short answer is that a decent notion of "quantum space" is a Hilbert space with some extra structure (e.g. a "spectral triple" of Connes et al). For the long answer, I will restrict to (nonlinear) quantum mechanics, which is the nonlinear quantum sigma model in one dimension.
The most basic form of quantum mechanics, after Feynman and well-tuned to be portable to qft, comes from the following picture. You are given a classical configuration space, which is an abstract manifold with some geometry (a metric to define "mass", a 1-form to define "external magnetic potential", a function to define "external electric potential", etc.). This geometry in particular determines for you an "action functional", which is a function on the paths in this manifold. Now you define a "quantum algebra of observables" as follows. There is a bijection between quantum observables and classical observables, where "classical observables" are functions on the tangent bundle to your space, aka the phase space – so observables are functions of position and velocity. But the quantum algebra has a much richer algebraic structure than the commutative algebra of functions. Namely, given two functions $f_1$ and $f_2$, their product depends on three numbers $(t_1, t_2;t_3)$, and is: $$ (f_1\star_{(t_1,t_2;t_3)} f_2)(x,v) = \int_{\text{paths }\gamma\text{ s.t. } (\gamma,\dot\gamma)(t_3) = (x,v)} f(\gamma(t_1),\dot\gamma(t_1)) \ f(\gamma(t_2),\dot\gamma(t_2))\ \exp(\text{action}(\gamma)) $$ The associativity of this algebra is somewhat subtle, and depends on the times. You should really think of this as "$f_1$ inserted at time $t_1$ multiplied by $f_2$ inserted at time $t_2$, measured at time $t_3$". There is a straightforward way to evolve a function inserted at some time $t_1$ to a different function inserted at any other time $t_3$: namely multiply $1$ at an arbitrary time $t_2$.
Now you can do the following. Since we can evolve functions, $t_3$ isn't much data – let's just decide $t_3 = (t_1 + t_2)/2$, say. Now let's just consider those situations when $t_2 = t_1 + \epsilon$ for very small epsilon. Dividing by $\epsilon$ and taking a limit, you get a usual associative noncommutative algebra.
There is a general notion in mathematics that a "space" is the same data as its (commutative) algebra of functions. Similarly, you can define a "noncommutative space" to be the same data as a noncommutative algebra of functions. Since we're doing integrals, the functions we're working with are the types of functions that appear in measure theory and functional analysis, rather than in geometry. Just knowing the algebra of measurable functions on a manifold only tells that manifold as a measure space, and all measure spaces are isomorphic, so you should also remember some data of the smooth structure and metric and so on. Similarly, the algebra you most naively get out of this construction doesn't know a lot; the extra data is that of a "spectral triple".
More precisely, algebras like to have representations, and the algebras of functional analysis like to be represented on Hilbert spaces. For QM (but not for higher-dimensional qft), this Hilbert space is essentially unique (similarly to the way that "the" measure space is essentially unique). The extra structure is what makes it "curved". In the case at hand, this essentially-unique Hilbert space arises in many ways: for example, the action picks out a symplectic form on the tangent bundle to your configuration space, identifying it with the cotangent bundle, and you can choose a way to identify functions on the cotangent bundle with differential operators, and then that algebra of differential operators can be naturally identified with the algebra we have constructed; in this way, the algebra acts on "wave functions" on your configuration space.
Is this a linear space? Not really. I've already mentioned one way that thinking of it as linear is wrong (it forgets the geometry). Another is that the representation is really projective, so the actual space of states is more like the space of lines-through-the-origin in the Hilbert space than the space of points. Really, this linear/nonlinear dichotomy is about like saying of your manifold "Manifold, I would rather you be linear, so I'm going to allow some linear combinations of your points".
I hope that helps clarify some things, and in particular the Schrodinger/Heisenberg dichotomy I alluded to earlier.
If I read your updated question correctly, you are asking whether people have considered non-linear modifications of quantum mechanics in order to accommodate interacting QFTs. I'm sure someone, somewhere has, but that's certainly not mainstream thought in QFT research, either on the mathematics or theoretical physics sides. Consider the analogous question in the quantum mechanics of particles: do non-linear equations of motion require a non-linear modification of quantum mechanics? The answer is most certainly No.
Without going into generalities, the Hydrogen atom and the double-well potential are prominent examples of systems with non-linear (Heisenberg) equations of motion that live perfectly well within the standard quantum formalism (states form a linear Hilbert space, observables are linear operators on states, time evolution is unitary on states in the Schroedinger picture and conjugation by unitary operators in the Heisenberg picture). When going from particle mechanics to field theory, what changes is the number of space-time dimensions, not the type of non-linearities in the equations of motion. So there is no mathematical reason to expect a non-linear modification of quantum mechanics in the transition.
Now, a few words about your intuition regarding states as solutions to the equations of motion. Unfortunately, it is somewhat off the mark. As you should be aware, relativistic QFT is usually discussed in the Heisenberg picture. This means that it is the field operators $\hat{\phi}(t,x)$ that obey the possibly non-linear equations of motion. For example, $\square\hat{\phi}(t,x) - \lambda{:}\hat{\phi}^3(t,x){:}=0$, where $\square$ is the wave operator and the colons denote normal ordering. On the other hand, states are just elements $|\Psi\rangle$ of an abstract Hilbert space (with the vacuum state $|0\rangle$ singled out by Poincaré invariance), entirely independent of spacetime coordinates. At this point, it should be clear why states have nothing to do with the equations of motion.
Your intuition is not entirely without basis, though. Spelling it out, also shows how the standard formalism of QFT (Wightman or any related one) already accommodates non-linear interactions. One can define the following hierarchy of $n$-point functions (sometimes called Wightman functions): \begin{align} W^0_\Psi &= \langle 0|\Psi\rangle \\ W^1_\Psi(t_1,x_1) &= \langle 0|\hat{\phi}(t_1,x_1)|\Psi\rangle \\ W^2_\Psi(t_1,x_1;t_2,x_2) &= \langle 0|\hat{\phi}(t_1,x_1)\hat{\phi}(t_2,x_2)|\Psi\rangle \\ & \cdots \end{align} It is a fundamental result in QFT (known under different names, such as the Wightman reconstruction theorem, multiparticle representation of states, or simply second quantization) that knowledge of all the $W^n_\Psi$ is completely equivalent to the knowledge of $|\Psi\rangle$.
These Wightman functions, by virtue of the Heisenberg equations of motion, satisfy the following infinite dimensional hierarchical system of equations \begin{align} \square_{t,x} W^1_\Psi(t,x) &= \lambda W^3_\Psi(t,x;t,x;t,x) + \text{(n-ord)} \\ \square_{t,x} W^2_\Psi(t,x;t_1,x_1) &= \lambda W^4_\Psi(t,x;t,x;t,x;t_1,x_1) + \text{(n-ord)} \\ \square_{t,x} W^2_\Psi(t_1,x_1;t,x) &= \lambda W^4_\Psi(t_1,x_1;t,x;t,x;t,x) + \text{(n-ord)} \\ & \cdots \end{align} I'm being a bit sloppy with coincidence limits here. The Wightman functions are singular if any two spacetime points in their arguments coincide, the terms labeled (n-ord) represent the necessary regulating subtractions to make this limit finite. This necessary regularization also explains why the non-linear terms in the equations of motion needed normal ordering.
If $\lambda=0$, the theory is non-interacting, then each of the above equations for the $W^n_\Psi$ becomes self-contained (independent of $n$-point functions of different order) and identical to the now linear equations of motion. At this point it should be clear how your intuition does in fact apply to the states of a non-interacting QFT. States $|\Psi\rangle$ can be put into correspondence with multiparticle "wave functions" solving the linear equations of motion (which are actually the Wightman functions $W^n_\Psi$).
Finally, when it comes to trying to construct models of QFT, people usually just concentrate on the Wightman functions associated to the vacuum state, $W^n_0 = \langle 0|\cdots|0\rangle$, which are sufficient to reconstruct the corresponding $n$-point functions for all other states. In short, the standard approaches to constructive QFT already incorporate non-linear interactions in a natural way. And non-linear modifications to the quantum mechanical formalism are simply a whole different, independent topic.
Essentially nothing like the what you describe in the more detailed form of your question exists for the important reason that your "intuitive conception" of a hypothetical non-linear state space is incorrect.
The linear structure in a quantum theory HAS NOTHING TO DO with whether or not the equations of motion are linear. It is an exact feature that persists independent of any interactions. Indeed the Hilbert space arises from the fact that a configuration of a quantum system is given as a complex valued linear functional from the vector space $V$ which has a basis in bijective correspondence with the possible classical states of the system. (Typically these functionals are required to be square normalizable in a suitable sense but this issue is secondary to the present discussion.) Linear functionals of course always have the structure of a vector space and as a result so too does the state space of a quantum mechanical theory including in particular quantum field theory.
As a specific example consider quantum mechanics of a single particle moving on a line $\mathbb{R}$. The space of classical configurations is of course the real line, meaning exactly that to specify the all classical information means to specify a function $x(t)$ which tells you at which point $x$ the particle can be found at time $t$. Quantum mechanically this is modified as follows. We introduce a vector space $V$ with a basis of states in bijective correspondence with the set of all allowed classical configurations. Thus in this example $V$ has a basis for each point in $\mathbb{R}$. The Hilbert space of the theory is then the dual space (over $\mathbb{C}$) to $V$. This is true no matter what non-linear interactions may be occurring, and a wide variety of exactly solvable non-linear models exist. In this example the dual space is spanned by Dirac delta functions, typically denoted by $|x\rangle$ which indicates the functional that takes a non-zero value on the basis element corresponding to point $x$ and zero at all other basis elements associated to points $y\neq x$. As a result the Hilbert space can be described as (suitably normalizable) distributions from $\mathbb{R}$ to $\mathbb{C}$. The interpretation of an element of this Hilbert space $\Psi(x): \mathbb{R}\longrightarrow \mathbb{C}$ is that the square of its norm specifies the probability distribution for the quantum mechanical particle to be observed at the position $x$. The time evolution of the wave function $\Psi(x)$ is then always governed by the linear Schrodinger equation.
In a quantum field theory the abstract structure is the same, however $V$ is typically a much larger vector space since it has a basis in correspondence with fields, i.e. functions from space to say $\mathbb{R}$.