Writing theory section of thesis - feels like I am just copying

Although I'm not in "physics/applied-math", but, rather, "math", I see the same or similar issue arise. Let's consider the point that, by this year, the "context/background/set-up" has very likely been highly optimized, both for effectiveness and for succinctness. Nothing wasted, nothing superfluous, etc. So it is likely to be hard to "improvise" much on its description without making it worse in some way. This may be the case even down to small notational conventions, wording, etc.

So, be absolutely scrupulous in citing, and try to have it all in your own head so that you can write it yourself (not literally copying), even while citing, ... and remarking something like "we recall some standard facts/ideas/set-up, e.g., from [X]."

Given that you've almost finished a PhD in your field, it's fair to say that you should have a solid grasp of the basic definitions and concepts you're trying to write up at the moment. So I suggest you do the following. Close all your textbooks and papers and write that section on your own, without looking at any other sources while you're writing it. That should ensure that you express the ideas mostly in your own words. Then, go back and check your version against the standard references to make sure you didn't get anything wrong and that you've included any relevant citations. If you see that one of the standard sources has explained one of the concepts much better than you did, consider explicitly quoting that source: e.g., "Symplectic widgets were defined by Widgetmeister [Wid83] as [...]".

At the end of the day, though, you don't need to worry too much about this. As long as you cite sources clearly, I don't think anyone's going to complain that your description of the standard material looks a lot like other peoples' descriptions of the same thing, especially if it's not word-for-word the same. Standard material always looks standard and a lot of readers will skip over it anyway. The real contribution of your thesis is in the later sections.

This is actually very simple ethically:

You did research work which - by your description - is experimental. You ran experiments, you're analyzing data, you're making conclusions based on the data.

Your research contribution is your conclusions. (and possibly your raw results and experimental setup as well.)

The theory section is not plagiarized because you're not claiming this is your original (theoretical) work. You're summing up what other people have done and discussed.

That's it. Other answers have good suggestions regarding how to approach writing this section.

PS - Sometimes, the cogent presentation of the theoretical background can itself be considered a contribution. But this happens more in textbooks or in review papers, less in theses.