Splitting a relatively long paper into two shorter papers

Well, in my opinion, the following fragment of your question outweighs everything else:

(2) My paper consists of two parts and each part addresses a different subject. Therefore, the set of reviewers who have expertise in both subjects and are willing to read and review my paper is very small.

I appreciate that you are able to imagine the problem that will be faced by the journal editors. So, splitting into two is certainly not a bad idea in this case.

I can mention some instances where this has been done in the past. In each of these cases, the problem wasn't as acute as yours (i.e. both could actually be reviewed by the same expert), but perhaps the authors chose to do this because of length considerations (or some other reason that I can't imagine). Also, in these cases, it wasn't a case of two different journals A and B - it was two sequential papers in the same journal. So, that creates an additional option for you, if you find it appropriate. (The context here is Physics, but I'm sure this can be generalized to Maths, if I'm right!)

Example 1: R. P. Feynman was a charismatic Nobel Laureate (Nobel, 1965), as you probably know. Here are his two significant contributions to QED, appearing back to back in Physical Review:

Paper1, Paper2 (both are free pdfs officially, given their landmark status.)

In particular, he began Paper 1 by writing:

This is the first of a set of papers dealing with the solution of problems in quantum electrodynamics.

and started Paper 2 with the sentence:

This paper should be considered as a direct continuation of the preceding one ...

He had developed the formalism in the former and applied it to the problem in the latter. That makes a candidate for splitting into two.

Example 2 Here are two papers by Sidney Coleman which form the backbone of phenomenological effective Lagrangian method in low-energy Nuclear Physics. (These aren't free and I'm not sure you will be able to get past the paywall here!)

Paper 1, paper 2.

Notice again, that they are consecutive papers in the same journal. The second paper also has an extra author, and that could be one reason for splitting into two. But once again, from the point of view of content, the general method was devised in paper 1 and applied to some context in paper 2. But here, the authors spent a section of paper 2 in explaining what they developed in Paper 1.

Thus, long story cut short - it should be possible to go ahead and split into two parts. If they are consecutive, you can carry over everything directly, if not, spend a few sentences explaining your notation etc.

PS - Congratulations for doing this sort of work which could put the editors into this type of a fix. That smells like a significant contribution to Maths, having applications elsewhere (other branches?), which is probably why you insist that it would be rare to find a referee who can ably judge both!

This problem sounds either:

• bad editorial choice and poorly structure paper or
• two papers that were not split in time

Generally a research program is not just growing like a blob, but follow some strategy, planing. If you split the two parts, you can briefly mention this strategy (even if it is retrospective) and the results of the other paper. It can be in the introduction and in discussion as well. Most journals allow to cite manuscripts under revision. If not, you can try to describe without citation that you had this and this motivation, and the results will be published soon.

If you don't want to split the paper, you can restructure it as one part to subordinate to the other. Eg. if you you have some important technical detail in the first part you use in the second, but boring in itself, you put the details in an appendix or SI. It is pretty common practice, especially in older paper, and it helps the reader orient themselves around.

Two notes:

• Again, I would consult with the editor at first place. They are busy, but generally nice people, so he may comment more in detail about your worries about size and structure. You may have completely different reasons of rejection.
• You always can try to send out another journal (not necessarily more specialist) without much re-editing. Different journals have different expectations for length, structure, scope.