Is it bad manners to mark your territory?
This is a great question, but I think it's slightly misformulated. What you are thinking of doing is neither "bad manners", nor can it really be considered "marking your territory" in all but an extremely weak (so weak as to be essentially meaningless in my opinion) sense. The correct question is the one you ask in the body of the question: is it a good idea? And specifically, is it a good way to build a solid reputation for yourself and to maximize the impact of your work?
The principle at work here is that in general it's a bad idea to brag about achievements you haven't yet attained. You might think that by doing so you are "marking your territory", but that's simply not the case: in reality it is fair game for anyone to publish a proof to a claim that you say you plan to prove in a future paper (and if they publish a proof using a different technique than the one you use, as you suggested, then that is even more legitimate and there is even less concern that they will be deterred by your "territory-marking", to the extent that such a concern exists at all). On the other hand, by trying to lay claim to results of which you do not currently have a written proof you undermine your own credibility: no one knows whether you will follow through on your promises, and everyone has seen cases of people making such claims and never following through, and would therefore be quite likely to be very skeptical of your claims (both the present claims and any claims by you they will encounter in future papers). So, by and large, bragging about as-of-yet-unattained achievements is not a winning strategy, and I'd advise against it in all but a handful of circumstances.
So when is it appropriate to advertise planned future follow-up work at the end of a math paper? I can think of two situations where it may not be a bad idea:
You have a 90%-complete manuscript of the future work and have a very high degree of confidence that you can complete it in a very short period of time, and you believe that advertising the future work helps make your current paper seem more attractive. (But be very careful: predictions of this sort tend to be wildly optimistic. I am speaking from personal experience, which I'm guessing is shared by almost everyone who has been publishing in pure math for more than a few years...)
Your intent is not to advertise the future work in a selfish, territory-marking sort of way, but rather to advertise directions for follow-up work that you believe are of interest to the community in an altruistic sort of way that makes it clear that anyone is free to pursue those directions. It's still perfectly fine to imply that you plan to do so yourself, just don't expect that this gives you any special rights that prevent others from beating you to the punch. For example, you can write something like:
To conclude, an interesting open problem is whether a non-Riemannian hypersquare exists. We believe the techniques of the current paper can be used to show that the answer is positive, and plan to address this in future work.
A "moral high ground" answer:
- It's not "your territory", it's everyone's. That is, scientific discoveries should not be seen as owned in any way by individuals or groups, but as common treasures of human civilization.
- It's good to let your fellow researchers know what you're working on, because:
- This prevents redundant parallel work (although of course sometimes parallel work on the same problem with different perspectives can be a good thing; it's just that you want to know whether that's the case or not).
- It increases the chances of collaboration with other researchers interested in the same problem.
- Papers should not reasonably be limited from the addition of a few short paragraphs indicating future research directions; and even if such a restriction is in place for some reason, such intentions can be published on a research group / individual researcher website or blog.
Thus if we ignore things like squabbles over credit, I believe the answer is pretty clear.
PS - I would always tone down what's expected, to avoid the deterring effect you mentioned and not to have to retract what I claimed. So "We're working on the problem" is preferable to "We believe we've obtained a solution" as long as it's not 100% sure-fire in-the-bag.
I think it is both expected and acceptable to let people know in a paper both what the conjectures and open questions are, and to let people know whether you are working on them, whether you hope to solve them and whether you intend to publish something on them.
However there is a subset of behavior which I think is bad manners when I've encountered it:
- Author(s) X publishes paper A.
- They cite by name currently unpublished paper B, by the same authors.
- In a particularly annoying case, some of the results in paper A depend on other results in paper B. Sometimes it's not even clear what the statement of these results is, just that if you knew what it was, you would believe the claims made in A.
- Researcher Y reads paper A.
- They unsuccessfully search for paper B which they expect has now been published.
- They contact X and request a preprint.
- X informs them that paper B is not quite ready to send out preprints since they are in the process of adding some even more exciting results.
- X promises to send out paper B when it is ready.
- <crickets>
- Y doesn't know whether to believe in the results in B, whether to work on their own proof of these results, whether the results in A are still correct, whether or how to cite the 'ghost paper' B in their own work.