What is the aim of modified GR theories?

First, there is no reason why all possible higher-order terms of the Einstein-Hilbert action could not in principle be there. At low energies, the effects of these higher-order terms would be less relevant, and to describe all of our astrophysical observations (on which General Relativity is based) we might suffice using only a the terms of lowest order in the Lagrangian, giving a low-energy effective description equivalent to normal GR.

When we go to higher energies, (a quantum theory of) gravity as described by the Einstein-Hilbert action, is non-renormalizable and would need all these possible higher order terms as counterterms, indicating again that all these higher order terms might be there in a quantum theory of gravity. This is part of the motivation for studying $f(R)$ theories of gravity. String theory also gives rise to higher order terms in the Einstein-Hilbert action, so most people in that field believe those terms could be there (for the above-mentioned reasons).

In addition, one of the more well-known inflation models (using a scalar field with a slow roll potential) known as the Starobinsky model, can be obtained from an extended theory of gravity that includes only the next leading cubic term in $R$. $$ S = \frac{1}{2} \int d^4 x \left(R + \frac{R^2}{6M^2} \right) $$ This adds to the suspicion that a proper quantum theory of gravity, describing the full quantum mechanical dynamics of spacetime, including cosmic inflation in the early universe, might indeed be one that contains (many) of these higher-order terms.

Using $f(R)$ theories in an attempt to explain dark matter is typically more difficult since $f(R)$ will at low curvature reduce to GR (which will again reduce to Newton's law of gravitation). Theories that attempt to explain dark matter by modified theories of gravity should give a different Newton's law of gravitation at low energies and are therefore known as MOND (Modified Newtonian Dynamics) theories.

However, I am unable to understand the appeal of these [modified gravity] theories. Are they studied purely for the sake of studying them, or are there underlying reasons?

People study such theories for many different reasons. Here is a list with some motivations, mainly from cosmology, for why this has been a popular research field over the last few decades:

• The accelerated expansion of the Universe. This is often called the biggest puzzle of modern cosmology. The observations so far is well described by adding a simple cosmological constant to General Relativity, but it's plagued by some theoretical problems like it's extremely fine-tuned value. This has motivated people to look at alternative explanations like modified gravity (among many other things). The accelerated expansion is a cosmological effect and GR is only really well tested on small scales so it's natural to ask if this effect could be due to modifications of gravity (though so far most of the models that have been proposed introduce more problems than they solve).

For a review see: Clifton, Ferreira, Padilla and Skordis "Modified Gravity and Cosmology"

• Testing gravity in new regimes. Gravity is very well testing the Solar-System, but the length-scales associated here are tiny compared to the size of the observable Universe. With current and upcoming large cosmological surveys we are finally in a position to be able to produce high precision (percent level) constraints on how gravity operates on the largest scales in the Universe. This will either strengthen the evidence for General Relativity or show evidence for new physics. Whenever we test a model it's always very useful to have alternative models available. This gives us something to test it against and alternative models might show new interesting effects which can lead us to completely new ways of testing our standard model.

For a review see: Will "The Confrontation between General Relativity and Experiment" ; Koyama "Cosmological Tests of Modified Gravity" ; Jain "Novel Probes of Gravity and Dark Energy"

• The discovery of screening mechanisms This is related to the point above. One of the first scalar-tensor theories (see Brans "The roots of scalar-tensor theory: an approximate history") discovered, Jordan-Brans-Dicke, is not so interesting anymore as it's constrained to be so close to GR that it's modifications are tiny in almost all situations (plus the unnaturally large value of the JBD paramter). One can however construct more elaborate scalar-tensor theories that have the property that they can hide their modifications in regions of (relative) high density thereby by evading the tight Solar-System constraints on gravity while at the same time give rise to large deviation from GR on cosmological scales. This opened the door for many different types of interesting signatures one could go and look for in observations (and also in laboratory experiments on Earth, see e.g. Burrage and Copeland "Using Atom Interferometry to Detect Dark Energy", "Numerical forecasts for lab experiments constraining modified gravity").

For a review see: Joyce, Jain, Khoury and Trodden "Beyond the Cosmological Standard Model" ; See 1312.2006 for a low-level introduction.

• Dark matter. This was perhaps the main motivation that started the modern interest in modified gravity. It was discovered in the 1980s that one could describe rotation curves of galaxies with a very simple modification of Newton's law of gravitation instead of particle dark matter. These proposals are not so popular anymore after it was found that one generally needs to add some real dark matter to explain all the observations, but there are still people working on this.

Some papers: Milgrom "A modification of the Newtonian dynamics as a possible alternative to the hidden mass hypothesis" ; Bekenstein "Relativistic gravitation theory for the MOND paradigm" ; "Mimetic dark matter"

• Theoretical motivations. Apart from the obvious ones: the cosmological constant problem and quantum gravity, there are other interesting theoretical problems where modifications of gravity could be relevant. For example massive gravity, GR with a mass-term for the graviton, was though to be plagued by instabilities for decades. About a decade ago the original theoretical obstacles that was in the way of deriving a consistent theory of massive gravity was overcome (but with new challenges appearing). This gave new hope for a consistent theory of a massive graviton. One could also add to this the discovery of theories with large extra dimensions (brane-world models) which was quite popular in the mid 2000s and lead to much research (mainly as they had mechanisms to give self-acceleration of the Universe, but also for it's purely theoretical properties).

For a review see: de Rham "Massive Gravity" ; Dvali, Gabadadze and Porrati "4D Gravity on a Brane in 5D Minkowski Space"