Abstract definition of convex set

There has been a bunch of work along these lines, and I think the idea has been rediscovered several times. I suggest looking at the papers of Anna Romanowska, who refers to them as "barycentric algebras", to get an idea of what's known. Her book with Smith, "Modes", covers this as well as generalizations where $t$ is not required to be in $[0,1]$. Here are some slides that cover the basics.

It's known that not all of them are representable as vector spaces. For example, if you mush everything in the interior of $[0,1]$ to a single point, your operations are still well-defined, but you can't embed it in a vector space. The Modes book has a structure theorem.

A good (but probably not up to date) reference on several abstractions of the notion of convexity is

Singer, Ivan. Abstract convex analysis. Vol. 20. John Wiley & Sons, 1997.

There you'll find notions like "order convexity" in posets, "metric convexity" for metric spaces, abstract convex combinations, but also notions based on approximation from the outside or based on abstraction of the "convex hull" operation. I am not sure if something along th lines of your ideas is in there, but it is a good read anyway.