What are these red triangle-shaped things?

Those are Rubies. In addition to appearing in stone, gems can be found attached to the sides of other blocks, particularly in a Gemstone Cave. You can harvest them with your pickaxe directly, or by breaking all the blocks next to them.