What is the difference between "public" games and "competitive" games? How are these "competitive" games located?

There are really two types of competitive in TF2.

The first is the more structured league play that has been done in leagues such as ESEA and UGC for 6+ years now. This involves a lot of coordination outside of the game itself to set up and, depending on the league, may have a lot of item restrictions in place that are enforced by TF2's built-in Tournament Mode settings.

The second is more ad hoc Competitive mode that Valve in the game itself. However, this game mode is currently in Beta, which is why you don't see it on the main menu. To gain access to this mode, you need to be given a TF2 Competitive Beta pass or be given one during a random giveaway to members in the Competitive steam group.

For Valve's ad hoc system, you basically join a game lobby (which looks a lot like MvM's lobbies) or join a team randomly. Competitive currently only supports 6v6 games, but may expand to the Highlander system later (9v9, 1 of each class).

Both the leagues and Valve's Competitive mode have ranks, but only Valve's system is done, more or less, with random people.

Beyond Powerlord's excellent answer, competitive play in TF2 tends to look a little different than the normal action on the public servers. Teams play and practice together for extended periods of time and rely heavily on voice chat to coordinate their pushes, as opposed to the random teams and random chatter on pubbies. Games usually follow the standard competitive format rules and 6-player teams will stick close to the standard competitive lineup; you don't just choose to play Pyro because it's fun or have three Heavies at once. Teams follow a basic competitive strategy focused on getting to the mid-point quickly through carefully rehearsed sequences of rocket and sticky jumps and the careful use of Ubercharges for strategic advantages.

The wiki links I included here, and the links therein, provide a further overview of the basics of competitive TF2 play.