How to apply uniqueness on Junction object?

Create a text field (External Id, Unique) on the junction object.

Create a workflow on C to fire Everytime the record is created or edited (to account for cases where junction object records are reparented) to set this field as the concatenation of the Ids of A and B records that the C record is relating.

This workflow trying to set the composite primary key field will generate an error if a duplicate record is being inserted.

You can add to the composite key to include any other uniqueness criteria such as relationship type if you have that notion.

The workflow will fire only for new records created, you will need to retrospectively populate for already existing records possibly by extracting and concatenation via data loader, or simply do a phantom update on the junction records and that should cause the workflow to fire and set the key. However this could also throw some errors for any existing duplicates which you may need to resolve manually.

I thought that junction objects were specifically for Many-to-Many relationships and as such the behaviour you describe is what I would expect.

You could enforce the uniqueness with a Trigger on Object C or as techtrekker says with workflow for a no-code solution.