How to get the display density in a lightning components controller?

You can get Display density information using Salesforce User Interface API. This Link will help you. I don't think you can directly access Salesforce UI-API from Lightning controller, but you can use Apex class to achieve this.

If you want your custom components to be responsive to the Display Density use the variable density slds classes.

For example variable classes for padding

slds-var-p-top_large
slds-var-p-bottom_large

Similarly for margin

slds-var-m-around_xxx-small
slds-var-m-around_small

Variable Density Classes Margin
Variable Density Classes Padding