How to monitor EC2 instances by memory?

Memory and Disk specific statistics require AWS to monitor at the OS level rather than the host level, so that is why they leave it out by default. It will probably be added at some point but since it has been on the wish list for about 7 years, we can assume it's a very low priority item.

The recommended way to monitor memory usage is to create a custom Cloudwatch metric by using your own monitoring scripts on your instances. AWS have published documentation on how to achieve this on Linux instances with a set of (unsupported) scripts.

Once your instances are publishing the custom metrics, you will be able to attach alarms to them in CloudWatch.

Well, now the new CloudWatch agent can collect metrics like memory and disk usage, see the docs.

With this you can monitor this kind of metrics, but you will need to install and configure the agent in the instances.