Drupal - How do I delete files the correct way?

However if I remove it on the basic page 'Edit' page, it will also still keep a record in the database.

Correct, however after editing, the status of the file should be set to 0. (Unless the same file is in use elsewhere.) Drupal has a built in system that should automatically clean up these stale files both from the filesystem and the database, which you can configure on example.com/admin/config/media/file-system. The default time is set to 6 hours (depending on when cron runs this could take up to the next cron run).

Be aware though that there are known issues with this automated deletion. See the File usage tracking causes unexpected file deletion section in the release notes.