What is the right way to query all Notes & Attachments?

Salesforce has two sorts of notes and attachments. There's the Classic type, represented by the Note and Attachment sObjects, and the Lightning type, represented by the Content suite of objects - ContentDocument, ContentVersion, ContentDocumentLink, and ContentNote. These two types are accessed in quite different ways.

The classic Note and Attachment records have a single polymorphic lookup to the record to which they're attached. You can query them simply:

List<Note> notes = [SELECT Id FROM Note WHERE ParentId = :myRecord];
List<Attachment> attachments = [SELECT Id FROM Attachment WHERE ParentId = :myRecord];

In the Lightning world of Content, the object model is quite a bit more complex. Both Notes and Attachments are represented as ContentDocument, a parent object over potentially multiple ContentVersion records. The ContentDocument is then linked to all of the locations in which it's been shared, which may be many, via ContentDocumentLink.

To make things even more confusing, there's a facade object, ContentNote, that acts like a ContentDocument with a single ContentVersion inherent in its Content field. You'll mostly have to worry about ContentNote if you're adding Notes, however.

Anyway - not to try to summarize the substantial complexity involved in the Content system - to query both Notes and Attachments in the Lightning style for some object whose Id is myRecord, you'd do

List<ContentDocumentLink> cdls = [SELECT ContentDocumentId FROM ContentDocumentLink WHERE LinkedEntityId = :myRecord];
Set<Id> documentIds = new Set<Id>();
for (ContentDocumentLink cdl : cdls) {
    documentIds.add(cdl.ContentDocumentId);
}
// ContentDocumentLink doesn't support semi-joins, which is frustrating.
List<ContentDocument> documents = [SELECT Id FROM ContentDocument WHERE Id IN :documentIds];

This will return both ContentNote and regular ContentDocument Attachments. If you want just Notes, query on ContentNote rather than ContentDocument. If you want just Attachments and not Notes, filter on FileType != 'SNOTE'.

To get the actual content, for ContentDocument, you can execute a ContentVersion query against ContentDocumentId and IsLatest = true. For ContentNote, you can simply ask for the Content field in your ContentNote query, which is a Blob field.

To delete records, you only need to delete the top-level ContentDocument record. That will take the ContentDocumentLink records with it. Deleting ContentDocumentLink records may make an Attachment inaccessible, but it won't be deleted entirely.