Whatsapp access to Google Drive
If you go to Google My Account Permissions and expand "WhatsApp Messenger":
View and manage its own configuration data in your Google Drive
View and manage Google Drive files and folders that you have opened or created with this app
I am not an expert on Google Apps permissions, but the wording would indicate that it is limited to files WhatsApp wrote to Google Drive itself: "opened or created with this app".