Whatsapp access to Google Drive

If you go to Google My Account Permissions and expand "WhatsApp Messenger":

View and manage its own configuration data in your Google Drive

View and manage Google Drive files and folders that you have opened or created with this app

I am not an expert on Google Apps permissions, but the wording would indicate that it is limited to files WhatsApp wrote to Google Drive itself: "opened or created with this app".