Problems from using Office 64bit
As you can see in this MS article Compatibility Between the x86 and x64 Versions of Office, there are not much issues between the normal operation of the x86 and x64 version for the normal user; there are though, issues when using VBA in office files but the article lists workaround for that, and that x86 addons (which are the most out there) would not work.
The benefits are that office x64 can use more memory for working with big files, and also the size limit of the files is not restricted to 2GB.
The files created with x64 version of Office are completely compatible with the x86 versions as long as their size is below 2GB and have compatibility for the new filetype (Office 2007 and up, docx, xlsx, etc.)
Some macros must be written differently, but it is usually the case of a single corrected declaration rather than having to rewrite the entire thing, and plugins for 32bit versions of Office cannot run in the 64bit version of Office.
Also, if you have any of the various other "Office" products such as Skype for Business or Microsoft Office Document Imaging or Project in the 32bit versions, you may encounter issues during installation. Typically, uninstalling the 32bit applications, installing the 64bit Office, and then reinstalling the 32bit applications again will work just fine.
Based on what you've described, I believe it is very unlikely you'll encounter any of these issues if you were to remove the 32bit Office and install the 64bit Office.
There are several possible issues
ODBC | going to 64 bit uses a different odbc connector. While this can be ok with you, it may cause problems for users who only have a 32 bit odbc system
Max Size | If you have a lot of memory, great for you. However, you can easily make workbooks that are so huge it will swamp the users with 32 bit systems without you having any issues.
Additional items dealing with older files and some add-ins. Because you do not have the old plug-ins does not mean the 32 bit users do not have them.