Drupal - What are the downsides of using PHP Filter code in blocks, nodes, views-args, etc?

Some reasons:

• Code in your database can not be version controlled and is also harder to find in general later on.
• Eval()'d code is much slower than something hardcoded in a file.
• If there is an error somewhere in that code, you will get a very unhelpful error message (error in eval()'d code at line 3) and you might even have go through your database manually to find and fix the error. If it is inside a block that is displayed on all pages and results in a fatal error all the time for example.
• The above is also true when upgrading from Drupal 6 to 7 and whatever API's you used were changed. So you will have to port your code while migrating. If the code is in a module, you can port it in advance, test it, and only deploy it on the new site. Inside a node or block, it will only work with either Drupal 6 or 7.
• Writing and maintaing that code is also harder, because you are working in a textfield inside your browser. Having it in a module allows you to use an editor/IDE with syntax highlighting, autocomplete and so on.
• There is always the possibility of a misconfiguration that gives people access to a text format/block/whatever with php execution enabled. If php.module (in D7, D6 is not so strict, for example for block access rules) isn't even enabled, that risk is much lower already.
• If your CMS allows execution of PHP then an attacker who finds a security vulnerability of XSS or privilege escalation can now use your server for enormously malicious things (as part of a DDOS, sending spam, hosting malware, hacking into other sites/databases on the server, hacking into other servers on the network that might be behind firewalls). In addition to making small vulnerabilities more painful, this makes the site a more likely target of attack if its known that it can be used to execute php.

There might be more reasons, but that should be enough :)

This code is difficult to debug and maintain. I don't know any way to use version control for such kind of php code.

And it's really a potential security risk for people new to Drupal or PHP,

Considering the case of the PHP filter used in a node, the reason not to use it is that you limit the users who can edit that node, if you don't want to allow all the users to use the PHP filter.
Rather than using the PHP filter, it is better to use a custom module that replaces specific text in the node content with the result of the code it executes (without using eval()), or that appends its own text to the body content of the nodes. In this case, any user could edit the node, without to have the permission to add arbitrary PHP code that is then run by the PHP filter.

Generally, it is better to avoid eval() because it decreases the readability of the code, the ability for you to predict the code path (and possible security implications of that) before runtime, and hence the ability to debug code.

Apart in a development or a test site, I would not enable the PHP filter, or use PHP code that is passed to eval().

The PHP filter has been removed from Drupal 8. It is now a third-party module, not covered from the security advisory policy. This is probably a reason more for not using it in production servers (if the already given reasons didn't convince you).