Wordpress - Is it mandatory to use $wpdb->prefix in custom tables

It is mandatory, though it is not enforced.

Consider the scenario when two Wordpress site has been setup in the same database. One with prefix wp_ and another with wp2_. If you install your plugin in both of the sites with the prefix, your created tables will be wp_liveshoutbox for first site and wp2_liveshoutbox for the second site. But if you omit the prefix, both of the site will be using the same table named liveshoutbox and the whole thing will break up.

Consider the following:

Your plugin is used on a wordpress network, which uses different table prefixes for each site. Your plugin could be running simultaneously on 836 different sites, all in the same database. wp_385677_liveshoutbox is a perfectly reasonable table name.

Your plugin is installed by a user who has some concept of security, and has changed the table prefix to block bots that try to inject select * from wp_users into the system. Even if they find a new vulnerability it won't work.

Taking shortcuts like hardcoding table names is a good way to get a product up and running, but not a good way to release it. in a very short time the plugin will have a pile of "doesn't work" comments on it, in the worst case you will break someone else's site.

If I have a complex query and I don't want to deal with the pain of writing 'select foo from ' . $wpdb->prefix . '_mytable left join ' . $wpdb->prefix . '_mytablemeta on ' . $wpdb->prefix . '.ID = ' . $wpdb->prefix . '.meta_id .... you can use replacers. For example:

$query = 'select foo from %table% left join %meta% on %table%.ID = %meta%.meta_id ... ';

$change = array (
    '%table%' => $wpdb->prefix . '_mytable',
    '%meta%'  => $wpdb->prefix . '_mytablemeta'
    );


$sql = str_replace( array_keys( $change ), array_values( $change ), $query );

$results = $wpdb->get_results( $sql );

Wordpress is constantly changing. What "works" today may not work tomorrow. That's why there are API functions. The Wordpress developers will make sure the public API behaviour is consistent (or they will depreciate the function). If you start using internal method calls because it's 'faster that way' it will usually come back to bite you. There are very few true shortcuts in software - they just move the required work from now to later, and like your credit card "later" usually costs more.