Transactions don't rollback

The key issue in this case was bad default in datasources in some JBoss versions. Original code was fine and was working correctly in other application servers (WebSphere App Server and lightweight WebSphere Liberty).

Datasources created in JBoss are not JTA - in admin console the Use JTA setting is unchecked and in xml related setting is <datasource jta="false" .... Changing this setting to true fixed the problem. (JohnB, you wrote that defining xa-datasource fixed that, but since I didn't see your original xml with datasource definition, I believe that during changing datasource you've also change this flawed jta="false" setting). It will work for non xa-datasources also, as Grzesiek tested.

This is a very bad default, since it causes transactions not to be managed by container and causes flawed transaction behavior in connections got in EJB components.

Big thanks to Grzesiek D. who helped me in diagnosing this issue.

Please try this

@Override
public void updateTest() throws CustomException {

    Connection connection = dataSource.getConnection();
    try {
        connection.setAutoCommit(false);  // this should be the key

        dao.updateRecord(connection);
        dao.saveRecord(connection);

        connection.commit();

    } catch(Exception ex) {
        connection.rollback();
        throw new CustomException(ex, ex.getMessage());

    } finally {
        if(connection != null) {
            connection.close();
        }
    }
}

Update

My answer above has a mistake, because above code assumes that BMT (Bean-Managed Transactions) is used. But as we can see, you are using CMT (Container-Managed Transactions). Because @TransactionManagement is equivalent of @TransactionManagement(TransactionManagementType.CONTAINER)).

Above code snippet will only work with BMT. With CMT you should get error like below:

Caused by: java.sql.SQLException: You cannot set autocommit during a managed transaction!

But, my mistake turned to someething good in the end, because when you wrote

This works very well (...)

then we found an answer: you think that your EJB bean use CMT with JTA, but due to some error, it does not.

In comments below, I've also advised you to use JPA, but in this simple case JDBC is good enough. CMT transactions can be freely used also with JDBC.

Also type of the data source doesn't matter here. CMT can be freely used with a non-XA datasource (also called local datasource) and XA datasource as well.

Update 2

User @Gas solved the problem in the following comment. Kudos for him.

Basically: there were nothing wrong with the original code. Problem lies in the configuration of the datasource (has to be JTA enabled). So edit Datasource configuration via JBoss Administration console and set a checkbox "Use JTA".

I'm almost sure that your problem is caused because you are creating your DAO by hand with new keyword:

@PostConstruct
public void init() {
    dao = new XxxxDAOImpl();
}

When you are doing such things, your ejb container cannot manage that object lifecycle and transaction boundaries. You should let the container to create and manage the dao (and inject it for you). In that way, you will gain a proper transaction propagation across all your EJB methods - and thus your problem will be solved.

To achieve that, you can simply annotate your DAO class with @Stateless and inject it in your TransactionTestServiceImpl class with:

@EJB
private XxxxDAO dao;

And then, of course, remove init and destroy methods.

Personal advice

Why to use a separate, additional dao layer at all? In Java EE world the most convenient option is to just use an EntityManager. Entity Manager plays role of dao very well in most use cases. Although JDBC in your example is good enough, JPA is (IMHO) simpler.

Update

This was a bad guess, see my other answer on this page with updates.