Computational number theory

First, this question is extremely broad and thus I hesitate to start answering it, but anyway some remarks.

Number therory is a broad field and there are many different types of problems to which "computers" can contribute in one form or another.

Since you ask about rings of integers let me focus on this.

Regarding literature: One book I can recommend is Henri Cohen "A Course in Computational Algebraic Number Theory" and there is also a follow-up "Advanced Topics in Computational Number Theory".

In this book the author explains, among others, how to solve the basic tasks of Comptuational Algebraic Number Theory. So how to calculate with algebraic numbers, calculating rings of integers, discriminant, Galois group and so on. Also, certain methods of factorisation are discussed as well as question on arithemtic of polynomials.

The author is/was a main contributor to the development of Pari.
Pari is specialized for number theory; opposed to the other programms you mention.

Roughly, the functionality of Matlab is not geared towards number theory. Mathematica and Maple offer more here, certainly useful for some things and for some even very good as far as I know, but not specialized for number theory. An important (non-free) other program is Magma, which is I think considered as leading for certain number theory (related) tasks.

And, last but certainly not least, there is a large free open-source project Sage http://www.sagemath.org that has a certain focus on number theory (the founder William Stein is a number theorist). It inculdes (more or less) Pari and much other free open-source math software; some directly or indirectly relevant for number theory.

If you search for a possibility to do computational number theory and to potentially do something of lasting value, I would recommend that you look into Sage. Its web page offers a lot of documentation but also (number theory) papers written with the help of Sage. Yet also (number theory) lecture notes and text books with a computational slant. The developpment process seems very open and there are plenty of tasks to be done (from small to large, from beginner friendly to research level). [Note: I did not contribute anything to Sage, I only followed its developpment from a distance but somewhat in detail.]

You might be interested in Project Euler (http://projecteuler.net/), which guides you through solving progressively more difficult math problems using computer programming. This is not exactly what you asked about, but is rather a complement to quid's excellent answer.

There's a different sort of number theory, combinatorial, that benefits tremendously from computation and experimentation. An excellent website to get you started is http://www.experimentalmath.info/. Borwein and Bailey (responsible for that website) are two of the champions of using computers to find new results, and Zeilberger is the godfather of using computers to prove combinatorial results.

Surprisingly, of the various branches of number theory (analytic, algebraic, combinatorial, additive), it seems to me that analytic number theory has benefited the least from experimentation.