What is the name of the number that is being added to another number?

Either addend or summand is correct. If you are adding more than two numbers, it might be more appropriate to use the term addend or summand to refer to any term being added, e.g. $$\text{summand} + \text{summand} + \dotsb + \text{summand} = \text{sum (or total)} $$ or $$\text{addend} + \text{addend} + \dotsb + \text{addend} = \text{sum (or total)}. $$ If you need to refer to a specific term, the phrase "the $n$-th summand (or addend)" gets the job done.

In a case when you are adding exactly two numbers, you can use augend and addend, to refer to the first and second terms in a summation, i.e. you can write $$\text{augend} + \text{addend} = \text{sum}.$$ That being said, this use of the term "addend" is not unambiguous: for example, Mathworld defines "summand" as a synonym of addend, with augend referring to the first summand (or addend) in any sum. Hence it might be inadvisable to use "addend" to refer to only the second term in a sum, as the term "addend" is not unambiguously understood in this way; it could refer to any term in a sum. Moreover, the term "augend" is somewhat esoteric, and its use will likely draw requests for clarification.

The word you're looking for is addend.

The word summand would denote $2$ as well as $4$ in your example, and is not what you want.

Then, augend would denote the $2$ alone, and addend the $4$ alone.