Simple function returning 'undefined' value

Other people have given good, correct answers but I want to be explicit about why, since it might not be obvious to some people (not directed at the OP).

A function is nothing more than a set of steps for the computer to take.

This is known as a function call:

getSmallestDivisor(121)

Anytime the return keyword is used, the function stops and replaces the function call with whatever comes after that return word (it could be nothing).

So in this case, the problem with the original function is that when the script reaches this line...

getSmallestDivisor(xSqrt);

...it returns 11 to that function call, which never gets returned to the original function call that happened inside of alert().

So the solution is simply to add a return before the one where it calls itself.

return getSmallestDivisor(xSqrt);

This is a common mistake when making recursive functions. A good way to help figure out what is going on is to make extensive use of the browser console.

function getSmallestDivisor(xVal) {    
    console.log("This is xVal: " + xVal);
    if (xVal % 2 === 0) {
        console.log("xVal % 2 === 0 was true");
        return 2;
    }
    else if (xVal % 3 === 0) {
        console.log("xVal % 3 === 0 was true");
        return 3;
    }
    else {
        console.log("This is else.");
        var xSqrt = Math.sqrt(xVal);
        console.log("This is xSqrt of xVal: " + xSqrt);
        if (xSqrt % 1 === 0) {
            console.log("xSqrt % 1 === 0 was true... recursing with xSqrt!!!");
            getSmallestDivisor(xSqrt);
        }
        else {
            console.log("This is the else inside of else. I am returning: " + xVal);
            return xVal;
        }
    }
}
var y = getSmallestDivisor(121);
console.log("This is y: " + y);

Now in your browser, you can open the console (Option + Command + I in most browsers on macOS) and watch what is happening - which parts get executed, etc.

if (xSqrt % 1 === 0) {
    return getSmallestDivisor(xSqrt); // missing return here
} else {
    return xVal;
}

Demo: Fiddle