How do higher-order functions, like .map(), work internally in JavaScript?

.map is just a method that accepts a callback, invokes the callback for every item of the array, and assigns the value to a new array. There's nothing very special about it. You can even implement it yourself quite easily:

Array.prototype.myMap = function(callback) {
  const newArr = [];
  for (let i = 0; i < this.length; i++) {
    newArr.push(callback(this[i], i, this));
  }
  return newArr;
}

var numbers = [16, 25, 36];
var results = numbers.myMap(Math.sqrt);
console.log(results); // [4, 5, 6]

To be fully spec-compliant, you'd also need to check, among other things, that the this is an object, that the callback is callable, and to .call the callback with the second parameter passed to myMap if there is one, but those are details not important to a beginning understanding of higher-order functions.

I guess every vendor is supposed to implement it as per the spec

The actual implementation, for example V8 can be a bit complex, refer this answer for a start. You can also refer v8 source in github but it may not be easy to understand only one part in isolation.

Quoted from above answer:

V8 developer here. We have several different implementation techniques for "builtins": some are written in C++, some in Torque, some in what we call CodeStubAssembler, and a few directly in assembly. In earlier versions of V8, some were implemented in JavaScript. Each of these strategies has its own strengths (trading off code complexity, debuggability, performance in various situations, binary size, and memory consumption); plus there is always the historical reason that code has evolved over time.

ES2015 spec:

1. Let O be ToObject(this value).
2. ReturnIfAbrupt(O).
3. Let len be ToLength(Get(O, "length")).
4. ReturnIfAbrupt(len).
5. If IsCallable(callbackfn) is false, throw a TypeError exception.
6. If thisArg was supplied, let T be thisArg; else let T be undefined.
7. Let A be ArraySpeciesCreate(O, len).
8. ReturnIfAbrupt(A).
9. Let k be 0.
10. Repeat, while k < len
    1. Let Pk be ToString(k).
    2. Let kPresent be HasProperty(O, Pk).
    3. ReturnIfAbrupt(kPresent).
    4. If kPresent is true, then
       1. Let kValue be Get(O, Pk).
       2. ReturnIfAbrupt(kValue).
       3. Let mappedValue be Call(callbackfn, T, «kValue, k, O»).
       4. ReturnIfAbrupt(mappedValue).
       5. Let status be CreateDataPropertyOrThrow (A, Pk, mappedValue).
       6. ReturnIfAbrupt(status).
    5. Increase k by 1.
11. Return A.