Unable to use "class" methods for callbacks in JavaScript

Am I trying to force JavaScript into a paradigm which it doesn't belong?

When you're talking about Classes yes.

So what is the right way to do this?

First off, you should learn how what kind of values the this keyword can contain.

1. Simple function call

   myFunc(); - this will refer to the global object (aka window) [1]

2. Function call as a property of an object (aka method)

   obj.method(); - this will refer to obj

3. Function call along wit the new operator

   new MyFunc(); - this will refer to the new instance being created

Now let's see how it applies to your case:

MyClass.prototype.open = function() {
    $.ajax({ // <-- an object literal starts here
        //...
        success: this.some_callback,  // <- this will refer to that object
    });      // <- object ends here
}

If you want to call some_callback method of the current instance you should save the reference to that instance (to a simple variable).

MyClass.prototype.open = function() {
    var self = this; // <- save reference to the current instance of MyClass
    $.ajax({ 
        //...
        success: function () {
            self.some_callback();  // <- use the saved reference
        }                          //    to access instance.some_callback
    });                             
}

[1] please note that in the new version (ES 5 Str.) Case 1 will cause this to be the value undefined

[2] There is yet another case where you use call or apply to invoke a function with a given this

Building on @gblazex's response, I use the following variation for methods that serve as both the origin and target of callbacks:

className.prototype.methodName = function(_callback, ...) {

    var self = (this.hasOwnProperty('instance_name'))?this.instance_name:this;

    if (_callback === true) {

        // code to be executed on callback

    } else {

        // code to set up callback
    
    }

};

on the initial call, "this" refers to the object instance. On the callback, "this" refers to your root document, requiring you to refer to the instance property (instance_name) of the root document.