How to declare a custom class in google scripts?

TypeScript can now be used with GAS. To use it, download the script files using clasp then convert them to TypeScript by changing the file suffixes to .ts. More info can be found at https://developers.google.com/apps-script/guides/typescript. Once this is done, class can be used.

An example of TypeScript being used to implement Google Sheets custom functions can be found at https://github.com/november-yankee/science-test-grading.

Update:

As of spring 2020 Google has introduced a new runtime for Apps Script which supports Classes.
New scripts use this runtime by default, while older scripts must be converted. A prompt is displayed in the script editor, from which you can convert your scripts.

// V8 runtime
class Rectangle {
  constructor(width, height) { // class constructor
    this.width = width;
    this.height = height;
  }

  logToConsole() { // class method
    console.log(`Rectangle(width=${this.width}, height=${this.height})`);
  }
}

const r = new Rectangle(10, 20);
r.logToConsole();  // Outputs Rectangle(width=10, height=20)

Original (old) answer:

Historically Javascript is a "classless" language, classes are a newer feature which haven't been widely adopted yet, and apparently are not yet supported by Apps Script.

Here's an example of how you can imitate class behaviour in Apps Script:

var Polygon = function(height, width){
  this.height = height;
  this.width = width;
  
  this.logDimension = function(){
    Logger.log(this.height);
    Logger.log(this.width);
  }
};

function testPoly(){
  var poly1 = new Polygon(1,2);
  var poly2 = new Polygon(3,4);
  
  Logger.log(poly1);
  Logger.log(poly2);
  poly2.logDimension();
}