How can you compute the maximum volume of an envelope(used to enclose a letter)?

Your question is a variant of the teabag problem. I don't believe an exact answer is known, but for the $1 \times 1$ square teabag, the maximum volume is about $0.2$:

---

         
          ^{(Image from Wikipedia article.)}

---

• The primary reference is Anthony Robin's 2004 article, "Paper Bag Problem". Mathematics Today—Bulletin of the Institute of Mathematics and its Applications. 40.3 (2004): 104-107.

• MathWorld article on "paper bags", which quotes Robin's 2004 volume bound.

• $13$-second YouTube Simulation of Inflating a Teabag, using a cloth simulation algorithm.

As the envelope is made of paper, a mathematical model of its deformation would be an isometric coordinate transformation and thus must have zero Gauss curvature almost everywhere; that restriction is clearly violated by the cited solution of the teabag problem.

I already pointed out in a comment, that there was an article in the German issue (Spektrum der Wissenschaft, June 1995) of Scientific American, where the topic has been discussed here (in German).
Maybe the same topic has also been treated a few month earlier in Scientific American.

The essential ressource is Edouard Baumann, especially this link (unfortunately in German only); there is also an image of such a polyhedral cushion.