Optimal Strategy for Rock Paper Scissors with different rewards

Let $(x_1,x_2,x_3)$ be the first player's strategy (i.e., his probabilities of playing rock, paper, and scissors respectively), and let $(y_1,y_2,y_3)$ be the second player's strategy. The expected payoff to the first player is $$ P(x,y)=9(x_1y_3-x_3y_1)+3(x_2y_1-x_1y_2)+5(x_3y_2-x_2y_3). $$ To constrain the probability sums to be $1$, we take $x_3=1-x_1-x_2$ and $y_3=1-y_1-y_2$. So $$ P(x,y)=9\left(x_1(1-y_1-y_2)-(1-x_1-x_2)y_1\right)+3(x_2y_1-x_1y_2)+5\left((1-x_1-x_2)y_2-x_2(1-y_1-y_2)\right) \\ =9(x_1-y_1)+ 17(x_2 y_1 -x_1y_2) + 5(y_2-x_2). $$ The first derivatives are zero when $$ \frac{\partial}{\partial x_1}P(x,y)= 9 -17y_2=0 \\ \frac{\partial}{\partial x_2}P(x,y)= 17y_1 -5=0 \\ \frac{\partial}{\partial y_1}P(x,y)=-9+17x_2 = 0\\ \frac{\partial}{\partial y_2}P(x,y)=-17x_1+5=0, $$ or at $(x_1,x_2,x_3)=(y_1,y_2,y_3)=(5/17, 9/17, 3/17)$. The Nash equilibrium is to play to beat each move with probability proportional to that move's reward.

To check that this is indeed a Nash equilibrium, suppose $y_1=5/17$ and $y_2=9/17$. Then $$ P(x)=9(x_1-5/17)+17\left(x_2 (5/17)-x_1 (9/17)\right)-5(x_2-9/17)=0; $$ that is, the first player's expected payoff is zero with any strategy. So the first player cannot improve his payoff by changing his strategy unilaterally, and by symmetry, neither can the second player; this is the definition of a Nash equilibrium.

This table summarizes the possible outcomes in playing this game once: $$ \begin{array}{c|c|c|c|c} \text{Hero Plays} & \text{Villain Plays} & \text{Hero's Earnings} \\ \hline R & R & +\$0 \\ R & P & -\$3 \\ R & S & +\$9 \\ P & R & +\$3 \\ P & P & +\$0 \\ P & S & -\$5 \\ S & R & -\$9 \\ S & P & +\$5 \\ S & S & +\$0 \end{array} $$ Assume that all outcomes are equally likely. Letting $X$ be our earnings we can compute our expected earnings given which choice we make \begin{align*} \Bbb E(X|R) &= \frac{1}{3}\cdot (\$0)+\frac{1}{3}\cdot (-\$3)+\frac{1}{3}\cdot(\$9) \\ &= -\$1+\$3 \\ &= \$2 \\ \Bbb E(X|P) &= \frac{1}{3}\cdot (\$3)+\frac{1}{3}\cdot (\$0)+\frac{1}{3}\cdot(-\$5) \\ &= \$1-\$\frac{5}{3} \\ &= -\$\frac{2}{3} \\ \Bbb E(X|S) &= \frac{1}{3}\cdot (-\$9)+\frac{1}{3}\cdot (\$5)+\frac{1}{3}\cdot(\$0) \\ &= -\$3+\$\frac{5}{3} \\ &= -\$\frac{4}{3} \\ \end{align*} According to these computations, the only winning strategy is to play rock in which case the expected value is $\$2$.

Note: The above assumes that hero makes a choice against a villain that is playing randomly.

Edit: I missed the "Nash-Equilibrium" tag when I first read this question. You're probably looking for something a bit more sophisticated than this! I'll keep the answer posted in case others find these computations useful.

If "optimal" means Nash equilibrium (i.e. a state that is stable wrt. small perturbations of strategies), than it can be computed. If you assume that $x_1$ is the probability of first player to play Rock, $x_2$ his probability to play Scissors and $1-x_1-x_2$ his probability to play Paper, and similarly for $y_i$, then the Payoff of the first player is $$f(x_1, x_2, y_1, y_2) = x_1 (9y_2 - 3 (1-y_2-y_3)) + x_2 (-9 y_1 + 5(1-y_2-y_3)) + (1-x_1-x_2)(3y_1-5y_2)$$ or something like that. The condition on Nash is that all partial derivatives vanish; you can probably easily compute the probabilities and check, whether you guessed the right solutions (the solution should be unique in this case with $x_i$ and $y_i$ nonzero).

However, in different circumstances, optimal may meen different things; if they are good friends and know that it's a zero sum game, they can also play both Rock all the time.