Finding the character table of this group

Since $G' = \langle (bab^{-1})a^{-1} \rangle = \langle a^3 \rangle$ we have $G/G' \cong C_3 \times C_3$. This gives $9$ characters of degree $1$, including three which have kernel containing $H$, so come from characters of $G/H \cong C_3$.

Your idea to consider the character $\rho : \langle a \rangle \rightarrow \mathbb{C}^\times$ defined by $\rho(a) = \exp(2\pi i/9) = \zeta$ is a good one. Let $H = \langle a \rangle$. By Frobenius reciprocity

$$\langle \rho\uparrow^G , \rho\uparrow^G \rangle_G = \langle \rho\uparrow^G\downarrow_H, \rho \rangle_H. $$

Now $\rho\uparrow^G\downarrow_H(a^i) = \rho(a^i) + \rho(ba^ib^{-1}) + \rho(b^2a^ib^{-2})$ so $\rho\uparrow^G\downarrow_H$ is the sum of three distinct conjugates of $\rho$. Hence the right-hand side above is $1$ and $\rho\uparrow^G$ is irreducible.

So $\rho\uparrow^G$ is an irreducible character of degree $3$, satisfying $$\rho\uparrow^G(a^3) = \zeta^3 + \zeta^{12} + \zeta^{48} = 3\zeta^3 = 3\exp (2\pi i/3).$$ Since $\rho(a^3) \not\in \mathbb{R}$, $\bar{\rho}$ must be the remaining irreducible character.

Alternatively, knowing the degrees and the nine linear characters, you could use row and column orthogonality relations to complete the character table. Since all elements of $G$ not in $Z(G) = \langle a^3 \rangle$ have centralizer of size $9$, the degree $3$ irreducible characters are zero off $Z(G)$. (In particular $\rho(a) = \zeta+\zeta^4+\zeta^{16} = \zeta + \zeta^4 + \zeta^7 = 0$.) This only leaves four entries to find.

I think my answer will be quite similar to Mark's, but I find it slightly more elementary.

As Mark said, the Abelianisation of $G$ is the Abelian group generated by two commuting order-3 elements $A$ and $B$ (and, obviously, the Abelianisation map sends $a$ to $A$ and $b$ to $B$). This gives you a complete list of $1$-dimensional irreducible representations: there are nine of them, and you will see in your character table the character table of $C_3\times C_3$.

By the way, your list of conjugacy classes is now easy to understand. Conjugate elements are mapped to the same element in the Abelianised group. Since the Abelianisation map is 3-to-1 and the number of elements in a conjugacy class must divide the order of the group, the conjugacy classes have 1 or 3 elements.

An element $g$ is alone in its conjugacy class iff it is in the center of $G$ : there is only three such elements (you can check it by hand or you can use small group theory exercises: the center of $G$ has to be nontrivial because $G$ is a $3$-group, and it cannot have more than $3$ elements because, else, $G/ZG$ would be cyclic, which forces $G$ to be Abelian). So the center of $G$ has three elements: $\mathrm{id}$, $a^3$ and $a^6$, and the other conjugacy classes are the 3-element sets which are sent to the same (nontrivial) element in the Abelianisation: $c_{i,j} = \{a^i b^j, a^{i+3} b^j, a^{i+6} b^j\}$. We then have three one-element classes and eight three-element ones.

Then, you can use the $\sum_\chi (\dim \chi)^2 = |G|$ formula to obtain that the $11 - 9 = 2$ remaining representations have dimensions $n_1, n_2 > 1$ such that $n_1^2 + n_2^2 = 27 - 9 = 18$. The only solution is $n_1 = n_2 = 3$.

Now, I will prove that the characters corresponding to these $3$-dimensional representations vanish on the $3$-element conjugacy classes. Indeed, suppose that for one of these classes, $\chi_\rho(c) = z \neq 0$. Then I can use a one-dimensional representation $\lambda$ to twist $\rho$: I get an irreducible representation $\rho \otimes \lambda$ such that $$\chi_{\rho \otimes \lambda}(c) = \lambda(c) \chi_{\rho}(c).$$ Because these three-element conjugacy classes are sent to a nontrivial element in the Abelianised group, I can find two one-dimensional representations such that $\lambda_1(c) = j$ and $\lambda_2(c) = j^2$. The three $3$-dimensional irreducibles representations $\rho$, $\rho \otimes \lambda_1$ and $\rho \otimes \lambda_2$ then have to be different (their characters take different values on the conjugacy class $c$), but that's absurd, because there are only two $3$-dimensional irreducible representations. (You can also prove this result using the orthogonality conditions).

We still have four undetermined entries (the values of the two 3-dimensional characters on the two nontrivial central conjugacy classes), but the orthonormality conditions are now enough to determine them.

If $\alpha = 3 e^{\frac{2 \pi i}3}$ and $\alpha' = \overline{\alpha}$, we then get the full character table (where $\mathrm{III}_k$ are the eight 3-element conjugacy classes).

$$\begin{array}{c||ccc|ccc|ccc|cc} &\{\mathrm{id}\} & \mathrm{III}_1 & \mathrm{III}_2 & \mathrm{III}_3 & \mathrm{III}_4 & \mathrm{III}_5 & \mathrm{III}_6 & \mathrm{III}_7 & \mathrm{III}_8 & \{a^3\} & \{a^6\}\\ \hline \hline 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\ \lambda_1 & 1 & 1 & 1 & j & j & j & j^2 & j^2 & j^2 & 1 & 1 \\ \lambda_2 & 1 & 1 & 1 & j^2 & j^2 & j^2 & j & j & j & 1 & 1 \\ \hline \lambda_3 & 1 & j & j^2 & 1 & j & j^2 & 1 & j & j^2 & 1 & 1\\ \lambda_4 & 1 & j & j^2 & j & j^2 & 1 & j^2 & 1 & j & 1 & 1 \\ \lambda_5 & 1 & j & j^2 & j^2 & 1 & j & j & j^2 & 1 & 1 & 1 \\ \hline \lambda_6 & 1 & j^2 & j & 1 & j^2 & j & 1 & j^2 & j & 1 & 1\\ \lambda_7 & 1 & j^2 & j & j & 1 & j^2 & j^2 & j & 1 & 1 & 1 \\ \lambda_8 & 1 & j^2 & j & j^2 & j & 1 & j & 1 & j^2 & 1 & 1 \\ \hline \rho_1 & 3 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \alpha & \alpha' \\ \rho_2 & 3 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \alpha' & \alpha \\ \end{array}.$$ I've chosen this strange order for the conjugacy classes so that the principal $8 \times 8$-submatrix is the character table of $C_3 \times C_3$. The simple lines are just there to emphasize the structure of this table.

I must say I'm still not satisfied... I spent some time to try and find an elegant definition of the elusive (but faithful!) three-dimensional representation (I say "the", because the two representations are conjugated by an element of the Galois group, so they are "essentially" the same) but I haven't much to show for it.

The only "concrete" manifestation I've found is that $G$ is the subgroup of $$\mathrm{Aff}_1(\mathbb Z/9) = \left\{ x \mapsto m x + p \middle| m \in (\mathbb Z/9)^\times, p\in \mathbb Z/9 \right\}$$ corresponding to the (index two) subgroup of $(\mathbb Z/9)^\times$ generated by $4$ (with this isomorphism, $a$ is $t\mapsto t+1$ and $b$ is $t\mapsto 4t$) and that the permutation representation on the 9-element set $\mathbb Z/9$ induced by this definition splits as $$\lambda_1 \oplus \lambda_2 \oplus \lambda_3 \oplus \rho_1 \oplus \rho_2,$$ where the three 1-dimensional $\lambda_i$ are the ones vanishing on $a$.

(Note that since $\alpha$ isn't an integer, there is no hope to find $\rho_1$ and $\rho_2$ as permutation representations. I think that they are likely to appear as something number-theoretic, perhaps in relation with the cyclotomic field $\mathbb Q(\zeta_9)$, but so far my attempts have been quite frustrated.)