Triangle and/or Square?

Japt, 18 15 11 bytes

[U8*UÄ]®¬v1

Try it online!

JavaScript (ES7), 40 32 bytes

Returns 1 for square, 2 for triangle, 3 for both or 0 for neither.

n=>!(n**.5%1)+2*!((8*n+1)**.5%1)

• 8 bytes saved in collaboration with ETHproductions and Neil.

Try It

Displays "Square/Triangle/Both/Neither" for clarity

f=
n=>!(n**.5%1)+2*!((8*n+1)**.5%1)
i.addEventListener("input",_=>o.innerText=["Neither","Square","Triangle","Both"][f(+i.value)])

<input id=i type=number><pre id=o>

Jelly, 9 7 bytes

×8‘,µÆ²

Try it online!

Outputs [1,1] for both, [0,1] for square, [1,0] for triangle, [0,0] for neither.

Explanation

×8‘,µÆ² - main link, input a
   ,    - a list of the following two:
×8‘     - a×8+1 (which is square when a is triangle)
        - (implicit) a
    µÆ² - On the previous, check if each is a square