Maximum number of vertices in intersection of triangle with box
I agree the maximum is $9$ vertices. It's clear that the points where the triangle enters the box will all be vertices of the polygon. There is a maximum of $6$ such entry points, achieved when each of the $3$ sides enters and exits the box.
As a triangle edge pass through the box, the segment of the edge lying inside the box must form one edge of the polygon. In other words, no extra vertices are gained as the triangle passes inside the box.
When the triangle exits the box, at $A$, say, and then re-enters it, at $B$, say, we want maximal box edges lying between $A$ and $B$ because each such box edge will add an extra vertex to the polygon. We would place $A$ and $B$ therefore on opposite faces of the box but $A$ and $B$ being on different sides of a triangle prevents this. So the next best is to have $A$ and $B$ on adjacent faces of the box, in which case there is one box edge between them and hence one extra vertex for the polygon where that box edge intersects the plane of the triangle.
So we have $6$ vertices where the triangle edges enter the box, plus $3$ extra vertices from the box edges giving us $9$ in total.
It can be seen that this is in fact achievable with a little thought about positioning the triangle relative to the box as just described.
Colour codes:
Green : sides of triangle inside the cube
Blue : sides of triangle outside the cube and visible
Orange: sides of triangle outside the cube and hidden behind the cube
Purple : sides of polygon on a cube face
The purple and green sides form the polygon.
An answer emerged from a couple of the comments, observing that
- The problem can be reduced to finding the intersection of the box with the triangle's plane, which produces a polygon, followed by the 2D problem of intersecting that polygon with the triangle.
- It's possible to cut a box with a plane in such a way that the intersection is a hexagon.
One can easily see that intersecting a triangle with a hexagon can give a nonagon, since each of the triangle's three edges can cut off one vertex of the hexagon. So the upper bound of 9 vertices is fulfilled in such a case.