I've found a bunch of answers to determining which side a point lays between a line, but only in 2D.
Here's an existing question of what I want, but it 2D: How to tell whether a point is to the right or left side of a line
How do I adapt the formula below that is currently only taking into account the X, Y positions, to also take into account the Z axis? Assuming the view point is top-down.
position = sign((Bx - Ax) * (Y - Ay) - (By - Ay) * (X - Ax))
A line in 3D can be viewed from any angle, so in order to pick 'sides' you first have to pick an axis that you'll view it from. Lets say we're viewing it on the Y axis looking straight down, and our line consists of these two points:
(1,4,5) -> (5,2,6)
As we're looking along that Y axis, we can just completely ignore the Y coordinates and treat the line as if it was simply this:
(1,0,5) -> (5,0,6)
Then perform a 2D sign test like any other, only using x/z here.
If your viewer isn't axis aligned then you'd just need to project your line into "screen space" by multiplying the points by your view matrix (implementation of which varies by use case). You'd then end up with a line that is typically being viewed along the Z axis so you can then safely ignore it.