Yes, it's the shape in the water that will cause a pull left or right. A tiny shift of weight on a small craft can make a huge difference!
It could be the fin alignment. It could be the fin shape. Did the fin get bent because you sore the board on it? Or, is it a manufacturing anomaly (defect)?
Same questions for the board. Or, is there a chunk out of it causing more drag on one side than the other?
Finally, look at your position on the board. Is one side lower than the other? Remember that small craft effect; are you too far forward? Too far back?