Definitely not, the more the merrier. I like 6 pack boats if you can find one. When I dive the beautiful reefs in Belize, we have four in the group plus the dive master. Ideal, but not always possible.
I absolutely try to avoid diveswith kids. My worst dive experience was with three awful kids, not the kids fault but the ignorant parents.