Hewlett-Packard's implementation of supplementary groups is nonstandard (for historical reasons). There are two group files, /etc/group and /etc/logingroup; the system maps UIDs to numbers using the former, but initgroups() reads the latter. Most system admins who know the ropes symlink /etc/group to /etc/logingroup (hard-link does not work for reasons too obtuse to go into here). initgroups() will complain if one of the groups you're in, in /etc/logingroup, has what it considers to be an invalid ID, which means outside the range [0..UID_MAX], where UID_MAX is 60000 currently on HP-UX. This precludes -2 and 65534, the usual nobody GIDs.

If you encounter this problem, make sure the programs that are failing to initgroups() are run as users, not in any groups with GIDs outside the allowed range.

This is documented in the HP manual pages under setgroups(2) and passwd(4).

On HP-UX you must use gcc or the HP ANSI compiler. The free compiler that comes with HP-UX is not ANSI compliant and cannot compile Samba.