sshd: fatal: initgroups: <User Private Group>: Invalid argument
01-13-2017 06:04 AM
I have a newly installed Ubuntu Xenaial 16.04 server running LXD with a fresh LXC container using the ubuntu:xenial image.
I used the centrify-suite-2016.1-deb7-x86_64.tgz download, unpacked and installed it joined to the domain with no reported errors. I have installed the Centrify SSHD. I can getent passwd and group with no problems.
The problem is when I SSH to the LXC container I get the login prompt, and on entering the username the SSH session disconnects with "Network error: Software caused connection abort". In the container auth.log I see a fatal error: initgroups: Invalid argument. The group causing the fatal error is the User Private Group.
Some system info:
root@osm:/etc/pam.d# uname -a
Linux osm 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@osm:/etc/pam.d# cat /etc/*release
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
NAME="Ubuntu" VERSION="16.04.1 LTS (Xenial Xerus)"
PRETTY_NAME="Ubuntu 16.04.1 LTS"
root@osm:/etc/pam.d# adinfo -v
adinfo (CentrifyDC 5.3.1-398)
root@osm:/etc/pam.d# adinfo -m
root@osm:/etc/pam.d# adinfo --sysinfo zone
======== Zone Information ========
root@osm:/etc/pam.d# adquery user i87000
The auth.log output:
Jan 13 11:35:43 osm sshd: Authorized to i87000, krb5 principal I87000@MYDOMAIN.COM (krb5_kuserok)
Jan 13 11:35:43 osm sshd: Accepted gssapi-with-mic for i87000 from 192.168.0.10 port 61199 ssh2
Jan 13 11:35:43 osm adclient: INFO AUDIT_TRAIL|Centrify Suite|PAM|1.0|300|PAM account management granted|5|user=i87000(type:ad,i87000@MYDOMAIN.COM) pid=2118 utc=1484307343515 centrifyEventID=24300 status=GRANTED service=sshd tty=ssh client=192.168.0.10
Jan 13 11:35:43 osm sshd: fatal: initgroups: i87000: Invalid argument
I notice that the primary gid for the user doesn't exist in getent group output.
root@osm:/etc/pam.d# getent passwd | grep i87000
root@osm:/etc/pam.d# getent group | grep 851444974
Checking the groups output for the user shows the User Private Group, but it doesn't appear in the getent group output:
root@osm:/etc/pam.d# groups i87000
i87000 : i87000 all_employees all_users centrify_mobile_users desktop_administrators domain_admins domain_users
root@osm:/etc/pam.d# getent group | grep ^i87000
This has always been the case for previous installs of Centrify Express on other containers, but this is the first install I have done using with the latest Centrify insaller using OpenSSH version 7
OpenSSH_7.2p2 (CentrifyDC build 5.3.1-391) , OpenSSL 1.0.2g 1 Mar 2016
Any ideas why sshd is giving the 'invalid argument' error?
01-13-2017 07:16 AM
Welcome to the Centrify Express community.
- You don't need Centrify OpenSSH (unless you have specific use cases), you can use Stock SSH.
- Remember that we implement the latest security measures like refusing keys smaller than 1024 bits. More info here:
Use the su command to verify that the issue is strictly related to OpenSSH. (e.g. su - i87000 and type the AD password), you'll note that you can switch succesfully and confirm that the issue is with OpenSSH. You an debug OpenSSH to pinpoint the issue (See below) or simply uninstall Centrify-enhanced OpenSSH in favor of stock SSH.
- Centrify Express for UNIX works in workstation mode and uses auto-private group (user's UID as the primary group) therefore if you use getent you won't see it.
This post has tips on troubleshooting OpenSSH: http://community.centrify.com/t5/Centrify-Express/Network-error-Connection-refused/td-p/22199
01-16-2017 02:49 AM
Thanks for the reply. I checked the key exchange issue, and it didn't seem to be that. I tried confirming that su worked to prove it was an OpneSSH issue, but su failed too. I uninstalled Centrify completely, added the "PasswordAuthentication yes" to /etc/ssh/sshd_config and confirmed that /usr/sbin/sshd works without problems for local users. I reinstalled Centrify Express and am getting a similar error using /usr/sbin/sshd.
Here's the auth.log from my first connection attempt using AD credentials (present in getent passwd) initiated from a fresh Putty session using Putty defaults. I was prompted for a password and the session terminated on me typing in the domain password:
Jan 16 09:59:17 osm sshd: Accepted keyboard-interactive/pam for i87000 from 192.168.0.10 port 55006 ssh2 Jan 16 09:59:17 osm adclient: INFO <fd:10 PAMCreateKrb5Creds > daemon.ipcclient2 Problem storing credentials into credentials cache file for user 'i87000': Problem setting the ownership of FILE:/tmp/krb5cc_851444974: error = -1765328188, error message = krb5_cc_chown: Internal credentials cache error Jan 16 09:59:17 osm adclient: WARN <fd:22 sshd(573)> Set credentials for user 'i87000': Problem storing credentials into credentials cache Jan 16 09:59:17 osm adclient: INFO AUDIT_TRAIL|Centrify Suite|PAM|1.0|201|PAM set credentials denied|5|user=i87000(type:ad,i87810@MYDOMAIN.COM) pid=573 utc=1484560757106 centrifyEventID=24201 status=DENIED service=sshd tty=ssh client=192.168.0.10 reason=Failed to set user credentials Jan 16 09:59:17 osm sshd: fatal: PAM: pam_setcred(): Authentication failure Jan 16 09:59:32 osm adclient: INFO AUDIT_TRAIL|Centrify Suite|Trusted Path|1.0|2700|Trusted path granted|5|user=osm$@MYDOMAIN.COM pid=364 utc=1484560772490 centrifyEventID=23700 status=GRANTED server=ldap/pdc.mydomain.com@MYDOMAIN.COM
Further connection attempts fail on providing the username and don't prompt for a password.
I can successfully SSH in as a local user. When I try to su from that local user to a domain user I get the following from auth.log:
Jan 16 10:26:19 osm adclient: INFO AUDIT_TRAIL|Centrify Suite|PAM|1.0|100|PAM authentication granted|5|user=i87000(type:ad,i87000@MYDOMAIN.COM) pid=709 utc=1484562379611 centrifyEventID=24100 status=GRANTED service=su tty=/dev/pts/0 client=(none) Jan 16 10:26:19 osm su: Successful su for i87000 by localuser Jan 16 10:26:19 osm adclient: INFO AUDIT_TRAIL|Centrify Suite|PAM|1.0|300|PAM account management granted|5|user=i87000(type:ad,i87000@MYDOMAIN.COM) pid=709 utc=1484562379614 centrifyEventID=24300 status=GRANTED service=su tty=/dev/pts/0 client=(none) Jan 16 10:26:19 osm su: + /dev/pts/0 localuser:i87000 Jan 16 10:26:19 osm su: bad group ID `851444974' for user `i87000': Invalid argument
I've tried running sshd in debug on a fresh port with '/usr/sbin/sshd -ddde -p 2222' then running SSH from another Ubuntu server using the domain ID. I've truncated the sshd debug output (it's error free above this point):
debug1: PAM: initializing for "i87000" debug1: PAM: setting PAM_RHOST to "192.168.0.11" debug1: PAM: setting PAM_TTY to "ssh" debug2: monitor_read: 100 used once, disabling now debug3: receive packet: type 50 [preauth] debug1: userauth-request for user i87810 service ssh-connection method publickey [preauth] debug1: attempt 1 failures 0 [preauth] debug2: input_userauth_request: try method publickey [preauth] debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for RSA SHA256:l4Ntxwcz+iWnxeaY1XjVvyfo18E/Xev4VUgaj5a/uWg [preauth] debug3: mm_key_allowed entering [preauth] debug3: mm_request_send entering: type 22 [preauth] debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth] debug3: mm_request_receive_expect entering: type 23 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 4 debug3: mm_answer_authserv: service=ssh-connection, style=, role= debug2: monitor_read: 4 used once, disabling now debug3: mm_request_receive entering debug3: monitor_read: checking request 22 debug3: mm_answer_keyallowed entering debug3: mm_answer_keyallowed: key_from_blob: 0x55bcb38df3c0 debug1: temporarily_use_uid: 851444974/851444974 (e=0/0) initgroups: i87000: Invalid argument debug1: do_cleanup debug1: PAM: cleanup debug3: PAM: sshpam_thread_cleanup entering debug1: Killing privsep child 712 debug1: audit_event: unhandled event 12
Thank you for the link to the troubleshooting tips for OpenSSH - it was very helpful and informative, but I'm not sure this is an OpenSSH issue. Neither the problem adclient has in storing credentials or the su error with invalid groups relate to OpenSSH. I'm not sure which of these is the underlying issue - is adclient failing to store the cache credentials because it's trying to use an invalid group, or is there an invalid group because adclient is unable to cache credentials?
01-16-2017 06:39 AM
Great job at debugging and providing output.
I think this part pinpoints the issue (file permissions):
Jan 16 09:59:17 osm adclient: INFO <fd:10 PAMCreateKrb5Creds > daemon.ipcclient2 Problem storing credentials into credentials cache file for user 'i87000':
Problem setting the ownership of FILE:/tmp/krb5cc_851444974: error = -1765328188, error message = krb5_cc_chown: Internal credentials cache error Jan 16 09:59:17 osm adclient: WARN <fd:22 sshd(573)> Set credentials for user 'i87000': Problem storing credentials into credentials cache
The Kerberos cache file for the user is in the /tmp filesystem with naming convention krb5cc_[UID]; I would check the permissions or any masking in that filesystem (unorthodox), this should be writable by root.
Please confirm if the issue is on a single host (that would explain unorthodox config) or all systems (that would point to a bug). Also, are there any DevOps solutions perhaps enforcing settings (Chef, Puppet, Ansible)?
01-30-2017 08:19 AM
Thanks very much for your help. The issue seems to be related to (lack of) privilege in the LXC container. The container was running in the default unprivileged mode which maps uids and gids (in /etc/subuid and /etc/subgid respectively) to uids of the parent host. I've set security.privileged to true on the container, which stops the id mapping, and the authentication via su and ssh succeeds.
This gives me a workaround for now and buys a bit of time to experiment with getting things working in an unprivileged container.