× Welcome to the Centrify Community! Looking for Express & Smart Card Help? Click Here

sshd: fatal: initgroups: <User Private Group>: Invalid argument

Showing results for 
Search instead for 
Do you mean 
Reply
DTR
Participant II
Posts: 3
Registered: ‎01-13-2017
#1 of 5 701

sshd: fatal: initgroups: <User Private Group>: Invalid argument

I have a newly installed Ubuntu Xenaial 16.04 server running LXD with a fresh LXC container using the ubuntu:xenial image.

 

I used the centrify-suite-2016.1-deb7-x86_64.tgz download, unpacked and installed it joined to the domain with no reported errors. I have installed the Centrify SSHD. I can getent passwd and group with no problems.

 

The problem is when I SSH to the LXC container I get the login prompt, and on entering the username the SSH session disconnects with "Network error: Software caused connection abort". In the container auth.log I see a fatal error: initgroups: Invalid argument. The group causing the fatal error is the User Private Group.

 

Some system info:

 

root@osm:/etc/pam.d# uname -a

Linux osm 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

root@osm:/etc/pam.d# cat /etc/*release

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=16.04

DISTRIB_CODENAME=xenial

DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

NAME="Ubuntu" VERSION="16.04.1 LTS (Xenial Xerus)"

ID=ubuntu ID_LIKE=debian

PRETTY_NAME="Ubuntu 16.04.1 LTS"

VERSION_ID="16.04"

HOME_URL="http://www.ubuntu.com/"

SUPPORT_URL="http://help.ubuntu.com/"

BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

VERSION_CODENAME=xenial

UBUNTU_CODENAME=xenial

root@osm:/etc/pam.d# adinfo -v

adinfo (CentrifyDC 5.3.1-398)

root@osm:/etc/pam.d# adinfo -m

connected

root@osm:/etc/pam.d# adinfo --sysinfo zone

System Diagnostic

======== Zone Information ========

Auto zone

root@osm:/etc/pam.d# adquery user i87000

i87000:x:851444974:851444974:DTR:/home/i87000:/bin/bash

 

The auth.log output:

 

Jan 13 11:35:43 osm sshd[2118]: Authorized to i87000, krb5 principal I87000@MYDOMAIN.COM (krb5_kuserok)

Jan 13 11:35:43 osm sshd[2118]: Accepted gssapi-with-mic for i87000 from 192.168.0.10 port 61199 ssh2

Jan 13 11:35:43 osm adclient[383]: INFO AUDIT_TRAIL|Centrify Suite|PAM|1.0|300|PAM account management granted|5|user=i87000(type:ad,i87000@MYDOMAIN.COM) pid=2118 utc=1484307343515 centrifyEventID=24300 status=GRANTED service=sshd tty=ssh client=192.168.0.10

Jan 13 11:35:43 osm sshd[2118]: fatal: initgroups: i87000: Invalid argument

 

I notice that the primary gid for the user doesn't exist in getent group output.

 

root@osm:/etc/pam.d# getent passwd | grep i87000

i87000:x:851444974:851444974:DTR:/home/i87000:/bin/bash

root@osm:/etc/pam.d# getent group | grep 851444974

root@osm:/etc/pam.d#

 

Checking the groups output for the user shows the User Private Group, but it doesn't appear in the getent group output:

 

root@osm:/etc/pam.d# groups i87000

i87000 : i87000 all_employees all_users centrify_mobile_users desktop_administrators domain_admins domain_users

root@osm:/etc/pam.d# getent group | grep ^i87000

root@osm:/etc/pam.d#

 

This has always been the case for previous installs of Centrify Express on other containers, but this is the first install I have done using with the latest Centrify insaller using OpenSSH version 7

 

OpenSSH_7.2p2 (CentrifyDC build 5.3.1-391) , OpenSSL 1.0.2g 1 Mar 2016

 

Any ideas why sshd is giving the 'invalid argument' error?

Centrify Guru I
Posts: 1,693
Registered: ‎07-26-2012
#2 of 5 695

Re: sshd: fatal: initgroups: <User Private Group>: Invalid argument

[ Edited ]

@DTR,

 

Welcome to the Centrify Express community.

 

Some notes:

  • You don't need Centrify OpenSSH (unless you have specific use cases), you can use Stock SSH.
  • Remember that we implement the latest security measures like refusing keys smaller than 1024 bits.  More info here:
    http://community.centrify.com/t5/TechBlog/How-to-avoid-authentication-issues-using-Centrify-OpenSSH-...
    Use the su command to verify that the issue is strictly related to OpenSSH.  (e.g. su - i87000 and type the AD password), you'll note that you can switch succesfully and confirm that the issue is with OpenSSH.   You an debug OpenSSH to pinpoint the issue (See below)  or simply uninstall Centrify-enhanced OpenSSH in favor of stock SSH.
  • Centrify Express for UNIX works in workstation mode and uses auto-private group (user's UID as the primary group) therefore if you use getent you won't see it.

 

This post has tips on troubleshooting OpenSSH:  http://community.centrify.com/t5/Centrify-Express/Network-error-Connection-refused/td-p/22199

 

R.P

Want to learn more about practical Centrify examples? Check out my blog at http://centrifying.blogspot.com
Follow Centrify:
DTR
Participant II
Posts: 3
Registered: ‎01-13-2017
#3 of 5 672

Re: sshd: fatal: initgroups: <User Private Group>: Invalid argument

Thanks for the reply.  I checked the key exchange issue, and it didn't seem to be that.  I tried confirming that su worked to prove it was an OpneSSH issue, but su failed too.  I uninstalled Centrify completely, added the "PasswordAuthentication yes" to /etc/ssh/sshd_config and confirmed that /usr/sbin/sshd works without problems for local users.  I reinstalled Centrify Express and am getting a similar error using /usr/sbin/sshd.

 

Here's the auth.log from my first connection attempt using AD credentials (present in getent passwd) initiated from a fresh Putty session using Putty defaults.  I was prompted for a password and the session terminated on me typing in the domain password:

 

Jan 16 09:59:17 osm sshd[573]: Accepted keyboard-interactive/pam for i87000 from 192.168.0.10 port 55006 ssh2
Jan 16 09:59:17 osm adclient[364]: INFO <fd:10 PAMCreateKrb5Creds > daemon.ipcclient2 Problem storing credentials into credentials cache file for user 'i87000': Problem setting the ownership of FILE:/tmp/krb5cc_851444974: error = -1765328188, error message = krb5_cc_chown: Internal credentials cache error
Jan 16 09:59:17 osm adclient[364]: WARN <fd:22 sshd(573)> Set credentials for user 'i87000': Problem storing credentials into credentials cache
Jan 16 09:59:17 osm adclient[364]: INFO AUDIT_TRAIL|Centrify Suite|PAM|1.0|201|PAM set credentials denied|5|user=i87000(type:ad,i87810@MYDOMAIN.COM) pid=573 utc=1484560757106 centrifyEventID=24201 status=DENIED service=sshd tty=ssh client=192.168.0.10 reason=Failed to set user credentials
Jan 16 09:59:17 osm sshd[573]: fatal: PAM: pam_setcred(): Authentication failure
Jan 16 09:59:32 osm adclient[364]: INFO AUDIT_TRAIL|Centrify Suite|Trusted Path|1.0|2700|Trusted path granted|5|user=osm$@MYDOMAIN.COM pid=364 utc=1484560772490 centrifyEventID=23700 status=GRANTED server=ldap/pdc.mydomain.com@MYDOMAIN.COM

 

Further connection attempts fail on providing the username and don't prompt for a password.

 

I can successfully SSH in as a local user.  When I try to su from that local user to a domain user I get the following from auth.log:

 

 

Jan 16 10:26:19 osm adclient[364]: INFO AUDIT_TRAIL|Centrify Suite|PAM|1.0|100|PAM authentication granted|5|user=i87000(type:ad,i87000@MYDOMAIN.COM) pid=709 utc=1484562379611 centrifyEventID=24100 status=GRANTED service=su tty=/dev/pts/0 client=(none)
Jan 16 10:26:19 osm su[709]: Successful su for i87000 by localuser
Jan 16 10:26:19 osm adclient[364]: INFO AUDIT_TRAIL|Centrify Suite|PAM|1.0|300|PAM account management granted|5|user=i87000(type:ad,i87000@MYDOMAIN.COM) pid=709 utc=1484562379614 centrifyEventID=24300 status=GRANTED service=su tty=/dev/pts/0 client=(none)
Jan 16 10:26:19 osm su[709]: + /dev/pts/0 localuser:i87000
Jan 16 10:26:19 osm su[709]: bad group ID `851444974' for user `i87000': Invalid argument

I've tried running sshd in debug on a fresh port with '/usr/sbin/sshd -ddde -p 2222' then running SSH from another Ubuntu server using the domain ID.  I've truncated the sshd debug output (it's error free above this point):

 

debug1: PAM: initializing for "i87000"
debug1: PAM: setting PAM_RHOST to "192.168.0.11"
debug1: PAM: setting PAM_TTY to "ssh"
debug2: monitor_read: 100 used once, disabling now
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user i87810 service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug2: input_userauth_request: try method publickey [preauth]
debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for RSA SHA256:l4Ntxwcz+iWnxeaY1XjVvyfo18E/Xev4VUgaj5a/uWg [preauth]
debug3: mm_key_allowed entering [preauth]
debug3: mm_request_send entering: type 22 [preauth]
debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
debug3: mm_request_receive_expect entering: type 23 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 4
debug3: mm_answer_authserv: service=ssh-connection, style=, role=
debug2: monitor_read: 4 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 22
debug3: mm_answer_keyallowed entering
debug3: mm_answer_keyallowed: key_from_blob: 0x55bcb38df3c0
debug1: temporarily_use_uid: 851444974/851444974 (e=0/0)
initgroups: i87000: Invalid argument
debug1: do_cleanup
debug1: PAM: cleanup
debug3: PAM: sshpam_thread_cleanup entering
debug1: Killing privsep child 712
debug1: audit_event: unhandled event 12

 

Thank you for the link to the troubleshooting tips for OpenSSH - it was very helpful and informative, but I'm not sure this is an OpenSSH issue.  Neither the  problem adclient has in storing credentials or the su error with invalid groups relate to OpenSSH.  I'm not sure which of these is the underlying issue - is adclient failing to store the cache credentials because it's trying to use an invalid group, or is there an invalid group because adclient is unable to cache credentials?

Centrify Guru I
Posts: 1,693
Registered: ‎07-26-2012
#4 of 5 665

Re: sshd: fatal: initgroups: <User Private Group>: Invalid argument

Great job at debugging and providing output.

 

I think this part pinpoints the issue (file permissions):

 

Jan 16 09:59:17 osm adclient[364]: INFO <fd:10 PAMCreateKrb5Creds > daemon.ipcclient2 Problem storing credentials into credentials cache file for user 'i87000': 
Problem setting the ownership of FILE:/tmp/krb5cc_851444974: error = -1765328188, error message = krb5_cc_chown: Internal credentials cache error Jan 16 09:59:17 osm adclient[364]: WARN <fd:22 sshd(573)> Set credentials for user 'i87000': Problem storing credentials into credentials cache

The Kerberos cache file for the user is in the /tmp filesystem with naming convention krb5cc_[UID];  I would check the permissions or any masking in that filesystem (unorthodox), this should be writable by root.

 

Please confirm if the issue is on a single host (that would explain unorthodox config) or all systems (that would point to a bug).  Also, are there any DevOps solutions perhaps enforcing settings (Chef, Puppet, Ansible)?

 

R.P

Want to learn more about practical Centrify examples? Check out my blog at http://centrifying.blogspot.com
Follow Centrify:
DTR
Participant II
Posts: 3
Registered: ‎01-13-2017
#5 of 5 600

Re: sshd: fatal: initgroups: <User Private Group>: Invalid argument

Thanks very much for your help.  The issue seems to be related to (lack of) privilege in the LXC container.  The container was running in the default unprivileged mode which maps uids and gids (in /etc/subuid and /etc/subgid respectively) to uids of the parent host.  I've set security.privileged to true on the container, which stops the id mapping, and the authentication via su and ssh succeeds.

 

This gives me a workaround for now and buys a bit of time to experiment with getting things working in an unprivileged container.