× Welcome to the Centrify Community! We are rolling out product name changes — click here to learn more.

Centrify Crash Dumps

Showing results for 
Search instead for 
Do you mean 
Reply
Participant III
Posts: 11
Registered: 2 weeks ago
#1 of 16 349
Accepted Solution

Centrify Crash Dumps

We use Centrify Express on several production servers that are a bit dated. On our file server (CentOS 6.3, Centrify DC 5.0.4), Centrify has been working great for years. Yesterday, we rebooted the server to troubleshoot an issue with our backup server, and Centrify never bounced back.

 

Whenever a user attempts to access a Samba share, Centrify would crash and create crash dump files.

 

We've been troubleshooting non-stop for about 36 hours now, so describing everything we've done is probably useless, so here's where we are now:

 

1) We used yum remove CentrifyDC to uninstall Centrify (after doing the adleave command).

2) We used yum update to update the server to CentOS 6.9.

3) We reinstalled Samba and were able to verify that we could connect to local Samba shares with local accounts with Centrify removed.

4) We downloaded Centrify Suite 2017 (Centrify DC 5.4.2) and installed it using instructions.

5) We used the adjoin command to joint to our domain without errors. The adinfo command shows valid info for our domain controller.

5) We are still using stock Samba and our backed-up config for that.

 

At this point, our symptoms are:

1) We can't login to the server with domain credentials. It accepts the username and password, then reverts back to the login screen (yes, home directories exist with the correct permissions).

2) We can't login to the server with local credentials that are not root, but root is working correctly.

3) We cannot connect to Samba shares without getting "Access denied" errors, though permissions have not been changed.

4) We don't know how to test to ensure that Centrify is working correctly or not.

 

At this point, we're really uncertain as to how to proceed. Given that Samba shares work fine without Centrify installed, we believe that Samba is configured correctly and functioning. We believe that Centrify is not processing authentication correctly or that we have a misconfiguration somewhere, but don't know where to begin.

 

Any advice is good advice! Thanks in advance for the help.

Centrify Guru I
Posts: 1,949
Registered: ‎07-26-2012
#2 of 16 329

Re: Centrify Crash Dumps

[ Edited ]

@KennySisco,

 

Welcome to the Centrify community. 

 

I will give you some rapid-fire comments.  Bottomline, due to the poor lifecycle management of the software, there are "many things going on" :

 

On our file server (CentOS 6.3, Centrify DC 5.0.4), Centrify has been working '
great for years. Yesterday, we rebooted the server to troubleshoot an issue
with our backup server, and Centrify never bounced back.

Thanks for the feedback. Unfortunately the " set it and forget it"  approach does not help with authentication software (because of proper IT practices, and most importantly, security practices).  CDC 5.0.4 has been EOL since September 2012, and you are using it in CentOS 6.3 that we started to support officially in 2015.  Surprisingly you were doing well.  

 

2) We used yum update to update the server to CentOS 6.9.
3) We reinstalled Samba and were able to verify that we could connect to 
local Samba shares with local accounts with Centrify removed. 4) We downloaded Centrify Suite 2017 (Centrify DC 5.4.2) and installed it
using instructions.

Several comments here.  I'm not sure how critical this server is (and there's nothing we can do about the past), but I think looking at change control before doing these steps could have helped better.  That being said:
- With the release of 5.4.0, many things happened with our software (we split many of the open source utilities), this means that the best bet (which was an upgrade in place) wasn't supported from such an older version.  By starting from scratch, you needed to understand how to reconfigure the system.  This is complicated because of the changes below.

- Samba changes.  Since BadLock, we EoL'd all the Centrify-enhanced Samba software, in favor of the adbindproxy Identity Mapper, this means that the deployment is slightly different.

 

1) We can't login to the server with domain credentials. 
It accepts the username and password, then reverts back to the login screen
See below on how to determine this issue. Watch for Samba/CDC clashes depending on
how you set it up.
(yes, home directories exist with the correct permissions).
That's to be expected. The files won't be destroyed unless a human being or a program does. 2) We can't login to the server with local credentials that are not root,
but root is working correctly.
root will work correctly. We don't mess with it. 3) We cannot connect to Samba shares without getting "Access denied" errors,
though permissions have not been changed.
You need to make sure adbindproxy is set up and configured (see below). Most
importantly, remove all Centrify-ehnhanced Samba servers from your environment
since they are EOL'd and susceptible to anything from BadLock on. 4) We don't know how to test to ensure that Centrify is working correctly
or not.
Use the adinfo or adcheck commands. The most basic test is to see if the system is
disconnected with adinfo -T; after that you can check to see if the users and groups
are resolvable (adquery user/adquery group). See the link below.

If I were to approach this, I'd simply start in different parts.

Make sure CDC is working correctly and allows login; once that's done I'd work on the Samba integration.

 

I'd start by disabling Samba; the system does not need to be double-joined.  The potential (this is an assumption) is that Samba and CDC may be having a conflict on UID/GID resolution.  Let's say that userX tries to log in (the UID/GID data from Centrify is there), but depending how the NSS/PAM stack is laid out, the Samba UID/GID is resolved first, then we have a problem.

 

Tips:

 

Please let us know of any progress you make. 

 

R.P

Want to learn more about practical Centrify examples? Check out my blog at http://centrifying.blogspot.com
Follow Centrify:
Participant III
Posts: 11
Registered: 2 weeks ago
#3 of 16 321

Re: Centrify Crash Dumps

@Robertson,

 

Thank you for the comments - they've been very helpful! I realize we made a rookie mistake with the updates, and we got started with the adbindproxy documentation last night. We believe we're very close to resolving this, even with all of the junk that happened in between.

 

In the Centrify log, with debug on (as you suggested), here's what we see when a domain user logs in:

 

Nov 08 13:41:33 sshd[32443] DEBUG: -> getpwuid_centrifydc_r  UID=0
Nov 08 13:41:33 sshd[32443] DEBUG: getpwuid: UID 0 is in 'nss.uid.ignore' list
Nov 08 13:41:33 sshd[32443] DEBUG: <- getpwuid_centrifydc_r, result=NSS_NOTFOUND(0)
Nov 08 13:41:33 sshd[32443] DEBUG: -> getgrnam_centrifydc_r  group="tty"
Nov 08 13:41:33 sshd[32443] DEBUG: getgrnam: Group 'tty' is in 'nss.group.ignore' list
Nov 08 13:41:33 sshd[32443] DEBUG: <- getgrnam_centrifydc_r, result=NSS_NOTFOUND(0)
Nov 08 13:41:33 sshd[32443] DEBUG: -> getgrnam_centrifydc_r  group="tty"
Nov 08 13:41:33 sshd[32443] DEBUG: getgrnam: Group 'tty' is in 'nss.group.ignore' list
Nov 08 13:41:33 sshd[32443] DEBUG: <- getgrnam_centrifydc_r, result=NSS_NOTFOUND(0)
Nov 08 13:41:33 sshd[32443] DEBUG: -> getpwuid_centrifydc_r  UID=0
Nov 08 13:41:33 sshd[32443] DEBUG: getpwuid: UID 0 is in 'nss.uid.ignore' list
Nov 08 13:41:33 sshd[32443] DEBUG: <- getpwuid_centrifydc_r, result=NSS_NOTFOUND(0)
Nov 08 13:41:33 sshd[32443] DEBUG: -> pam_sm_setcred
Nov 08 13:41:33 sshd[32443] DEBUG: PAM Options: (none)
Nov 08 13:41:33 sshd[32443] DEBUG: PAM Flags: (none)
Nov 08 13:41:33 sshd[32443] DEBUG: Flag PAM_ESTABLISH_CRED or PAM_REINITIALIZE_CRED or PAM_REFRESH_CRED is not given, ignored!
Nov 08 13:41:33 sshd[32443] DEBUG: <- pam_sm_setcred, result=PAM_IGNORE(25)
Nov 08 13:41:33 sshd[32443] DEBUG: -> pam_sm_setcred
Nov 08 13:41:33 sshd[32443] DEBUG: PAM Options: deny
Nov 08 13:41:33 sshd[32443] DEBUG: PAM Flags: (none)
Nov 08 13:41:33 sshd[32443] DEBUG: Flag PAM_ESTABLISH_CRED or PAM_REINITIALIZE_CRED or PAM_REFRESH_CRED is not given, ignored!
Nov 08 13:41:33 sshd[32443] DEBUG: <- pam_sm_setcred, result=PAM_IGNORE(25)

We've been seeing a lot of errors in Samba regarding UIDs not being found, so I believe you're right about the UID/GID conflict (potentially). Here's the log we see in Samba when trying to connect (there's a lot more data here, but this is what I've found to be relevant):

 

[2017/11/08 13:44:54.814265,  1] ../source3/auth/token_util.c:430(add_local_groups)
  SID S-1-5-21-3931225680-1871015619-2963001510-1368939 -> getpwuid(1001) failed
[2017/11/08 13:44:54.814304,  3] ../source3/auth/token_util.c:316(create_local_nt_token_from_info3)
  Failed to finalize nt token
[2017/11/08 13:44:54.814337,  1] ../source3/auth/auth_generic.c:127(auth3_generate_session_info_pac)
  Failed to map kerberos pac to server info (NT_STATUS_UNSUCCESSFUL)
[2017/11/08 13:44:54.815371,  3] ../source3/smbd/server_exit.c:249(exit_server_common)
  Server exit (NT_STATUS_CONNECTION_RESET)

Any ideas on what to look at to get CDC working first?

Centrify Guru I
Posts: 1,949
Registered: ‎07-26-2012
#4 of 16 319

Re: Centrify Crash Dumps

@KennySisco,

 

Impressive. I'm sorry you had to go through this.

However, it looks like you're close.

 

I would disable (or remove) samba and just focus on CDC

 

Is the client connected (adinfo should display)

What's the de-identified output of adinfo -T?

What happens when you do an adquery user or adquery group command?

 

It would be good to also know the output of the passwd and group stanza on /etc/nsswitch.conf

 

Some tips on testing raw authentication:

 

adquery user -A -u test.user

helps you test using just the agent, bypassing PAM, NSS or any app like Login or SSH.  It will prompt you for the password of the user and if you type it correctly, that's an end-to-end test of the client connectivity.

 

Keep this handy:  https://community.centrify.com/t5/TechBlog/TIPS-A-Centrify-Server-Suite-Cheat-Sheet/ba-p/22568

 

Most folks find what they need there.

 

R.P

 

Want to learn more about practical Centrify examples? Check out my blog at http://centrifying.blogspot.com
Follow Centrify:
Participant III
Posts: 11
Registered: 2 weeks ago
#5 of 16 317

Re: Centrify Crash Dumps

@Robertson Great tips! A lot of these give good output - the connection with the domain seems to be fine. My guess is it's related to NSS or PAM like you mentioned (those have come up a lot in our research), we just don't know what we're looking at.

 

Here's the output of the commands you suggested.

 

adinfo -T (unidentified):

Domain Diagnostics:
  Domain: [correct.domain.name]
    DNS query for: _ldap._tcp.[correct.domain.name]
    DNS query for: _gc._tcp.[correct.domain.name]
  Testing Active Directory connectivity:
    Global Catalog: lares.[correct.domain.name]      gc:       3268/tcp - good
    Global Catalog: zeus.[correct.domain.name]
      gc:       3268/tcp - good
    Global Catalog: aphrodite.[correct.domain.name]
      gc:       3268/tcp - good
    Global Catalog: flora.[correct.domain.name]
      gc:       3268/tcp - timeout
      No TCP LDAP response, giving up on flora.[correct.domain.name]    Global Catalog: fauna.[correct.domain.name]     gc:       3268/tcp - timeout
      No TCP LDAP response, giving up on fauna.[correct.domain.name]
    Global Catalog: ares.[correct.domain.name]
      gc:       3268/tcp - good
    Global Catalog: artemis.[correct.domain.name]      gc:       3268/tcp - good
    Global Catalog: helios.[correct.domain.name]
      gc:       3268/tcp - good
    Global Catalog: ceres.[correct.domain.name]      gc:       3268/tcp - good
    Domain Controller: helios.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: artemis.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: flora.[correct.domain.name]
      ldap:      389/tcp - timeout
      No TCP LDAP response, giving up on flora.[correct.domain.name]
    Domain Controller: lares.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: ceres.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: fauna.[correct.domain.name]
      ldap:      389/tcp - timeout
      No TCP LDAP response, giving up on fauna.[correct.domain.name]
    Domain Controller: ares.[correct.domain.name]      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: zeus.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: aphrodite.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good

Adquery user works fine as well. In fact, running just that command tries to list every user in AD, which is a lot. Here's an example of one (with PHI removed):

[username]:x:819327050:817889793:[Last Name], [First Name] [Initial]:/export/homes/[username]:/bin/bash

NSSwitch.conf (sections you asked for):

passwd: centrifydc files
shadow: centrifydc files
group: centrifydc files

Adquery user -A -u [username] gave a ton of output that is correct, including name, uid, gid, shell, home, dn, sid, userPrincipalName, guid, account info, group memberships, etc. I don't want to copy all of that in here due to the personally identifiable information, but it's working fantastically.

 

Interestingly enough, however, that command does not prommpt for a password like you mentioned. It just gives output.

 

So you're onto something here in that NSS and/or PAM is likely where the mixup is happening. Do you know where we can go next?

Centrify Guru I
Posts: 1,949
Registered: ‎07-26-2012
#6 of 16 315

Re: Centrify Crash Dumps

@KennySisco,

 

My bad!

 

It was
adinfo -A -u username

Sorry.

 

Do me a favor,  as root, switch user (su) to an ad user  (you'll be able to do it), then try to su to another known user (type the correct password).  If you are able to su from the normal account to the other user with the password, then your issue is with OpenSSH or the GUI.

 

I have to continue working on something else now.  Ciao.

 

R.P

 

Want to learn more about practical Centrify examples? Check out my blog at http://centrifying.blogspot.com
Follow Centrify:
Participant III
Posts: 11
Registered: 2 weeks ago
#7 of 16 313

Re: Centrify Crash Dumps

@Robertson Thanks for all of your support! You've been super helpful.

 

I ran the command adinfo -A -u [username] with my domain credentials, and it DID prompt for a password and said the password is correct. So that works great.

 

The su is where we broke here. I tried the command "su - [username]" with a domain user, and got the following errors:

su: warning: cannot change directory to /export/homes/[username]: Permission denied
su: /bin/bash: Permission denied

This is basically what we've been seeing across the board since getting this far and where we've been stuck.

 

We'll continue digging, I think you gave us a lot to read and a lot to work on. I appreciate the support and would, of course, love to hear your feedback when you have time.

 

Will post updates as we make progress.

Centrify Guru I
Posts: 1,949
Registered: ‎07-26-2012
#8 of 16 310

Re: Centrify Crash Dumps

[ Edited ]

Do an adquery user for that user and note the UID/GID.

Now do an ls -ln on that home directory .

 

If the numbers are different, you have an identity mismatch.  This could be due to something changing in the domain (long shot, but perhaps migrated??? or conflict with another source like local file? most likely human intervention???).

 

chmoding will solve the issue for that user and we provide a tool called adfixid to do this in bulk.

*** HANDLE WITH CARE ***

 

https://community.centrify.com/t5/TechBlog/HOWTO-Using-ADFIXID-and-ADRMLOCAL-to-clean-up-the-local-u...

 

 

Want to learn more about practical Centrify examples? Check out my blog at http://centrifying.blogspot.com
Follow Centrify:
Participant III
Posts: 11
Registered: 2 weeks ago
#9 of 16 302

Re: Centrify Crash Dumps

@Robertson thanks again for pointing us in that direction!

 

We found that the adquery and the permissions showed the same UID, but the logs I posted above don't - they look like they're showing that the uid=0? Not sure what that means from a log perspective, but it's weird for sure.

 

Our local Linux guy found that permissions on our server seem to be really messed up. He made a permissions change on / to 0555 and that got SSH somewhat working - it allows the user to login, gives access denied on the home directory, but gives a bash shell nonetheless. It's really strange.

 

I think there's a good chance that Centrify is no longer our problem (though there are some config changes we can do to clean up some of the services). We're gonna keep digging on this permissions thing.

What are your thoughts on that?

Participant III
Posts: 11
Registered: 2 weeks ago
#10 of 16 299

Re: Centrify Crash Dumps

So we conducted another test and we found that the UID=0 is what's actually occurring here, but we're not sure why or how to change that.

 

Here's what we did to find this out:
We changed permissions to 755 on one user's home directory, and it allowed us to login without any Access Denied errors, but it's dropping us in the /root folder, which is indicative of the UID=0 (which is root).

 

Any idea why that might be happening? The logs I posted earlier show it occurring with Centrify, but my guess is it has to do with NSS or PAM like you mentioned before.

 

Also, I ran the adfixid command just out of curiosity, and it picked up 0 errors immediately - didn't really appear to scan at all...