Centrify does not start after AIX Update to TL5

Showing results for 
Search instead for 
Do you mean 
Reply
Highlighted
Contributor II
Posts: 17
Registered: ‎04-25-2017
#1 of 13 5,715
Accepted Solution

Centrify does not start after AIX Update to TL5

Updated our system to AIX TL5 and after every reboot, Centrify errors out and does not startup.

If we stop it and start it again, sometimes it comes online properly. At other times, it takes more than one stop/start because adinfo hangs and so does centrifydc status..

 

errpt is posting the following:

LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3

Date/Time: Wed Dec 13 12:00:41 CST 2017
Sequence Number: 804
Machine Id: 00F83C874C00
Node Id: psa
Class: S
Type: PERM
WPAR: Global
Resource Name: SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
393222
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'376'
FAILING MODULE
centrifydc

The following related services do make it ok:

 

root 2883924 2163006   0 12:09:08      -  0:00 /usr/share/centrifydc/libexec/adinfo -m
root 1573764 1246190 174 12:09:03      -  5:25 /usr/sbin/adclient -F -M

Anything that can be done to fix it's startup?

 

 

 

Centrify Guru I
Posts: 2,388
Registered: ‎07-26-2012
#2 of 13 5,717

Re: Centrify does not start after AIX Update to TL5

@PFS37273,

 

Welcome back.

 

What is the version of Centrify?  (adinfo --version)

 

Please note that AIX 7.1 TL5 was released on 11/2017 and the last version of adclient was released in the summer.  

AIX 7.1 TL5
October 2017 30 April 2022 (End of Service)

 

Are you a commercial customer?  We can have someone from support reach-out to you.

 

R.P

Want to learn more about practical Centrify examples? Check out my blog at http://centrifying.blogspot.com
Follow Centrify:
Contributor II
Posts: 17
Registered: ‎04-25-2017
#3 of 13 5,714

Re: Centrify does not start after AIX Update to TL5

bash-4.3# adinfo --version
adinfo (CentrifyDC 5.4.2-668)

No, not commercial. Using Express.

Centrify Guru I
Posts: 2,388
Registered: ‎07-26-2012
#4 of 13 5,703

Re: Centrify does not start after AIX Update to TL5

[ Edited ]

@PFS37273,

 

I have opened an issue to track this internally (CSSSUP-9021).  

We will call you out in this thread once we have investigated.

 

Please also private-message me your information (email and phone number) in case we need to reach out to you directly.

 

R.P

Want to learn more about practical Centrify examples? Check out my blog at http://centrifying.blogspot.com
Follow Centrify:
Contributor II
Posts: 17
Registered: ‎04-25-2017
#5 of 13 5,676

Re: Centrify does not start after AIX Update to TL5

Thanks for opening the issue internally.

Here is an update.

We opted to uninstall (unistall.sh and removed all related subsystems entries) and re-install Centrify Express and Samba Integration then we saw your message.Hopefully, what has happened can help your folks help us.

The entire process was done as root. No sudo. No su. Straight root.

During the reinstall it hung in the same place as before.

Here is the result:

WARNING: Centrify adcheck exited with warning(s).
Unzipping ./centrifydc-5.4.2-aix7.1-ppc-bff.gz to /var/centrify/install-tmp* ...
Unzipping ./centrifydc-openssl-5.4.2-aix7.1-ppc-bff.gz to /var/centrify/install-tmp* ...
Unzipping ./centrifydc-openldap-5.4.2-aix7.1-ppc-bff.gz to /var/centrify/install-tmp* ...
Unzipping ./centrifydc-curl-5.4.2-aix7.1-ppc-bff.gz to /var/centrify/install-tmp* ...
Installing all from spool dir ...
Joining the Active Directory domain pg.local ...
Using domain controller: dc3.pg.local writable=true
Join to domain:pg.local, zone:Auto Zone successful

Centrify DirectControl started.

The warning came from DOMNAME and using.local as part of the FDQN.

 

Here are the errpt entries:

They begin with

 

---------------------------------------------------------------------------
LABEL:          CORE_DUMP
IDENTIFIER:     A924A5FC

Date/Time:       Thu Dec 14 12:02:47 CST 2017
Sequence Number: 869
Node Id:         psa
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSPROC

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

        Recommended Actions
        CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        RERUN THE APPLICATION PROGRAM
        IF PROBLEM PERSISTS THEN DO THE FOLLOWING
        CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
           6
USER'S PROCESS ID:
               2490692
FILE SYSTEM SERIAL NUMBER
           4
INODE NUMBER
                 65536
CORE FILE NAME
/var/centrifydc/core
PROGRAM NAME
adclient
STACK EXECUTION DISABLED
           0
COME FROM ADDRESS REGISTER
thread_se 1FC

PROCESSOR ID
  hw_fru_id: N/A
  hw_cpu_id: N/A

ADDITIONAL INFORMATION
_set_canc 48
??
pthread_s 64
pthread_s 14
_ZN4cims4 E0
_ZN4cims4 20
_ZN4cims1 24
_ZN4cims8 544
_ZN4cimsL 148
_ZN4cims6 1C4
_ZN4cims9 16AC
main 4D70
__start 74

Symptom Data
REPORTABLE
1
INTERNAL ERROR
0
SYMPTOM CODE
PCSS/SPI2 FLDS/adclient SIG/6 FLDS/_set_canc VALU/48

Followed by..

 

---------------------------------------------------------------------------
LABEL:          SRC_SVKO
IDENTIFIER:     BC3BE5A3

Date/Time:       Thu Dec 14 12:20:08 CST 2017
Sequence Number: 872
Node Id:         psa
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
      393222
SOFTWARE ERROR CODE
       -9017
ERROR CODE
           0
DETECTING MODULE
'srchevn.c'@line:'376'
FAILING MODULE
centrifydc
---------------------------------------------------------------------------
LABEL:          SRC_SVKO
IDENTIFIER:     BC3BE5A3

Date/Time:       Thu Dec 14 12:11:18 CST 2017
Sequence Number: 871
Node Id:         psa
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
      393222
SOFTWARE ERROR CODE
       -9017
ERROR CODE
           0
DETECTING MODULE
'srchevn.c'@line:'376'
FAILING MODULE
centrifydc
---------------------------------------------------------------------------
LABEL:          SRC_SVKO
IDENTIFIER:     BC3BE5A3

Date/Time:       Thu Dec 14 12:02:47 CST 2017
Sequence Number: 870
Node Id:         psa
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SRC

Description
SOFTWARE PROGRAM ERROR

Probable Causes
APPLICATION PROGRAM

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        MANUALLY RESTART SUBSYSTEM IF NEEDED

Detail Data
SYMPTOM CODE
      393350
SOFTWARE ERROR CODE
       -9017
ERROR CODE
           0
DETECTING MODULE
'srchevn.c'@line:'376'
FAILING MODULE
centrifydc

Looks like the program is trying to recover/restart but just can't.

 

 

Contributor II
Posts: 17
Registered: ‎04-25-2017
#6 of 13 5,670

Re: Centrify does not start after AIX Update to TL5

Hour and a half later...


Ran centrifydc stop on a different SSH session because I wanted to see if the system would spit out on the original session any messages once I stopped whatever was running. Usually breaking/stopping the hung prompt did not print anything.

 

Lo and behold. Original session did have something to say after all.

Error: Failed to start adclient within given wait time (in seconds) - 60
Could not communicate with adclient.
Initializing cache
Exception during cache load ipc socket connect: No such file or directory
.join failed


Error detected.

Error detected. More information may be found in the logfile
(location is /var/log/centrifydc-install.log).
Exiting ...

The install log mentioned ended with an ERROR CODE: 26.

 

The following process was removed after the centrifydc stop was called.

root 1311156 1573424   0 11:54:15  pts/0  0:00 /usr/share/centrifydc/libexec/adjoin -p * -n psa -c OU=Servers,DC=PG,DC=local -w pg.local
Centrify Guru I
Posts: 2,388
Registered: ‎07-26-2012
#7 of 13 5,663

Re: Centrify does not start after AIX Update to TL5

@PFS37273,

 

Thanks for the heads-up.

 

We have some folks in paralell looking at this.  We'll keep this thread updated.

 

R.P

Want to learn more about practical Centrify examples? Check out my blog at http://centrifying.blogspot.com
Follow Centrify:
Centrify
Posts: 7
Registered: ‎03-23-2017
#8 of 13 5,655

Re: Centrify does not start after AIX Update to TL5

[ Edited ]

We would also like to request you turn on debug and collect debug logs, it will be very helpful for engineering to investigate this issue.

On the Centrify Unix server, as root or sudo, please run the following commands:

1. Turn on Centrify debug:
# /usr/share/centrifydc/bin/addebug on
# /usr/share/centrifydc/bin/addebug clear
Make sure /var/log/centrifydc.log is growing in size.

2. Reproduce the issue. <restart adclient>

3. Collect adinfo:
# adinfo -t

4. Turn off debug:
# /usr/share/centrifydc/bin/addebug off


Please attach the following:
a) /var/centrify/tmp/adinfo_support.tar.gz


Thank you.

Contributor II
Posts: 17
Registered: ‎04-25-2017
#9 of 13 5,554

Re: Centrify does not start after AIX Update to TL5

Done. Files collected and sent.

Centrify
Posts: 7
Registered: ‎03-23-2017
#10 of 13 5,535

Re: Centrify does not start after AIX Update to TL5

Got it, thank you!