Showing results for 
Search instead for 
Do you mean 
Reply
Frequent Visitor
torstenn
Posts: 5
Registered: ‎06-01-2011

NFS server problem with rpc.svcgssd

I have followed the guide:
http://www.centrify.com/blogs/tomkemp/nfsv4_security_via_kerberos.asp

Step 4: Join the AD Domain: adjoin centrify.dev -z nfszone
I'm joining the machine lintst1.nfit.au.dk to the domain ad.nfit.au.dk
using my ad-account torstenn:
adjoin --user torstenn --name lintst1 --force -w ad.nfit.au.dk
I cannot use -z nfszone parameter since this is the Express version of
Centrify.

When I follow step 7:
Edit /etc/sysconfig/nfs and set: SECURE_NFS="yes"
and restart NFS (step 8) I get the error:
Shutting down RPC svcgssd:                                 [FAILED]
Starting RPC svcgssd:                                      [FAILED]

[root@lintst1 ~]# /usr/sbin/rpc.svcgssd -f -v
ERROR: GSS-API: error in gss_acquire_cred(): Unspecified GSS failure.
Minor code may provide more information - No principal in keytab matches
desired name
Unable to obtain credentials for 'nfs'
unable to obtain root (machine) credentials
do you have a keytab entry for nfs/<your.host>@<YOUR.REALM> in
/etc/krb5.keytab?

[root@lintst1 ~]# klist -ke /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
----
--------------------------------------------------------------------------
   6 nfs/lintst1.ad.nfit.au.dk@AD.NFIT.AU.DK (AES-256 CTS mode with
96-bit SHA-1 HMAC)
   6 nfs/lintst1.ad.nfit.au.dk@AD.NFIT.AU.DK (AES-128 CTS mode with
96-bit SHA-1 HMAC)
   6 nfs/lintst1.ad.nfit.au.dk@AD.NFIT.AU.DK (ArcFour with HMAC/md5)
   6 nfs/lintst1.ad.nfit.au.dk@AD.NFIT.AU.DK (DES cbc mode with RSA-MD5)
   6 nfs/lintst1.ad.nfit.au.dk@AD.NFIT.AU.DK (DES cbc mode with CRC-32)
   6 nfs/lintst1@AD.NFIT.AU.DK (AES-256 CTS mode with 96-bit SHA-1 HMAC)
   6 nfs/lintst1@AD.NFIT.AU.DK (AES-128 CTS mode with 96-bit SHA-1 HMAC)
   6 nfs/lintst1@AD.NFIT.AU.DK (ArcFour with HMAC/md5)
   6 nfs/lintst1@AD.NFIT.AU.DK (DES cbc mode with RSA-MD5)
   6 nfs/lintst1@AD.NFIT.AU.DK (DES cbc mode with CRC-32)

Maybe there should be a nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK entry?


Please use plain text.
Centrify
Centrify
Fel
Posts: 631
Registered: ‎07-06-2010

Re: NFS server problem with rpc.svcgssd

 

Most likely that's the problem.

 

When joining, if youre UNIX DNS domain name (ie. nfit.au.dk) does NOT match your Active Directory DNS domain name (AD.NFIT.AU.DK) you should always make sure you join the domain with an alias for the machine's fully qualified domain name.  This will assure that the appropriate SPNs for all of the principals are created.

 

Give this a try. 

 

1) Leave the domain.  Run "adleave -u torstenn -r -V".

2) Join the domain.  Run "adjoin --user torstenn --name lintst1 -a lintst1.nfit.au.dk --force -w ad.nfit.au.dk"

3) Verify there's an NFS principal in the keytab for lintst1.nfit.au.dk by running "/usr/share/centrifydc/kerberos/bin/klist -kt" as a root equivalent user.

 

Let us know the results. 

 

Also curious what platforms you're trying to get Kerberized NFS to work on.  Just Redhat or other platforms?

 

Regards,

 

 

 

Felderi Santiago
Technical Manager - LATAM
Centrify Corporation
Found my response helpful? Click the Kudos button!
Please use plain text.
Frequent Visitor
torstenn
Posts: 5
Registered: ‎06-01-2011

Re: NFS server problem with rpc.svcgssd

[ Edited ]

It worked - thanks!

 

So far so good:

[root@lintst1 ~]# /usr/share/centrifydc/kerberos/bin/klist -kt
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 07/06/11 11:03:15 nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK


[root@lintst1 ~]# service nfs restart
Shutting down RPC svcgssd:                                 [  OK  ]
Starting RPC svcgssd:                                      [  OK  ]


With regards to your question about platform:

Most likely the NFS servers are going to be RedHat and the clients Ubuntu.

Server platform could be something else - not set in stone.

Client platform most likely will be Ubuntu, but could include other platforms.

Please use plain text.
Frequent Visitor
torstenn
Posts: 5
Registered: ‎06-01-2011

Re: NFS server problem with rpc.svcgssd

On to the client part - using same server for testing.

 

First test (establish default/original behaviour)

[root@lintst1 ~]# rpc.gssd -v -f
Using keytab file '/etc/krb5.keytab'
WARNING: Client not found in Kerberos database while getting initial ticket for principal 'nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK' from keytab 'FILE:/etc/krb5.keytab'
ERROR: No usable machine credentials obtained

exiting on signal 2

 

Fix things on the AD:

"use ADSIEdit from Windows and go to the properties for the client machine and set the userPrincipleName to nfs/nfsclient.centrify.dev"

 

Real/Second test

Testing fails, when trying to mount as root:

 

Terminal1:
[root@lintst1 ~]# rpc.gssd -f -v
Using keytab file '/etc/krb5.keytab'
Using (machine) credentials cache: 'MEMORY:/tmp/krb5cc_machine_AD.NFIT.AU.DK'

 

(Running mount command in Terminal2)

 

handling krb5 upcall
Using keytab file '/etc/krb5.keytab'
WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server lintst1.nfit.au.dk
doing error downcall

Terminal2:
[root@lintst1 ~]# mount -t nfs4 -o sec=krb5p lintst1:/ /mnt/nfs/
mount.nfs4: Permission denied


Please use plain text.
Advisor
tonyp
Posts: 19
Registered: ‎07-08-2011

Re: NFS server problem with rpc.svcgssd

 

Torsten,

 

I am trying to follow the same blog by Tom Kemp. Like you, I am waiting for a response. Is this problem due to a lack of zoning with Centrify Express?

 

Tony P.

Please use plain text.
Centrify
Centrify
Fel
Posts: 631
Registered: ‎07-06-2010

Re: NFS server problem with rpc.svcgssd

 

Hi,

 

Can you outline which set of steps you ran on the client side and which versions of nfs-utils you have?

 

I also want to make sure everyone understands that Centrify's job for this use case starts and ends at joining the UNIX system to AD and to provide a robust and automatically managed Kerberos environment across all of the UNIX and Linux systems we support.

 

Also keep in mind that we've tested Kerberized NFS only on Redhat and not on any other Operating Systems at this point. We've done our best to provide a set of instructions that should work, but you should also be aware that the NFS stack across different OSes works differently.  I've personally seen various cases where the NFS stack doesn't work correctly across different OSes, hence my question to you about which OSes you're using.  I've also seen many customers that have gotten this to work.

 

With all of that said, we as Kerberos "experts" :), we'll do our best to help. Looks like everything on the server side is working OK.  On the client, can you please detail what steps you followed thus far and what the version of nfs-utils is?  Did you notice that depending on the version of nfs-utils, the set of steps differs?

 

Regards,

 

Felderi Santiago
Technical Manager - LATAM
Centrify Corporation
Found my response helpful? Click the Kudos button!
Please use plain text.
Frequent Visitor
torstenn
Posts: 5
Registered: ‎06-01-2011

Re: NFS server problem with rpc.svcgssd

[ Edited ]

Hi,

 

I appreciate all the help I get - I am a novice in Kerberos, AD and NFS, which is probably not the best skill-set to get this working ;-) But I am learning as I go along.

 

I am testing this on a "live" AD - maybe I should change that, but so far no major changes on the AD side - I am just dependent on others to do the changes for me, and they are on holliday now :-(

 

I have a few test machines - I have primarily worked with RedHat machines. Actually I just want it to work on one machine. Using lintst1 as client as well as server.

 

Tested NFSv4 without kerberos - this worked.

 

The Kerberos/NFS-server side is ok - and the client-side has the version-dependent step 5.

I am running version 1.0.9-50:

[root@lintst1 ~]# rpm -q nfs-utils
nfs-utils-1.0.9-50.el5
The AD is a version 2008, so I got one of the AD-admins to

set the userPrincipleName for the machine to nfs/lintst1.nfit.au.dk

 

I am running the mount command as a local root, and kerberos complains

about uid 0

 

I have changed /etc/exports from

/a/home/      <ip>(rw,sync,fsid=0)

to

/a/home/        gss/krb5p(rw,sync,fsid=0)
to get kerberos enabled nfs-mount

 

In the end, I would like AD-users to mount their home-dir using automount on Ubuntu - maybe even get something like a roaming profile for linux- and mac-laptops.

Please use plain text.
Advisor
tonyp
Posts: 19
Registered: ‎07-08-2011

Re: NFS server problem with rpc.svcgssd

Fel,

 

I am using Centos 5.6 on both server and client with both machines joined to the AD using Centrify Express. Like Torsten, I am using nfs-utils 1.0.9-50.el5 on Centos. I have made sure the userPrincipalName for the client contains

 

nfs/bio878.bham.ac.uk

 

AD domain = ADF.BHAM.AC.UK AD level = 2008

client machine FQDN = bio878.bham.ac.uk

 

Regards,

 

Tony P.

 

Please use plain text.
Centrify
Centrify
Fel
Posts: 631
Registered: ‎07-06-2010

Re: NFS server problem with rpc.svcgssd

Hi,

 

I decided to go through the steps outlined in our Blog http://www.centrify.com/blogs/tomkemp/nfsv4_security_via_kerberos.asp to see if I could get an NFS Server and client to use Kerberized NFS v4.

 

The short answer is that I was successful which is good news.  However I had to modify the instructions a bit as well as take an additional step that was not documented.

 

Let me start out with my environment:

1) NFS Server and Client running on: Redhat 5.4

2) Windows Active Directory Environment: Windows 2008 R2, forest and domain forest functional level

3) NFS Server and client are on the same system

4) Linux system was joined to my Active Directory environment, centrify.dt as outlined in the instructions

5) nfs-utils-1.0.9-42.el5

 

Following the directions step by step, the first problem I came across was:

 

Jul 13 11:22:25 rhe5-bos rpc.gssd[19320]: WARNING: Client not found in Kerberos database while getting initial ticket for principal 'nfs/rhe5-bos.centrify.dt@CENTRIFY.DT' from keytab 'FILE:/etc/krb5.keytab'
Jul 13 11:22:25 rhe5-bos rpc.gssd[19320]: ERROR: No usable machine credentials obtained


Since I had a Windows 2008 environment, I set the UPN to be nfs/rhe5-bos.centrify.dt as stated in the instructions.  After receiving this error, I went back and set the UPN to nfs/rhe5-bos.centrify.dt@CENTRIFY.DT.  After doing so, I executed rpc.gssd and voila, no errors returned.  Something that helped me troubleshoot was setting the following two parameters in /etc/sysconfig/nfs which gave additional verbose messages in /var/log/messages

 

# Set to turn on Secure NFS mounts.
SECURE_NFS="yes"
# Optional arguments passed to rpc.gssd. See rpc.gssd(8)
RPCGSSDARGS="-vvv"
# Optional arguments passed to rpc.svcgssd. See rpc.svcgssd(8)
RPCSVCGSSDARGS="-vvv"

 

I also before spending too much time with NFS, I ran the following commands to make sure everything from a Kerberos perpective was setup properly.

 

# /usr/share/centrifydc/kerberos/bin/kinit -k "nfs/rhe5-bos.centrify.dt@CENTRIFY.DT"

# /usr/share/centrifydc/kerberos/bin/klist

 

If you're unable to get a Kerberos ticket, then the UPN was probably not setup correctly.  This is what led me to make the change in UPN in my environment that I mentioned above.

 

Next I tried to mount the directory, mount -t nfs4 -o sec=krb5p nfsserver.centrify.dev:/ /nfsmnt, and received the following error:

 

Jul 13 11:40:11 rhe5-bos rpc.gssd[19324]: WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server rhe5-bos.centrify.dt


Given that I am not an NFS expert, I turned to search engines and came across the following thread:  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521878.  I implemented the suggestion mentioned in Message 8.  Basically the work around requires that the idmapd.conf file to be modified to include a domain/realm mapping line under the General section as seen below.

 

# cat /etc/idmapd.conf
[General]

Verbosity = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
centrify.dt = CENTRIFY.DT

[Mapping]

Nobody-User = nobody
Nobody-Group = nobody

[Translation]
Method = nsswitch


 

Once I implemented this change and tried the mount command, I was able to successfully mount and access the directory.

 

Give some of these tips a try and let us know if you're able to get things to work.

 

Regards,

 

 

 

Felderi Santiago
Technical Manager - LATAM
Centrify Corporation
Found my response helpful? Click the Kudos button!
Please use plain text.
Frequent Visitor
torstenn
Posts: 5
Registered: ‎06-01-2011

Re: NFS server problem with rpc.svcgssd

[ Edited ]

It did not work for me.

Tried to put: nfit.au.dk = AD.NFIT.AU.DK (and variations) in idmapd.conf:

 

[root@lintst1]# cat /etc/idmapd.conf
[General]

Verbosity = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
nfit.au.dk = AD.NFIT.AU.DK

[Mapping]

Nobody-User = nobody
Nobody-Group = nobody

[Translation]
Method = nsswitch

 

I also tried:

ad.nfit.au.dk = AD.NFIT.AU.DK

nfit.au.dk = NFIT.AU.DK

 

 

I seem to be getting a kerberos-key ok:


[root@lintst1 etc]# /usr/share/centrifydc/kerberos/bin/kinit -k "nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK"
[root@lintst1 etc]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK

Valid starting     Expires            Service principal
07/14/11 10:43:47  07/14/11 20:43:47  krbtgt/AD.NFIT.AU.DK@AD.NFIT.AU.DK
        renew until 07/15/11 10:43:47


Kerberos 4 ticket cache: /tmp/tkt0
klist: You have no tickets cached
[root@lintst1 etc]# mount -t nfs4 -o sec=krb5p lintst1:/a/home /mnt/nfs
mount.nfs4: Permission denied

 

I set the verbose options in /etc/sysconfig/nfs, and this is the output to /var/log/messages:

[root@lintst1 etc]# tail /var/log/messages
Jul 14 10:42:21 lintst1 rpc.gssd[29242]: rpcsec_gss: gss_init_sec_context: (major) Unspecified GSS failure.  Minor code may provide more information - (minor) Unknown code krb5 14
Jul 14 10:42:21 lintst1 rpc.gssd[29242]: WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server lintst1.nfit.au.dk
Jul 14 10:47:17 lintst1 rpc.gssd[29242]: rpcsec_gss: gss_init_sec_context: (major) Unspecified GSS failure.  Minor code may provide more information - (minor) Unknown code krb5 14
Jul 14 10:47:17 lintst1 rpc.gssd[29242]: WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server lintst1.nfit.au.dk
Jul 14 10:51:29 lintst1 adclient[3200]: WARN  <fd:10 ldap search> base.bind.cache LDAP search cn=nismaps,DC=ad,DC=nfit,DC=au,DC=dk:(objectClass=*) threw unexpected exception: ldap search, no such object : No such object : 0000208D: NameErr: DSID-0310020A, problem 2001 (NO_OBJECT), data 0, best match of:        'DC=ad,DC=nfit,DC=au,DC=dk'  matched DC=ad,DC=nfit,DC=au,DC=dk
Jul 14 10:51:29 lintst1 adnisd[3509]: WARN  <main> adnisd No NIS maps found on server dc2.ad.nfit.au.dk
Jul 14 10:51:29 lintst1 adclient[3200]: WARN  <fd:10 get object> base.schema nextItem(Person) got exception 'ADAttribute '_server' is empty' -- moving to next domain


PS I also tried the tip in Message #13 from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521878

and set "allow_crypto_weak = true" in /etc/krb5.conf - did not help.

 

/var/log/messages with rpc.gssd -v -f running, while trying to mount in other terminal:

Jul 14 11:47:34 lintst1 rpc.gssd[3443]: handling krb5 upcall
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: Using keytab file '/etc/krb5.keytab'
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: INFO: Credentials in CC 'MEMORY:/tmp/krb5cc_machine_AD.NFIT.AU.DK' are good until 1310672836
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: using MEMORY:/tmp/krb5cc_machine_AD.NFIT.AU.DK as credentials cache for machine creds
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: using environment variable to select krb5 ccache MEMORY:/tmp/krb5cc_machine_AD.NFIT.AU.DK
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: creating context using fsuid 0 (save_uid 0)
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: creating tcp client for server lintst1.nfit.au.dk
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: creating context with server nfs@lintst1.nfit.au.dk
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: rpcsec_gss: gss_init_sec_context: (major) Unspecified GSS failure.  Minor code may provide more information - (minor) Unknown code krb5 14
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: WARNING: Failed to create krb5 context for user with uid 0 for server lintst1.nfit.au.dk
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: WARNING: Failed to create krb5 context for user with uid 0 with credentials cache MEMORY:/tmp/krb5cc_machine_AD.NFIT.AU.DK for server lintst1.nfit.au.dk
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server lintst1.nfit.au.dk
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: doing error downcall
Jul 14 11:47:34 lintst1 rpc.gssd[3443]: destroying client clnt1
Jul 14 11:47:36 lintst1 rpc.gssd[3443]: destroying client clnt0


There is also this stuff in /var/log/messages:

Jul 14 11:47:10 lintst1 adclient[3201]: INFO  <main> daemon.main adclient starting with no arguments
Jul 14 11:47:10 lintst1 adclient[3201]: INFO  <main> daemon.main uname: Linux lintst1.nfit.au.dk 2.6.18-194.32.1.el5 #1 SMP Mon Dec 20 10:52:42 EST 2010 x86_64
Jul 14 11:47:10 lintst1 adclient[3202]: INFO  <main> daemon.main Starting /usr/sbin/adclient. lrpc pipe='/var/centrifydc/daemon', lrpc2 pipe='/var/centrifydc/daemon2'
Jul 14 11:47:10 lintst1 adinfo[3204]: INFO  lrpc.session process authentication request failed: ipc socket connect: No such file or directory
Jul 14 11:47:10 lintst1 adinfo[3204]: INFO  lrpc.session process authentication request failed: ipc socket connect: No such file or directory
Jul 14 11:47:10 lintst1 adclient[3202]: INFO  <main> base.adagent Version: CentrifyDC 4.4.3-424
Jul 14 11:47:10 lintst1 adclient[3202]: INFO  <main> base.adagent DirectAuthorized not enabled on this computer.
Jul 14 11:47:10 lintst1 adclient[3202]: INFO  <bg:krb5.conf> daemon.main Start trusted domain discovery
Jul 14 11:47:10 lintst1 adclient[3202]: INFO  <bg:krb5.conf> daemon.main Trusted domain discovery complete: 2 domains found
Jul 14 11:47:13 lintst1 adbindd[3263]: INFO  samba.adbindd2 adbindd version 1.1 (CentrifyDC-adbindproxy 4.3.1-145) Starting up
Jul 14 11:47:13 lintst1 adbindd[3263]: INFO  samba.adbindd2 Listening for requests, socket = '/tmp/.winbindd/pipe', version = 20
Jul 14 11:47:13 lintst1 adbindd[3263]: INFO  samba.adbindd2 Connect to adclient successfully.
Jul 14 11:47:14 lintst1 rpc.statd[3361]: Version 1.0.9 Starting
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: Using keytab file '/etc/krb5.keytab'
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: Processing keytab entry for principal 'nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK'
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: We will use this entry (nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK)
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: Processing keytab entry for principal 'nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK'
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: We will NOT use this entry (nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK)
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: Processing keytab entry for principal 'nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK'
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: We will NOT use this entry (nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK)
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: Processing keytab entry for principal 'nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK'
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: We will NOT use this entry (nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK)
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: Processing keytab entry for principal 'nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK'
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: We will NOT use this entry (nfs/lintst1.nfit.au.dk@AD.NFIT.AU.DK)
Jul 14 11:47:14 lintst1 rpc.gssd[3443]: Processing keytab entry for principal 'nfs/lintst1.ad.nfit.au.dk@AD.NFIT.AU.DK'
.....

 

It seems a little strange to get a message "We will use this entry ..." followed some "We will NOT use this entry ...",

when it seems to be the same entry. This is follow by a lot of  "We will NOT use this entry ..", where the entry changes.

 

I will try to find some time to read up on kerberos, to hopefully make a bit of sence out of this.

Best regards,

Torsten Nielsen

Please use plain text.