Linux: Upgrading Debian woes (1)
So I decided to give my server, running Debian testing, an upgrade. Normally this works like a breeze. I have never encountered any problems during an upgrade, even though I am following testing.
This time was different and things were very b0rken. This is about fixing my first issue: LDAP + libnss + udev.
LDAP
During the upgrade, the script didn't manage to create a decent backup of the running LDAP server. This LDAP is serving users for my home network and is also the backend for my Samba domain controller. So the LDAP was very b0rken. The only fix I could see was repopulating the entire tree, starting from a fresh, empty LDAP. To help me with this, I found this a very interesting and thorough guide. Since the LDAP is also my Samba backend I am using smbldap-tools to maintain users and such. So repopulating the LDAP wasn't such a big task.
Lucky.
udev
What is udev?
udev works entirely in userspace, using hotplug events the kernel sends whenever a device is added or removed from the kernel. Details about the devices are exported by the kernel to the sysfs filesystem at /sys All device naming policy permission control and event handling is done in userspace. devfs is operated from within the kernel. (read).
This thing is started way before my LDAP boots. Why? Because this thing is necessary to build the /dev tree and modprobing drivers, such as drivers for my network card. So udev starts very early in the boot process looking for users and groups to assign. E.g. /dev/audio is assigned to the audio group.
If a user or group is not found local, it tries to search the LDAP, which is ... not there. Before the upgrade udev tried a few times to reach the LDAP, and finally gave up. Now it didn't. It just kept trying to reach the damn LDAP. This link describes the problem I was having (bug #375077). The solution to this problem was to change the bind_policy so it only tries to reach the LDAP 3 times (or less if you want):
Libnss
After messing around a little, I suddenly got the following error:
Now that's what I call a decent error ;) This happened when trying to log in with a user known in the LDAP, but not on the system. After pulling my hair a little and googling a while, I discovered I accidentally messed up the libnss-ldap.conf. This configuration file is necessary for the name switch service to be able to access the ldap. This file contains the root DN and credentials to access/search the LDAP. Therefore this file should be:
Since I must be able to add and change users on this particular machine, the file contains administrator credentials and is readable for root only. So I need the nscd running which was throwing the above error while asking the name switch service for a user existing in the LDAP. Reverting the libnss-ldap.conf to its original state solved my problems. If you store your /etc in a subversion repository, this is easy ;)
In a next post I will be discussing my next problem: "Fixing authentication in Apache 2, after doing an upgrade"
This time was different and things were very b0rken. This is about fixing my first issue: LDAP + libnss + udev.
LDAP
During the upgrade, the script didn't manage to create a decent backup of the running LDAP server. This LDAP is serving users for my home network and is also the backend for my Samba domain controller. So the LDAP was very b0rken. The only fix I could see was repopulating the entire tree, starting from a fresh, empty LDAP. To help me with this, I found this a very interesting and thorough guide. Since the LDAP is also my Samba backend I am using smbldap-tools to maintain users and such. So repopulating the LDAP wasn't such a big task.
Lucky.
udev
What is udev?
udev works entirely in userspace, using hotplug events the kernel sends whenever a device is added or removed from the kernel. Details about the devices are exported by the kernel to the sysfs filesystem at /sys All device naming policy permission control and event handling is done in userspace. devfs is operated from within the kernel. (read).
This thing is started way before my LDAP boots. Why? Because this thing is necessary to build the /dev tree and modprobing drivers, such as drivers for my network card. So udev starts very early in the boot process looking for users and groups to assign. E.g. /dev/audio is assigned to the audio group.
If a user or group is not found local, it tries to search the LDAP, which is ... not there. Before the upgrade udev tried a few times to reach the LDAP, and finally gave up. Now it didn't. It just kept trying to reach the damn LDAP. This link describes the problem I was having (bug #375077). The solution to this problem was to change the bind_policy so it only tries to reach the LDAP 3 times (or less if you want):
bind_policy hard
nss_reconnect_tries 3
nss_reconnect_sleeptime 1
nss_reconnect_maxconntries 3
Libnss
After messing around a little, I suddenly got the following error:
ldap-nss.c:1323: do_init: Assertion `cfg->ldc_uris[__session.ls_current_uri] != ((void *)0)' failed.
Now that's what I call a decent error ;) This happened when trying to log in with a user known in the LDAP, but not on the system. After pulling my hair a little and googling a while, I discovered I accidentally messed up the libnss-ldap.conf. This configuration file is necessary for the name switch service to be able to access the ldap. This file contains the root DN and credentials to access/search the LDAP. Therefore this file should be:
- readable by everyone;
- or readable by root only, but then you'll need the nscd (name switch caching daemon).
Since I must be able to add and change users on this particular machine, the file contains administrator credentials and is readable for root only. So I need the nscd running which was throwing the above error while asking the name switch service for a user existing in the LDAP. Reverting the libnss-ldap.conf to its original state solved my problems. If you store your /etc in a subversion repository, this is easy ;)
In a next post I will be discussing my next problem: "Fixing authentication in Apache 2, after doing an upgrade"
Comments