Monday, December 29, 2008

The end of vinyl records

Vinyl records have been there for ages. My dad still has a couple of old records but he has no turntable anymore to play them. This is because _I_ claimed his turntable a long time ago, which he thought had become old fashioned.
I started buying vinyl records about 12 years ago. I remember saving all my pocket money and buying the latest records with it, instead of visiting pubs with friends. I remember everyone thinking vinyl records belonged to the past and MP3's and CD's were taking over. In fact, this was not true. DJ's playing at parties and clubs never stopped buying records and from my experience, about 80% of their music came from plain old vinyl and a tiny 20% from other media like CD's or MP3's.
The reasoning behind this is simple, if you wanted something new, something not popular (yet), you bought it on vinyl. If it became popular, some record company would probably be so kind to put it on some CD and sell it. But then again, ages, if not decades ( :) ) could pass between the song only being available on vinyl and the song being available to the public on CD (or MP3).
From my experience, playing vinyl records was also much easier then playing CD's or other media. The CD disappears in a box, if you're playing it, and there's no way you can touch it or fiddle with it unless you're using some stupid buttons which can do fancy stuff, like bending the pitch, or playing it reverse.
Companies building CD players for the DJ market (like Pioneer, Denon and others) have done their very best to build something that comes very close to a vinyl experience. Pioneer's CDJ-1000, for example, comes very close, but people used to vinyl know this is just fake.
Serato and Traktor have looked at this from a different perspective. Suppose we have a record playing a wave only understood by software (called time-coded vinyl), running on a PC or Apple, and then this software would sync an MP3 or any other media to this wave then we could bring together the best of both worlds. DJ's can still use vinyl records and they could use time-coded vinyl together with their collection of MP3's (which saves your back if you can't find someone to carry your cases).
I used this software the other night and I must say, even though it is not real vinyl, it feels very much the same. The only thing I missed were 400 large and heavy records I used to browse through, in search of the next great song to play. Instead, I just had to type in parts of the name of the artist or parts of the name of the song, and then select it.
This, my dear friends, could mean the end of true vinyl records. Lots of small shops, that survived on selling vinyl, already had to close down and go bankrupt. Only the larger and online stores like juno seem to be able to survive for now, but I wonder how long it will take.
If you want something new today, something not popular (yet), chances are that this song won't be available on vinyl, but available on MP3 instead.
I like buying records and I like spending a whole afternoon in a record shop, sniffing through a large collection just to find some great song I don't already have. I guess I am becoming old fashioned now, getting old, being surpassed by young people with fancy software. Is this a midlife crisis?

EDIT: As seen in a comment by someone reading this blog, one of Belgium's biggest dance stores (USA Import) is closing down. You can read all about it here.
This hurts :(

Sunday, December 28, 2008

KDE4: Panel artifacts when using Thunderbird or OOo using NVIDIA drivers

NVIDIA has had the best drivers for Linux users for several years. Since early releases of KDE 4, however, some nasty artifacts appeared when running some Gnome based applications (like Open Office and Thunderbird) on systems with NVIDIA cards. These bugs have been reported to NVIDIA but haven't been addressed until now. According to this very long thread, the latest beta drivers from NVIDIA (180.x) the nasty artifacts should have disappeared. I compiled and installed this latest driver and indeed, the panel corruption when, for example, composing a new mail in Thunderbird does no longer appear. Finally, people running KDE 4 with NVIDIA cards are no longer nvidia-victims :)

Saturday, December 27, 2008

Easy backups on Linux, putting it to the test

In an earlier post I was talking about how I've been using mondo as preferred backup solution for quite some time. However, I have never used it to restore stuff, since I've never broken any install before and hardware hasn't failed on me yet.
Since I wanted to replace the smaller 10GB hard drive, I've installed in the Wyse, with a bigger 500GB one, this would be an ideal case to try and restore everything with mondo.
So I created a backup of the 10GB drive with mondo to an NFS share on another PC (using the script posted earlier). Next, burned the resulting ISO to a CD and finally booted from the CD and interactively restored my data.
mondo has 2 main restore modes; interactive and nuke. Since my new drive was bigger than the original drive, mondo suggested to drop to interactive mode, which allowed me to create a whole new partition scheme (which is nice). The only thing that failed was grub installing itself on the master boot record. I'm not sure why, maybe a bug in the ancient mondo that ships with Etch, who knows. I had to install grub using knoppix and then the system booted just fine.

Wednesday, December 24, 2008

Easy backups on Linux

Ubuntu has been a fantastic operating system for my desktop and has been my preferred Desktop OS for more than 3 years now. The cool thing about Ubuntu is it's Debian based. So you install once and dist-upgrade to every new release without ever having to reinstall the whole thing. The downside is a dist-upgrade in Ubuntu sometimes fails leaving you with a very b0rken system. That's why I stick with LTS releases for now, because I have little spare time to reinstall my desktop every few months or so. Nevertheless there are brave people out there always keeping track of the latest, more or less stable, version and don't mind the dist-upgrade problems.
Anyway if your install is broken after a dist-upgrade (or any upgrade) it's always nice to have a backup at hand. In search of the ideal, most flexible, backup tool out there I found mondo. So, people, stop asking for a backup tool for Linux on forums, this one will suite your needs.
Unlike some other tools, mondo creates a backup of a live system. It is also capable of creating a set of CD's or DVD's which are bootable and will help you to easily restore your data.
mondo comes with an ncurses based UI to help you easily create backups. I still like the command line version though, so here's my script:
#!/bin/sh
BACKUP_HOME=/home/bu_operator/bkp
BACKUP_TEMP=/home/bu_operator/tmp
BACKUP_PATH=$BACKUP_HOME/`date +%Y%m%d`_myHost
BACKUP_EXCLUDE='/mnt /dev /proc /tmp /home/bu_operator'

mkdir -p $BACKUP_PATH
mondoarchive -Oi -9 -s 4200m -d $BACKUP_PATH -E "$BACKUP_EXCLUDE" -T $BACKUP_TEMP -S $BACKUP_TEMP

The above script will create a live backup (compressed) of the whole system on DVD ISO images. The BACKUP_EXCLUDE variable contains some volatile folders we don't want to be included in the backup.
Restoring the backup is easy. Burn the ISO images on a CD/DVD (or set of CD's/DVD's), boot from the first CD/DVD, select RESTORE and go grab a cup of coffee while mondo is doing the hard work.

Installing the Wyse 6: adding LDAP and Samba for centralized user management

I like to have my users, that will be logging in on different types of machines on the network, stored in a central place. The ideal solution for this is an LDAP repository and configuring Samba to talk to this repository and configure it as a Primary Domain Controller.
This solution will enable users running Linux to authenticate using pam_ldap and users running Windows (NT and up) authenticate against a Samba Domain Controller. Linux users can also join their machines on the Samba domain, and create shares accessible to other users known by the Domain Controller.
This setup caused me a second headache, since the Samba server running on Ubuntu Hardy could not join the Samba Domain Controller running on Etch. This is because there is a compatibility issue between Samba 3.0.28a (Hardy) and Samba 3.0.24a (Etch). I had to fetch some more recent source packages here and compile and build them on Etch. This is not a procedure for the faint of heart, so if you haven't upgraded to Lenny yet, now is the time (at the time of this writing, Lenny has Samba 3.2.5).
I was planning on providing a step by step procedure to install and configure all of the necessary stuff, but apparently, someone already did. Following these instructions to the letter, will get you up and running in no time. The only thing you have to do is add these lines to libnss-ldap.conf:

bind_policy soft
nss_reconnect_tries 3
nss_reconnect_sleeptime 1
nss_reconnect_maxconntries 3

This is necessary, because otherwise the machine will try forever finding accounts in the LDAP server when it boots and is populating /dev. Populating /dev happens very early in the boot stage when no network or LDAP server is started yet.
I also created an additional LDAP user (next to the admin user) that is only capable of reading entries (instead of reading _and_ writing) as suggested here. This user is called nss and is used to configure pam_ldap on the client Linux machines.
My pam configuration file are also a bit different, so here they are for completeness:

common-session


session required pam_mkhomedir.so skel=/etc/skel umask=0077
session optional pam_ldap.so

common-password


password required pam_passwdqc.so
password sufficient pam_ldap.so
password required pam_unix_passwd.so use_first_pass md5 shadow

common-account


account sufficient pam_ldap.so
account required pam_unix_acct.so

common-auth


auth sufficient pam_ldap.so
auth required pam_unix_auth.so use_first_pass

Please note that messing with pam modules could be dangerous and could lead to a system where you're unable to login. The above configuration definitely works, but it's always safe to have backups of your original pam files and have a bootable medium (CD, DVD, floppy, USB) at hand.

If you want to have a graphical user interface to be able to browse the LDAP you can use Apache Directory Studio (advanced but heavyweight application) or LDAP Browser (basic lightweight application). Both are Java based, so should run on anything that provides a Sun JDK or JRE.

If you find errors in this post, in fact, if you find errors in any post, please leave a comment or send me a mail.

Tuesday, December 23, 2008

Installing the Wyse 5: configure and install cups

If you're still following these series of posts and installed Debian Etch on your firewall, please go back and install Lenny instead. The problem with Etch is that it has a lot of old, outdated, software packages, containing bugs that _will_ cause you headaches.
While configuring cups should be a breeze, there are a number of bugs requiring you to do some manual interventions on the server side.
First there is a compatibility issue which causes cups to hang when configuring printers using Firefox 3. Due to export regulations, Debian does not ship Open SSL by default, causing cups to hang while generating SSL keys. The new admin interface now requires HTTPS, so you will need to install Open SSL by hand. The default configuration for cups, generated bu Debian's package management, also contains a few flaws.
All the above things are probably resolved in Lenny, so you might want to consider upgrading to Lenny. I chose to stick with Etch and here are the things I did to get cups up and running.
First you need to install the necessary packages:

ii hp-ppd 0.8 HP Postscript Printer Definition (PPD) files
ii hpijs-ppds 2.6.10+1.6.10-3etch1 HP Linux Printing and Imaging - HPIJS PPD fi
ii linuxprinting.org-ppds 20061031-1 linuxprinting.org printer support - PostScri
ii foomatic-db 20061031-1 linuxprinting.org printer support - database
ii foomatic-db-hpijs 20061031-1 linuxprinting.org printer support - database
ii foomatic-filters 3.0.2-20061031-1.2 linuxprinting.org printer support - filters
ii cupsys 1.2.7-4etch6 Common UNIX Printing System(tm) - server
ii cupsys-common 1.2.7-4etch6 Common UNIX Printing System(tm) - common fil
ii libcupsimage2 1.2.7-4etch6 Common UNIX Printing System(tm) - image libs
ii libcupsys2 1.2.7-4etch6 Common UNIX Printing System(tm) - libs
ii libcupsys2-dev 1.2.7-4etch6 Common UNIX Printing System(tm) - developmen
ii openssl 0.9.8c-4etch3 Secure Socket Layer (SSL) binary and related
ii hplip 1.6.10-3etch1 HP Linux Printing and Imaging System (HPLIP)
ii hplip-data 1.6.10-3etch1 HP Linux Printing and Imaging - data files
The above list contains the packages I installed to get cups up and running. I am not sure this is an exhaustive list, but it should be more or less complete.
After installing the necessary packages, we need to tell cups to accept calls from the internal network, so we're able to configure it using its web interface. Below is cupsd.conf containing all my modifications.
# Show troubleshooting information in error_log.
LogLevel debug
SystemGroup lpadmin
# Allow remote access
Port 631
Listen /var/run/cups/cups.sock
# Enable printer sharing and shared printers.
Browsing On
BrowseOrder allow,deny
BrowseAllow @LOCAL
BrowseAddress @LOCAL
DefaultAuthType Basic
<Location />
# Allow shared printing and remote administration...
Order allow,deny
Allow @LOCAL
</Location>
<Location /admin>
AuthType Basic
AuthClass System
Encryption Required
# Allow remote administration...
Order allow,deny
Allow @LOCAL
</Location>
<Location /admin/conf>
AuthType Basic
Require user @SYSTEM
# Allow remote access to the configuration files...
Order allow,deny
Allow @LOCAL
</Location>
<Policy default>
<Limit Send-Document Send-URI Hold-Job Release-Job Restart-Job Purge-Jobs Set-Job-Attributes Create-Job-Subscription Renew-Subscription Cancel-Subscription Get-Notifications Reprocess-Job Cancel-Current-Job Suspend-Current-Job Resume-Job CUPS-Move-Job>
Require user @OWNER @SYSTEM
Order deny,allow
</Limit>
<Limit Pause-Printer Resume-Printer Set-Printer-Attributes Enable-Printer Disable-Printer Pause-Printer-After-Current-Job Hold-New-Jobs Release-Held-New-Jobs Deactivate-Printer Activate-Printer Restart-Printer Shutdown-Printer Startup-Printer Promote-Job Schedule-Job-After CUPS-Add-Printer CUPS-Delete-Printer CUPS-Add-Class CUPS-Delete-Class CUPS-Accept-Jobs CUPS-Reject-Jobs CUPS-Set-Default>
AuthType Basic
Require user @SYSTEM
Order deny,allow
</Limit>
<Limit CUPS-Authenticate-Job>
Require user @OWNER @SYSTEM
Order deny,allow
</Limit>
# Only the owner or an administrator can cancel a job...
<Limit Cancel-Job>
Order deny,allow
Require user @OWNER @SYSTEM
</Limit>
<Limit All>
Order deny,allow
</Limit>
</Policy>
Printcap /var/run/cups/printcap

Next, create a folder called ssl underneath /etc/cups to hold the ssl keys. Generate the keys, by running the following command:

openssl req -new -x509 -nodes -days 365 -out server.crt -keyout server.key

underneath /etc/cups/ssl
Now stop and start the cups daemon to let the changes have effect.
You should be able now to login and configure your printer. As I already said, don't use Firefox 3, but use another browser like safari or konqueror instead.
Below screenshot shows the completed configuration of my printer.

Installing the Wyse 4: configure the firewall

The Wyse will be used as my gateway to the internet. There are a lot of out of the box Linux (and BSD) firewall distributions out there. Some of them run from a bootable CD, others can even run from a single floppy. The problem with these distributions is they don't have all the necessary software I need. Next to the firewall software, I will need the following additional services:
  • samba: to use as server for potential windows machines on the network
  • LDAP: to serve as central storage for user accounts and their metadata
  • cups: to use as centralized printing server for all kinds of machines on the network
  • other services I might have forgotten (like subversion, squid, dansguardian, ....)
I have to admit, though, that Endian comes close to what I need, but I just like to do stuff all myself :)
My old Etch firewall was running some custom firewall script I created by hand, but this time, I wanted to use a tool called fwbuilder to create the script.
fwbuilder is a GUI available for most popular OS'es out there (including Mac OS X), capable of generating firewall scripts for a variety of platforms. Platforms include Linux (even old kernels), BSD, CISCO, ... . It also has some pre built templates, to get you started quickly.
So, first you need to download and install fwbuilder for your OS. In Ubuntu (or any Debian based distro), just type apt-get install fwbuilder to download and install the software.
Fire up fwbuilder and create a new object file, which I called home.fwb. Now, right click on Firewalls and select new Firewall from the pop-up menu.
In the next screen, select or enter the following:
  • name: your firewall its name
  • firewall software: select iptables if your firewall is running Linux 2.4 or higher
  • firewall OS: select Linux 2.4/2.6
Be sure to also check Use preconfigured template for firewall objects to get a basic set of rules.
In the next screen, I selected fw template 2 since this one suits my needs. Now click finish.
That's it. You now have a set of rules for a machine running Linux 2.4 or higher with 2 network cards. eth0 should be connected to the internet and has a dynamically assigned network address. eth1 should be connected to the internal network and has a static assigned address. By default, this is 192.168.1.1/255.255.255.0, which I changed to 192.168.1.254/255.255.255.0. You can change this underneath User -> Firewalls -> Your Firewall -> inside -> eth1:ip.
By default, access from the internal network is limited to DNS queries and SSH (Rule 2). Access from the firewall to the internet is limited to DNS queries only (Rule 5). We will need to change this, because we need to be able to upgrade Debian every now and then and we want to have access from the internal network to some of the services mentioned above.
The screenshot below shows my modifications. Rule 5 now allows requests to ntp(to be able to synchronize our clock) and HTTP (to be able to fetch new Debian packages from the internet). Rules 6 and 7 make sure that our logs are not flooded by broadcasts and multicasts from the outside world. Rule 2 was changed, as I already said, to allow access to additional services running on the firewall.

To generate an iptables script from the rules we just created, just select Rules -> Compile. If you transfer the script to the firewall, make it executable (chmod +x) and launch it, you should be able to connect from any machine connected to eth1 with the internet.
Below are some additional scripts I created to start and stop the firwall from /etc/init.d.
The firewall startup script (/etc/init.d/firewall):
#! /bin/sh
#

set -e

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="firewall service"
NAME="firewall"
SCRIPT_START="/etc/init.d/rc.start-firewall"
SCRIPT_STOP="/etc/init.d/rc.flush-firewall"

# Gracefully exit if the package has been removed.
test -x $SCRIPT_START || exit 0
test -x $SCRIPT_STOP || exit 0

#
# Function that starts the daemon/service.
#
d_start() {
$SCRIPT_START
}

#
# Function that stops the daemon/service.
#
d_stop() {
$SCRIPT_STOP
}

case "$1" in
start)
echo "Starting $DESC: $NAME"
d_start
;;
stop)
echo "Stopping $DESC: $NAME"
d_stop
;;
restart|force-reload)
#
# If the "reload" option is implemented, move the "force-reload"
# option to the "reload" entry above. If not, "force-reload" is
# just the same as "restart".
#
echo "Restarting $DESC: $NAME"
d_stop
sleep 1
d_start
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
exit 1
;;
esac

exit 0

The script referenced in the above script to start the firewall (/etc/init.d/rc.start-firewall) is the script created by fwbuilder.
The script referenced in the above script to stop the firewall (/etc/init.d/rc.flush-firewall):
#!/bin/sh
#
# Stop firewall.
#

. /etc/default/firewall

log "Flushing firewall ..."

#
# reset the default policies in the filter table.
#
$IPTABLES -P INPUT ACCEPT
$IPTABLES -P FORWARD ACCEPT
$IPTABLES -P OUTPUT ACCEPT

#
# reset the default policies in the nat table.
#
$IPTABLES -t nat -P PREROUTING ACCEPT
$IPTABLES -t nat -P POSTROUTING ACCEPT
$IPTABLES -t nat -P OUTPUT ACCEPT

#
# reset the default policies in the mangle table.
#
$IPTABLES -t mangle -P PREROUTING ACCEPT
$IPTABLES -t mangle -P OUTPUT ACCEPT

#
# flush all the rules in the filter and nat tables.
#
$IPTABLES -F
$IPTABLES -t nat -F
$IPTABLES -t mangle -F
#
# erase all chains that's not default in filter and nat table.
#
$IPTABLES -X
$IPTABLES -t nat -X
$IPTABLES -t mangle -X

log "... flushed"

The common script used for logging and constants (/etc/default/firewall):
#!/bin/sh
#
# Firewall resources.
#

LOG_FILE="/var/log/firewall.log"

DEBUG="OFF"
debug () {
[ $DEBUG == "ON" ] && echo "[$0]: $1" | $TEE -a $LOG_FILE
}

log () {
echo "[$0]: $1" | $TEE -a $LOG_FILE
}

export PATH="/sbin:/bin:/usr/sbin:/usr/bin"

# Check if all the commands are there.
#
alias which='type -p'
IPTABLES=`which iptables` || IPTABLES="/sbin/iptables"
IFCONFIG=`which ifconfig` || IFCONFIG="/sbin/ifconfig"
IPSC=`which ipsc` || IPSC="/usr/bin/ipsc"
AWK=`which awk` || AWK="/usr/bin/awk"
SED=`which sed` || SED="/bin/sed"
TEE=`which tee` || TEE="/usr/bin/tee"

REQCMDS="$IPTABLES $IFCONFIG $AWK $SED $TEE"
for i in $REQCMDS; do
if [ -x $i ]; then
debug " found $i"
else
echo "I need $i for executing this script, bailing out."
exit 0
fi
done

# Get the IP address for a specified interface. IP is returned in $IP.
#
get_ip_for_interface () {
IP=""
IP=`$IFCONFIG $1 2>/dev/null \
| $AWK '/inet addr:/ {print $2}' \
| $SED 's/addr://'`
debug "found IP: $IP for $1"
}

# Get the subnet mask for a specified interface. Mask is returned in $MASK.
#
get_mask_for_interface () {
MASK=""
MASK=`$IFCONFIG $1 2>/dev/null \
| $AWK '/ Mask:/ {print $4}' \
| $SED 's/Mask://'`
debug "found MASK: $MASK for $1"
}

# Get the net for a specified interface. Net is returned in $NET.
#
get_net_for_interface () {
NET=""
NET=`$IPSC -i $1 2>/dev/null \
| $AWK '/Network address:/{print $3}'`
debug "found NET: $NET for $1"
}

In the next posts, we will get our hands dirty and start configuring cups, LDAP and samba.

Monday, December 01, 2008

Installing the Wyse 3: dhcp server and client

In a previous post I showed how to install a Domain Name Relay Daemon to serve as a caching DNS server. Since the firewall will get its address from the DHCP server from your ISP, there's one small configuration we still need to do.
In /etc/dhcp3/dhclient.conf we need to uncomment (or add, if its not there) the following line:

prepend domain-name-servers 192.168.1.254;

This will make sure the firewall itself will query the name server on the firewall to resolve host names on the local network and the Internet.
The firewall should also run a DHCP server for the internal network. The internal network addresses range from 192.168.1.1/255.255.255.0 to 192.168.1.253/255.255.255.0. So, first we need to install a DHCP server:

# apt-get install dhcp3-server

Next, use the following configuration for /etc/dhcp3/dhcpd.conf:

allow bootp;
ddns-update-style none;
subnet 192.168.1.0 netmask 255.255.255.0 {
default-lease-time 600;
max-lease-time 7200;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.1.255;
option domain-name-servers 192.168.1.254;
option domain-name "earth";
option routers 192.168.1.254;
group {
host internalhost {
hardware ethernet 11:11:11:11:11:11;
fixed-address 192.168.1.1;
}
}
pool {
range 192.168.1.10 192.168.1.20;
}
}

The above configuration will configure the DHCP server to serve addresses from 192.168.1.10/255.255.255.0 to 192.168.1.20/255.255.255.0. This will allow any PC to connect to the local network and get an address.
We also configured a host, called internalhost to receive a fixed address (192.168.1.1), since we want to add this fixed address to our dnrd configuration as well.
The DHCP server will also propagate its own address to the clients as preferred name server.
The bootp flag is used to be able to net-boot some old UNIX machines I have (a Sun Ultra and an HP 712/60). For now, no boot images are configured or served, but this will change in the future.
In a next post, we will start configuring the firewall itself using a GUI called fwbuilder and iptables.