Uptime, QoS, SLA

Found this very interesting  
post and I liked the emphasis on learnings and an iterative process of improvement.
The 99.99 or greater availability promise is a good one for the business but unless I have a rock solid plan for recovery and have meticulously documented past outages, history is bound to repeat.

Add swap space on a running system

I found a few systems that were setup with low swap [= to original RAM] on the system and now all my alerts for swap usage are going off.

On debian the steps to add a regular file for swapping were as follows:

1. make the swap file - create a contiguous file and label it as such



dd if=/dev/zero of=fourGfile count=4000000 bs=1024
chmod 0600 fourGfile
mkswap fourGfile



2. Mount it as a swap partition



swapon -v fourGfile



3. Add it to /etc/fstab



/var/cache/swapfile/fourGfile          none            swap    sw              0       0

This was it and now we have enough swap on the systems,

bash-3.1# free -l
           total       used       free     shared    buffers     cached
Mem:       8180284    7866476     313808          0      84660    7323408
Low:       8180284    7866476     313808
High:            0          0          0
-/+ buffers/cache:     458408    7721876
Swap:      7951784      95304    7856480

NOTE: Swapping on a system is not a good thing for performance but it adds to total virtual memory that a process can access on the system.

NFS exports

Installation and configuration was a snap ; and in 10 minutes we had the filesystem exported and mounted on the target system.

It involved the following:

1. package - nfs-user-server – the description from the
   package had these notes. hasn't been an issue for us so far.

   This package contains all necessary programs to make your Linux machine act as an NFS server, being an NFS daemon (rpc.nfsd), a mount daemon (rpc.mountd).Unlike other NFS daemons, this NFS server runs entirely in user space. This
   makes it a tad slower than other NFS implementations, and also introduces some awkwardnesses in the semantics (for instance, moving a file to a different directory will render its file handle
   invalid). There is currently no support for file locking.

2. Portmap had to be reconfigured to take out the loopback.
   

   • File changed:/etc/default/portmap
     

   • #OPTIONS="-i 127.0.0.1"

3. Edited /etc/exports to add directory to export

   • /mnt/data3 target.mount.host (rw,sync,no_root_squash)

   • no_root_squash is to allow for root on the remote system to
     be root on the exported fs; we needed this to be able to write from
     tape extraction.

4. Restarted nfs-user-server and portmap.

5. showmount -e to confirm exports were successful

Bios upgrade on a Debian system

This one specifically refers to Dell and Debian - which is what we run.
Since Dell supports Windows and RHEL as its standard platforms, it seems like we are on our own.

DRAC firmware upgrade
Bhushan found that the DRAC firmware upgrades can be done using the DRAC interface itself.

1. Login to DRAC web
2. Goto "Media:Virtual Flash"
3. Upload Image file

Prerequisites:
Download the firmware image from dell.com for the system.

CLI - go racadm
the racadm command line - ssh to the DRAC IP and run racadm help fwupdate. This one I haven't run and will update some day in the future.
A very detailed reference to set of steps from Marius are available here.

First a list of references

Starting from where to get the CD image everything has been new to me. So the first place to find was, the download location. This one is for the so called amd64 which Marius educated me was just a common moniker for all 64 bit [including Intel processors].
And have to start this one with a big Thank You to Marius Ducea for showing me around this OS. I have been on FreeBSD most recently @ Y.
I am working in an environment running the etch release (Debian GNU/Linux 4.0)

Some basics that helped me get on firm ground:

1. wrong time zone : I installed the OS and set my server in the wrong timezone. Use /usr/sbin/tzconfig.




• Use ntpdate to keep date in sync with a time server.

Suggest a simple script like this,


#!/bin/sh
/usr/sbin/ntpdate ntp.somedomain.com


Pay attention to the note in the setup - do not query the level 1 NTP servers.



2. wrong hostname: /bin/hostname ; and set it in /etc/hostname, /etc/hosts for permanence across reboot. Good howto in Marius' blog post.
3. Changing IP: ifconfig, and /etc/network/interfaces.
   
4. redundant NIC setup - if you 2 NICs on board then ifenslave [version 2.6] is the way to go. A good howto. We did it slightly differently.
   


• Took out the iface definitions from /etc/network/interfaces
• Added ifenslave lines to /etc/rc.local
• And added sysctl -p in rc.local after ifenslave setup.
  

modprobe bonding mode=active-backup miimon=100
ifconfig bond0 10.1.2.3 netmask  255.255.255.0 up
ifenslave bond0 eth0
ifenslave bond0 eth1   ## or eth2 as the case maybe
route add default gw 10.31.1.254
sysctl -p > /dev/null
exit 0    ### important to come out cleanly for boot sequence





5. Setup your /etc/hosts, /etc/resolv.conf right ; otherwise be prepared for name lookup timeouts. SImple rules to follow there:




• Add local hosts to /etc/hosts - no need to look up name servers for those. NOTE: Does mean you have to maintain it when you change names/IPs


• Find a good name server that is local - normally your ISP can help here. for critical services, good idea is to setup your own dns cache server like djbdns
  


6. Debian support for aptitude is one of the biggest advantages in my opinion - having come from the yinst world at Y.



# apt-get update ## first step after installation to get security patches
# apt-get openssh-server openssh-client sudo ntpdate