Application operations

logging, but just enough to be useful to troubleshoot

2008-11-18T08:21:00.000-08:00

... to making operations engineering a real task

Take for instance a situation where there are reported timeouts on connections to servers - there could be several culprits here.

server taking too long to process requests ; resulting in non-acceptance of connections
name lookup failure [if client is using names, not ip]
network layer drops

Each of these 3 can be rat's hole to debug if you don't know where to start. In every debug operation in a production environment i have started with logs; logging comes at a price and needs centralization for ease of consumption. The key point of this post it to emphasize the need to have awareness inculcated into all forms of development to focus on leaving a usable "rice trail" so these troubleshooting exercises can actually be done after that event occurence.
Basic principles of logging:

log key entry and exit functions in your application
seed in time information where it'd be useful - for instance if a connection is critical to the performance of the next step; have a try-catch block that captures return codes.
always have a timeout on a remote call - i.e. know what you need and how fast [do not rely on the server's sensitivity always] - and its a great piece of feedback to the server
network flows are important and put that into place ** this area is new to me, more on this later.

Don't :

log DEBUG/INFO messages on production systems - it eats into precious IO resources

An operations environment aware development engineer is one that has an instinct for these; for the others i am still looking for the quickstart book!

Uptime, QoS, SLA

2008-01-23T22:42:00.000-08:00

Found this very interesting
post and I liked the emphasis on learnings and an iterative process of improvement.
The 99.99 or greater availability promise is a good one for the business but unless I have a rock solid plan for recovery and have meticulously documented past outages, history is bound to repeat.

Add swap space on a running system

2007-11-28T14:48:00.000-08:00

I found a few systems that were setup with low swap [= to original RAM] on the system and now all my alerts for swap usage are going off.

On debian the steps to add a regular file for swapping were as follows:

make the swap file - create a contiguous file and label it as such


dd if=/dev/zero of=fourGfile count=4000000 bs=1024
chmod 0600 fourGfile
mkswap fourGfile

Mount it as a swap partition


swapon -v fourGfile

Add it to /etc/fstab


/var/cache/swapfile/fourGfile          none            swap    sw              0       0

This was it and now we have enough swap on the systems,


bash-3.1# free -l
           total       used       free     shared    buffers     cached
Mem:       8180284    7866476     313808          0      84660    7323408
Low:       8180284    7866476     313808
High:            0          0          0
-/+ buffers/cache:     458408    7721876
Swap:      7951784      95304    7856480

NOTE: Swapping on a system is not a good thing for performance but it adds to total virtual memory that a process can access on the system.

NFS exports

2007-09-13T11:24:00.000-07:00

Installation and configuration was a snap ; and in 10 minutes we had the filesystem exported and mounted on the target system.

It involved the following:

package - nfs-user-server – the description from the
package had these notes. hasn't been an issue for us so far.

This package contains all necessary programs to make your Linux machine act as an NFS server, being an NFS daemon (rpc.nfsd), a mount daemon (rpc.mountd).Unlike other NFS daemons, this NFS server runs entirely in user space. This
makes it a tad slower than other NFS implementations, and also introduces some awkwardnesses in the semantics (for instance, moving a file to a different directory will render its file handle
invalid). There is currently no support for file locking.
Portmap had to be reconfigured to take out the loopback.
Edited /etc/exports to add directory to export
- /mnt/data3 target.mount.host (rw,sync,no_root_squash)
- no_root_squash is to allow for root on the remote system to
  be root on the exported fs; we needed this to be able to write from
  tape extraction.
Restarted nfs-user-server and portmap.
showmount -e to confirm exports were successful

Bios upgrade on a Debian system

2007-08-30T16:13:00.000-07:00

This one specifically refers to Dell and Debian - which is what we run.
Since Dell supports Windows and RHEL as its standard platforms, it seems like we are on our own.

DRAC firmware upgrade
Bhushan found that the DRAC firmware upgrades can be done using the DRAC interface itself.

Login to DRAC web
Goto "Media:Virtual Flash"
Upload Image file

Prerequisites:
Download the firmware image from dell.com for the system.

CLI - go racadm
the racadm command line - ssh to the DRAC IP and run racadm help fwupdate. This one I haven't run and will update some day in the future.
A very detailed reference to set of steps from Marius are available here.

First a list of references

2007-07-27T09:25:00.000-07:00

Starting from where to get the CD image everything has been new to me. So the first place to find was, the download location. This one is for the so called amd64 which Marius educated me was just a common moniker for all 64 bit [including Intel processors].
And have to start this one with a big Thank You to Marius Ducea for showing me around this OS. I have been on FreeBSD most recently @ Y.
I am working in an environment running the etch release (Debian GNU/Linux 4.0)

Some basics that helped me get on firm ground:

wrong time zone : I installed the OS and set my server in the wrong timezone. Use /usr/sbin/tzconfig.

Use ntpdate to keep date in sync with a time server.


#!/bin/sh
/usr/sbin/ntpdate ntp.somedomain.com

do not

wrong hostname: /bin/hostname ; and set it in /etc/hostname, /etc/hosts for permanence across reboot. Good howto in Marius' blog post.
Changing IP: ifconfig, and /etc/network/interfaces.
redundant NIC setup - if you 2 NICs on board then ifenslave [version 2.6] is the way to go. A good howto. We did it slightly differently.

Took out the iface definitions from /etc/network/interfaces
Added ifenslave lines to /etc/rc.local
And added sysctl -p in rc.local after ifenslave setup.

modprobe bonding mode=active-backup miimon=100
ifconfig bond0 10.1.2.3 netmask  255.255.255.0 up
ifenslave bond0 eth0
ifenslave bond0 eth1   ## or eth2 as the case maybe
route add default gw 10.31.1.254
sysctl -p > /dev/null
exit 0    ### important to come out cleanly for boot sequence

Setup your /etc/hosts, /etc/resolv.conf right ; otherwise be prepared for name lookup timeouts. SImple rules to follow there:

Add local hosts to /etc/hosts - no need to look up name servers for those. NOTE: Does mean you have to maintain it when you change names/IPs

Find a good name server that is local - normally your ISP can help here. for critical services, good idea is to setup your own dns cache server like djbdns

Debian support for aptitude is one of the biggest advantages in my opinion - having come from the yinst world at Y.


# apt-get update ## first step after installation to get security patches
# apt-get openssh-server openssh-client sudo ntpdate