evolution of appops

goal of monitoring: no news is good news and...

This post started out in 2011 to summarize the operating system that helped run admob operations successfully as a high availability system. I was part of the team that created and ran these systems and so had a good view of these. And had a part to play in setting direction of the evolution of the system.

how did we get there?

Started simple - built a health checking system that can be trusted to throw an alert at the first sign of trouble. This meant every check will go off if there is a widespread issue, so at the outset we decoupled the alerting from the checking systems. This allowed us to silence alerts enmasse without suppressing alerts.
This helped build confidence in the monitoring system.
Starting from a simple no-news-is-good-news which is trustworthy, we added a time series db (rrd), moved to self-service monitoring and then to event correlation. The job's not done, now the foundation is there to empower developers in the production ecosystem and also help us all comprehend a common language that consists of metrics such as latency, utilization, wait times and timeouts.
In the following there is a detailed sketch of each of these discrete steps that formed the foundation.
Simple lessons that endured along the way - don't optimize prematurely; build for the near future not the super-distant imagined future.

V1: Central Monitoring and Alerting systems

A central server checking health of all servers and services that are required to run the network.

key elements

• ping check - reachable node
• process existence check
• logs check
• port check
• black box test

pros

1. simple to implement
2. a central place to control access- checks and checked nodes
3. easy to snapshot and recover
4. checks the health of the network, albeit from a single point

cons

limited by polling frequency - 300 services 1 per second

single point of failure

can be mitigated by watching from a 3rd party location

V2: Historical data

As number of servers and services increased as the product got more successful. This led to a substantial (20 to 200) increase in servers and by design there was an increase in network distance. There came a need to optimize resource utilization.
The question to answer: What was the behavior last week? Has there been an improvement wi this change?

We worked  to get a timeseries database of system + application events. 

At this time quantile latency metrics entered  admob common parlance.
This meant for all latency, utilization metrics median, 95th, 99th and variance of a time series distribution were being picked out.
Knowing these metrics made for focused resource planning. The key benefit was realized because we were able to question the expectations of performance in situations where a certain resource was getting starved way ahead of the others.

This helped drive a very numbers driven canary process for all new releases - application servers and OS packages could be subjected to the same measuring yardstick.

This allowed us to focus time on seeking out better ethernet drivers for support higher packet rates. And also meant reduction of systems that we didn't need, like software raid on webservers.

V3: Decentralize checks

Passive checks: Decentralize checks; in this version we moved checks pertaining to process (existence, health and logs) to the local machine. The central checks were kept in place for aggregate servers like load balancers, cross correlation rsyslog, database replication and such.

This resulted in 2 wins

1. reduced the traffic between the central server and the leaf nodes - which got over a scaling bottlneck in the monitoring service
2. Allowed us to use the same template for cross datacenter monitoring

V4: High Frequency Event

There timeseries database was becoming a bottleneck for how much data we could collect. At this time we started looking for a solution that could handle higher sampling rates - until this the sampling interval was at 5minutes.

1. rabbitmq for event collection for rapid collection 
2. separate set of consumers to process and store events

Graphite emerged as a winner for visualization and whisper database as an upgrade from rrd...

in conclusion

The work is ongoing and in this time two of the underlying systems were also open sourced thanks to support from google.

• rocksteady - the event correlation framework
• astmob - asset database, a source of truth

logging, but just enough to be useful to troubleshoot

... to making operations engineering a real task

Take for instance a situation where there are reported timeouts on connections to servers - there could be several culprits here.

1. server taking too long to process requests ; resulting in non-acceptance of connections
2. name lookup failure [if client is using names, not ip]
3. network layer drops

Each of these 3 can be rat's hole to debug if you don't know where to start. In every debug operation in a production environment i have started with logs; logging comes at a price and needs centralization for ease of consumption. The key point of this post it to emphasize the need to have awareness inculcated into all forms of development to focus on leaving a usable "rice trail" so these troubleshooting exercises can actually be done after that event occurence.
Basic principles of logging:

• log key entry and exit functions in your application
• seed in time information where it'd be useful - for instance if a connection is critical to the performance of the next step; have a try-catch block that captures return codes.
• always have a timeout on a remote call - i.e. know what you need and how fast [do not rely on the server's sensitivity always] - and its a great piece of feedback to the server
• network flows are important and put that into place ** this area is new to me, more on this later.

Don't :

• log DEBUG/INFO messages on production systems - it eats into precious IO resources

An operations environment aware development engineer is one that has an instinct for these; for the others i am still looking for the quickstart book!

Uptime, QoS, SLA

Found this very interesting  
post and I liked the emphasis on learnings and an iterative process of improvement.
The 99.99 or greater availability promise is a good one for the business but unless I have a rock solid plan for recovery and have meticulously documented past outages, history is bound to repeat.

Add swap space on a running system

I found a few systems that were setup with low swap [= to original RAM] on the system and now all my alerts for swap usage are going off.

On debian the steps to add a regular file for swapping were as follows:

1. make the swap file - create a contiguous file and label it as such



dd if=/dev/zero of=fourGfile count=4000000 bs=1024
chmod 0600 fourGfile
mkswap fourGfile



2. Mount it as a swap partition



swapon -v fourGfile



3. Add it to /etc/fstab



/var/cache/swapfile/fourGfile          none            swap    sw              0       0

This was it and now we have enough swap on the systems,

bash-3.1# free -l
           total       used       free     shared    buffers     cached
Mem:       8180284    7866476     313808          0      84660    7323408
Low:       8180284    7866476     313808
High:            0          0          0
-/+ buffers/cache:     458408    7721876
Swap:      7951784      95304    7856480

NOTE: Swapping on a system is not a good thing for performance but it adds to total virtual memory that a process can access on the system.

NFS exports

Installation and configuration was a snap ; and in 10 minutes we had the filesystem exported and mounted on the target system.

It involved the following:

1. package - nfs-user-server – the description from the
   package had these notes. hasn't been an issue for us so far.

   This package contains all necessary programs to make your Linux machine act as an NFS server, being an NFS daemon (rpc.nfsd), a mount daemon (rpc.mountd).Unlike other NFS daemons, this NFS server runs entirely in user space. This
   makes it a tad slower than other NFS implementations, and also introduces some awkwardnesses in the semantics (for instance, moving a file to a different directory will render its file handle
   invalid). There is currently no support for file locking.

2. Portmap had to be reconfigured to take out the loopback.
   

   • File changed:/etc/default/portmap
     

   • #OPTIONS="-i 127.0.0.1"

3. Edited /etc/exports to add directory to export

   • /mnt/data3 target.mount.host (rw,sync,no_root_squash)

   • no_root_squash is to allow for root on the remote system to
     be root on the exported fs; we needed this to be able to write from
     tape extraction.

4. Restarted nfs-user-server and portmap.

5. showmount -e to confirm exports were successful

Bios upgrade on a Debian system

This one specifically refers to Dell and Debian - which is what we run.
Since Dell supports Windows and RHEL as its standard platforms, it seems like we are on our own.

DRAC firmware upgrade
Bhushan found that the DRAC firmware upgrades can be done using the DRAC interface itself.

1. Login to DRAC web
2. Goto "Media:Virtual Flash"
3. Upload Image file

Prerequisites:
Download the firmware image from dell.com for the system.

CLI - go racadm
the racadm command line - ssh to the DRAC IP and run racadm help fwupdate. This one I haven't run and will update some day in the future.
A very detailed reference to set of steps from Marius are available here.

First a list of references

Starting from where to get the CD image everything has been new to me. So the first place to find was, the download location. This one is for the so called amd64 which Marius educated me was just a common moniker for all 64 bit [including Intel processors].
And have to start this one with a big Thank You to Marius Ducea for showing me around this OS. I have been on FreeBSD most recently @ Y.
I am working in an environment running the etch release (Debian GNU/Linux 4.0)

Some basics that helped me get on firm ground:

1. wrong time zone : I installed the OS and set my server in the wrong timezone. Use /usr/sbin/tzconfig.




• Use ntpdate to keep date in sync with a time server.

Suggest a simple script like this,


#!/bin/sh
/usr/sbin/ntpdate ntp.somedomain.com


Pay attention to the note in the setup - do not query the level 1 NTP servers.



2. wrong hostname: /bin/hostname ; and set it in /etc/hostname, /etc/hosts for permanence across reboot. Good howto in Marius' blog post.
3. Changing IP: ifconfig, and /etc/network/interfaces.
   
4. redundant NIC setup - if you 2 NICs on board then ifenslave [version 2.6] is the way to go. A good howto. We did it slightly differently.
   


• Took out the iface definitions from /etc/network/interfaces
• Added ifenslave lines to /etc/rc.local
• And added sysctl -p in rc.local after ifenslave setup.
  

modprobe bonding mode=active-backup miimon=100
ifconfig bond0 10.1.2.3 netmask  255.255.255.0 up
ifenslave bond0 eth0
ifenslave bond0 eth1   ## or eth2 as the case maybe
route add default gw 10.31.1.254
sysctl -p > /dev/null
exit 0    ### important to come out cleanly for boot sequence





5. Setup your /etc/hosts, /etc/resolv.conf right ; otherwise be prepared for name lookup timeouts. SImple rules to follow there:




• Add local hosts to /etc/hosts - no need to look up name servers for those. NOTE: Does mean you have to maintain it when you change names/IPs


• Find a good name server that is local - normally your ISP can help here. for critical services, good idea is to setup your own dns cache server like djbdns
  


6. Debian support for aptitude is one of the biggest advantages in my opinion - having come from the yinst world at Y.



# apt-get update ## first step after installation to get security patches
# apt-get openssh-server openssh-client sudo ntpdate