Category Archives: Linux

Posts about Linux.

Create Online KVM Backups

Posted on by
Reply

Running eCommerce sites, downtime is considered a bad deal… at the same time, not having backups is even worse. Here is a simple script that I’ve put together to execute an online backup (not a snapshot) of a Virtual Machine managed by KVM without having to shut down the VM.

Create a new file.

$vi snapshot_backup.sh

And copy and paste the following code.

#!/bin/bash
debug=1

function backup_snapshot {
  date=$1
  filename=$2
  if [ `qemu-img info $filename.qcow2 | grep  --count "file format: qcow2"` -eq 1 ] then
    echo "Backing up $filename"
    # create the snapshot
    if [ $debug -eq 1 ] then
      echo "Creating snapshot"
    fi
    qemu-img snapshot -c $date $filename.qcow2
    #copy the snapshot out
    if [ $debug -eq 1 ] then
      echo "Converting snapshot to standalone qcow2 file"
    fi
    qemu-img convert -c -f qcow2 -O qcow2 $filename.qcow2 $filename-$date.qcow2
    #delete the old snapshot
    if [ $debug -eq 1 ] then
      echo "Deleting snapshot"
    fi
    qemu-img snapshot -d $date $filename.qcow2
  else
    echo "Snapshot backups only support qcow2 files"
 fi
}
date=`date +%d%m%Y`
filename=`echo $1 | sed s/.qcow2//`
backup_snapshot $date $filename

Exit vi, saving the script.

Now make your script executable.

$chmod +x snapshot_backup.sh

To run the script pass it the name of your VM.

$./snapshot_backup.sh /path/to/your/vm/hostname.qcow2

From my experiance, the performance impact to the VM is fairly minimal.

Hope this helps.

Raspberry Pi Slideshow

Posted on by
4

Wanting to do something productive with my Raspberry Pi, I decided to at least get it showing me new art each day. As it turns out… its a little more challenging than I had originally expected. The main reason was resources on the Pi. Many image viewers, both command line and x windows based like to do some crazy things that tend to blow up the memory consumption on my Raspberry. As a result, the kernel will, in turn, enjoy killing the offending process. The following is how I overcame the challenges.

The Equipment:
Raspberry Pi running Raspbian “wheezy” on a 4GB SD Card
HDMI Monitor
An Internet connection

The Overview:
The idea is to get updated images from Deviantart nightly and display them via slide show over the course of the day.

The Steps:
With a fresh install of Raspbian I booted up the Pi. The initial wizard gives you a few options, here is what I did.
1. Expand the root partition to fill the SD Card
2. Changed the keyboard to the US layout
3. Changed the timezone to reflect where I live
4. Changed the boot into desktop to yes (meaning it will start the desktop after boot).
5. As a matter of security, I also updated the password for the pi user.

Reboot the Pi and you should find yourself looking at the desktop once you log in.

Add Auto Login:
You can skip this part if you don’t want to auto login, it wont affect the outcome unless you reboot your machine.
Making this as automated as possible, we need to set up the Pi to auto login the pi user. Here’s how we do that.

Edit /etc/inittab using vi. Of course vi is just my preferred text editor, feel free to use your own favorite. I’ve heard quite a few people like nano.

sudo vi /etc/inittab

Scroll down till you see the following line:

1:2345:respawn:/sbin/getty --noclear 38400 tty1

Comment it out by adding a pound sign (#) in front of it. Insert a new line and add the following.

1:2345:respawn:/bin/login -f pi tty1 </dev/tty1 >/dev/tty1 2>&1

Your file should now look something like this:

#1:2345:respawn:/sbin/getty --noclear 38400 tty1
1:2345:respawn:/bin/login -f pi tty1 </dev/tty1 >/dev/tty1 2>&1

NOTE: If your using a user different than ‘pi’ you will want to change the user, pi, to your user.

Close and save the file.

Go ahead and reboot your Pi just to make sure your changes have properly taken affect.

sudo shutdown -r now

Update and Install pqiv
Once the machine comes back up, it should automatically boot you into LXDE (the default desktop environment). Go ahead and open a terminal window and update the apt-get repository.

sudo apt-get update

Normally we would issue a simple apt-get install pqiv to install pqiv; however, due to issues with version in the raspbian repository we will need to download, fix, compile, and install pqiv manually. Don’t forget to remove pqiv if you’ve already installed it at some point.

sudo apt-get remove pqiv

Install git and the dependency gtk2 library for pqiv.

sudo apt-get install git libgtk2.0-dev

We have what we need to build the application, now we need to download the source.

cd
git clone https://github.com/phillipberndt/pqiv.git -b 0.12

Now that you’ve downloaded the pqiv source, you will notice a new directory in your home directory called pqiv. Go into this directory and run ./configure to validate all preconditions are met and create a platform specific make file.

cd pqiv
./configure

Now we need to fix the source code by replacing the gchar option; with gint option; in the main() function in the pqiv.c file. The line we are looking for is around 2233.

/*gchar option;*/
gint option;

Save and exit vi. Now you’ve fixed the bug, we will run make and install the application.

sudo make all install

You now have the fixed version of pqiv running on your Pi.

Get the slide show images:
Now we need to create the process to download the images from Deviant art. To do this, first create a directory in your home directory. Open a terminal session and type in the following commands.

cd
mkdir images
mkdir scripts
cd scripts

Now we need to create the script to get the images from DeviantArt. Create a new file in the scripts directory.

vi ~/scripts/get_images.sh

Inside the script copy and paste the following code.

NOTE: Updated wget line to adapt to the change in RSS format from Deviantart.

#!/bin/bash

cd /home/pi/images

rm *.jpg *.png

# go get the files
#landscape files
wget -q -O- "http://backend.deviantart.com/rss.xml?q=boost%3Apopular+in%3Adigitalart%2Fpaintings%2Flandscapes+max_age%3A72h&type=deviation" | grep -o '<media:content url="[^"]*' | grep -o '[^"]*$' | xargs wget -c

FILES=/home/pi/images/*
count=0
one=1

for f in $FILES
  do
    file -b $f
    echo "Checking file number $count"
    count=$(($count + $one))
    if [ "$(file $f|grep JPEG)" ]; then
      mv ${f} $RANDOM.jpg
    elif [ "$(file $f|grep PNG)" ]; then
      mv ${f} $RANDOM.png
    elif [ "$(file $f|grep ASCII)" ]; then
      echo ”Skip Script file”
    else
      echo "unknown file type $f"
    fi
  done
Once you've added the script, save your file and make it executable.
chmod +x ~/scripts/get_images.sh

Just to be sure your script is working execute it via the command line.

cd ~/scripts
./get_images.sh

So… whats this script do you might ask? For starters, using wget, it reaches out and pulls down images from DeviantArt via their RSS feeds. Using grep and sed we save the resulting downloaded file to the images directory. The second part of the script looks at all the files downloaded (some of them end up not being images) and gives them random file names. These random file names are what make the slide show seem a little more “random”.

But we want this pull to be a automated. To do this we add the script to cron. To edit cron:

crontab -e

And then add the following line at the end of the file.

00 01 * * * /home/pi/scripts/get_images.sh > /home/pi/get_images.log 2>&1

The line above kicks off your script every night at 1:00 am and outputs both output and error to a log file in the home directory of pi. (this is really only to help with troubleshooting if needed).

Slide show Image Swapping
Next we need to create a set of scripts to refresh the slide show file and make sure the slide show continues to run.

First up is a really simple script to continually refresh the slide show image. Create a new script file in your scripts directory.

cd ~/scripts
vi update_slideshow_image.sh

Copy and paste this script into your new file.

#!/bin/bash

cd /home/pi/images
while true;
  do for i in *;
    do cp $i /home/pi/slideshow.jpg;
      echo $i;
      sleep 30;
    done;
  done

Save your file and make it executable.

chmod +x update_slideshow_images.sh

Here is a quick outline of what the above script does.

It first changes its current directory to the images directory. Once there, it goes into an infinite loop. Once inside the loop it gets a list of files from your images directory. It then copies the first file to your home directory calling the file slideshow.jpg. The script then waits 30 seconds and then copies the second file, overwriting the existing slideshow.jpg with this new one. The script will do this for each file until it goes through all of them. Then it starts from the begging and does it all over again. At 1:00 am the get_images script will download a new set of images and because the update_slideshow_image get a new file list each time it loops, these new files will get added to the list to be displayed. This may not make much sense now, but it will after in a moment.

Using pqiv to display images:
Now we need to create the script to display the images.

cd ~/scripts
vi start_slideshow.sh

Copy and past this script into your new file.

#!/bin/bash

# Lets start and restart the slideshow if needed
until /usr/bin/pqiv -i -f -s -w /home/pi/slideshow.jpg; do
  echo "Slideshow crashed with exit code $?. Restarting..." >&2
  sleep 2
done

Save your script and make it executable.

chmod +x start_slideshow.sh

This script is really quite simple. Most of the actual slide show work is done by pqiv. This script’s only duty is to kick off pqiv. If the program crashes the script will echo the reason it crashed and then wait a few seconds before restarting the slide show. Why do I do this you might ask? Memory resources on a Raspberry Pi are limited and image viewing software isn’t all that great at being memory friendly, especially when trying to load larger files, scale them, and then display them. Do that over a hundred times and you will quickly get killed by the kernel.

As a side note, I tried several other image viewing programs like fbi, fim, and feh. While they worked, they each had some small issues and more importantly, they were not fond of refreshing the image when the underlying file changed.

Autostart the slide show:
To have our slide show kick off automaticly once the Pi reboots. To do this we need to add two .desktop files to autostart. One file will run our update_slideshow_image script that the other will run the start_slideshow script.

Here is how you create the first file.

cd /etc/xdg/autostart
sudo vi slideshow_image_changer.desktop

Inside this file add the following.

[Desktop Entry]
Type=Application
Name=Slideshow Image Changer
Comment=Slideshow Image Changer
Exec=/home/pi/scripts/update_slideshow_image.sh
Terminal=true

Save this file and create the second file.

sudo vi slideshow_start.desktop

Add the following to this new desktop file.

[Desktop Entry]
Type=Application
Name=Slideshow Image Changer
Comment=Slideshow Image Changer
Exec=/home/pi/scripts/start_slideshow.sh
Terminal=true

Save and close this file.

Prevent the Raspberry from going to sleep:
We are almost done… one thing that drove me nuts for a little while was how the Raspberry would occasionally turn off the monitor. I couldn’t really narrow down what I did to fix the issue. I updated the following config settings and they seem to work as expected.

Update the kbd config file.

sudo vi /etc/kbd/config

Scroll down till you see the BLANK_TIME setting and change it to 0.

# screen blanking timeout. monitor remains on, but the screen is cleared to
# range: 0-60 min (0==never) kernels I've looked at default to 10 minutes.
# (see linux/drivers/char/console.c)
#BLANK_TIME=30
BLANK_TIME=0

Scroll down a little more till you see the POWERDOWN_TIME setting and change it to 0

# Powerdown time. The console will go to DPMS Off mode POWERDOWN_TIME
# minutes _after_ blanking. (POWERDOWN_TIME + BLANK_TIME after the last input)
#POWERDOWN_TIME=30
POWERDOWN_TIME=0

Save and close the config file.

Next I ended updating the lightdm.conf file.

sudo vi /etc/lightdm/lightdm.conf

Scroll down till you see the [SeatDefault] section and add a line towards the end of it.

xserver-command=X -s 0 dpms

My screen no longer goes blank and I have the pleasure of having a dynamic slide show all day.

Reboot your Raspberry Pi. Provided everything goes as planned, your slide show should start shortly after booting into the LXDE desktop… congratulations!

Simple Bash Script To Email Server Status

Posted on by
1

I didn’t want to constantly have to log into my servers in order to check on key performance indicators so I decided to write a simple script that would do the checking for me. After collecting results, the script emails them to me.

There are a few tools called within the script you might need to install. I also convert any tabs into spaces in order to make sure things line up nicely inside my email.

#!/bin/bash

SERVER="myserver001"
TOEMAIL="admin@myservers.com"
FROMEMAIL="myserver001@myserverscom"
# Who is logged in and what are they up to
WHO=`w`
# Hows my processor doing
MPSTAT=`mpstat`
# Hows my virtual memory doing
VMSTAT=`vmstat`
# Top 10 memory consumers
PS_MEM=`ps -A -o pid,pcpu,pmem,start_time,state,time,comm | perl -e '($_ = join "",<>) =~ s/(\t)/     /g; print;' |sort -g -k 3 -r | head -10`
# Top 10 cpu consumers
PS_CPU=`ps -A -o pid,pcpu,pmem,start_time,state,time,comm | perl -e '($_ = join "",<>) =~ s/(\t)/     /g; print;' | sort -g -k 2 -r | head -10`
#  Memory usage in MB
FREE=`free -m`
# Procinfo
PROCINFO=`procinfo`
# IPTables status
IPTABLES=`iptables -nL`
# Established connections
NETSTAT=`netstat -na |grep -i esta |grep -v 127.0.0.1 |sort -n -t. -k2`
# Filesystem space
FILESYSTEM=`df -h`
# Line divider
DL="~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
# Put the email together
BODY="${DL}
`date`
${DL}
${SERVER}
${DL}
${WHO}
${DL}
${FREE}
${DL}
${MPSTAT}
${DL}
${VMSTAT}
${DL}
${PROCINFO}
${DL}
Top 10 CPU processes
${PS_CPU}
${DL}
Top 10 Memory processes
${PS_MEM}
${DL}
${IPTABLES}
${DL}
${NETSTAT}
${DL}
${FILESYSTEM}
${DL}
"

echo "${BODY}" | perl -e '($_ = join "",<>) =~ s/(\t)/     /g; print;' | sendEmail -f "${FROMEMAIL}" -u "${SERVER} Status Update" -t ${TOEMAIL}

Adding the script to cron, I now get a daily update on how my server is doing.

Using SSH To Reach Servers Through Gateways

Posted on by
Reply

I use SSH a lot and having to log into all of my machines individually and then run a command gets tedious. In many cases I need to go through a gateway machine before accessing my target machine. Here are a few simple methods for running commands from your machine to a target machine.

Simple two hop

This will put you at the command prompt for serverB.

$ ssh -t user@serverA "ssh user@serverB"

Simple three hop

This command will put you at the prompt three hosts into a network on serverC.

$ ssh -t user@serverA "ssh -t user@serverB 'ssh user@serverC'"

You can replace the ssh user@serverC command with another command such as ping targetserverC and you will get the response ping as if you were sitting at serverB executing the ping.

$ ssh -t user@serverA "ssh -t user@serverB 'ping google.com'"
*******************************************************************
user@serverA's password:
*******************************************************************
user@serverB's password:
*******************************************************************
PING google.com (74.125.45.138) 56(84) bytes of data.
64 bytes from serverB (74.125.45.138): icmp_seq=1 ttl=54 time=10.9 ms
64 bytes from serverB (74.125.45.138): icmp_seq=2 ttl=54 time=2.40 ms
64 bytes from serverB (74.125.45.138): icmp_seq=3 ttl=54 time=2.83 ms
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 15ms
rtt min/avg/max/mdev = 2.154/3.533/10.952/1.935 ms

Good luck!

Adding and Removing Available Apache Sites in Ubuntu

Posted on by
Reply

To add or remove site configurations within Ubuntu’s /etc/apache2/sites-available use these commands.

# enable site
sudo a2ensite SITECONFIG.FILE

# disable site
sudo a2dissite SITECONFIG.FILE

# restart apache
sudo  /etc/init.d/apache2 force-reload

 

If you’ve added a module to apache use these commands to build all of the correct symlinks in mods-enabled.

# enable apache2 module
sudo a2enmod MODULENAME

# disable apache2 module
sudo a2dismod MODULENAME

# restart apache
sudo  /etc/init.d/apache2 force-reload

Exporting and Reinstalling Debian Packages in Ubuntu

Posted on by
Reply

If your like me, you have several servers running linux and aren’t very keen on upgrading every time Ubuntu releases a new version. Time progresses and you kinda forget all of the packages you’ve installed. This post covers how to easily generate a list of currently installed Debian packages; and with a few easy steps, re-installing them after you’ve rebuilt your server.

First, generate the list.

$ sudo dpkg --get-selections | grep -v deinstall > installed-pkg.txt

You now have the list of all installed packages. Make sure to save it somewhere safe if you are on the same server your about to flatten. The file is generally very small and can be emailed to yourself using the following command.

$ sudo dpkg --get-selections | grep -v deinstall > ubuntu-files; cat ubuntu-files | mailx -s “installed-pkg.txt” email@address.com

Once you’ve got your server up and running with a fresh base install, move your installed-pkg.txt file into your home directory and run the following commands.

$ sudo apt-get update
$ sudo apt-get dist-upgrade
$ sudo dpkg --set-selections  < installed-pkg.txt

Now you’ve told your system what it needs to install, so let’s install it all.

$ sudo dselect

Follow these steps when the dselect session starts.

  1. Type ‘I‘ and allow dselect to install of the the packages listed in your installed-pkg.txt document.
  2. When its done type ‘Q‘ and hit the ENTER key to exit dselect.

There you have it… your new system with all the same applications you had on your older system. I’ve also used this process when building a production server image based on a existing physical development machine.

Updating a hostname Without Rebooting on RHEL

Posted on by
1

I’ve been taking full advantage of my companies cloud environment and spinning up virtual machines for all kinds of uses. Something that always bothered me was the long meaningless default host names each new machine came with. What bothered me even more was the apparent need to restart after updating the various places with the new hostname.

So, as usual, I took this as an opportunity and found the following easy four steps to update the hostname without the need to reboot!

NOTE: You must have root privileges to make these changes.

1. Change the HOSTNAME line in /etc/sysconfig/network

HOSTNAME=yourNewName

2. Change the current hostname FQDN and any aliases in /etc/hosts

127.0.0.1    localhost    yourNewName
::1          localhost    yourNewName

3. Run /bin/hostname yourNewName for the hostname change to take effect immediately.

# /bin/hostname yourNewName

4. Run /sbin/service rsyslog restart for syslog to log using the new hostname.

# /sbin/service rsyslog restart
Shutting down system logger:                       [  OK  ]
Starting system logger:                            [  OK  ]

And there you have it! Simply run /bin/hostname to verify the changes took effect or log out and back in to see your new hostname take effect in your session.

[richard@yourNewName ~]$

 

Increasing the Speed of ssh Logins

Posted on by
Reply

Have you ever run into instances after a fresh install of a Linux server and when trying to log into the box vis SSH you noticed a significant delay? On average this delay can last for about 15 seconds. In most cases this is due SSH attempting to verify DNS by way of Revers DNS records.

If your server is inside a private LAN then there is a chance the domain your using is not properly resolving and the nameserver found in /etc/resolv.conf is timing out (about 15 seconds).

Possible Fix #1

Make sure the domain specified in side your /etc/resolv.conf is responding and can properly resolve your server’s host name. To test this use dig, a bind utility.

# time dig ptr 192.168.0.202
; <<>> DiG 9.7.0-P2-RedHat-9.7.0-5.P2.el6_0.1 <<>> ptr 192.168.0.202
;; global options: printcmd
;;  connection timed out; no servers could be reached

real 0m15.020s
user  0m0.000s
sys 0m0.004s

If you get a timeout message, update your /etc/resolv.conf file with valid nameservers and re-test. As a word of warning, make sure each nameserver you add can be properly resolved. If not, expect an additional 15 second delay for each server in the list.

Possible Fix #2

Updating SSH to no longer use DNS by editing the /etc/ssh/sshd_config file.

Change the following line:

#UseDNS yes

After your update, it should look like this:

UseDNS no

Save your updates and restart ssh.

/etc/init.d/sshd restart

Possible Fix #3

There are some instances where ssh takes a long time resolving IPv6 or IPv4 addresses from domain names. To work around this issue update the /etc/ssh/sshd_config file, updating the AddressFamily option.

Change the following line:

#AddressFamily any

After your update, it should look like this:

AddressFamily inet

As always, make sure you know what your doing or at a minimum can get back to the server after you restart ssh. If your config file is messed up, you wont be able to gain remote access again.

Common Linux date ‘+%’ Formats

Posted on by
Reply

The following table is a breakdown of the common date formats used with the date command in linux and most unix operating systems.

To use the formats, do something like this:

$ date '+Today is %A %B %d %Y'
Today is Sunday October 02 2011

%% A literal percent sign %
%a locale’s abbreviated weekday name (e.g., Sun)
%A locale’s full weekday name (e.g., Sunday)
%b locale’s abbreviated month name (e.g., Jan)
%B locale’s full month name (e.g., January)
%c locale’s date and time (e.g., Thu Mar 3 23:05:25 2005)
%C century; like %Y, except omit last two digits (e.g., 21)
%d day of month (e.g, 01)
%D date; same as %m/%d/%y
%e day of month, space padded; same as %_d
%F full date; same as %Y-%m-%d
%g last two digits of year of ISO week number (see %G)
%G year of ISO week number (see %V); normally useful only with %V
%h same as %b
%H hour (00..23)
%I hour (01..12)
%j day of year (001..366)
%k hour ( 0..23)
%l hour ( 1..12)
%m month (01..12)
%M minute (00..59)
%n a newline
%N nanoseconds (000000000..999999999)
%p locale’s equivalent of either AM or PM; blank if not known
%P like %p, but lower case
%r locale’s 12-hour clock time (e.g., 11:11:04 PM)
%R 24-hour hour and minute; same as %H:%M
%s seconds since 1970-01-01 00:00:00 UTC
%S second (00..60)
%t a tab
%T time; same as %H:%M:%S
%u day of week (1..7); 1 is Monday
%U week number of year, with Sunday as first day of week (00..53)
%V ISO week number, with Monday as first day of week (01..53)
%w day of week (0..6); 0 is Sunday
%W week number of year, with Monday as first day of week (00..53)
%x locale’s date representation (e.g., 12/31/99)
%X locale’s time representation (e.g., 23:13:48)
%y last two digits of year (00..99)
%Y year (e.g., 2011)
%z +hhmm numeric timezone (e.g., -0400)
%:z +hh:mm numeric timezone (e.g., -04:00)
%::z +hh:mm:ss numeric time zone (e.g., -04:00:00)
%:::z numeric time zone with : to necessary precision (e.g., -04, +05:30)
%Z alphabetic time zone abbreviation (e.g., EDT)

 

Installing Ubuntu 10.10 packages on Atom processors

Posted on by
Reply

Running Ubuntu 10.10 on a Netbook with an Atom (low power) processor can cause a few headaches. Seems like a lot of package installs look for the firmware-b32-installer package that doesn’t explicitly allow installs on the low power CPUs.

The error message often received is:

Not supported low-power chip with PCI id 14e4:4315!
Aborting.

Thankfully to resolve the issue is fairly simple and straight forward. Open a command line and type

$ sudo apt-get install firmware-b43-lpphy-installer

If you prefer the GUI install:

  1. Click on the System menu
  2. Click on Administration
  3. Click on Synaptic Package Manager
  4. Search for firmware-b43-lpphy-installer
  5. Click on Mark
  6. Click Install
  7. Close the window

And your done… Now go back and try to re-install (remove and install) the previously failing package.