post

Fixing Docker pull timeout errors in Jenkins

Jenkins has a Docker plugin that allows you to authenticate with a docker image registry, pull the container image that you want, and then run tests inside the container. The plugin works well, and you’re guaranteed a “known starting state” for your test since you’re running in a freshly-instantiated container.

The issue that I ran into is that my docker registry is in a different data center than my test environment, there are tests running 24×7, and occasionally there are temporary outages on the Internet in-between the two data centers. When that happens the plugin’s docker.image().pull() command will fail. Even if the image is stored locally it will fail because it attempts to verify the locally-stored image’s signature against the registry image’s signature.

Since the Jenkinsfile is written in Groovy my solution was to add a Groovy function to the top of the Jenkinsfile that wrapped the docker.image().pull() command inside of a retry loop:

// Since docker.image().pull() will fail during
// intermittent or temporary network outages wrap
// it in a retry loop.
def public static docker_image_pull(Object docker, String imageName) {
    def int attempt = 0
    def int max_attempts = 10
    def boolean success = false
    while ((! success) && (attempt < 10)) {
        try {
            docker.image(imageName).pull()
            success = true
        } catch (Exception err) {
            attempt += 1
            def sleep_sec = attempt * 30
            println("Attempt #${attempt}: Failed to pull ${imageName}. Sleeping ${sleep_sec}s")
            sleep(sleep_sec)
        }
    }
    if (! success) {
        throw new Exception("Failed to pull ${imageName}")
    }
}

The function will attempt to pull the image up to 10 times. It has a back-off delay so it first sleeps 30s and retries, then 60s, then 90s, etc. For temporary / intermittent network outages this eventually succeeds. For longer network outages, or of the problem is something else such as “image doesn’t exist” the function will eventually time out and exit with an error.

After that I just had to replace every instance of docker.image('myImageName:latest').pull() inside the Jenkinsfile with:

docker_image_pull(docker, 'myImageName:latest')

Hope you find this useful.

Share Button
post

How to make the best drip-brewed coffee every time

My sister was visiting and I made a pot of coffee. My sister had a cup and said “That’s really good coffee. How did you make it?”

I’ve gotten that same response from many, many people who drink my coffee and I finally realized that there’s a lot of people who just don’t know how to make a pot of coffee. There’s no magic to it: start with good coffee and measure what you put in the coffee pot.

Pretentious coffee snobs are annoying. No one cares about your perfect cup of espresso. If you’re one of those people just go away and die somewhere. Really.

When I bought my first drip coffee maker years ago — a Cuisinart 12-Cup Brew Central Programmable Coffee Maker that’s still being manufactured and sold today — I wanted to make coffee like Starbucks, so I went to the Starbucks web site and read their instructions.

Step Zero – Buy good coffee. They don’t actually say that on the Starbucks site, but they’re assuming you’re buying Starbucks ground coffee. You can’t make good food from bad ingredients, and you can’t make good coffee from inferior beans.

Buy some good quality coffee. It doesn’t have to be from Starbucks, but it can’t be dry brown dust with bits of bean husk in it. The grounds should smell like the best cup of coffee you ever had and look like rich black loam. If you’re looking for suggestions try Starbucks Casi Cielo or Peet’s Sulawesi Kalosi.

Step One – Choose the right grind
For a flat bottom filter, use a medium grind that resembles sea salt. Cone filters use a finer grind that resembles granulated sugar.

Starbucks – How to brew coffee at home


That seems clear enough. My Cuisinart has a cone filter, so get ground coffee that looks like granulated sugar. Easy. Done.

Trigger alert for pretentious coffee snobs: I tend to buy Peet’s coffee and grind the whole pound at once. A pound lasts about a week at my house, so it’s not going to lose any of its vital essence. Get over yourself.

Step Two – Measure
Use 2 tablespoons of freshly ground coffee for every 6 ounces of water.

— Starbucks – How to brew coffee at home


I think that this is where they lose people. 6 ounces of water? My coffee pot says it makes 12 cups. How many tablespoons for a pot of coffee? What if I want to make a half a pot? Or 4 cups?

First off, if you try to make a pot of coffee and you’re measuring grounds by the tablespoon your measurements will be off every time. You will add too much coffee or not enough. Don’t use a tablespoon.

Second, the cups on a drip coffee pot are not the same as a 1 cup measure. A one cup measure is 8 fluid ounces. My “12 cup” Cuisinart pot holds 64 fluid ounces of coffee, so “1 Cuisinart cup” = 5-1/3 fluid ounces. WTF?

Using the Starbucks measuring method I’d need 10-2/3 tablespoons per pot. Try it and you’ll get weak, underwhelming coffee. That’s not how they make coffee at Starbucks.

Make a perfect pot of coffee

My method is simple: Use the lines on the pot to measure water, use measuring cups to measure coffee grounds.

Full pot of coffee – fill the pot to 12 “cups” of water, use 1 cup of ground coffee.

Half pot – 6 “cups” water, 1/2 cup ground coffee.

Third pot – 4 “cups” water, 1/3 cup ground coffee.

Quarter pot – 3 “cups” water, 1/4 cup ground coffee.

Easy, right? Full pot, one cup of ground coffee. Half pot, 1/2 cup of ground coffee. Quarter pot, 1/4 cup of ground coffee.

For advanced coffee preparers:

Two-thirds of a pot – 8 cups of water, add 1/3 cup of ground coffee TWICE!

10 cups of coffee – 10 cups of water, 1/2 and 1/3 cups of ground coffee.

Try this method and and I GUARANTEE you’ll never go back to whatever you were doing to make coffee before or NO MONEY BACK. You too will hear the words “That’s really good coffee. How did you make it?”

Hope you find this useful.

Share Button
post

Automatically decrypt multiple LUKS-encrypted volumes

I’ve written in the past on Adding an external encrypted drive with LVM to Ubuntu Linux and Adding a LUKS-encrypted iSCSI volume to Synology DS414 NAS but I neglected to mention how to automatically decrypt additional volumes.

When installing a fresh copy of Ubuntu one of the options is to install with a LUKS-encrypted Logical Volume Manager Volume Group (LVM VG). This puts your root volume on the encrypted LVM VG. When you power up your machine Ubuntu prompts you to enter the decryption passphrase in order to decrypt the VG and start your computer. Without the passphrase the contents of your hard drive are unreadable.

If you add encrypted external drives and/or additional VGs you will end up with multiple encrypted volumes. Ubuntu will prompt you for the passphrase of each additional encrypted volume when you boot up the machine.

If you don’t want to enter multiple, different passphrases each time you boot, you can store the passphrases for additional volumes on the encrypted root filesystem of your first drive using the /etc/crypttab file. You’ll just be prompted for one passphrase, of the first VG, and that decrypts the passphrases needed to decrypt the additional volumes.

Here’s how it works.

The /etc/crypttab file contains 4 fields per line: the name of the encrypted volume, a UUID identifying the storage device, the name of a file with the decryption passphrase, and encryption options.

nvme0n1p5   UUID=405d8c73-1cf9-4b2c-9b8e-c76b90d27c67 none                        luks,discard
datastorage UUID=f2d73ac8-1ef1-4735-9dd4-9e778fc9e781 /root/.luks-datastorage     luks,discard
external1   UUID=0140476b-dd0b-4aab-b7d4-2f5fa14d1a0c /root/.luks-backupexternal1 luks
external2   UUID=610a67d4-c4f6-4b73-a824-a437971e8d24 /root/.luks-backupexternal2 luks
iscsi       UUID=b106b749-f4ab-44be-8962-6ff867dc074e /root/.luks-backupiscsi     luks

The first volume, nvme0n1p5, is the encrypted boot volume. It contains the root filesystem and the /root home directory. The third field is “none” which means that Ubuntu will prompt you for a decryption passphrase in order to unlock and decrypt the drive.

The remaining volumes have files defined that contain the decryption passphrase for each volume. Those files are hidden files in the /root home directory. Once the nvme0n1p5 volume is decrypted and mounted, the remaining volumes are automatically decrypted using the passphrases stored in the hidden files.

The end result is that all of your drives are encrypted, but you only have to enter one passphrase to unlock all of your drives.

Hope you find this useful.

Share Button
post

Too many authentication failures

I was working with a new Linux distro and after creating a brand-new VM with a single login I attempted to ssh into the VM only to be greeted with:

Received disconnect from 10.0.0.180 port 22:2: Too many authentication failures
Disconnected from 10.0.0.180 port 22

It was a new VM, and I hadn’t loaded an ssh key (there was no option to do so in the install). I’d set up a user and password, so I expected to get a password prompt. I didn’t get to a password prompt, just an immediate disconnect.

I used ssh -vvv to connect and found that my ssh client was attempting to use my ssh keys, as ssh is supposed to, and on the third key the VM spat back the error:

Received disconnect from 10.0.0.180 port 22:2: Too many authentication failures
Disconnected from 10.0.0.180 port 22

Well, I wanted to connect with a password anyhow, so I tried:

ssh -o PubkeyAuthentication=no username@10.0.0.180

I was greeted with a password: prompt.

I checked the /etc/ssh/sshd_config and found that someone who’d built the install image had changed the default setting for MaxAuthTries from 6 to 2.

The MaxAuthTries setting tells the ssh daemon how many different authentication attempts a user can try before it disconnects. Each ssh key loaded into ssh-agent counts as one authentication attempt. The default is 6 because many users (like me) have multiple ssh keys loaded into ssh-agent so that we can automatically log into different hosts that use different ssh keys. Trying more than one ssh key isn’t the same as thumb-fingering a password — ssh is designed to allow for multiple key attempts. After the ssh connection attempts all of your ssh keys and you haven’t run out of attempts and passwords are enabled you’ll eventually get a password prompt.

Setting MaxAuthTries back to the more reasonable default of 6 and reloading the sshd daemon fixed the issue. Apparently whoever tested the setup only has one ssh key and wasn’t aware of what changing the MaxAuthTries setting does when people with more than one key attempt to log in.

If you’re concerned about ssh security sshd_config allows you to control what versions of the ssh protocol are supported, which ciphers you trust (or don’t trust), and to tune other settings that lock down what you will or won’t allow ssh to do in your environment. It may be that for some applications in some environments setting MaxAuthTries 2 makes sense, but using it for an out of the box installation just breaks ssh for no good reason.

Hope you find this useful.

Share Button
post

Install a local .deb file and its dependencies

To install a local deb file and its dependencies use apt, not dpkg:

sudo apt install ./foo-1.2.3.deb

You’ll automatically get all of the dependencies installed with the package. (dpkg doesn’t understand dependencies or repos, apt does.)

The leading ./, or a full or relative path to the deb file, is required. The path is what tells apt that it’s a local file.

Hope you find this useful.

Share Button