Lampros - Weird Bricks

Building OpenSSH 7.5 for CentOS 7.3

22 July, 2017 | CentOS

Interwebs,

I've built RPM packages before, my love for FPM after all cannot be overstated. However recently I needed to build the latest version of OpenSSH portable (at the time of writing the version was 7.5) for CentOS 7. That turned out to be a little more complicated than I expected. 

The Google search:

Searching on Google led me here - http://www.arvinep.com/2015/12/building-rpm-openssh-71p1-on-rhelcentos.html - this turned out to be a great resource but it was written for CentOS 6.5. When I tried the same instructions on CentOS 7.3 I ended up running into some issues which I have documented at the end of this post.

How I got it done in a high level

  1. Download OpenSSH 7.5 portable
  2. Modified the SPEC file that came with it
  3. Built the RPM
  4. Extracted the RPM to a temporary directory
  5. Downloaded the CentOS OpenSSH RPM and extracted it to a different temporary directory
  6. Rsynced the CentOS OpenSSH file over the portable OpenSSH files using the `--ignore-existing` option
  7. Filled in the blanks
  8. No goats were harmed during this post

1. Download OpenSSH 7.5 portable:

cd /tmp
curl -LO ftp://mirrors.sonic.net/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz
tar -xzf openssh-*.tar.gz
cd openssh-*

Create the specs directory and the sources directory:

mkdir -p /usr/src/redhat/SPECS/ /root/rpmbuild/SOURCES/

Copy the downloaded source to sources directory:

cp /tmp/openssh-*.gz /root/rpmbuild/SOURCES/

2. (Optional) - Download my SPEC file and skip step 3:

cd /usr/src/redhat/SPECS/
curl -LO https://raw.githubusercontent.com/weirdbricks/openssh-portable/patch-1/contrib/redhat/openssh.spec

3. (Skip if you did step 2) - Modify the SPEC file that came with it:

Disable x11-askpass as that's for a host with a desktop environment:

sed -i 's/%define no_x11_askpass 0/%define no_x11_askpass 1/' /usr/src/redhat/SPECS/openssh.spec
sed -i 's/%define no_gnome_askpass 0/%define no_gnome_askpass 1/' /usr/src/redhat/SPECS/openssh.spec

Disable kerberos5 support:

sed -i 's/%define kerberos5 1/%define kerberos5 0/' /usr/src/redhat/SPECS/openssh.spec

Add the pam-devel package as a dependency:

sed -i 's/BuildRequires: perl, openssl-devel/BuildRequires: perl, openssl-devel, pam-devel/' /usr/src/redhat/SPECS/openssh.spec

Change the make line to use all cores:

sed -i 's/make/make -j\`nproc\`/' /usr/src/redhat/SPECS/openssh.spec

Change the configure bit to match the CentOS one minus some incompatible options with OpenSSH version 7.5:

%configure \
--sysconfdir=%{_sysconfdir}/ssh \
--libexecdir=%{_libexecdir}/openssh \
--datadir=%{_datadir}/openssh \
--with-default-path=/usr/local/bin:/usr/bin \
--with-superuser-path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
--with-privsep-path=%{_var}/empty/sshd \
--disable-strip \
--without-zlib-version-check \
--with-ssl-engine \
--with-ipaddr-display

4. Install build dependencies:

Install any build dependencies that are specified in the SPEC file

yum-builddep -y /usr/src/redhat/SPECS/openssh.spec

Install the RPM build tools along with Ruby so that we can install FPM later:

yum install -y rpm-build yum-utils rpmdevtools gcc tree ruby ruby-devel

Also install net-tools - we'll need netstat later for verification:

yum install -y net-tools

5. Build the RPM from source:

rpmbuild -bb /usr/src/redhat/SPECS/openssh.spec

The new packages are under /root/rpmbuild/RPMS/x86_64/ :

ls -l /root/rpmbuild/RPMS/x86_64/
total 1320
-rw-r--r-- 1 root root 470792 Jul 14 05:14 openssh-7.5p1-1.x86_64.rpm
-rw-r--r-- 1 root root 490680 Jul 14 05:14 openssh-clients-7.5p1-1.x86_64.rpm
-rw-r--r-- 1 root root 16984 Jul 14 05:14 openssh-debuginfo-7.5p1-1.x86_64.rpm
-rw-r--r-- 1 root root 367460 Jul 14 05:14 openssh-server-7.5p1-1.x86_64.rpm

6. Extracted the OpenSSH server RPM we just built to a temporary directory:

mkdir -pv /openssh-server

Extract the files:

cd /openssh-server
rpm2cpio /root/rpmbuild/RPMS/x86_64/openssh-server-7.5p1-1.x86_64.rpm | cpio -idmv

Output:

./etc/pam.d/sshd
./etc/rc.d/init.d/sshd
./etc/ssh
./etc/ssh/sshd_config
./usr/libexec/openssh/sftp-server
./usr/sbin/sshd
./usr/share/man/man5/moduli.5.gz
./usr/share/man/man5/sshd_config.5.gz
./usr/share/man/man8/sftp-server.8.gz
./usr/share/man/man8/sshd.8.gz
./var/empty/sshd

7. How do you figure that the RPMs contain different files?

Before our new RPM package is identical to the one that CentOS installs, we need to make sure that all files in the CentOS package are also present in ours. To figure out what files we need, we ask rpm:

Figure out what files the current OpenSSH server package includes by using the 'repoquery' command:

repoquery -l openssh-server
/etc/pam.d/sshd
/etc/ssh/sshd_config
/etc/sysconfig/sshd
/usr/lib/systemd/system/sshd-keygen.service
/usr/lib/systemd/system/sshd.service
/usr/lib/systemd/system/sshd.socket
/usr/lib/systemd/system/sshd@.service
/usr/lib64/fipscheck/sshd.hmac
/usr/libexec/openssh/sftp-server
/usr/sbin/sshd
/usr/sbin/sshd-keygen
/usr/share/man/man5/moduli.5.gz
/usr/share/man/man5/sshd_config.5.gz
/usr/share/man/man8/sftp-server.8.gz
/usr/share/man/man8/sshd.8.gz
/var/empty/sshd

As you can see we're missing some files - let's extract the openssh-server package (the official one that CentOS built) and get any files we're missing from there:

mkdir /openssh-server-centos
yumdownloader --destdir=/tmp openssh-server
cd /openssh-server-centos
rpm2cpio /tmp/openssh-server-6.6.1p1-35.el7_3.x86_64.rpm | cpio -idmv

8. rsync playing the role of Frankenstein:

Now how do we do the merge you ask? we use rsync of course!!!

Install it first:

yum install rsync -y

Now merge the files - note the '--ignore-existing' switch:

rsync -HAXa --ignore-existing -vv --log-file=/tmp/rsync.log /openssh-server-centos/ /openssh-server/

Sample output:

sending incremental file list
delta-transmission disabled for local transfer or --whole-file
./
etc/
etc/pam.d/
etc/pam.d/sshd exists
etc/ssh/
etc/ssh/sshd_config exists
etc/sysconfig/
etc/sysconfig/sshd
usr/
usr/lib/
usr/lib/systemd/
usr/lib/systemd/system/
usr/lib/systemd/system/sshd-keygen.service
usr/lib/systemd/system/sshd.service
usr/lib/systemd/system/sshd.socket
usr/lib/systemd/system/sshd@.service
usr/lib64/
usr/lib64/fipscheck/
usr/lib64/fipscheck/sshd.hmac
usr/libexec/
usr/libexec/openssh/
usr/libexec/openssh/sftp-server exists
usr/sbin/
usr/sbin/sshd exists
usr/sbin/sshd-keygen
usr/share/
usr/share/man/
usr/share/man/man5/
usr/share/man/man5/moduli.5.gz exists
usr/share/man/man5/sshd_config.5.gz exists
usr/share/man/man8/
usr/share/man/man8/sftp-server.8.gz exists
usr/share/man/man8/sshd.8.gz exists
var/
var/empty/
var/empty/sshd/

See in the above output how some files say 'exists'? those are the ones that were NOT overwritten. What we want to make sure of is that the files that were NOT overwritten are going to work for us:

You can also get the same log from /tmp/rsync.log:

grep exists /tmp/rsync.log
2017/07/17 04:33:53 [10570] etc/pam.d/sshd exists
2017/07/17 04:33:53 [10570] etc/ssh/sshd_config exists
2017/07/17 04:33:53 [10570] usr/libexec/openssh/sftp-server exists
2017/07/17 04:33:53 [10570] usr/sbin/sshd exists
2017/07/17 04:33:53 [10570] usr/share/man/man5/moduli.5.gz exists
2017/07/17 04:33:53 [10570] usr/share/man/man5/sshd_config.5.gz exists
2017/07/17 04:33:53 [10570] usr/share/man/man8/sftp-server.8.gz exists
2017/07/17 04:33:53 [10570] usr/share/man/man8/sshd.8.gz exists

9. Filling in the blanks:

Some of the above we obviously want to leave as is since those are the new binaries. There are a couple of files though that we do want to overwrite:

etc/pam.d/sshd - we want to keep the CentOS one as the authentication methods listed in this one do not exist in CentOS and this will result in breaking up pretty much everything:

/bin/cp -f /openssh-server-centos/etc/pam.d/sshd /openssh-server/etc/pam.d/sshd

sshd_config - we do want the CentOS modifications, but we need to take out the Kerberos lines as those will not work since we disabled 'kerberos5' support earlier:

/bin/cp -f /openssh-server-centos/etc/ssh/sshd_config /openssh-server/etc/ssh/sshd_config 
sed -i '/GSSAPI/d' /openssh-server/etc/ssh/sshd_config

We also want to remove the 'UsePrivilegeSeparation' line as it's a deprecated option:

sed -i '/UsePrivilegeSeparation/d' /openssh-server/etc/ssh/sshd_config

Also .. systemd :) I couldn't get systemd to work with the new version of OpenSSH, so I "told" systemd to use the contributed '/etc/rc.d/init.d/sshd' -- by simply editing the systemd file:

vi /openssh-server/usr/lib/systemd/system/sshd.service

Delete *everything* and replace with this:

[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.service
Wants=sshd-keygen.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/init.d/sshd start
ExecReload=/etc/init.d/sshd reload
ExecStop=/etc/init.d/sshd stop
Environment=BOOTUP=serial
Environment=CONSOLETYPE=serial
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=basic.target

Before we call this good to go, we need to steal one more thing from CentOS - the openssh-server packages that it depends on - there's more than you would expect!! To get the dependencies we use the 'repoquery --requires --resolve' command:

repoquery --requires --resolve openssh-server
krb5-libs-0:1.14.1-27.el7_3.x86_64
bash-0:4.2.46-20.el7_2.x86_64
openldap-0:2.4.40-13.el7.x86_64
audit-libs-0:2.6.5-3.el7.x86_64
systemd-libs-0:219-30.el7_3.8.x86_64
openssh-0:6.6.1p1-35.el7_3.x86_64
libcap-ng-0:0.7.5-4.el7.x86_64
pam-0:1.1.8-18.el7.i686
systemd-0:219-30.el7_3.8.x86_64
shadow-utils-2:4.1.5.1-24.el7.x86_64
pam-0:1.1.8-18.el7.x86_64
libselinux-0:2.5-6.el7.x86_64
zlib-0:1.2.7-17.el7.x86_64
glibc-0:2.17-157.el7_3.2.x86_64
tcp_wrappers-libs-0:7.6-77.el7.x86_64
glibc-0:2.17-157.el7_3.2.i686
openssl-libs-1:1.0.1e-60.el7_3.1.x86_64
libcom_err-0:1.42.9-9.el7.x86_64
fipscheck-lib-0:1.4.1-5.el7.x86_64

We're going to need to instruct FPM to add those as dependencies for our package as well. 

10. Install FPM:

Easy:

gem install fpm --no-ri --no-rdoc

11. Unleash it upon mankind!!!! - uhm, I mean,  use FPM to package it all up:

cd /openssh-server
fpm --verbose -v 7.5p1 -n openssh-server \
--replaces openssh-server-\* \
--url="http://lampros.chaidas.com" \
-d bash \
-d openssl-libs \
-d audit-libs \
-d systemd-libs \
-d openssh \
-d libcap-ng \
-d pam \
-d systemd \
-d shadow-utils \
-d libselinux \
-d zlib \
-d glibc \
-d libcom_err \
-d fipscheck-lib \
--after-upgrade=/tmp/post-install.sh \
-s dir -t rpm .=/

The above does a few things: It will create a new package called "openssh-server" with version "7.5p1" a bunch of dependencies AND will execute the script `/tmp/post-install.sh` right after the RPM is installed.

Here's the contents of `/tmp/post-install.sh`:

#!/bin/env bash

echo "A requirement of upgrading to SSH 7.5 is to make sure that the permissions for your host's SSH keys are not too open."
echo "This script will execute the command: \"chmod 600 /etc/ssh/ssh_host_*\"..."
printf "Running the command..."
chmod 600 /etc/ssh/ssh_host_*
if [ $? != 0 ]; then
printf "failed!! - exiting\n"
exit 1
fi
printf "OK!\n"

After you've run the FPM command you should now have output similar to this:

Wrote: /tmp/package-rpm-build-48189d8642baa33b3ec4904b2b34a6ef9b0e7035123f0a564e44a1e210b7/RPMS/x86_64/openssh-server-7.5p1-1.x86_64.rpm {:level=>:info}
Executing(%clean): /bin/sh -e /tmp/rpm-tmp.eZ6RCw {:level=>:info}

12. Install your new package:

yum install openssh-server-7.5p1-1.x86_64.rpm

You should see output like this - I've highlighted the good stuff in red:

yum install openssh-server-7.5p1-1.x86_64.rpm
Loaded plugins: fastestmirror
Examining openssh-server-7.5p1-1.x86_64.rpm: openssh-server-7.5p1-1.x86_64
Marking openssh-server-7.5p1-1.x86_64.rpm as an update to openssh-server-6.6.1p1-35.el7_3.x86_64
Resolving Dependencies
--> Running transaction check
---> Package openssh-server.x86_64 0:6.6.1p1-35.el7_3 will be updated
---> Package openssh-server.x86_64 0:6.6.1p1-35.el7_3 will be obsoleted
---> Package openssh-server.x86_64 0:7.5p1-1 will be obsoleting
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================
Installing:
openssh-server x86_64 7.5p1-1 /openssh-server-7.5p1-1.x86_64 889 k
replacing openssh-server.x86_64 6.6.1p1-35.el7_3

 Check that OpenSSH is listening:

netstat -ntlp | grep sshd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 13342/sshd
tcp6 0 0 :::22 :::* LISTEN 13342/sshd

Check your package version:

/usr/sbin/sshd --version &>/dev/stdout | grep OpenSSH
OpenSSH_7.5p1, OpenSSL 1.0.1e-fips 11 Feb 2013

13. Make sure OpenSSH starts on boot:

systemctl is-enabled sshd
enabled

Hey yoooooooo!!!  

14. Want to install my package?

yum -y install http://rpm.chaidas.com/rpm.chaidas.com-0.1-1.x86_64.rpm
yum -y install openssh-server

Sources: