Adventures in Perl Packaging

Jon Jensen
End Point Corporation
endpoint.com

YAPC::Europe, Kiev, Ukraine

12 August 2013

(See also video of this talk.)

Everything comes with Perl

Every major Linux server OS, anyway

That’s what I’ll focus on here

And usually some
prepackaged CPAN modules

That is very convenient

RHEL/CentOS example

yum install 'perl(DBD::Pg)'

Package “provides”

Actual package name was

perl-DBD-Pg

But this fails:

yum install perl-HTTP-Date

Don’t need to know package or parent module name:

yum install 'perl(HTTP::Date)'

Installs perl-libwww-perl which provides it.

Predictable

Prevents module incompatibilities

Make your own meta-RPM

An RPM that requires
everything you need,
including specific versions
where necessary.

Install it all with one command:

yum install myapp-deps

Changed requirements between
myapp-deps-1.3 and myapp-deps-1.4?

Roll back may work:
yum downgrade myapp-deps-1.3

All my CPAN modules there?

RHEL has a reputation of not having
many CPAN modules available packaged.

Not true with EPEL, and in
Debian there are plenty in the main repo.

It only takes one

Of course if the one CPAN module you want
happens not to be packaged yet, it’s all vain.

Other snags

What version of Perl?

And CPAN module versions?

Enterprise Linux distributions

The whole point is long-term support.

Without updates breaking things.

Often by the time you get
the chance to use this distro …

the packages will be old

How old?

(Debian is not a long-term support distro,
but it’s widely used, so I show it for comparison.)

OS Released Perl EOL
RHEL 5 2007 5.8.8 2017
RHEL 6 2010 5.10.1 2020
Ubuntu 10.04 LTS 2010 5.10.1 2015
Debian 6 “squeeze” 2011 5.10.1 2014
Ubuntu 12.04 LTS 2012 5.14.2 2017
Debian 7 “wheezy” 2013 5.14.2 ?

Falling further behind

Perl releases now more frequent.

Distros not keeping up.

And not just Linux

OpenBSD 5.3 was released last month.
It includes Perl 5.12.2.

Let’s consider only CPAN modules for a moment

Packaging not just for distro maintainers

But packaging is harder
than not packaging!

Different skill set than
Perl developers usually have

Quick & dirty

Break the distro?

Yes, it’s possible.
Some system packages
depend on perl.

Which ones?

repoquery --whatrequires perl apt-cache rdepends perl

Install new perl into /usr/local!

That’s the default when building from source.

Yes, a long tradition.

Simple, easy, cheap (in the moment).

If only ever this one server
running one Perl app
(or a few that use the
same libraries), this will work.

Poor repeatability

Multiple servers?

I’ll do the same thing
on each server!

Things that start the same
won’t stay the same

Install a CPAN module on one server during development.

When you later install on production,
you forget to install it there.

Site’s down!

Hurry and install it, but the cpan(m) client installs
a different (newer) version from CPAN.
Maybe that matters now, maybe later, maybe not at all.

Identical setup? Clone it.

If the exact same distro, version, and architecture,
you can just clone /usr/local over.

perlbrew, plenv, Carton

Very nice, and solve some
of these problems.

Carton + plenv per-project perl

Carton bundling

Cache CPAN tarballs and thus avoid install-time
dependency on public Internet & CPAN.

But still building on the fly

System libraries

If you install needed system libraries from source,
things get even more complicated to reproduce.

Fine for desktop

But I’m talking here about server environments.
Need reliable, supportable, reproducible over years.

Our story

We needed:

We also needed many, many
CPAN modules, and wanted their
versions frozen.

So we made our own yum repos

packages.endpoint.com

RHEL 5: local-perl 5.10.0 and over 700 CPAN packages

RHEL 6: local-perl 5.14.1 and over 300 CPAN packages

Where?

In /usr/local/bin/perl so it’s in default PATH already.

Easy install

Going back to `yum install myapp-deps`
— now that can install our custom perl
and all the CPAN modules.

Sounds great, right?

It is, once it’s done.

Let’s see what it takes to get there.

New Yum repo

Need a build server

RPM-building is its own talk or two

Our ~/.rpmmacros file

%HOME %{expand:%%(cd; pwd)} %_topdir %{HOME}/rpm %debug_package %nil %_signature gpg %_gpg_name hosting@endpoint.com %packager End Point Hosting Team %vendor End Point Corporation https://packages.endpoint.com/ %dist .ep %__perl_provides %{HOME}/bin/local-perl.prov %__perl_requires %{HOME}/bin/local-perl.req %__find_provides %{HOME}/bin/find-provides %__find_requires %{HOME}/bin/find-requires %_prefix /usr/local %_exec_prefix %{_prefix} %_bindir %{_exec_prefix}/bin %_sbindir %{_exec_prefix}/sbin %_libexecdir %{_exec_prefix}/libexec %_datadir %{_prefix}/share %_sysconfdir %{_prefix}/etc %_sharedstatedir %{_prefix}/com %_localstatedir %{_prefix}/var %_lib lib64 %_libdir %{_exec_prefix}/%{_lib} %_oldlibdir /usr/lib64 %_includedir %{_prefix}/include %_oldincludedir /usr/include %_infodir %{_prefix}/info %_mandir %{_datadir}/man %__perl /usr/local/bin/perl

Use a tool to make specfiles

We use cpan2rpm.

Ours is customized and called
local-cpan2rpm to support our
nonstandard installation location
and package naming convention.

Other options

Details of using cpan2rpm are too much
to cover in this talk.

Other similar tools:

Automatic dependency finders

Watch for:

But usually it works ok

Build initial local-perl package

Take RHEL’s source RPM and adapt for
local-perl location and naming, and
whatever other changes you’ll make.

For each distro & arch

The prominence of x86_64 is nice.
We aren’t supporting RHEL 6 32-bit.
Could automate but arch often requires manual attention.

Yum repo RPM

Build a “Yum repo RPM”, which adds support
for your custom yum repository.

Easier installation, upgrades, removal.

Things to do better

Use mock to build.

We didn’t use mock, but plan to for RHEL 7.

Don’t install into /usr/local

We wanted to be kinda lazy and
take advantage of the fact that
/usr/local/bin was usually in the path,
so if we installed perl there, it would work.

We no longer think that’s so important,
since you still have to make the shebang line
point to /usr/local/bin/perl.

Alternatives?

Could use the alternatives system borrowed
from Debian and available in RHEL.

Different name

EPEL did this with python26
alongside system python (2.4).
Usual filesystem locations, different names.

Then not in PATH

Just add your weird /opt/$your_org/perl-$version/bin
manually to PATH.

Because we now need multiple custom perls on a machine too.

We plan to do that for RHEL/CentOS 7.

#! line

Watch out using [/usr]/bin/env.

Better to hardcode to a specific perl.

Or use nonversioned wrappers or shims.

It’s easy to lose your custom PATH, e.g. on re-exec.

Red Hat Software Collections

Check out Red Hat Software Collections
and Fedora Software Collections.

Brings to RHEL 6 Perl 5.16.3
installed in /opt/rh/perl516.

Red Hat Software Collections

Includes other newer software:
PHP, Python, Ruby, databases.

Still called a beta.

Community’s good, but…

It would be good to join forces with Fedora on this.
Then you don’t do all the work or hosting yourself.

OTOH I doubt they would want our unthreaded perl.
We were on our own with that.

Software Collections standards

We’ll be looking to Software Collections for
better ways to do our custom packaging.

Still no multiple simultaneous versions

Except maybe in site_perl path.

Examples: Module::Build or Sys::Syslog
core module conflict.

Impatience a virtue or no?

Even if we can work under the umbrella of a group,
such as EPEL, Software Collections, or some other repo,
will a new package be available fast enough for
the developer who needs it?

No.

It’s never fast enough

Developers have little patience waiting for
you to build packages for CPAN modules.
Can’t I just install this one from source?

Quite understandable

And experimentation speed is important.

May try several CPAN modules
before finding the right one.

Why package those that weren’t selected?

local::lib

We’re starting to recommend developers
use local::lib for experiments,
and we only build RPMs once they’re
needed for production.

Software is hard

Packaging software is hard

This is why words like
“DevOps” are coined

Software is about production services

At least business software is.

So this is all worth the bother.

One can dream

Ideal would be to have RPM-installed CPAN modules
of any version simultaneously installable,
that Carton can selectively make visible to your app.
Initial experiment failed; will try more later.

There are analogous problems, techniques,
trade-offs, and tools for everything I’ve
talked about for Python, Ruby, and PHP.

RPM references

Source references

See also

Questions?

Conference talk page

Talk video recording

Twitter: @jonjensen0

jon@endpoint.com