Chapter 14. The build process

Table of Contents

14.1. Introduction
14.2. Program location
14.3. Directories used during the build process
14.4. Running a phase
14.5. The fetch phase
14.6. The checksum phase
14.7. The extract phase
14.8. The patch phase
14.9. The tools phase
14.10. The wrapper phase
14.11. The configure phase
14.12. The build phase
14.13. The test phase
14.14. The install phase
14.15. The package phase
14.16. Other helpful targets

14.1. Introduction

This chapter gives a detailed description on how a package is built. Building a package is separated into different phases (for example fetch, build, install), all of which are described in the following sections. Each phase is splitted into so-called stages, which take the name of the containing phase, prefixed by one of pre-, do- or post-. (Examples are pre-configure, post-build.) Most of the actual work is done in the do-* stages.

The basic steps for building a program are always the same. First the program's source (distfile) must be brought to the local system and then extracted. After any pkgsrc-specific patches to compile properly are applied, the software can be configured, then built (usually by compiling), and finally the generated binaries, etc. can be put into place on the system.

14.2. Program location

Before outlining the process performed by the NetBSD package system in the next section, here's a brief discussion on where programs are installed, and which variables influence this.

The automatic variable PREFIX indicates where all files of the final program shall be installed. It is usually set to LOCALBASE (/usr/pkg), or CROSSBASE for pkgs in the “cross” category. The value of PREFIX needs to be put into the various places in the program's source where paths to these files are encoded. See Section 8.3, “patches/*” and Section 16.3.1, “Shared libraries - libtool” for more details.

When choosing which of these variables to use, follow the following rules:

  • PREFIX always points to the location where the current pkg will be installed. When referring to a pkg's own installation path, use “${PREFIX}”.

  • LOCALBASE is where all non-X11 pkgs are installed. If you need to construct a -I or -L argument to the compiler to find includes and libraries installed by another non-X11 pkg, use “${LOCALBASE}”.

  • X11BASE is where the actual X11 distribution (from xsrc, etc.) is installed. When looking for standard X11 includes (not those installed by a pkg), use “${X11BASE}”.

  • X11-based packages are special in that they may be installed in either X11BASE or LOCALBASE.

    Usually, X11 packages should be installed under LOCALBASE whenever possible. Note that you will need to include ../../mk/x11.buildlink3.mk in them to request the presence of X11 and to get the right compilation flags.

    Even though, there are some packages that cannot be installed under LOCALBASE: those that come with app-defaults files. These packages are special and they must be placed under X11BASE. To accomplish this, set either USE_X11BASE or USE_IMAKE in your package.

    Some notes: If you need to find includes or libraries installed by a pkg that has USE_IMAKE or USE_X11BASE in its pkg Makefile, you need to look in both ${X11BASE} and ${LOCALBASE}. To force installation of all X11 packages in LOCALBASE, the pkgtools/xpkgwedge package is enabled by default.

  • X11PREFIX should be used to refer to the installed location of an X11 package. X11PREFIX will be set to X11BASE if xpkgwedge is not installed, and to LOCALBASE if xpkgwedge is installed.

  • If xpkgwedge is installed, it is possible to have some packages installed in X11BASE and some in LOCALBASE. To determine the prefix of an installed package, the EVAL_PREFIX definition can be used. It takes pairs in the format “DIRNAME=<package>”, and the make(1) variable DIRNAME will be set to the prefix of the installed package <package>, or “${X11PREFIX}” if the package is not installed.

    This is best illustrated by example.

    The following lines are taken from pkgsrc/wm/scwm/Makefile:

        EVAL_PREFIX+=           GTKDIR=gtk+
        CONFIGURE_ARGS+=        --with-guile-prefix=${LOCALBASE:Q}
        CONFIGURE_ARGS+=        --with-gtk-prefix=${GTKDIR:Q}
        CONFIGURE_ARGS+=        --enable-multibyte
    

    Specific defaults can be defined for the packages evaluated using EVAL_PREFIX, by using a definition of the form:

        GTKDIR_DEFAULT= ${LOCALBASE}
    

    where GTKDIR corresponds to the first definition in the EVAL_PREFIX pair.

  • Within ${PREFIX}, packages should install files according to hier(7), with the exception that manual pages go into ${PREFIX}/man, not ${PREFIX}/share/man.

14.3. Directories used during the build process

When building a package, a number of directories is used to store source files, temporary files, pkgsrc-internal files, and so on. These directories are explained here.

Some of the directory variables contain relative pathnames. There are two common base directories for these relative directories: PKGSRCDIR/PKGPATH is used for directories that are pkgsrc-specific. WRKSRC is used for directories inside the package itself.

PKGSRCDIR

This is an absolute pathname that points to the pkgsrc root directory. Generally, you don't need it.

PKGPATH

This is a pathname relative to PKGSRCDIR that points to the current package.

WRKDIR

This is an absolute pathname pointing to the directory where all work takes place. The distfiles are extraced to this directory. It also contains temporary directories and log files used by the various pkgsrc frameworks, like buildlink or the wrappers.

WRKSRC

This is an absolute pathname pointing to the directory where the distfiles are extracted. It is usually a direct subdirectory of WRKDIR, and often it's the only directory entry that isn't hidden. This variable may be changed by a package Makefile.

14.4. Running a phase

You can run a particular phase by typing make phase, where phase is the name of the phase. This will automatically run all phases that are required for this phase. The default phase is build, that is, when you run make without parameters in a package directory, the package will be built, but not installed.

14.5. The fetch phase

This will check if the file(s) given in the variables DISTFILES and PATCHFILES (as defined in the package's Makefile) are present on the local system in /usr/pkgsrc/distfiles. If they are not present, an attempt will be made to fetch them using commands of the form:

    ${FETCH_CMD} ${FETCH_BEFORE_ARGS} ${site}${file} ${FETCH_AFTER_ARGS}

where ${site} varies through several possibilities in turn: first, MASTER_SITE_OVERRIDE is tried, then the sites specified in either SITES_file if defined, else MASTER_SITES or PATCH_SITES, as applies, then finally the value of MASTER_SITE_BACKUP. The order of all except the first can be optionally sorted by the user, via setting either MASTER_SORT_AWK or MASTER_SORT_REGEX.

14.6. The checksum phase

After the distfile(s) are fetched, their checksum is generated and compared with the checksums stored in the distinfo file. If the checksums don't match, the build is aborted. This is to ensure the same distfile is used for building, and that the distfile wasn't changed, e.g. by some malign force, deliberately changed distfiles on the master distribution site or network lossage.

14.7. The extract phase

When the distfiles are present on the local system, they need to be extracted, as they usually come in the form of some compressed archive format.

By default, all DISTFILES are extracted. If you only need some of them, you can set the EXTRACT_ONLY variable to the list of those files.

Extracting the files is usually done by a little program, mk/scripts/extract, which already knows how to extract various archive formats, so most likely you will not need to change anything here. But if you need, the following variables may help you:

EXTRACT_OPTS_{BIN,LHA,PAX,RAR,TAR,ZIP,ZOO}

Use these variables to override the default options for an extract command, which are defined in mk/scripts/extract.

EXTRACT_USING

This variable can be set to pax, tar or an absolute pathname pointing to the command with which tar archives should be extracted.

If the extract program doesn't serve your needs, you can also override the EXTRACT_CMD variable, which holds the command used for extracting the files. This command is executed in the ${WRKSRC} directory. During execution of this command, the shell variable extract_file holds the absolute pathname of the file that is going to be extracted.

And if that still does not suffice, you can override the do-extract target in the package Makefile.

14.8. The patch phase

After extraction, all the patches named by the PATCHFILES, those present in the patches subdirectory of the package as well as in $LOCALPATCHES/$PKGPATH (e.g. /usr/local/patches/graphics/png) are applied. Patchfiles ending in .Z or .gz are uncompressed before they are applied, files ending in .orig or .rej are ignored. Any special options to patch(1) can be handed in PATCH_DIST_ARGS. See Section 8.3, “patches/*” for more details.

By default patch(1) is given special args to make it fail if the patches apply with some lines of fuzz. Please fix (regen) the patches so that they apply cleanly. The rationale behind this is that patches that don't apply cleanly may end up being applied in the wrong place, and cause severe harm there.

14.9. The tools phase

This is covered in Chapter 15, Tools needed for building or running.

14.10. The wrapper phase

[TODO]

14.11. The configure phase

Most pieces of software need information on the header files, system calls, and library routines which are available on the platform they run on. The process of determining this information is known as configuration, and is usually automated. In most cases, a script is supplied with the distfiles, and its invocation results in generation of header files, Makefiles, etc.

If the package contains a configure script, this can be invoked by setting HAS_CONFIGURE to “yes”. If the configure script is a GNU autoconf script, you should set GNU_CONFIGURE to “yes” instead. What happens in the configure phase is roughly:

    .for d in ${CONFIGURE_DIRS}
            cd ${WRKSRC} && cd ${d} && env ${CONFIGURE_ENV} \
                ${CONFIGURE_SCRIPT} ${CONFIGURE_ARGS}
    .endfor

CONFIGURE_DIRS (default: “.”) is a list of pathnames relative to WRKSRC. In each of these directories, the configure script is run with the environment CONFIGURE_ENV and arguments CONFIGURE_ARGS. The variables CONFIGURE_ENV, CONFIGURE_SCRIPT (default: “./configure”) and CONFIGURE_ARGS may all be changed by the package.

If the program uses an Imakefile for configuration, the appropriate steps can be invoked by setting USE_IMAKE to “yes”. (If you only want the package installed in ${X11PREFIX} but xmkmf not being run, set USE_X11BASE instead.)

14.12. The build phase

For building a package, a rough equivalent of the following code is executed.

    .for d in ${BUILD_DIRS}
            cd ${WRKSRC} && cd ${d} && env ${MAKE_ENV} \
                ${MAKE_PROGRAM} ${BUILD_MAKE_FLAGS} \
                    -f ${MAKEFILE} ${BUILD_TARGET}
    .endfor

BUILD_DIRS (default: “.”) is a list of pathnames relative to WRKSRC. In each of these directories, MAKE_PROGRAM is run with the environment MAKE_ENV and arguments BUILD_MAKE_FLAGS. The variables MAKE_ENV, BUILD_MAKE_FLAGS, MAKEFILE and BUILD_TARGET may all be changed by the package.

The default value of MAKE_PROGRAM is “gmake” if USE_TOOLS contains “gmake”, “make” otherwise. The default value of MAKEFILE is “Makefile”, and BUILD_TARGET defaults to “all”.

14.13. The test phase

[TODO]

14.14. The install phase

Once the build stage has completed, the final step is to install the software in public directories, so users can access the programs and files.

In the install phase, a rough equivalent of the following code is executed. Additionally, before and after this code, much magic is performed to do consistency checks, registering the package, and so on.

    .for d in ${INSTALL_DIRS}
            cd ${WRKSRC} && cd ${d} && env ${MAKE_ENV} \
                ${MAKE_PROGRAM} ${INSTALL_MAKE_FLAGS} \
                    -f ${MAKEFILE} ${BUILD_TARGET}
    .endfor

The variable's meanings are analogous to the ones in the build phase. INSTALL_DIRS defaults to BUILD_DIRS. INSTALL_TARGET is “install” by default, plus “install.man” if USE_IMAKE is defined.

In the install phase, the following variables are useful. They are all variations of the install(1) command that have the owner, group and permissions preset. INSTALL is the plain install command. The specialized variants, together with their intended use, are:

INSTALL_PROGRAM_DIR

directories that contain binaries

INSTALL_SCRIPT_DIR

directories that contain scripts

INSTALL_LIB_DIR

directories that contain shared and static libraries

INSTALL_DATA_DIR

directories that contain data files

INSTALL_MAN_DIR

directories that contain man pages

INSTALL_PROGRAM

binaries that can be stripped from debugging symbols

INSTALL_SCRIPT

binaries that cannot be stripped

INSTALL_GAME

game binaries

INSTALL_LIB

shared and static libraries

INSTALL_DATA

data files

INSTALL_GAME_DATA

data files for games

INSTALL_MAN

man pages

Some other variables are:

INSTALLATION_DIRS

A list of directories relative to PREFIX that are created by pkgsrc at the beginning of the install phase. If this variable is set, NO_MTREE=“yes” is assumed, which means that the package claims to create all needed directories itself before installing files to it. Therefore this variable should only be set in Makefiles that are under control of the package's author.

14.15. The package phase

[TODO]

14.16. Other helpful targets

pre/post-*

For any of the main targets described in the previous section, two auxiliary targets exist with “pre-” and “post-” used as a prefix for the main target's name. These targets are invoked before and after the main target is called, allowing extra configuration or installation steps be performed from a package's Makefile, for example, which a program's configure script or install target omitted.

do-*

Should one of the main targets do the wrong thing, and should there be no variable to fix this, you can redefine it with the do-* target. (Note that redefining the target itself instead of the do-* target is a bad idea, as the pre-* and post-* targets won't be called anymore, etc.) You will not usually need to do this.

reinstall

If you did a make install and you noticed some file was not installed properly, you can repeat the installation with this target, which will ignore the “already installed” flag.

deinstall

This target does a pkg_delete(1) in the current directory, effectively de-installing the package. The following variables can be used to tune the behaviour:

PKG_VERBOSE

Add a "-v" to the pkg_delete(1) command.

DEINSTALLDEPENDS

Remove all packages that require (depend on) the given package. This can be used to remove any packages that may have been pulled in by a given package, e.g. if make deinstall DEINSTALLDEPENDS=1 is done in pkgsrc/x11/kde, this is likely to remove whole KDE. Works by adding “-R” to the pkg_delete(1) command line.

update

This target causes the current package to be updated to the latest version. The package and all depending packages first get de-installed, then current versions of the corresponding packages get compiled and installed. This is similar to manually noting which packages are currently installed, then performing a series of make deinstall and make install (or whatever UPDATE_TARGET is set to) for these packages.

You can use the “update” target to resume package updating in case a previous make update was interrupted for some reason. However, in this case, make sure you don't call make clean or otherwise remove the list of dependent packages in WRKDIR. Otherwise, you lose the ability to automatically update the current package along with the dependent packages you have installed.

Resuming an interrupted make update will only work as long as the package tree remains unchanged. If the source code for one of the packages to be updated has been changed, resuming make update will most certainly fail!

The following variables can be used either on the command line or in /etc/mk.conf to alter the behaviour of make update:

UPDATE_TARGET

Install target to recursively use for the updated package and the dependent packages. Defaults to DEPENDS_TARGET if set, “install” otherwise for make update. e.g. make update UPDATE_TARGET=package

NOCLEAN

Don't clean up after updating. Useful if you want to leave the work sources of the updated packages around for inspection or other purposes. Be sure you eventually clean up the source tree (see the “clean-update” target below) or you may run into troubles with old source code still lying around on your next make or make update.

REINSTALL

Deinstall each package before installing (making DEPENDS_TARGET). This may be necessary if the “clean-update” target (see below) was called after interrupting a running make update.

DEPENDS_TARGET

Allows you to disable recursion and hardcode the target for packages. The default is “update” for the update target, facilitating a recursive update of prerequisite packages. Only set DEPENDS_TARGET if you want to disable recursive updates. Use UPDATE_TARGET instead to just set a specific target for each package to be installed during make update (see above).

clean-update

Clean the source tree for all packages that would get updated if make update was called from the current directory. This target should not be used if the current package (or any of its depending packages) have already been de-installed (e.g., after calling make update) or you may lose some packages you intended to update. As a rule of thumb: only use this target before the first time you run make update and only if you have a dirty package tree (e.g., if you used NOCLEAN).

If you are unsure about whether your tree is clean, you can either perform a make clean at the top of the tree, or use the following sequence of commands from the directory of the package you want to update (before running make update for the first time, otherwise you lose all the packages you wanted to update!):

# make clean-update
# make clean CLEANDEPENDS=YES
# make update

The following variables can be used either on the command line or in /etc/mk.conf to alter the behaviour of make clean-update:

CLEAR_DIRLIST

After make clean, do not reconstruct the list of directories to update for this package. Only use this if make update successfully installed all packages you wanted to update. Normally, this is done automatically on make update, but may have been suppressed by the NOCLEAN variable (see above).

info

This target invokes pkg_info(1) for the current package. You can use this to check which version of a package is installed.

readme

This target generates a README.html file, which can be viewed using a browser such as www/mozilla or www/links. The generated files contain references to any packages which are in the PACKAGES directory on the local host. The generated files can be made to refer to URLs based on FTP_PKG_URL_HOST and FTP_PKG_URL_DIR. For example, if I wanted to generate README.html files which pointed to binary packages on the local machine, in the directory /usr/packages, set FTP_PKG_URL_HOST=file://localhost and FTP_PKG_URL_DIR=/usr/packages. The ${PACKAGES} directory and its subdirectories will be searched for all the binary packages.

readme-all

Use this target to create a file README-all.html which contains a list of all packages currently available in the NetBSD Packages Collection, together with the category they belong to and a short description. This file is compiled from the pkgsrc/*/README.html files, so be sure to run this after a make readme.

cdrom-readme

This is very much the same as the “readme” target (see above), but is to be used when generating a pkgsrc tree to be written to a CD-ROM. This target also produces README.html files, and can be made to refer to URLs based on CDROM_PKG_URL_HOST and CDROM_PKG_URL_DIR.

show-distfiles

This target shows which distfiles and patchfiles are needed to build the package. (DISTFILES and PATCHFILES, but not patches/*)

show-downlevel

This target shows nothing if the package is not installed. If a version of this package is installed, but is not the version provided in this version of pkgsrc, then a warning message is displayed. This target can be used to show which of your installed packages are downlevel, and so the old versions can be deleted, and the current ones added.

show-pkgsrc-dir

This target shows the directory in the pkgsrc hierarchy from which the package can be built and installed. This may not be the same directory as the one from which the package was installed. This target is intended to be used by people who may wish to upgrade many packages on a single host, and can be invoked from the top-level pkgsrc Makefile by using the “show-host-specific-pkgs” target.

show-installed-depends

This target shows which installed packages match the current package's DEPENDS. Useful if out of date dependencies are causing build problems.

check-shlibs

After a package is installed, check all its binaries and (on ELF platforms) shared libraries to see if they find the shared libs they need. Run by default if PKG_DEVELOPER is set in /etc/mk.conf.

print-PLIST

After a “make install” from a new or upgraded pkg, this prints out an attempt to generate a new PLIST from a find -newer work/.extract_done. An attempt is made to care for shared libs etc., but it is strongly recommended to review the result before putting it into PLIST. On upgrades, it's useful to diff the output of this command against an already existing PLIST file.

If the package installs files via tar(1) or other methods that don't update file access times, be sure to add these files manually to your PLIST, as the “find -newer” command used by this target won't catch them!

See Section 10.3, “Tweaking output of make print-PLIST for more information on this target.

bulk-package

Used to do bulk builds. If an appropriate binary package already exists, no action is taken. If not, this target will compile, install and package it (and its depends, if PKG_DEPENDS is set properly. See Section 6.3.1, “Configuration”). After creating the binary package, the sources, the just-installed package and its required packages are removed, preserving free disk space.

Beware that this target may deinstall all packages installed on a system!

bulk-install

Used during bulk-installs to install required packages. If an up-to-date binary package is available, it will be installed via pkg_add(1). If not, make bulk-package will be executed, but the installed binary won't be removed.

A binary package is considered “up-to-date” to be installed via pkg_add(1) if:

  • None of the package's files (Makefile, ...) were modified since it was built.

  • None of the package's required (binary) packages were modified since it was built.

Beware that this target may deinstall all packages installed on a system!