# JPackage Java infrastructure design and packaging policy

Nicolas Mailhot
JPackage Project
Ville Skyttä
JPackage Project

This document describes the packaging policy followed by the JPackage cross-distribution RPM Java packaging project.

$Id: jpackage-1.5-policy.xml,v 1.1 2004/12/15 16:23:45 robert Exp $

## Why ?

Clean Java packaging has historically been a daunting task. Lack of any standard addressing the physical location of files on the system combined with the common use of licensing terms that only allow free redistribution of key components as a part of a greater ensemble has let to the systematic release of self-sufficient applications with built-in copies of external components.

As a consequence applications are only tested with the versions of the components they bundle, a complete Java system suffers from endless duplication of the same modules, and integrating multiple parts can be a nightmare since they are bound to depend on the same elements - only with different and subtly incompatible versions

Different requirements, different bugs.

. Any security or compatibility

For example when the system kernel or C core changes Java assumptions.

upgrade must be performed for each of those duplicated elements.

This problem is compounded by the current practice of folding extensions proper in the JVM itself after a while ; an element that could safely be embedded in a application will suddenly conflict with a JVM part and cause subtle failures.

It is not surprising then that complex Java systems tend to fossilize very quickly, with the cost of maintaining dependencies current growing too high so fast people basically give up on it.

This situation is incompatible with your typical fast-evolving Linux platform. To attain its aim of user- and administrator-friendly rpm packaging of Java applications the JPackage Project had to evolve its own system infrastructure and strict packaging rules.

## General rules

### Be modular

Whenever possible a module shall be broken out in its constituting elements. A given element will have a single name and location on the system. Only a single version of this element shall be installed. Applications sharing the same dependencies shall depend on external packages, not built-in versions.

### Be vendor and implementation agnostic

The user shall be able to choose between the different available implementations of the same element, especially when their licensing terms are not equivalent or their level of completeness unequal. Whenever possible it should be possible to install them in parallel, with the user able to select each of them specifically or let the system choose the best one

Based on licensing terms, known completeness, relative bugginess, market penetration and so on.

.

### Be distribution agnostic

Packages should work on most major rpm-based distributions. This means following standards like the Linux Standard Base, the Filesystem Hierarchy Standard, or freedesktop.org specifications.

Above all usage of distribution-specific facilities

Like Mandrake's `gprintf` function.

is forbidden.

Note this is a pragmatic reading of the standards. Facilities not specified but commonly found are readily used by the project

For example RPM 4 extensions, Debian's &alternatives; system...

and distribution-specific adaptations are tolerated provided they do not interfere with common usage or a generic version is also available.

## Naming

### Package naming

Packages shall be named after the common name of their origin project in small caps. When a package provides an extension that was folded at some point in the Java standard, the -ext (as in external) suffix shall be appended to distinguish between the package name and the extension name. Both the JVMs including this extension and the standalone extension package shall then have the original name as a virtual Provides.

JVM packages shall be named java-standard_version-vendor. Original on-site names are not followed since practice varies wildly from vendor to vendor and version to version.

### Jar file naming

1. If a package provides a single jar it shall have the same name as the package itself, with appended product version.

       jaf-1.0.2.jar

2. A non-versioned symbolic link pointing on the original file shall also be provided.

       jaf-1.0.2.jar
       jaf.jar → jaf-1.0.2.jar

3. If the project name and the commonly used jar name differ a symbolic link with the usual name shall also be provided.

   Single jar complete naming

       jaf-1.0.2.jar
       jaf.jar → jaf-1.0.2.jar
       activation.jar → jaf-1.0.2.jar

4. If the package provides several jars their usual names will be used.

       ant-1.5.3.jar
       ant.jar → ant-1.5.3.jar
       ant-optional-1.5.3.jar
       ant-optional.jar → ant-optional-1.5.3.jar

5. If the number of provided jars exceeds two or if they were at a point provided in a single monolithic jar the files should be placed in a sub-directory named after the package (as for a single jar).

   Bunch of jars naming

       javamail
       javamail/imap-1.3.jar
       javamail/imap.jar → imap-1.3.jar
       javamail/mailapi-1.3.jar
       javamail/mailapi.jar → mailapi-1.3.jar
       javamail/pop3-1.3.jar
       javamail/pop3.jar → pop3-1.3.jar
       javamail/smtp-1.3.jar
       javamail/smtp.jar → smtp-1.3.jar

6. If a project offers the choice of packaging it as a single monolithic jar or several ones, the split packaging will be preferred.

## Directory structure

Core directory structure is provided by the jpackage-utils package. It consists of:

### %{_javadir}

#### `/usr/share/java`

The %{_javadir} rpm macro defines the main jar repository. Historically it was the sole directory used by the 1.0 JPackage distribution, before packaging constraints forced a more complex system. It usually expands into `/usr/share/java`.

All jar files and directories of jar files that do not depend on a particular Java standard version or JNI shall be installed in `%{_javadir}`.

#### `/usr/share/java-ext`

From %{_javadir} we derive `%{_javadir}-ext`. All jar files and directories of jar files that depend on a particular Java standard version but not JNI shall be installed in `%{_javadir}-ext`.

#### `/usr/share/java-x.y.z`

We also derive the `%{_javadir}-x.y.z` directories. They contain symlinks to files or directories of `%{_javadir}-ext` for all elements that are valid for the x.y.z Java standard version. Since an implementation is usually valid for a range of Java standard iterations, a file or directory in `%{_javadir}-ext` will usually have several symbolic links pointing to it. Also of note are the non-versioned symlinks : for two jars named `foo13.jar` and `foo14.jar`, the `foo.jar` symlink will point to `foo13.jar` in `%{_javadir}-1.3.0` and `%{_javadir}-1.3.1`, and `foo14.jar` in `%{_javadir}-1.4.0`, `%{_javadir}-1.4.1` and `%{_javadir}-1.4.2`.

Unfortunately the Java standard has been known to change enough between minor versions that we must take into account the full version and distinguish between `%{_javadir}-1.4.0` and `%{_javadir}-1.4.1`.

Java standard version-specific repositories usage

    /usr/share/java-ext/jsse
    /usr/share/java-ext/jsse/jcert-1.0.3.01.jar
    /usr/share/java-ext/jsse/jcert.jar → jcert-1.0.3.01.jar
    /usr/share/java-ext/jsse/jnet-1.0.3.01.jar
    /usr/share/java-ext/jsse/jnet.jar → jnet-1.0.3.01.jar
    /usr/share/java-ext/jsse/jsse-1.0.3.01.jar
    /usr/share/java-ext/jsse/jsse.jar → jsse-1.0.3.01.jar
    /usr/share/java-1.3.0/jsse → /usr/share/java-ext/jsse
    /usr/share/java-1.3.1/jsse → /usr/share/java-ext/jsse

#### `/usr/share/java-utils`

We also derive `%{_javadir}-utils`. It is used to host java-related scripts and functions, including the main shell-script function library `java-functions`.

### %{_jnidir}: `/usr/lib/java`...

The %{_jnidir} rpm macro defines the main JNI jar repository. Like %{_javadir} it is declined in -ext and -x.y.z variants. It follows exactly the same rules as the %{_javadir}-derived tree structure, except that it hosts jars that use JNI.

%{_jnidir} usually expands into `/usr/lib/java`.

### %{_jvmdir}

#### %{_libdir}/jvm: `/usr/lib/jvm`

The %{_jvmdir} rpm macro defines the root directory under which the different system JVMs are installed. It usually expands into `/usr/lib/jvm`.

#### %{_libdir}/jvm-exports: `/usr/lib/jvm-exports`

From %{_jvmdir} we derive `%{_jvmdir}-exports`. Each subdirectory of `%{_jvmdir}` must have a corresponding one in `%{_jvmdir}-exports`. It is used to register Java extensions bundled with the SDK or RE with symbolic links pointing inside the JVM structure in `%{_jvmdir}`.

The symbolic links shall point to the actual in-JVM jar file providing the extension (though that is not strictly necessary for the system to work), be available in versioned and non-versioned versions and follow general naming rules.

#### %{_libdir}/jvm-private: `/usr/lib/jvm-private`

The `%{_jvmdir}-private` directory contains files that are private and internal to JVMs, but for some reason need to be relocated from their normal location in the JVM dirs. No scripts should reference these files. The contents below this directory are laid out so that there are appropriately versioned directories depending on their purpose, and the actual files are below those.

One example of private files are the Java Cryptography Extension (JCE) policy files; the ones that come with various 1.4.x JVMs are limited in strength and the vendors supply unlimited strength jurisdiction policy files as separate downloads. Typically these files are Java standard version and vendor specific (eg. `java-1.4.2-sun`).

JCE policy file layout for Sun's J2SE 1.4.2

    /usr/lib/jvm-private/java-1.4.2-sun
    /usr/lib/jvm-private/java-1.4.2-sun/jce
    /usr/lib/jvm-private/java-1.4.2-sun/jce/vanilla
    /usr/lib/jvm-private/java-1.4.2-sun/jce/vanilla/US_export_policy.jar
    /usr/lib/jvm-private/java-1.4.2-sun/jce/vanilla/local_policy.jar
    /usr/lib/jvm-private/java-1.4.2-sun/jce/unlimited
    /usr/lib/jvm-private/java-1.4.2-sun/jce/unlimited/US_export_policy.jar
    /usr/lib/jvm-private/java-1.4.2-sun/jce/unlimited/local_policy.jar

The various versions of JCE policy jars are further controlled by the &alternatives; system, using a link pointing to the corresponding jars in the JVM's `jre/lib/security` directory, with the unlimited versions having higher priority than the ones shipped with the JVM (vanilla).

### %{_sysconfdir}/java: `/etc/java`

`%{_sysconfdir}/java` hosts the general configuration files related to the java subsystem, mainly `java.conf`.

### %{_javadocdir}

This is the root of all installed javadoc documentation. It's location and intended usage are debated right now.

### Application-specific directories

1. If an application needs private directories they should be laid out on the system following Filesystem Hierarchy Standard rules.

2. If an application needs to simulate a single-root private tree like on other operating systems (when the FHS requires the subdirectories to be installed on different parts of the system), `%{_datadir}/appname` shall be used as application home with symbolic links pointing to the real locations of subdirectories on the system. Of course it is better to fix the application to understand proper split file layout and avoid the symbolic link indirection altogether.

   FHS and homedir-centered file layout

       /usr/share/tomcat4
       /usr/share/tomcat4/bin
       /usr/share/tomcat4/common → /var/lib/tomcat4/common
       /usr/share/tomcat4/conf → /etc/tomcat4
       /usr/share/tomcat4/logs → /var/log/tomcat4
       /usr/share/tomcat4/server → /var/lib/tomcat4/server
       /usr/share/tomcat4/shared → /var/lib/tomcat4/shared
       /usr/share/tomcat4/temp → /var/cache/tomcat4/temp
       /usr/share/tomcat4/webapps → /var/lib/tomcat4/webapps
       /usr/share/tomcat4/work → /var/cache/tomcat4/work

3. Gratuitous complexity is eschewed. If a directory will always have only a single subdirectory, collapse it

   ie. do not create a single lib subdirectory in `%{_datadir}/appname` just because more complex apps do so. If it will always remain single, use `%{_datadir}/appname` directly.

   .

4. If an application uses classpath elements that are not jar files they should be installed in a private application directory.

   zip, war and other class archive files

   One may encounter other types of archives used in classpaths. If the application has no special need of them being a particular file type, they should be converted to jars and used the usual way. This is particularly true of zip files that were in wide usage at a time but have been deprecated since.

5. If an application provides private jars files it will always be the only user of, they may be installed in private application directories.

   Private jar files

   This implies generic routines can not be used to build the application classpath, so this part must be handled by the application itself or its packager

   Hardcoding classpath bits for example.

   . However when an application can and will read its private jar repository, we do provide the means to maintain the repository parts that are shared with other applications.

## Scripts and actual run-time classpath resolution

We assume the correct environment variables have already been set, ie at least `$JAVA_HOME` to select a JVM in `%{_jvmdir}`, and eventually the relevant `$JAVACMD`, `$LD_ASSUME_KERNEL`, `$LANG`, and `$JAVA_COMPILER` variables.

FIXME

The current implementation is not sane and carries a lot of legacy garbage. A clean implementation would be to read these values in `~/.apprc`, falling back on `/etc/app.conf`, the user environment, `~/.java` and `/etc/java/java.conf`. If `$JAVA_HOME` was still undefined then, use`/usr/lib/jvm/java` as default.

Unfortunately we are not doing so right now.

### General resolution rules

When queried for foo/bar-x.y we search for the `foo/bar-x.y.jar` jar file, then the `foo/bar-x.y` jar directory in the following locations:

- `%{_jvmdir}-exports/name`

  The shadow of the JVM defined by `$JAVA_HOME`=`%{_jvmdir}/name`. This is the JVM-specific repository where we register the Java extensions it includes.

- `%{_jnidir}-java_version`

  Where java_version is the JVM standard Java compliance level as deduced from

      [bob@sys dir]$ $JAVACMD -version

- `%{_javadir}-java_version`

  The java-version specific non-JNI jar repository.

- `%{_jnidir}`

  The generic JNI jar repository.

- `%{_javadir}`

  The generic jar repository.

If we didn't find anything, the search is repeated for `foo/bar.jar`, `foo/bar`, `foo.jar` then at last the `foo` directory. Note the scan is performed for jar/directory pairs ie a subdirectory located in a more specific repository will always have precedence on a jar with the same name located in a less specific repository.

### `find-jar`

`find-jar` element

The `find-classpath` command tests the resolution of a given element. If successful it will return a jar file or a directory. It is solely intended for testing resolution and should not be used in scripts. Even for single-element classpath building the following command is preferred, since single-element searches can produce multiple file results when resolving into a directory.

`find-jar` output

    [bob@sys dir]$ find-jar jndi
    /usr/lib/jvm-exports/java-1.3.1-blackdown/jndi.jar

### `build-classpath`

`build-classpath` element

The `build-classpath` command builds a standard classpath following the general resolution rules. It takes a list of elements as arguments. If an element resolves in a directory, all its children jar files are included in the classpath.

`build-classpath` output

    [bob@sys dir]$ export JAVA_HOME=/usr/lib/jvm/java-1.3.1-blackdown
    [bob@sys dir]$ build-classpath jsse javamail/mailapi jaxp_parser_impl
    /usr/share/java-1.3.1/jsse/jcert-1.0.3.01.jar:/usr/share/java-1.3.1/jsse/jnet-1.0.3.01.jar:/usr/share/java-1.3.1/jsse/jsse-1.0.3.01.jar:/usr/share/java/javamail/mailapi.jar:/usr/share/java/jaxp_parser_impl.jar
    [bob@sys dir]$ export JAVA_HOME=/usr/lib/jvm/java-1.4.1-sun
    [bob@sys dir]$ build-classpath jsse javamail/mailapi jaxp_parser_impl
    /usr/lib/jvm-exports/java-1.4.1-sun/jsse.jar:/usr/share/java/javamail/mailapi.jar:/usr/share/java/jaxp_parser_impl.jar

Like the following command it will fail, return an non-zero error code and write to the error console if one or more of the requested elements were not found. Therefore for a classpath composed of required and optional parts recommended building practice is the following:

Constructing a classpath with mandatory and optional parts

    CLASSPATH=$(build-classpath list_of_required_elements):$(build-classpath list_of_optional_elements 2> /dev/null)

### `build-jar-repository`

`build-jar-repository` -s --soft --symbolic -h --hard -c --copy -p --preserve-naming directory element

The `build-jar-repository` command constructs a flat directory of jar file symlinks following the same rules. It takes a directory name followed by a list of elements as arguments. It will try to create a `[foo][bar]xxx.jar` set of symbolic links in the target directory for each foo/bar requested element

Remember, a single element can resolve in a full directory of jar files.

. The special symlink naming makes it possible to recognize they were created by this command and is required for the next command. One can optionally specify if created files must be hardlinks, softlinks or file copies. The default is symbolic links, and the switch should only be needed for very broken software.

Incremental use

`build-jar-repository` can be safely used on the same directory several times. Previous symlinks won't be removed. If one wants to reset the directory the symbolic links must be cleaned up manually.

Preserving naming

If you just want to populate a static directory with jar links or copies you can use the --preserve-naming `build-jar-repository` argument. This will create files with a name closer to the original ones. Note however this will inhibit any future repository refresh using the next command. Do not use this switch unless you know what you are doing.

--preserve-naming implies --copy unless specified otherwise.

`build-jar-repository` results

    [bob@sys dir]$ export JAVA_HOME=/usr/lib/jvm/java-1.3.1-blackdown
    [bob@sys dir]$ build-jar-repository lib jsse javamail/mailapi
    [bob@sys dir]$ build-jar-repository lib jaxp_parser_impl
    [bob@sys dir]$ tree lib
    [nim@rousalka nim]$ tree lib
    lib
    |-- [javamail][mailapi].jar -> /usr/share/java/javamail/mailapi.jar
    |-- [jaxp_parser_impl].jar -> /usr/share/java/jaxp_parser_impl.jar
    |-- [jsse]jcert-1.0.3.01.jar -> /usr/share/java-1.3.1/jsse/jcert-1.0.3.01.jar
    |-- [jsse]jcert.jar -> /usr/share/java-1.3.1/jsse/jcert.jar
    |-- [jsse]jnet-1.0.3.01.jar -> /usr/share/java-1.3.1/jsse/jnet-1.0.3.01.jar
    |-- [jsse]jnet.jar -> /usr/share/java-1.3.1/jsse/jnet.jar
    |-- [jsse]jsse-1.0.3.01.jar -> /usr/share/java-1.3.1/jsse/jsse-1.0.3.01.jar
    `-- [jsse]jsse.jar -> /usr/share/java-1.3.1/jsse/jsse.jar

### `rebuild-jar-repository`

`rebuild-jar-repository` -s --soft --symbolic -h --hard -c --copy directory

The `rebuild-jar-repository` command refreshes a jar repository created using `build-jar-repository`. It takes a directory name as argument, and is intended to be run after `$JAVA_HOME` was changed to insure all symlinked jar files are still compatible with the new JVM. The same optional arguments as `build-jar-repository` can be used to specify the kind of link to use.

Mixed usage

The very specific naming of the symbolic links used by `build-jar-repository` and `rebuild-jar-repository` makes it possible to mix automated links with manual links and normal files in directories handled by those commands. The scripts will only change their own links.

`rebuild-jar-repository` results

    [bob@sys dir]$ export export JAVA_HOME=/usr/lib/jvm/java-1.4.1-sun
    [bob@sys dir]$ rebuild-jar-repository lib
    [bob@sys dir]$ tree lib
    lib
    |-- [javamail][mailapi].jar -> /usr/share/java/javamail/mailapi.jar
    |-- [jaxp_parser_impl].jar -> /usr/share/java/jaxp_parser_impl.jar
    `-- [jsse].jar -> /usr/lib/jvm-exports/java-1.4.1-sun/jsse.jar

When element resolution fails...

`rebuild-jar-repository` will still create a symbolic link unlike `build-jar-repository`. This choice was made to avoid corrupting the element list when selecting an incomplete java system

Therefore another `rebuild-jar-repository` after the necessary jars were installed or a more complete JVM was selected will work as if the incomplete system was never tried.

. The created symbolic link points to a bogus location and should always be "broken".