JEP 220: Modular Run-Time Images

Summary

Restructure the JDK and JRE run-time images to accommodate modules and to improve performance, security, and maintainability. Define a new URI scheme for naming the modules, classes, and resources stored in a run-time image without revealing the internal structure or format of the image. Revise existing specifications as required to accommodate these changes.

Goals

• Adopt a run-time format for stored class and resource files that:

  • Is more time- and space-efficient than the legacy jar format, which in turn is based on the ancient zip format;

  • Can locate and load class and resource files on a per-module basis;

  • Can store class and resource files from JDK modules and from library and application modules; and

  • Can be extended to accommodate additional kinds of data going forward, such as precomputed JVM data structures and precompiled native code for Java classes.

• Restructure the JDK and JRE run-time images to draw a clear distinction between files that developers, deployers, and end-users can rely upon and, when appropriate, modify, in contrast to files that are internal to the implementation and subject to change without notice.

• Provide supported ways to perform common operations that today can only be done by inspecting the internal structure of a run-time image such as, e.g., enumerating all of the classes present in an image.

• Enable the selective de-privileging of JDK classes that today are granted all security permissions but do not actually require those permissions.

• Preserve the existing behavior of well-behaved applications, i.e., applications that do not depend upon internal aspects of JRE and JDK run-time images.

Success Metrics

Modular run-time images equivalent to the JRE, JDK, and Compact Profile images of the immediately-preceding JDK 9 build must not regress on a representative set of startup, static footprint, and dynamic footprint benchmarks.

Non-Goals

• It is not a goal to preserve all aspects of the current run-time image structure.

• It is not a goal to preserve the exact current behavior of all existing APIs.

Motivation

Project Jigsaw aims to design and implement a standard module system for the Java SE Platform and to apply that system to the Platform itself, and to the JDK. Its primary goals are to make implementations of the Platform more easily scalable down to small devices, improve the security and maintainability, enable improved application performance, and provide developers with better tools for programming in the large.

This JEP is the third of four JEPs planned for Project Jigsaw. The earlier JEP 200 defines the structure of the modular JDK, and JEP 201 reorganized the JDK source code into modules. A later JEP will introduce the actual module system.

Description

Current run-time image structure

The JDK build system presently produces two types of run-time images: A Java Runtime Environment (JRE), which is a complete implementation of the Java SE Platform, and a Java Development Kit (JDK), which embeds a JRE and includes development tools and libraries. (The three Compact Profile builds are subsets of the JRE.)

The root directory of a JRE image contains two directories, bin and lib, with the following content:

• The bin directory contains essential executable binaries, and in particular the java command for launching the run-time system. (On the Windows operating system it also contains the run-time system's dynamically-linked native libraries.)

• The lib directory contains a variety of files and subdirectories:

  • Various .properties and .policy files, most of which may be, though rarely are, edited by developers, deployers, and end users;

  • The endorsed directory, which does not exist by default, into which jar files containing implementations of endorsed standards and standalone technologies may be placed;

  • The ext directory, into which jar files containing extensions or optional packages may be placed;

  • Various implementation-internal data files in assorted binary formats, e.g., fonts, color profiles, and time-zone data;

  • Various jar files, including rt.jar, which contain the run-time system's Java class and resource files.

  • The run-time system's dynamically-linked native libraries on the Mac OS, Linux, and Solaris operating systems. (In the latter two cases the libraries are placed in the lib/$ARCH subdirectory where $ARCH is the name of the CPU-architecture family supported by the image, e.g., amd64.)

A JDK image includes a copy of the JRE in its jre subdirectory and contains additional subdirectories:

• The bin directory contains command-line development and debugging tools, e.g., javac, javadoc, and jconsole, along with duplicates of the binaries in the jre/bin directory for convenience;

• The demo and sample directories contain demonstration programs and sample code, respectively;

• The man directory contains UNIX-style manual pages;

• The include directory contains C/C++ header files for use when compiling native code that will interface directly with the run-time system; and

• The lib directory contains various jar files and other types of files comprising the implementations of the JDK's tools, among them tools.jar, which contains the classes of the javac compiler.

The root directory of a JDK image, or of a JRE image that is not embedded in a JDK image, also contains various COPYRIGHT, LICENSE and README files and also a release file that describes the image in terms of simple key/value property pairs, e.g.,

JAVA_VERSION="1.9.0"
OS_NAME="Linux"
OS_VERSION="2.6"
OS_ARCH="amd64"

New run-time image structure

The present distinction between JRE and JDK images is purely historical, a consequence of an implementation decision made late in the development of the JDK 1.2 release and never revisited. The new image structure will eliminate this distinction: A JDK image will simply be a run-time image that happens to contain the full set of development tools and other items historically found in the JDK.

A modular run-time image will contain the following directories:

• The bin directory will contain any command-line launchers defined by the modules linked into the image. (On Windows it will continue to contain the run-time system's dynamically-linked native libraries.)

• The conf directory will contain the .properties, .policy, and other kinds of files intended to be edited by developers, deployers, and end users, which were formerly found in the lib directory or subdirectories thereof.

• The lib directory on Mac OS, or the lib/$ARCH directory on Linux and Solaris, will contain the run-time system's dynamically-linked native libraries, as it does today. These may be linked against by programs that embed the run-time system.

• All other files and directories in the lib directory must be treated as private implementation details of the run-time system. They are not intended for external use and their names, format, and content will be subject to change without notice.

• A full JDK image will, additionally, contain the demo, sample, man, and include directories, as it does today.

The root directory of a modular run-time image will also, of course, contain the necessary COPYRIGHT, LICENSE, README, and release files. To make it easy to tell which modules are present in a run-time image the release file will be augmented with a new property, MODULES, which will be a space-separated list of the names of those modules. The list will be topologically ordered according to the modules' dependence relationships, so the java.base module will always be first.

Removed: The endorsed-standards override mechanism

The endorsed-standards override mechanism allows implementations of newer versions of standards maintained outside of the Java Community Process, or of standalone APIs that are part of the Java SE Platform yet continue to evolve independently, to be installed into a run-time image.

The endorsed-standards mechanism is presently defined in terms of a path-like system property, java.endorsed.dirs, and a default value for that property, $JAVA_HOME/lib/endorsed. A jar file containing a newer implementation of an endorsed standard or standalone API can be installed into a run-time image by placing it in one of the directories named by the system property, or by placing it in the default lib/endorsed directory if the system property is not defined. Such jar files are prepended to the JVM's bootstrap class path at run time, thereby overriding any definitions stored in the run-time system itself.

A modular image is composed of modules rather than jar files. Going forward we expect to support endorsed standards and standalone APIs in modular form only, via the concept of upgradeable modules. We therefore propose to remove the java.endorsed.dirs system property, the lib/endorsed directory, and the code that implements this mechanism. To help identify any existing uses of this mechanism we will modify the compiler and the launcher to fail if this system property is set or if the lib/endorsed directory exists.

Removed: The extension mechanism

The extension mechanism allows jar files containing APIs that extend the Java SE Platform to be installed into a run-time image so that their contents are visible to every application that is compiled with or runs on that image.

The mechanism is defined in terms of a path-like system property, java.ext.dirs, and a default value for that property composed of $JAVA_HOME/lib/ext and a platform-specific system-wide directory (e.g, /usr/java/packages/lib/ext on Linux). It works in much the same manner as the endorsed-standards mechanism except that jar files placed in an extension directory are loaded by the run-time environment's extension class loader, which is a child of the bootstrap class loader and the parent of the system class loader, which actually loads the application to be run from the class path. Extension classes therefore cannot override the JDK classes loaded by the bootstrap loader but they are loaded in preference to classes defined by the system loader and its descendants.

The extension mechanism was introduced in JDK 1.2, which was released in 1998, but in modern times we have seen little evidence of its use. This is not surprising, since most Java applications today place the libraries that they need directly on the class path rather than require that those libraries be installed as extensions of the run-time system.

It is technically possible, though awkward, to continue to support the extension mechanism in the modular JDK. To simplify both the Java SE Platform and the JDK we propose to remove the java.ext.dirs system property, the lib/ext directory, and the code that implements this mechanism. To help identify any existing uses of this mechanism we will modify the compiler and the launcher to fail if this system property is set or if the lib/ext directory exists. The compiler and the launcher will ignore the platform-specific system-wide extension directory by default, but if the -XX:+CheckEndorsedAndExtDirs command-line option is specified then they will fail if that directory exists and is not empty.

Several features associated with the extension mechanism will be retained, since they are useful in their own right:

• The Class-Path manifest attribute, which specifies jar files required by another jar file;

• The {Specification,Implementation}-{Title,Version,Vendor} manifest attributes, which specify package and jar-file version information;

• The Sealed manifest attribute, which seals a package or a jar file; and

• The extension class loader itself.

The extension class loader will be retained in order to maintain compatibility. For a class Foo loaded by the system class loader, in particular, the expression

Foo.class.getClassLoader().getParent() != null

will remain true.

Removed: rt.jar and tools.jar

The class and resource files previously stored in lib/rt.jar, lib/tools.jar, lib/dt.jar, and various other internal jar files will now be stored in a more efficient format in implementation-specific files in the lib directory. The format of these files will not be specified and is subject to change without notice.

The removal of rt.jar and similar files leads to three distinct problems:

1. Existing standard APIs such as the ClassLoader::getSystemResource method return URL objects to name class and resource files inside the run-time image. For example, when run on JDK 8 the code

    ClassLoader.getSystemResource("java/lang/Class.class");

   returns a jar URL of the form

    jar:file:/usr/local/jdk8/jre/lib/rt.jar!/java/lang/Class.class

   which, as can be seen, embeds a file URL to name the actual jar file within the run-time image. The getContent method of that URL object can be used to retrieve the content of the class file, via the built-in protocol handler for the jar URL scheme.

   A modular image will not contain any jar files, so URLs of the above form make no sense. The specifications of getSystemResource and related methods, fortunately, do not require the URL objects returned by these methods actually to use the jar scheme. They do, however, require that it be possible to load the content of a stored class or resource file via these URL objects.

2. The java.security.CodeSource API and security-policy files use URLs to name the locations of code bases that are to be granted specified permissions. Components of the run-time system that require specific permissions are currently identified in the lib/security/java.policy file via file URLs. The elliptic-curve cryptography provider, e.g., is identified as

    file:${java.home}/lib/ext/sunec.jar

   which, obviously, will have no meaning in a modular image.

3. IDEs and other kinds of development tools require the ability to enumerate the class and resource files stored in a run-time image, and to read their contents. Today they often do this directly by opening and reading rt.jar and similar files. This will, of course, not be possible with a modular image.

New URI scheme for naming stored modules, classes, and resources

To address the above three problems we propose to define a new URL scheme, jrt, for naming the modules, classes, and resources stored in a run-time image without revealing the internal structure or format of the image.

A jrt URL is a hierarchical URI, per RFC 3986, with the syntax

jrt:/[$MODULE[/$PATH]]

where $MODULE is an optional module name and $PATH, if present, is the path to a specific class or resource file within that module. The meaning of a jrt URL depends upon its structure:

• jrt:/$MODULE/$PATH refers to the specific class or resource file named $PATH within the given $MODULE.

• jrt:/$MODULE refers to all of the class and resource files in the module $MODULE.

• jrt:/ refers to the entire collection of class and resource files stored in the current run-time image.

These three forms of jrt URLs address the above problems as follows:

1. APIs that presently return jar URLs will now return jrt URLs. The above invocation of ClassLoader::getSystemResource, e.g., will now return the URL

    jrt:/java.base/java/lang/Class.class

   A built-in protocol handler for the jrt scheme will be defined so that the getContent method of such URL objects retrieves the content of the named class or resource file.

2. Security-policy files and other uses of the CodeSource API can use jrt URLs to name specific modules for the purpose of granting permissions. The elliptic-curve cryptography provider, e.g., can now be identified by the jrt URL

    jrt:/jdk.crypto.ec

   Other modules that are currently granted all permissions but do not actually require them can trivially be de-privileged, i.e., given precisely the permissions they require.

3. A built-in NIO FileSystem provider will be defined for the jrt URL scheme so that development tools can enumerate and read the class and resource files in a run-time image by loading the FileSystem named by the URL jrt:/, as follows:

   FileSystem fs = FileSystems.getFileSystem(URI.create("jrt:/"));
   byte[] jlo = Files.readAllBytes(fs.getPath("java.base",
                                              "java/lang/Object.class"));

   For tools that support the development of code for JDK 9 but which themselves run on JDK 8, a copy of this filesystem provider suitable for use on JDK 8 will be placed in the root directory of JDK 9 run-time images, in a file named jrt-fs.jar.

(The jrt URL protocol handler will not return any content for URLs of the second and third forms.)

Build-system changes

The build system will be modified to produce the new run-time image format described above. We will also take the opportunity here, finally, to rename the images/j2sdk-image, images/j2re-image, and images/j2re-compact$N-image directories to images/jdk, images/jre, and images/jre-compact$N, respectively.

Minor specification changes

JEP 162, implemented in JDK 8, made a number of changes to prepare the Java SE Platform and the JDK for the modularization work proposed here and in related JEPs. Among those changes were the removal of normative specification statements that require certain configuration files to be looked up in the lib directory of run-time images, since those files will now be placed in the conf directory. Most of the SE-only APIs with such statements were so revised as part of Java SE 8, but some APIs shared across the Java SE and EE Platforms still contain such statements:

• javax.xml.stream.XMLInputFactory specifies ${java.home}/lib/stax.properties (JSR 173).

• javax.xml.ws.spi.Provider specifies ${java.home}/lib/jaxws.properties (JSR 224).

• javax.xml.soap.MessageFactory, and related classes, specify ${java.home}/lib/jaxm.properties (JSR 67).

These statements will be revised so as not to mandate the lib directory as they do now.

Open issues

1. Some changes to how fonts are configured may yet be required.

2. The lib/security directory still contains two jar files, whose contents are simply the local and US-export cryptography policy files; we intend to replace these jar files with their content (8061842).

3. The lib/$ARCH directory is only present in Linux and Solaris builds. It is a vestigial remnant of images that could support multiple CPU architectures, which is no longer a requirement. We will investigate whether its content can be placed directly in the lib directory, as is done on Mac OS and Windows, in which case the lib/$ARCH directory will no longer be needed (8066474).

4. The content of the demo, sample, and man directories should ideally be derived from appropriate modules; we will investigate how best to do this (8066476).

5. The javax.activation.MailcapCommandMap class, and related classes, specify ${java.home}/lib/mailmap (JSR 925); this needs to be revised so as not to mandate the lib directory.

6. Early feedback from tool maintainers has suggested that it would be very useful to have a way to locate class and resource files in the jrt file system without having to know which modules contain them; we have a proposal in mind for this but it will be implemented after the initial integration of this JEP (8066492).

Testing

Some existing tests make direct use of run-time image internals (e.g., rt.jar) or refer to system properties (e.g., java.ext.dirs) that no longer exist. These tests will be fixed.

We plan to publish early-access builds containing these changes and then encourage members of the wider Java community to test their tools, libraries, and applications against these builds to help tease out any remaining compatibility issues.

Risks and Assumptions

The central risks of this proposal are ones of compatibility, summarized as follows:

• A JDK image will, as noted above, no longer contain a jre subdirectory. Existing code that assumes the existence of that directory might not work correctly.

• The system properties java.endorsed.dirs and java.ext.dirs will, as noted above, no longer be defined. Existing code that assumes these properties to have non-null values might not work correctly.

• JDK and JRE images will, as noted above, no longer contain the files lib/rt.jar, lib/tools.jar, lib/dt.jar, and other internal jar files. Existing code that assumes the existence of these files might not work correctly.

• Existing standard APIs that return URL objects to name class and resource files inside the run-time image will, as noted above, now return jrt URLs. Existing code that expects these APIs to return jar URLs might not work correctly. Such code was, e.g., already found in the Glassfish application server.

• The internal system property sun.boot.class.path will no longer name a sequence of jar files and directories. Existing code that depends upon this property might not work correctly.

• Class and resource files previously found in lib/tools.jar and visible only when that file was added to the class path will now, in a JDK image, be visible via the system class loader or, in some cases, the bootstrap class loader. The modules containing these files will not, however, be mentioned in the application class path, i.e., in the value of the system property java.class.path.

• Class and resource files previously found in lib/dt.jar and visible only when that file was added to the class path will now be visible via the bootstrap class loader and present in both the JRE and the JDK.

• The defining class loader of the types in some existing packages will change. Existing code that makes assumptions about the class loaders of these types might not work correctly. The specific changes are:

  Package                          Old loader     New loader
  -----------------------------    -----------    -----------
  com.sun.crypto                   extension      boot
  com.sun.jndi.dns                 boot           extension
  com.sun.jndi.url.dns             boot           extension
  com.sun.tools.corba.se.idl       application    boot
  com.sun.tools.script             application    boot
  com.sun.tracing                  boot           application
  sun.security.tools.policytool    boot           application
  sun.tools.jar                    boot           application
  sun.tracing.dtrace               boot           application

  These changes are a consequence of the way in which components that contain both APIs and tools were modularized. The classes of such a component were historically split between rt.jar and tools.jar, but now all such classes will be in a single module.

• The bin directory in a JRE image will contain a few commands that previously were found only in JDK images, namely appletviewer, idlj, java-rmi.cgi, jrunscript, and jstatd. As with the previous item, these changes are a consequence of the way in which components that contain both APIs and tools were modularized.

It is impossible to determine the full impact of these changes in the abstract. We must therefore rely upon extensive internal and—especially—external testing. Sophisticated applications such as IDEs are more likely to be affected by these changes than are straightforward libraries and simpler applications. If some of these changes prove to be insurmountable hurdles for developers, deployers, or end users then we will investigate ways to mitigate their impact.

Dependences

This JEP is the third of four JEPs for Project Jigsaw. It depends upon JEP 201, which reorganized the JDK source code into modules and upgraded the build system to compile modules. It also depends upon earlier preparatory work done in JEP 162, implemented in JDK 8.