Class Loading
In Java, a ClassLoader is responsible for laoding classes. When a ClassLoader X loaded Class A, Class A will ever ask ClassLoader X to load classes. If ClassLoader Y loads Class A (let's call it A'), A and A' are incompatible with each other, because they were loaded using different ClassLoaders and therefore the Class' versions may be different.
ClassLoaders are organized hierarchial: ClassLoaders can have one parent. A ClassLoader always asks its parent to load a class before trying to load a class itself. This mechanism makes sure that only one version of a class is loaded in an application.
For more information of Java class loading, see [1], [2], [3].
Observation: Java supports only one version of a class by default which is not suitable for a dynamic peer2peer environment.
las2peer's goal is it to simplify service deployment by making services self-deploying and self-replicating when more capacity is needed. This means, that las2peer needs to be able to load services from the network and hot-plug those services. As services should not be trusted by default and to avoid version conflicts (i.e. same class names or libraries used of a different version), services need to be isolated from the las2peer core and other services.
Our first approach was it to manage libraries (depending on other libraries) in a so called repository and see las2peer services as a library which exposes a service class. Las2peer resolved all dependencies on the fly before starting a service. It has turned out that is infeasible to manage each library itself in the network, because some questions are raised:
- Who manages a library? Most likely, the original author may not upload his library itself to the las2peer network.
- What about compatibility? How to ensure that the requested library version is the one that is compatible with a service?
To circumvent those issues, we shifted the dependency resolution and maintaining dependencies to the build step: The developer packages all used libraries together with his service into a single jar, called a service bundle. A service bundle exposes one and only one service to the outside. When las2peer loads a service from the network, it only needs to load a sigle archive and deploy it on a node.
To load such a service bundle securely, it needs to fulfill the following requirements:
- a service should not have access to sensible information (i.e. unlocked private keys) on the node
- a service should not have influence on another service
- service-to-service communication should be done via RMI calls only
We distinguish between two types of classes: The first one is the platform containing Java, las2peer core and connectors, such as the WebConnector and dependencies. The platform is installed and updated manually. The platform classes need to be unique, therefore Java's default AppClassLoader is used to load platform classes. The second types are the service bundles, each loaded within its own ServiceClassLoader, which has access to all classes in the service bundle plus some platform classes.
To sum up:
- The las2peer core is loaded using the regular Java class loader.
- To start a service, the core asks the ClassManager for the class. If the class is not instantiated yet, the Repository is asked for a service bundle, which is then loaded into a ServiceClassLoader.
- If a service class wants to load a service, the platform class loader is first asked for the class. If the class is on a whitelist, then this class is taken. If the class is blacklisted or not available via the platform class loader, the class is loaded from the service bundle.
Legacy document: As of las2peer 0.7, dependencies of a service are bundled and distributed in a single jar file. A service now has a single class loader for the service and all its dependencies. Furthermore, this class loader blocks access to some classes to improve security.
Since las2peer wants to provide automatic deployment of services, and services do not know each other, it needs to support multiple versions of a class dependend on the service, allow hotplugging of services and needs to be flexible enough to allow class loading via the network in future. Also, if two services use the same library and version, classes should not be loaded twice into memory.
We distinguish between two types of classes: The first one is the platform containing Java, las2peer core and connectors, such as the WebConnector and dependencies. The platform is installed and updated manually. The platform classes need to be unique, therefore Java's default AppClassLoader is used to load platform classes.
The second types are libraries, which are both services and their dependencies. These classes should be dynamically loadable from different sources. Before getting into class loading of these classes we have a look on how dependencies are organized:
Each library is stored in a jar file containing dependency information as suggested by OSGi: The file META-INF/MANIFEST.MF contains a list of libraries and a range of versions in the format MAJOR.MINOR.MICRO-BUILDNUMBER.
All libraries are loaded from a repository. At the moment, the only supported repository is the FileSystemRepository which loads files from the local "service" directory. However, implementing repositories loading classes over the network are straight-forward to implement.
In the design, a few facts should be considered:
- A library loaded by a service must load classes from the service's dependencies.
- A service should not influence any other service, thus do not break any dependencies.
- A malicious service should not have access to classes of another service. This means that static classes need to be unique per service.
- Services do not communicate directly (for example using 3rd part libraries), communication is only done through the core. As consequence, classes loaded in one service need not to be compatible with classes loaded in any other service.
- Classes are unique in all dependencies of a service.
- The core needs to have access to (and only to) the service class.
This means, that if two services use the same library of the same version, each service needs an own classloader for this library. Sharing the LibraryClassLoader is not possible because if a library wants to load a class it would be indeterministic from which service the class has to be loaded.
When loading a service, the core asks the L2pClassManager to return the service class. The L2pClassManager resolves dependencies asks the Repository to return a LoadedLibrary which can return binary definitions; also binary definitions are stored here. The L2pClassManager caches each LoadedLibrary using a LoadedLibraryCache. Then, a BundleClassManager (having the core's ClassLoader as parent) for the service is created. The BundleClassManager has multiple LibraryClassLoaders as children, each responsible for loading one library. To save memory, the binary definition of the LoadedLibrary is passed (as pointer) to the LibraryClassLoader. Note that only LibraryClassLoader extends Java's ClassLoader.
All classes loaded from a LibraryClassLoader ask the LibraryClassLoader to load classes. The LibraryClassLoader first asks the BundleClassManager to load classes, then it tries to load the class itself. The BundleClassManager first asks the core's ClassLoader, then it asks its LibraryClassLoaders (except the one that asked for the class).
Class loading is transparent to the service developer. Core devs should be aware that the core does not have access to service classes. If the core needs to instantiate a service class, take a Class as parameter, Class.forName will throw a ClassNotFoundException.
A Repository needs to contain all services that should be started and all of their dependencies. As mentioned above, currently only a FileSystemRepository is implemented, which loads jars from a local folder.
For more information see [4].
[1] http://docs.oracle.com/javase/7/docs/api/java/lang/ClassLoader.html
[2] https://docs.oracle.com/javase/specs/jls/se8/html/jls-12.html#jls-12.3
[3] https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-5.html#jvms-5.4.3
[4] LAS2peer - A Framework for Distributing Community Services in a Peer-to-Peer Infrastructure, Holger Janßen, 2013 (not publicly available)