add developmer_manual

docs/developer_manual/00-creating-a-new-layer.md

@@ -0,0 +1,149 @@
+# Creating a new layer
+
+The different functionality of the emulator is separated into different layers. `Base` layer, for example, describes the base of the emulation: what autonomous systems (AS) and internet exchanges (IX) are in the emulator, what nodes and networks are in each of the AS, and how nodes are connected with networks. `Bgp` layer, on the other hands, describes how ASes and where are peered with each other, and what's the relation between the peers.
+
+To add new functionality into the emulator that works with the entire emulation as a whole, you can create a new layer. Note that if you are creating something that only works with a single node, like a service (like a webapp), you should create a service instead.
+
+This guide is going to cover the following topics:
+
+- Render and configuration.
+- Why layers.
+- Implementing and working with the `Layer` interface.
+    - Implementing the `Configurable` interface.
+    - Implementing the `Printable` interface.
+    - Working with layer dependencies.
+- Working with the `Registry`.
+- Working with other layers.
+- Working with merging.
+
+## Render and configuration
+
+Render and configuration are two separate steps in the emulator.
+
+Layers will have no knowledge of what or how other layers are configured prior to the render stage. Layers must keep track of the changes they try to make internally within the class. The process of actually makeing changes, like adding nodes, networks, peering configurations, or other fancy stuff like DNS, web server, etc., into the emulation is called rendering.
+
+During the render stage, the `render` method of each layer will be called in order of their dependencies, and a `Emulator` object will be passed in. The `Emulator` object allows the different layers to collaboratively build a single emulation. 
+
+Configuration is an optional step. In the configure stage, layers will be provided with access to the emulator object. The configure stage allows the layer developer to have more control over the emulation construction.
+
+In the configure stage, layers should register the data that other layers might need. For example, in the `Base` layer, we register the nodes, networks, etc., and resolv all pending networks joins. Layers, however, should not make irreversible changes, as there are chances that other layers will add new data to the emulator.
+
+The configure stage is especially useful if a layer wants to make changes to another layer but still requires the other layer to have configured the emulator first. Currently, this is used in the `ReverseDomainName` and `CymruIpOrigin` service, both of which create a new zone in the `DomainName` service. They do so in the configure stage, as `DomainName` compiles the `Zone` data structure to zone files in the render stage, and additional zone added after the render stage won't be included in the final output. 
+
+## Why layers?
+
+In the old design, layers are allowed to access the emulator object and registry outside the render stage, which makes layers deeply coupled with a particular emulator scenario code.
+
+One example of this is the DNS layer. We once bulit a DNS infrastructure with the DNS layer. The old DNS layer uses the host node object to keep track of what nodes are hosting what zone (root zone, com TLD, net TLD, etc.). The `Base` layer creates the node objects, which means now the DNS layer is coupled with this particular `Base` layer. To use this DNS infrastructure in another emulation, we have to copy the `Base` layer along with it, which is not ideal as the base layer contains other ASes and IXes, which might cause conflicts in the other emulation where we try to port to.
+
+Therefore, we make the design decision to have layers keep track of the changes they try to make internally and only make changes during the render stage, so that the individual layers can be easily taken out and moved to other emulations. In the new design, all services layers (DNS is one of the service layers) keep track of nodes they try to install on with IP address or node name, and they find the node to install the server during render, therefore removes the dependencies on the node objects and `Base` layer.
+
+Just note that the configuration stage is completely optional. If you don't feel like you need it, there is no need to have it.
+
+## Implementing the `Layer` interface
+
+To create a new layer, one will need to implement the `Layer` interface. The `Layer` interface is actually fairly simple. It only has two methods:
+
+### `Layer::getName`
+
+All layers must implement the `Layer::getName` method. This method takes no input and returns a string indicating the name of this layer. The name of the layer must be unique. The name of the layer will be used as the identifier for the layer object in the emulator.
+
+### `Layer::render`
+
+All layers must implement the `Layer::Render` method. This method takes one parameter, the reference to the `Emulator` class instance, as input. It does not need to return anything. Layers should make changes to the objects in the emulation here.
+
+In the render stage, layers can safely assume that no further API calls will be made on the `Layer` class itself. An example of this is the DNS service layer. The DNS layer will start collecting nameservers for zones, finalize glue records, convert the internal tree structure to zone files, etc. No more changes can be made to the layer - meaning no new domains can be added, no more name servers can be added. 
+
+## Implementing the `Configurable` interface
+
+The `Layer` interface is derived from the `Configurable` interface. The `Configurable` class has only one method:
+
+### `Configurable::configure`
+
+Layers may optionally implement the `Configurable::configure` method. This method takes one parameter, the reference to the `Emulator` class instance, as input. It does not need to return anything. A default implementation of `Configurable::configure` is included in the `Configurable` class - it just returns when called. Layers can make changes to the objects, or even another layer, in the emulation here.
+
+Unlike the render stage, the layer should allow future changes to the layer, as other layers may make changes during their configuration stage. An example of this is, again, the DNS service layer. Layers like `ReverseDomainNameService` collect IP addresses in the emulator and assign reverse hostnames to them by creating a new `in-addr.arpa` zone in the DNS layer. 
+
+In the configuration stage, a layer should register objects that may be useful to other layers in the emulator. An example of this is the `Base` layers. `Node` and `Network` objects are, in fact, registered in the emulator in the configuration stage by the `Base` layer. 
+
+A `Node` object represents a network device in the emulator. It can be a server, a router, a user machine, or just any other device connected to the network. A `Network` object represents a network in the emulator. It can be a private local network within an AS, or a public global network like an internet exchange. For details on how to work with them, please consult the API documentation.
+
+## Working with the `Registry`
+
+To access another object in the emulator, one will use the `Registry`. Consider `Registry` as a database of objects in the emulator. Nodes, networks, and even other layers are registered in the `Registry`. To retrieve the `Registry`, use `Emulator::getRegistry`. The `Emulator` object is passed to layers in both render and configuration stages. 
+
+To retrieve an object from the `Registry`, one will need to know the scope, type, and name of the object. Scope is usually the name of the owner of the object. For private `Node` and `Network`, it's usually their ASN. Type, as the name suggested, specifies the type of the object. Examples are `network` for `Network`, `hnode` for `Node` with host role, and `rnode` for `Node` with router role. And name defines the name of the object.
+
+For example, to get a router node with name `router0` from `AS150`, one can do:
+
+```python
+r0_150: Router = emulator.getRegistry().get('150', 'rnode', 'router0')
+```
+
+Example of locations of some other objects in the emulator (in the format of `scope/type/name`):
+
+- AS150's host node with name `web_server`: `150/hnode/web_server`.
+- IX100's peering LAN: `ix/net/ix100`.
+- IX100's router server node: `ix/rs/ix100`.
+- The `Base` layer: `seedemu/layer/Base`.
+
+To test if an object exist, use `has`:
+
+```python
+if registry.has('150', 'rnode', 'router0'):
+    # do something...
+```
+
+To iterate over all objects, use `getAll`:
+
+```python
+for ((scope, type, name), obj) in registry.getAll().items():
+    # do something...
+```
+
+To iterate over all objects of a type in a scope, use `getByType`:
+
+```python
+for router in registry.getByType('150', 'rnode'):
+    # do something...
+```
+
+To register an object, use `register`:
+
+```python
+registry.register('some_scope', 'some_type', 'some_name', some_object)
+```
+
+For an object to be registrable, it must implement the `Registrable` interface. Refer to the API documentation for details.
+
+## Working with other layers
+
+As mentioned earlier, layers may access each other and make changes to each other. While having the two-stage process helps with making changes in order, sometimes it may be necessary to have some layers to be configured before or after another layer. The most obvious example of this is the `Base` layer. The `Base` layer must be configured before all other layers. Otherwise, they will not be able to retrieve the `Node` and `Network` objects from the `Registry`.
+
+A layer can require itself to be rendered before or after another layer or ask the emulator to error out when another layer does not exist. This mechanism is called layer dependency. To add a dependency, layer can call `Layer::addDependency`.
+
+`addDependency` takes three parametners:
+
+- `layerName`: string, name of the layer to target.
+- `reverse`: bool, when `True`, this `addDependency` creates a reverse dependency. Regular dependency requires the target layer to be rendered/configured before the current layer, while a reverse dependency requires the current layer to be rendered/configured before the target layer.
+- `optional`: bool, when `True`, the emulator will contine render even if the target layer does not exist in the emulation.
+
+## Working with merging
+
+Merging is an important feature of the emulator. It enables the re-use of existing layers in another emulation. One can build and public a full DNS infrastructure without the base layer. Other users can then merge the DNS infrastructure with their own emulation that has the base layer without having to re-build the DNS infrastructure themself.
+
+During merging, the same layer may exist in both emulators. A `Merger` needs to be implemented to handle the merging. The `Merger` interface are as follow:
+
+### `Merger::getName`
+
+The `getName` call takes no parameter and should return the name of the merger. This should be a unique identifier of the merger. 
+
+### `Merger::getTargetType`
+
+The `getTargetType` call takes no parameter and should return the name of type that this merger targers. For example, a merger that targets the `Base` layer should return `BaseLayer` here.
+
+### `Merger::doMerge`
+
+The `doMerge` call takes two parameters. Call them `objectA` and `objectB`. When user performs a merge by calling `newEmulator = emulatorA.merge(emulatorB)`, `objectA` will be the object from `emulatorA`, and `objectB` will be the object from `emulatorB`. 
+
+The call should return a new, merged object of the same type with the `objectA` and `objectB`. 

docs/developer_manual/01-creating-a-new-service.md

@@ -0,0 +1,56 @@
+# Creating new services
+
+Services are special kinds of layers. One way to differentiate the regular layers and services layer is the way they make changes to the emulation. A regular Layer makes changes to the entire emulation (like BGP, which configure peeing across multiple different autonomous systems, exchanges, and routers). In contrast, a service only makes changes to individual nodes (like `Web`, which install `nginx` on the given node). 
+
+In the old design, services are installed by providing the node object to the services. In the new design, we decided not to use any node object to identify which node to install the services on, as that creates dependencies to the base layer, as mentioned in the previous section. This design is also revised. In the new design, services do not directly install a web server on a physical host. Instead, services create virtual nodes and install services on them. Virtual nodes are not real nodes; consider them as a "blueprint" of a physical node. Services keeps track of changes made to each virtual node internally. Eventually, virtual nodes are mapped to physical nodes with the binding mechanism.
+
+For details on how to create virtual nodes and binding, see example 00 (simple peering) and the "Virtual node binding & filtering" section of the manual. One does not actually need to know how to create and manage virtual nodes when creating a new service - all of the heavy liftings are handled by the `Service` class.
+
+Before proceeding with this guide, please go over the "creating a new layer" guide first. This guide is going to cover the following topics:
+
+- Implementing the `Server` interface.
+- Implementing and working with the `Service` interface.
+
+## Implementing the `Server` interface
+
+The first step of creating a new service is to create a new class implementing the `Server` interface. The `Server` interface represents a server software running on a physical node. Only one method is required b the `Server` interface:
+
+### `Server:install`
+
+All server must implement this method. This call installs a service onto a physical node. One parameter, the reference to the physical node, will be passed in, and the method does not need to return anything. Generally, one will make changes to the physical node here. (e.g., call `node.addSoftware('some_software')`). For details on API on the `Node` class, refer to the API documentation. 
+
+Other methods to configure the service should also be implemented in this class. For example, the `WebServer` class implements `WebServer::setPort` to allow changing the listening port number of the nginx web server. However, if a setting will be affecting all instances of servers, the method to change such a setting should be implemented directly in the `Service` class instead. 
+
+## Implementing and working with the `Service` interface
+
+After finish working with the `Server` class, one will also need to create a new class implementing the `Service` interface. The `Service` interface is derived from the `Layer` interface. `Service` interface will handle the virtual node resolving internally.
+
+Only one method is required in the `Service` interface:
+
+### `Service::_createServer`
+
+All services must implement this method. This call takes no parameter, and should create and return an instance of the `Server` of the `Service` (the one created in the last section). This instance will eventually be returned to the user.
+
+One may optionally implement the following methods to customize the configuration and render of a server:
+
+### `Service::_doConfigure`
+
+The `_doConfigure` method will be called to configure a server on a node (during the configuration stage). Two parameters will be passed in. The first is a reference to the physical node, and the second is the instance of the `Server` that has been bound to this node. The default implementation of this method is to just return when called. 
+
+### `Service::_doInstall`
+
+The `_doInstall` method will be called to install a server on a node (during the render stage). Two parameters will be passed in. The first is a reference to the physical node, and the second is the instance of the `Server` that has been bound to this node. The default implementation of this method is to call `server.install(node)`.
+
+### Retrieving list of virtual nodes and physical nodes
+
+It is possible to get a list of virtual node names and their corresponding server objects by calling `Service::getPendingTargets`. It will return a dictionary, where the keys are virtual node names, and the values are server objects.
+
+To get a list of physical nodes, one may call `Service::getTargets`. Note that `getTargets` only work in the render stage, as virtual nodes are resolved in the configuration stage. It will return a set, where the elements are set of tuples of `(Server, Node)`.
+
+### Change render and configuration behaviour 
+
+Sometimes a service may need to do some extra configuration on the service itself as a whole. Examples of this are `DomainNameService` and  `CymruIpOriginService`.
+
+`DomainNameService` needs to add NS records to zone files after the virtual nodes are bound to physical nodes since before that, it does not know what IP address the servers will have. This is done by overriding the `render` implementation. Remember that the `Service` is just a special kind of `Layer`, so it still has all methods from the `Layer` interface. Services can still add logics to the render method. Just remember to call `super().render(emulator)`, so that the service interface can handle the rest of the installation process. 
+
+`CymruIpOriginService`, on the other hand, needs to collect IP addresses in the emulator, create a new zone in the DNS layer. It does so by overriding the `configure` method of the `Layer` interface and collect IP there. It also calls `super().configure(emulator)` so the servers are properly configured and bound to physical nodes.

docs/developer_manual/02-creating-a-new-component.md

@@ -0,0 +1,17 @@
+# Creating a new component
+
+A component is a collection of nodes, layers, and services. It does not necessarily have all of them. Consider it as a partially-built emulator template. An example of this is the BGP attacker component. The BGP attacker component allows users to create a new emulator with a BGP attack inside. The emulator contains a base layer for the attacker's autonomous system, a router in the autonomous system, and static route entries to route the prefix specified by the user to the black hole.
+
+With the BGP attacker component, to create a hijacker, all the user needs to do is to set an ASN, set a list of prefixes, set the exchange to join, and merge the output of the component to the user's emulator. 
+
+To create a new customizable component, one will need to implement the `Component` interface. However, if a component does not take any input (no customizable options), it is recommended to simply dump the emulation with the `Emulator::dump` API and share it that way instead.
+
+The `Component` interface has only two methods:
+
+## `Component::get`
+
+The `get` call takes no input and should return an `Emulator` instance. 
+
+## `Component::getVirtualNodes`
+
+The `getVirtualNodes` takes no input and should return a list name of unbound virtual nodes in the emulator. Users may use this list to bind the virtual nodes to physical nodes. The default implementation returns an empty list on invoked.