Orchestrator, Executor and Executor Spawners

.gitignore

@@ -1,2 +1 @@
 /target
-/src/data

Cargo.toml

@@ -11,7 +11,7 @@ const_env = "0.1.2"
 num-bigint = "0.4.4"
 serde = { version = "1.0.192", features = ["derive"] }
 serde_json = "1.0.108"
-summa-backend = { git = "https://github.com/summa-dev/summa-solvency", branch = "v1-for-summa-aggregation", version = "0.1.0" }
+summa-backend = { git = "https://github.com/summa-dev/summa-solvency", branch = "v1-improvements-and-consolidation" }
 halo2_proofs = { git = "https://github.com/privacy-scaling-explorations/halo2", tag = "v2023_04_20"}
 tokio = { version = "1.34.0", features = ["full"] }
 reqwest = "0.11.22"

README.md

@@ -1,160 +1,10 @@
 # Summa Aggregation
 
-Summa Aggregation is a scalable solution specifically designed to accelerate the process of building Merkle sum tree.
-
-Our benchmarks in Summa Solvency revealed that constructing a merkle sum tree is a major time-consuming aspect of proof generation.
-
-The primary objective of Summa Aggregation is to enable efficient scaling in the construction of the tree by parallelization and distributed computation in multiple machines.
-
-For further optimization in Summa Aggregation, we introduced the AggregationMerkleSumTree component. This component is designed for efficiently constructing large Merkle sum trees by utilizing smaller-sized Merkle sum tree.
-
-## Diagram of Parallel Merkle Sum Tree Construction
-
-The diagram illustrates a distributed approach to constructing the `AggregatedMerkleSumTree`, where an orchestrator delegates tasks to executors, which are then processed in parallel by workers. The following sections will explain the roles of the Orchestrator, Executor, Worker and ExecutorSpawner.
-
-![diagram](/Orchestrator-diagram.png)
-
-## Orchestrator
-
-The Orchestrator in the Summa Aggregation serves as the central management component, coordinating the data processing activities. It plays a pivotal role in coordinating the activities of Executors and Workers, improving efficiency of tasks of building Merkle sum tree.
-The final result of the Orchestrator is the construction of the `AggregationMerkleSumTree`. This is achieved by aggregating the mini-trees constructed by the Workers. The Worker in here refers to a container running `mini-tree-server`.
-
-Key functions of the Orchestrator include:
-
-- **Dynamic Executor Spawning**: The Orchestrator dynamically spawns Executors in numbers set by the user. Each Executor is then connected to a dedicated Worker for efficient task execution.
-
-- **Task Management and Distribution**: It oversees the overall task flow, loading tasks and distributing them to Executors.
-
-- **Error Management and Pipeline Control**: The Orchestrator handles basic pipeline control and responds to errors by initiating the cancellation of all tasks.
-
-- **Build AggregationMerkleSumTree**: Its final result is that evaluate `AggregationMerkleSumTree` by aggregating the mini-trees generated by the Workers.
-
-## Executor and Worker
-
-The Executor acts as a crucial intermediary between the Orchestrator and Workers, facilitating the data processing workflow. Spawned by the Orchestrator, each Executor operates in a one-to-one relationship with a Worker. Its primary function is to generate a segment of the AggregationMerkleSumTree, known as a `mini-tree`, by processing entry data. These mini-trees are then aggregated by the Orchestrator to form the complete AggregationMerkleSumTree.
-
-Key aspects of the Executor's role include:
-
-- **Spawning and Connection**: Executors are dynamically spawned by the Orchestrator as part of the system's scalability. Each Executor is designed to connect with a Worker for task execution.
-
-- **Data Handling and Task Distribution**: A primary function of the Executor is to receive data entries, often parsed and prepared by the Orchestrator. Upon receiving these entries, the Executor is responsible for forwarding them to its connected Worker.
-
-- **Communication Bridge**: The Executor serves as a communication bridge within the data pipeline. It relays processed data, `mini-tree`, from Workers back to the Orchestrator.
-
-## ExecutorSpawner
-
-The `ExecutorSpawner` is responsible for initializing and terminating Executors. It can serve as the management point for creating `Executor` instances and Workers. The latter function as the `mini-tree-server`.
-
-In the Summa-Aggregation, there are three types of `ExecutorSpawner` provided:
-
-- **MockSpawner**: Primarily used for testing, this spawner initializes Executors suitable for various test scenarios, including negative test cases. The Worker spawned by this spawner runs a `mini-tree-server` locally.
-
-- **LocalSpawner**: It is close to actual use cases, this spawner enables users to initialize Executors and Workers in local Docker environments.
-
-- **CloudSpawner**: Ideal for scenarios where cloud resources are accessed. This spawner functions similarly to the `LocalSpawner`, but it initializes workers on remote machines. In particular, it can be run on a Swarm network using the `docker-compose` file, which is a specific configuration for the Swarm network. Additionally, it can run using existing worker node URLs if the configuration file is not set.
-
-The Docker Swarm transforms multiple Docker hosts into a single virtual host, providing crucial capabilities for high availability and scalability. For more details about Docker Swarm mode, refer to the [official documentation](https://docs.docker.com/engine/swarm/).
-
-While both `LocalSpawner` and `CloudSpawner` manage Docker containers, they differ in operational context. `LocalSpawner` handles individual containers directly, providing simplicity but limited scalability. In contrast, `CloudSpawner` may employs Docker Swarm to manage containers as services, thereby offering enhanced scalability and resilience, crucial for larger workloads.
-
-It's important to note, however, that managing workers through these three type of  `ExecutorSpawner`, is not mandatory. Technically, the `ExecutorSpawner` is a trait with minimal requirements for the Orchestrator, specifically the methods `spawn_executor` and `terminate_executor`. You can create your own spawner and use it with the Orchestrator.
-
-## Orchestrating on Swarm
-
-For Summa-Aggregation purposes, you need to prepare a distributed environment where Workers can run on remote machines (referred to as 'Nodes'). An example of this is using Swarm, as mentioned in the previous section. This section will introduce how to set up Swarm mode and test it using Docker CLI.
-
-### Preparing Docker Swarm Mode
-
-In Summa-Aggregation, the `CloudSpawner` is designed to operate on Docker Swarm. It requires the URLs of Workers for initiation, which are the IP addresses of the Workers joining the Swarm network as per the instructions below.
-
-You can initialize your Docker environment in Swarm mode, which is essential for managing a cluster of Docker nodes as a single virtual system.
-
-Note that setting up Swarm mode may not work well depending on the OS, as network configurations differ across operating systems.
-
-1. **Activate Swarm Mode on the Main Machine**:
-
-    Run the following command to initialize Swarm mode:
-
-    ```bash
-    Main $ docker swarm init
-    ```
-
-      This command will output information about the Swarm, including a join token.
-
-2. **Join Node to the Swarm**:
-
-      Use the join token provided by the main machine to add nodes to the swarm. On each node, run like:
-
-      ```bash
-      Worker_1 $ docker swarm join --token <YOUR_JOIN_TOKEN> <MAIN_MACHINE_IP>:2377
-      ```
-
-      Replace `<YOUR_JOIN_TOKEN>` with the actual token and `<MAIN_MACHINE_IP>` with the IP address of your main machine.
-
-3. **Check Node Status**:
-
-      To confirm that the nodes are successfully joined to the swarm, check the node status on the main machine:
-
-      ```bash
-      Main $ docker node ls
-      ```
-
-      You should see a list of all nodes in the swarm, including their status, roles, and other details like this:
-
-      ```bash
-      ID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
-      kby50cicvqd5d95o9pgt4puo9 *   main       Ready     Active         Leader           20.10.12
-      2adikgxr2l1zp9oqo4kowvw7n     worker_1   Ready     Active                          20.10.12
-      dz2z2v7o06h6gazmjlspyr5c8     worker_2   Ready     Active                          20.10.12
-      ````
-
-      You are ready to spawn more workers!
-
-### Spawning More Workers with CloudSpawner
-
-You can spawning or managing worker by `CloudSpawner` without using Docker CLI command, which will explain in here. However, even with a well-configured swarm network in the previous section, workers may not be created properly in various reason.
-
-In this section, you can verify that you can create a 'mini-tree-server' on any node. Before introducing the specific instructions, it's important to understand that in Docker Swarm mode, containers are managed as services rather than by individual names.
-
-To spawn more workers, follow these steps:
-
-1. Deploy the Stack:
-
-    First, deploy your stack using the `docker-compose.yml` file if you haven't already:
-
-    ```bash
-    Main $ docker stack deploy -c docker-compose.yml summa_aggregation
-    ```
-
-2. Scale the Service:
-
-    Utilize the 'scale' sub-command within the Docker 'service' command to adjust the number of replicas (workers) for your mini-tree service.
-
-    'mini-tree' refers to the name of the service, which is configured in the 'docker-compose.yml' file. Think of the number of replicas as the number of workers.
-
-    For example, to scale up to 5 workers, run:
-
-    ```bash
-    Main $ docker service scale summa_aggregation_mini-tree=5
-    ```
-
-    Since each worker has access to all of the node's resources, it would be appropriate to set the scale number based on the number of node.
-
-3. Verify the Scaling:
-
-    Check that the service has been scaled properly with:
-
-    ```bash
-    Main $ docker service ls
-    ```
-
-    This command shows the number of replicas running for each service in the swarm.
-
-Scaling service allows you to adjust the number of workers more easily than spawning each worker individually.
+Summa Aggregation is a scalable solution specifically designed to accelerate the process of building Merkle sum tree. It addresses the time-intensive challenge of constructing these trees by enabling efficient scaling through parallelization and distributed computation across multiple machines.
 
 ## Running test
 
-You can run the tests using following command:
+Tests can be run using the following command:
 
 ```bash
 cargo test --release
@@ -165,9 +15,8 @@ Please ensure that this port is not already in use to avoid errors.
 
 ## Running Additional Tests Involving Docker and Docker Swarm
 
-To run additional tests involving Docker and Docker Swarm mode, ensure that your Docker registry contains the "summadev/summa-aggregation-mini-tree" image.
+For additional tests involving Docker and Docker Swarm mode, the presence of the "summadev/summa-aggregation-mini-tree" image in the local Docker registry is required.
 
-If you do not have this image, you can either build it or download it.
 
 ### Building the docker image
 
@@ -179,31 +28,32 @@ docker build . -t summadev/summa-aggregation-mini-tree
 
 ### Downloading the Docker Image
 
-Alternatively, you can download the image from Docker Hub using the following command:
+Alternatively, the image can be downloaded from Docker Hub:
 
 ```bash
 docker pull summadev/summa-aggregation-mini-tree
 ```
 
-Ensure that the "summadev/summa-aggregation-mini-tree" image exists in your local Docker registry.
-
 ### Testing with LocalSpawner
 
 The following command runs an additional test case using the LocalSpawner, which spawns worker containers in the local Docker environment. This extra test case involves running two containers during the testing process:
 
+
 ```bash
 cargo test --features docker
 ```
 
 ### Testing with CloudSpawner
 
-If your Docker environment is successfully running in Swarm mode, you can run an additional test case that spawns workers on Swarm nodes using the `CloudSpawner`. Before running this test, please refer to the section "Spawning More Workers with CloudSpawner" to check your Docker Swarm setup.
+For Summa-Aggregation, it's necessary to prepare a distributed environment where Workers can operate on remote machines, referred to as 'Nodes'. For guidance on setting up swarm nodes, please see [Getting Started with swarm mode](https://docs.docker.com/engine/swarm/swarm-tutorial)
+
+When the Docker environment is running successfully in Swarm mode, an additional test case that spawns workers on Swarm nodes using the `CloudSpawner` can be run:
 
 ```bash
 cargo test --features docker-swarm
 ```
 
-Please ensure that your Docker Swarm has at least one node connected to the manager node. Also, verify that each worker node in the swarm has the "summadev/summa-aggregation-mini-tree" image in its Docker registry. If a node connected to the manager node does not have the image, it will not be able to spawn workers on that node.
+It is critical to ensure that the Docker Swarm includes at least one node connected to the manager node. Additionally, each worker node in the swarm must have the "summadev/summa-aggregation-mini-tree" image in its Docker registry. Without this image on nodes connected to the manager node, spawning workers on that node is not possible.
 
 ## Summa Aggregation Example
 

bin/README.md

@@ -1,24 +1,32 @@
 # Mini Tree Server
 
-Mini tree server is wrapped 
+Mini Tree Server is an Axum-based server that encapsulates the functionality of the Mini Tree Generator.
 
-### Test Mini Tree Server
+## Test Mini Tree Server
 
-You can manually test the `Mini Tree Server` with running container.
+First, to start the Mini Tree Server, use the command:
 
-First, Use the command below to start the Mini Tree Server container:
+```bash
+  cargo run --release --bin mini-tree-server
+```
+
+Alternatively, if you have the summa-aggregation-mini-tree image locally, can run the server with this command:
 
   ```bash
-  docker run -d -p 4000:4000 --name mini-tree-server summa-aggretaion/mini-tree
+  docker run -d -p 4000:4000 --name mini-tree-server-test summadev/summa-aggregation-mini-tree
   ```
 
-Second, to send two entries to the `Mini Tree Server`, use this script:
+For details on obtaining the summadev/summa-aggregation-mini-tree image, please refer to the [Building Image](../README.md#building-the-docker-image) and [Downloading Image](../README.md#downloading-the-docker-image) sections in the README.
+
+Second, send two entries to the Mini Tree Server, execute the following script:
 
   ```bash
   bash ./scripts/test_sending_entry.sh
   ```
 
-Upon successful execution, you will receive a response similar to the following
+Note: Execute this command from the project's root folder to ensure proper functioning of scripts.
+
+Upon successful execution, you will receive a response similar to the following:
 <details>
 <summary>Click View Response</summary>
 
@@ -66,3 +74,4 @@ Upon successful execution, you will receive a response similar to the following
 this JSON output is prettified for clarity
 
 </details>
+

examples/aggregation_flow.rs

@@ -73,8 +73,8 @@ async fn main() -> Result<(), Box<dyn Error>> {
     let orchestrator = Orchestrator::<N_CURRENCIES, N_BYTES>::new(
         Box::new(spawner),
         vec![
-            "./src/orchestrator/csv/entry_16_1.csv".to_string(),
-            "./src/orchestrator/csv/entry_16_2.csv".to_string(),
+            "csv/entry_16_1.csv".to_string(),
+            "csv/entry_16_2.csv".to_string(),
         ],
     );
 
@@ -99,7 +99,7 @@ async fn main() -> Result<(), Box<dyn Error>> {
     //
     // Here, we demonstrate generating the proof of inclusion for User 0.
     let inclusion_proof_of_user0 = round.get_proof_of_inclusion(0).unwrap();
-    assert!(inclusion_proof_of_user0.get_public_inputs().len() > 0); // Check public input counts
+    assert!(!inclusion_proof_of_user0.get_public_inputs().is_empty()); // Check public input counts
 
     println!("Generated User 0 proof of inclusion");
     Ok(())