testing: Set Up End-to-End Tests for Askeladd DVM in CI (GitHub Actions)

[AbdelStark]

Description

We need to implement comprehensive end-to-end (e2e) tests for our Data Vending Machine (DVM) system and integrate them into our CI pipeline using GitHub Actions. These tests will ensure that all components of our system work together correctly under various scenarios.

Requirements

1. CI Environment Setup:

   • Configure a GitHub Actions workflow for e2e testing.
   • Set up the necessary environment variables and secrets.

2. Component Deployment:

   • Deploy a Nostr relayer for test communication.
   • Deploy a DVM service provider (prover agent).
   • Deploy multiple DVM customers (users).

3. Test Scenarios:

   • Implement multiple test scenarios covering both success and error cases.
   • Include tests for different payment flows (upfront, after completion).

4. Test Execution:

   • Run all components in isolated environments.
   • Execute test scenarios in parallel where possible.
   • Collect and report test results.

Implementation Steps

1. GitHub Actions Workflow:

   • Create a new workflow file: .github/workflows/e2e-tests.yml
   • Define the trigger events (e.g., push to main, pull requests)
   • Set up the test environment (e.g., Ubuntu latest)

2. Relayer Setup:

   • Use a lightweight Nostr relayer implementation suitable for testing.
   • Configure the relayer with test-specific settings.

3. DVM Service Provider (Prover Agent) Setup:

   • Deploy the prover agent with test configurations.
   • Ensure it connects to the test relayer.

4. DVM Customers Setup:

   • Create multiple customer instances with different configurations.
   • Implement customer behavior scripts for various scenarios.

5. Test Scenarios Implementation:

   • Develop a test suite covering the following scenarios:
     a. Successful job request and completion (with upfront payment)
     b. Successful job request and completion (with payment after completion)
     c. Job request with insufficient payment
     d. Job request with invalid input
     e. Service provider unavailability
     f. Relayer disconnection and reconnection
     g. Partial results and progressive payments (if supported)
     h. Concurrent job requests from multiple customers

6. Test Execution Script:

   • Create a main test execution script that:
     a. Starts all components (relayer, prover agent, customers)
     b. Runs all test scenarios
     c. Collects test results and logs
     d. Shuts down components after test completion

7. Result Reporting:

   • Implement a mechanism to collect and format test results.
   • Configure GitHub Actions to report test outcomes and upload artifacts (logs, results).

Testing

1. Local Testing:

   • Develop and test the e2e setup locally before implementing in CI.
   • Ensure all components can be started and stopped reliably.

2. CI Integration:

   • Test the GitHub Actions workflow with a subset of scenarios.
   • Gradually add more scenarios and verify CI behavior.

3. Performance Testing:

   • Monitor the execution time of the e2e tests in CI.
   • Optimize parallel execution and resource usage if necessary.

Documentation

1. Update the project README with information about the e2e tests.
2. Document the structure and purpose of each test scenario.
3. Provide instructions for running e2e tests locally for developers.

Considerations

• Ensure that the e2e tests don't interfere with production systems (use separate relayers, isolated environments).
• Implement proper cleanup procedures to remove all test data after execution.
• Consider using docker containers for consistent environment setup across different systems.

Definition of Done

• All specified components (relayer, prover agent, customers) can be deployed in the CI environment.
• All test scenarios are implemented and running successfully in CI.
• Test results are properly reported in GitHub Actions UI.
• Documentation is updated with e2e testing information.
• Developers can run e2e tests locally following provided instructions.