Merge pull request #277 from grycap/dev-slangarita

Dev slangarita

.github/ISSUE_TEMPLATE/bug_report.md

@@ -23,4 +23,5 @@ about: Create a report to help us improve
 
 ### Possible Solution
 
-Any thoughts as to potential solutions or ideas to go about finding one. Please include links to any research.
+Any thoughts as to potential solutions or ideas to go about finding one.
+Please include links to any research.

CODE_OF_CONDUCT.md

@@ -14,19 +14,22 @@ orientation.
 Examples of behavior that contributes to creating a positive environment
 include:
 
--   Using welcoming and inclusive language
--   Being respectful of differing viewpoints and experiences
--   Gracefully accepting constructive criticism
--   Focusing on what is best for the community
--   Showing empathy towards other community members
+- Using welcoming and inclusive language
+- Being respectful of differing viewpoints and experiences
+- Gracefully accepting constructive criticism
+- Focusing on what is best for the community
+- Showing empathy towards other community members
 
 Examples of unacceptable behavior by participants include:
 
--   The use of sexualized language or imagery and unwelcome sexual attention or advances
--   Trolling, insulting/derogatory comments, and personal or political attacks
--   Public or private harassment
--   Publishing others' private information, such as a physical or electronic address, without explicit permission
--   Other conduct which could reasonably be considered inappropriate in a professional setting
+- The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+- Trolling, insulting/derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+- Other conduct which could reasonably be considered inappropriate in a
+  professional setting
 
 ## Our Responsibilities
 
@@ -52,7 +55,7 @@ further defined and clarified by project maintainers.
 ## Enforcement
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at [email protected]. All
+reported by contacting the project team at <[email protected]>. All
 complaints will be reviewed and investigated and will result in a response that
 is deemed necessary and appropriate to the circumstances. The project team is
 obligated to maintain confidentiality with regard to the reporter of an incident.

README.md

@@ -1,7 +1,7 @@
 # OSCAR - Open Source Serverless Computing for Data-Processing Applications
 
-[![Go Report Card](https://goreportcard.com/badge/github.com/grycap/oscar)](https://goreportcard.com/report/github.com/grycap/oscar)
-[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/8145efdfb9d24af1b5b53e21c6e2df99)](https://www.codacy.com/gh/grycap/oscar/dashboard?utm_source=github.com&utm_medium=referral&utm_content=grycap/oscar&utm_campaign=Badge_Coverage)
+[![Go Report Card](https://goreportcard.com/badge/github.com/grycap/oscar/v3)](https://goreportcard.com/report/github.com/grycap/oscar/v3)
+[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/8145efdfb9d24af1b5b53e21c6e2df99)](https://app.codacy.com/gh/grycap/oscar/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_coverage)
 [![tests](https://github.com/grycap/oscar/actions/workflows/tests.yaml/badge.svg?branch=master)](https://github.com/grycap/oscar/actions/workflows/tests.yaml)
 [![build](https://github.com/grycap/oscar/workflows/build/badge.svg)](https://github.com/grycap/oscar/actions?query=workflow%3Abuild)
 [![GitHub release (latest by date)](https://img.shields.io/github/v/release/grycap/oscar)](https://github.com/grycap/oscar/pkgs/container/oscar)

deploy/ansible/README.md

@@ -1,3 +1,3 @@
 # Ansible playbook to deploy K3s and the OSCAR platform
 
-Please refer to the [docs](https://docs.oscar.grycap.net/deploy-ansible/) for instructions.
+Please refer to the [docs](https://docs.oscar.grycap.net/deploy-ansible/) for instructions.

docs/api.md

@@ -4,4 +4,8 @@ OSCAR exposes a secure REST API available at the Kubernetes master's node IP
 through an Ingress Controller. This API has been described following the
 [OpenAPI Specification](https://www.openapis.org/) and it is available below.
 
+> ℹ️
+>
+> The bearer token used to run a service can be either the OSCAR [service access token](invoking-sync.md#service-access-tokens) or the [user's Access Token](integration-egi.md#obtaining-an-access-token) if the OSCAR cluster is integrated with EGI Check-in.
+
 !!swagger api.yaml!!

docs/api.yaml

@@ -33,6 +33,7 @@ paths:
       description: List all created services
       security:
         - basicAuth: []
+        - token: []
       tags:
         - services
     post:
@@ -50,6 +51,7 @@ paths:
       description: Create a service
       security:
         - basicAuth: []
+        - token: []
       requestBody:
         content:
           application/json:
@@ -74,6 +76,7 @@ paths:
       description: Update a service
       security:
         - basicAuth: []
+        - token: []
       requestBody:
         content:
           application/json:
@@ -108,6 +111,7 @@ paths:
       operationId: ReadService
       security:
         - basicAuth: []
+        - token: []
       description: Read a service
     delete:
       summary: Delete service
@@ -124,6 +128,7 @@ paths:
       description: Delete a service
       security:
         - basicAuth: []
+        - token: []
       tags:
         - services
   '/system/logs/{serviceName}':
@@ -157,6 +162,7 @@ paths:
       operationId: ListJobs
       security:
         - basicAuth: []
+        - token: []
       description: List all jobs with their status
     delete:
       summary: Delete jobs
@@ -173,6 +179,7 @@ paths:
       description: Delete all jobs from a service.
       security:
         - basicAuth: []
+        - token: []
       parameters:
         - schema:
             type: boolean
@@ -214,6 +221,7 @@ paths:
       description: Get the logs from a job
       security:
         - basicAuth: []
+        - token: []
       parameters:
         - schema:
             type: boolean
@@ -234,6 +242,7 @@ paths:
       description: Delete a job
       security:
         - basicAuth: []
+        - token: []
       tags:
         - logs
   /system/info:
@@ -256,6 +265,7 @@ paths:
       description: Get system info
       security:
         - basicAuth: []
+        - token: []
   /health:
     get:
       summary: Health
@@ -316,6 +326,7 @@ paths:
       description: Get system configuration
       security:
         - basicAuth: []
+        - token: []
   '/run/{serviceName}':
     parameters:
       - schema:
@@ -607,4 +618,6 @@ servers:
   - url: 'https://localhost'
     description: 'Local testing'
   - url: 'https://inference.cloud.ai4eosc.eu'
-    description: 'AI4EOSC OSCAR cluster'
+    description: 'AI4EOSC OSCAR cluster'
+  - url: 'https://inference-walton.cloud.imagine-ai.eu'
+    description: 'iMagine OSCAR cluster'

docs/integration-egi.md

@@ -67,7 +67,7 @@ grant access for all users from that VO.
 
 The static web interface of OSCAR has been integrated with EGI Check-in and
 published in [ui.oscar.grycap.net](https://ui.oscar.grycap.net) to facilitate
-the authorization of users. To login through EGI Checkín using OIDC tokens,
+the authorization of users. To login through EGI Check-In using OIDC tokens,
 users only have to put the endpoint of its OSCAR cluster and click on the
 "EGI CHECK-IN" button.
 
@@ -87,3 +87,17 @@ create a new account configuration for the
 After that, clusters can be
 added with the command [`oscar-cli cluster add`](oscar-cli.md#add) specifying
 the oidc-agent account name with the `--oidc-account-name` flag.
+
+### Obtaining an Access Token
+
+Once logged in via EGI Check-In you can obtain an Access Token with one of this approaches:
+
+* From the command-line, using `oidc-agent` with the following command:
+
+    ```sh
+    oidc-token <account-short-name>
+    ```
+    where `account-short-name` is the name of your account configuration.
+* From the EGI Check-In Token Portal: [https://aai.egi.eu/token](https://aai.egi.eu/token)
+
+![egi-checkin-token-portal.png](images/oidc/egi-checkin-token-portal.png)

docs/invoking-async.md

@@ -2,11 +2,11 @@
 
 For event-driven file processing, OSCAR automatically manages the creation
 and [notification system](https://docs.min.io/minio/baremetal/monitoring/bucket-notifications/bucket-notifications.html#minio-bucket-notifications)
-of MinIO buckets in order to allow the event-driven invocation of services
-using asynchronous requests, generating a Kubernetes job for every file to be
-processed.
-
+of MinIO buckets. This allow the event-driven invocation of services
+using asynchronous requests for every file uploaded to the bucket, which generates a Kubernetes job for every file to be processed.
 
 ![oscar-async.png](images/oscar-async.png)
 
+These jobs will be queued up in the Kubernetes scheduler and will be processed whenever there are resources available. If OSCAR cluster has been deployed as an elastic Kubernetes cluster (see [Deployment with IM](https://docs.oscar.grycap.net/deploy-im-dashboard/)), then new Virtual Machines will be provisioned (up to the maximum number of nodes defined) in the underlying Cloud platform and seamlessly integrated in the Kubernetes clusters to proceed with the execution of jobs. These nodes will be terminated as the worload is reduced. Notice that the output files can be stores in MinIO or in any other storage back-end supported by the [FaaS supervisor](oscar-service.md#faas-supervisor). 
 
+If you want to process a large number of data files, consider using [OSCAR Batch](https://github.com/grycap/oscar-batch), a tool designed to perform batch-based processing in OSCAR clusters. It includes a coordinator tool where the user provides a MinIO bucket containing files for processing. This service calculates the optimal number of parallel service invocations that can be accommodated within the cluster, according to its current status, and distributes the image processing workload accordingly among the service invocations. This is mainly intended to process large amounts of files, for example, historical data.

docs/invoking-sync.md

@@ -83,8 +83,8 @@ base64 input.png | curl -X POST -H "Authorization: Bearer <TOKEN>" \
 
 ## Service access tokens
 
-As detailed in the [API specification](api.md), invocation paths require the
-service access token in the request header for authentication. Service access
+As detailed in the [API specification](api.md), invocation paths require either the
+service access token or the Access Token of the user when the cluster is integrated with EGI Check-in, in the request header for authentication (any of them is valid). Service access
 tokens are auto-generated in service creation and update, and MinIO eventing
 system is automatically configured to use them for event-driven file
 processing. Tokens can be obtained through the API, using the

docs/invoking.md

@@ -2,7 +2,16 @@
 
 OSCAR services can be executed:
 
-  - [Synchronously](invoking-sync.md), so that the invocation to the service blocks the client until the response is obtained. Useful for short-lived service invocations.
+  - [Synchronously](invoking-sync.md), so that the invocation to the service blocks the client until the response is obtained. 
   - [Asynchronously](invoking-async.md), typically in response to a file upload to MinIO or via the OSCAR API.
-  - As an [exposed service](exposed-services.md), where the application executed already provides its own API or user interface (e.g. a Jupyter Notebook)
+  - As an [exposed service](exposed-services.md), where the application executed already provides its own API or user interface (e.g. Jupyter Notebook)
+
+
+After reading the different service execution types, take into account the following considerations to better decide the most appropriate execution type for your use case:
+
+* **Scalability**: Asynchronous invocations provide the best throughput when dealing with multiple concurrent data processing requests, since these are processed by independent jobs which are managed by the Kubernetes scheduler. A two-level elasticity approach is used (increase in the number of pods and increase in the number of Virtual Machines, if the OSCAR cluster was configured to be elastic). This is the recommended approach when each processing request exceeds the order of tens of seconds. 
+
+* **Reduced Latency** Synchronous invocations are oriented for short-lived (< tens of seconds) bursty requests. A certain number of containers can be configured to be kept alive to avoid the performance penalty of spawning new ones while providing an upper bound limit (see [`min_scale` and `max_scale` in the FDL](fdl.md#synchronoussettings), at the expense of always consuming resources in the OSCAR cluster. If the processing file is in the order of several MBytes it may not fit in the payload of the HTTP request.
+
+* **Easy Access** For services that provide their own user interface or their own API, exposed services provide the ability to execute them in OSCAR and benefit for an auto-scaled configuration in case they are [stateless](https://en.wikipedia.org/wiki/Service_statelessness_principle). This way, users can directly access the service using its well-known interfaces by the users. 
 

docs/minio_usage.md → docs/minio-usage.md

@@ -1,4 +1,4 @@
-# Accessing and managing MinIO storage provider
+# Using the MinIO Storage Provider
 
 Each OSCAR cluster includes a deployed MinIO instance, which is used to trigger service executions. When a service is configured to use MinIO as its storage provider, it monitors a specified input folder for new data. Whenever new data is added to this folder, it triggers the associated service to execute. 
 
@@ -30,19 +30,21 @@ MinIO buckets can also be managed through [oscar-cli command-line](https://githu
   - [put-file](https://docs.oscar.grycap.net/oscar-cli/#put-file): Upload a file on a service storage provider.
 
   An example of a put-file operation:
-  ``` sh
+
+  ``` bash
   oscar-cli service put-file fish-detector.yaml minio .path/to/your/images ./fish-detector/input/
   ```
 
 - **mc**: If a user wants to use the MinIO client it needs to follow some previous steps.
   - *Install the client*: Detailed instructions for installing the MinIO client (mc) are available in [the official documentation](https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc).
   - *Configure the MinIO instance*: The client requires credentials to connect and interact with the MinIO instance. This configuration can be set with the following command:
 
-    ``` sh
+    ``` bash
     mc alias set myminio https://minio.gracious-varahamihira6.im.grycap.net YOUR-ACCESS-KEY YOUR-SECRET-KEY
     ```
 
   Once the client is configured, users can perform various operations supported by the MinIO client. For a complete list of available commands and their usage, refer to the [MinIO client reference](https://min.io/docs/minio/linux/reference/minio-mc.html#command-quick-reference). The following example demonstrates a PUT operation, where a file is uploaded to a specific folder within a bucket.
-  ``` sh
+
+  ```bash
   mc cp /path/to/your/images/*.jpg myminio/fish-detector/input/
   ```

docs/oscar-service.md

@@ -15,7 +15,7 @@ is in charge of:
 
 
 
-### Input/Output
+### FaaS Supervisor
 
 [FaaS Supervisor](https://github.com/grycap/faas-supervisor), the component in
 charge of managing the input and output of services, allows JSON or base64
@@ -37,6 +37,12 @@ The output of synchronous invocations will depend on the application itself:
 
 This way users can adapt OSCAR's services to their own needs.
 
+The FaaS Supervisor supports the following storage back-ends:
+* [MinIO](https://min.io)
+* [Amazon S3](https://aws.amazon.com/s3/)
+* Webdav (and, therefore, [dCache](https://dcache.org))
+* Onedata (and, therefore, [EGI DataHub](https://www.egi.eu/service/datahub/))
+
 ### Container images
 
 Container images on asynchronous services use the tag `imagePullPolicy: Always`, which means that Kubernetes will check for the image digest on the image registry and download it if it is not present.

examples/plant-classification-theano/README.md

@@ -63,5 +63,5 @@ To run this example you need:
 
 1. Once the function is executed, the output is automatically copied to the
   output bucket in minio, in this case `plant-classifier-out`. You can
-  download the ouput from here for further processing.
+  download the output from here for further processing.
   ![minio-out.png](img/Minio-OUT.png)

examples/plants-classification-tensorflow/script.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-IMAGE_NAME=`basename "$INPUT_FILE_PATH"`
+IMAGE_NAME=`basename "$INPUT_FILE_PATH" | cut -d. -f1`
 OUTPUT_FILE="$TMP_OUTPUT_DIR/output.json"
 
 deepaas-cli predict --files "$INPUT_FILE_PATH" 2>&1 | grep -Po '{.*}' > "$OUTPUT_FILE" 

examples/yolov8/README.md

@@ -0,0 +1,24 @@
+# Object Detection with YOLOv8
+
+Detect objects in images using the state-of-the-art YOLOv8 model.
+
+## About YOLO
+
+This node utilizes the YOLOv8 (You Only Look Once version 8) model to detect objects within images. YOLOv8 is a cutting-edge, real-time object detection system known for its speed and accuracy, capable of identifying thousands of object categories efficiently.
+
+## About YOLOV8 Service in OSCAR
+
+This service uses the pre-trained YOLOv8 model provided by DEEP-Hybrid-DataCloud for object detection. It is designed to handle synchronous invocations and real-time image processing with high scalability, managed automatically by an elastic Kubernetes cluster.
+
+In order to invoke the function, first you have to create a service, either by the OSCAR UI or by using the FDL within the following command.
+
+
+``` sh
+oscar-cli apply yolov8.yaml
+```
+
+Once the service is created you can make the invocation with the following
+command, which will store the output on a minio bucket.
+
+``` sh
+oscar-cli service put-file yolov8.yaml minio img/cat.jpg yolov8/input/cat.jpg

examples/yolov8/img/cat.jpg:Zone.Identifier

@@ -0,0 +1,4 @@
+[ZoneTransfer]
+ZoneId=3
+ReferrerUrl=https://www.google.com/
+HostUrl=https://upload.wikimedia.org/wikipedia/commons/thumb/4/4d/Cat_November_2010-1a.jpg/220px-Cat_November_2010-1a.jpg

examples/yolov8/script.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+IMAGE_NAME=`basename "$INPUT_FILE_PATH"`
+OUTPUT_FILE="$TMP_OUTPUT_DIR/output.png"
+
+deepaas-cli --deepaas_method_output="$OUTPUT_FILE" predict --files "$INPUT_FILE_PATH" --accept image/png 2>&1
+
+echo "Prediction was saved in: $OUTPUT_FILE"

examples/yolov8/yolov8.yaml

@@ -0,0 +1,17 @@
+functions:
+  oscar:
+  - oscar-cluster:
+     name: yolov8
+     memory: 4Gi
+     cpu: '2.0'
+     image: ai4oshub/ai4os-yolov8-torch:latest
+     script: script.sh
+     vo: vo.imagine-ai.eu
+     allowed_users: []
+     log_level: CRITICAL		
+     input:
+     - storage_provider: minio.default
+       path: yolov8/input
+     output:
+     - storage_provider: minio.default
+       path: yolov8/output

go.mod

@@ -52,6 +52,7 @@ require (
 )
 
 require (
+	bou.ke/monkey v1.0.2
 	github.com/GehirnInc/crypt v0.0.0-20190301055215-6c0105aabd46 // indirect
 	github.com/apache/yunikorn-scheduler-interface v1.2.0 // indirect
 	github.com/blendle/zapdriver v1.3.1 // indirect