[Cherry Pick - 3250] feat: CA bundle mount options for storage initializer

[Jooho]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #kserve#3250

Type of changes
Please delete options that are not relevant.
Cherry-pick

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

You can find original tests from this PR(kserve#3250).

I updated the test scenario according to the platform change.

Pre-requirements

• install openshift cluster (above 4.12)
• login to openshift cluster
• install ServiceMesh/Serverless operators
• install RHODS 2.5 operator
  • Deploy RHODS 2.5
    ODH 2.5 is not released yet so this test will use RHODS 2.5 with 0.11.1 latest images.
  • Create a default DSC
  • If you are using rosa, please add this into SMCP

    security:
      identity:
        type: ThirdParty  #required setting for ROSA
    

Update images

Please wait until all pods are running in knative-serving namespace.

# Create demo folder
export DEMO_HOME=/tmp/demo
mkdir $DEMO_HOME
cd $DEMO_HOME

# update manifests and use right images
cat <<EOF> ${DEMO_HOME}/dsc-custom.yaml
apiVersion: datasciencecluster.opendatahub.io/v1
kind: DataScienceCluster
metadata:
  name: default-dsc
spec:
  components:
    kserve:
      devFlags:
        manifests:
          - contextDir: config
            sourcePath: overlays/odh
            uri: >-
              https://github.com/jooho/kserve/tarball/odh_0.11.1_pr3250-manifests-test
      managementState: Managed
      serving:
        ingressGateway:
          certificate:
            type: SelfSigned
        managementState: Managed
        name: knative-serving
EOF

oc patch dsc/default-dsc --patch-file ${DEMO_HOME}/dsc-custom.yaml  --type=merge
oc delete deploy/kserve-controller-manager --force --grace-period=0 -n redhat-ods-applications

Deploy Minio with SSL

git clone [email protected]:Jooho/jhouse_openshift.git
cd jhouse_openshift/Minio/minio-tls-kserve/
source env.sh
./1.setup.sh 
./2.generate-cert.sh 

sed 's+quay.io/opendatahub/modelmesh-minio-examples:caikit-flan-t5+kserve/modelmesh-minio-examples+g'  -i ${DEMO_HOME}/minio.yaml
./3.deploy-minio.sh 
cp /tmp/minio/minio_certs/root.crt /tmp/minio/minio_certs/cabundle.crt

kserve setup for test

cd /tmp/demo 

git clone [email protected]:Jooho/kserve.git
cd kserve

# Create the cabundle secret in kserve namespace
oc create configmap cabundle --from-file=/tmp/minio/minio_certs/cabundle.crt -n redhat-ods-applications

What you should see in the tests is the runtime should be Running.

1. namespace scope test

• 1-1. using json style storage-config

oc new-project kserve-demo

kustomize build ./config/runtimes/| sed 's/ClusterServingRuntime/ServingRuntime/g' |oc create -n kserve-demo -f -

oc create configmap local-cabundle --from-file=/tmp/minio/minio_certs/cabundle.crt -n kserve-demo

cat <<EOF|oc apply -f -
apiVersion: v1
stringData:
  localMinIO: |
    {
      "type": "s3",
      "access_key_id": "THEACCESSKEY",
      "secret_access_key": "THEPASSWORD",
      "endpoint_url": "https://minio.minio.svc:9000",
      "bucket": "modelmesh-example-models",
      "region": "us-south",
      "cabundle_configmap": "local-cabundle"
    }
kind: Secret
metadata:
  name: storage-config
type: Opaque   
EOF

cat<<EOF | oc create -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    serving.knative.openshift.io/enablePassthrough: "true"
    sidecar.istio.io/inject: "true"
    sidecar.istio.io/rewriteAppHTTPProbers: "true"
  name: sklearn-iris-v2-rest
  namespace: kserve-demo
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      runtime: kserve-sklearnserver
      storage:
        key: localMinIO
        path: sklearn/mnist-svm.joblib
EOF

Check point

• Successfully pull a model

• Pod is running

• Check these Volumes

     volumeMounts:
      - mountPath: /mnt/models
        name: kserve-provision-location
      - mountPath: /etc/ssl/custom-certs
        name: cabundle-cert
      volumes:
      - configMap:
        defaultMode: 420
        name: local-cabundle
      name: cabundle-cert
  

• 1-2. using annotation style storage-config

# Clean up previous test
oc delete isvc --all --force
oc delete secret storage-config

cat <<EOF|oc apply -f -
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: VEhFQUNDRVNTS0VZ
  AWS_SECRET_ACCESS_KEY: VEhFUEFTU1dPUkQ=
kind: Secret
metadata:
  annotations:
    serving.kserve.io/s3-endpoint: minio.minio.svc:9000
    serving.kserve.io/s3-region: us-east-2
    serving.kserve.io/s3-useanoncredential: "false"
    serving.kserve.io/s3-usehttps: "1"
    serving.kserve.io/s3-cabundle-configmap: "local-cabundle"
  name: storage-config
  namespace: kserve-demo
type: Opaque
EOF

cat<<EOF | oc create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
secrets:
- name: storage-config
EOF


cat<<EOF | oc create -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    serving.knative.openshift.io/enablePassthrough: "true"
    sidecar.istio.io/inject: "true"
    sidecar.istio.io/rewriteAppHTTPProbers: "true"
  name: sklearn-iris-v2-rest
  namespace: kserve-demo
spec:
  predictor:
    serviceAccountName: sa
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      runtime: kserve-sklearnserver
      storageUri: s3://modelmesh-example-models/sklearn/mnist-svm.joblib
EOF

• 2. global scope test
• 2-1. using inferenceservice-config configmap
  Setup configmap

oc delete isvc --all --force
oc delete secret storage-config

# Scale down rhods operator to customize configmap
oc scale deploy/rhods-operator --replicas=0 -n redhat-ods-operator

oc edit configmap inferenceservice-config -n redhat-ods-applications
  storageInitializer: |-
    {
        ...
        "caBundleConfigMapName": "cabundle",
        "enableDirectPvcVolumeMount": false
    }
oc delete pod -l control-plane=kserve-controller-manager --force -n redhat-ods-applications

oc project kserve-demo 

cat <<EOF|oc apply -f -
apiVersion: v1
stringData:
  localMinIO: |
    {
      "type": "s3",
      "access_key_id": "THEACCESSKEY",
      "secret_access_key": "THEPASSWORD",
      "endpoint_url": "https://minio.minio.svc:9000",
      "bucket": "modelmesh-example-models",
      "region": "us-south"
    }
kind: Secret
metadata:
  name: storage-config
type: Opaque   
EOF

cat<<EOF | oc create -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    serving.knative.openshift.io/enablePassthrough: "true"
    sidecar.istio.io/inject: "true"
    sidecar.istio.io/rewriteAppHTTPProbers: "true"
  name: sklearn-iris-v2-rest
  namespace: kserve-demo
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      runtime: kserve-sklearnserver
      storage:
        key: localMinIO
        path: sklearn/mnist-svm.joblib
EOF

Check Point
with global setting, global-ca-bundle will be created in the namespace

oc get cm
NAME                       DATA   AGE
global-ca-bundle           1      4s

2-2. override the global cabundle by cabundle_configmap in StorageSpec

oc delete isvc --all --force
oc delete cm global-ca-bundle
oc delete secret storage-config

cat <<EOF|oc apply -f -
apiVersion: v1
stringData:
  localMinIO: |
    {
      "type": "s3",
      "access_key_id": "THEACCESSKEY",
      "secret_access_key": "THEPASSWORD",
      "endpoint_url": "https://minio.minio.svc:9000",
      "bucket": "modelmesh-example-models",
      "region": "us-south",
      "cabundle_configmap": "local-cabundle"
    }
kind: Secret
metadata:
  name: storage-config
type: Opaque   
EOF

cat<<EOF | oc create -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    serving.knative.openshift.io/enablePassthrough: "true"
    sidecar.istio.io/inject: "true"
    sidecar.istio.io/rewriteAppHTTPProbers: "true"
  name: sklearn-iris-v2-rest
  namespace: kserve-demo
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      runtime: kserve-sklearnserver
      storage:
        key: localMinIO
        path: sklearn/mnist-svm.joblib
EOF

2-3. override the global cabundle by serving.kserve.io/s3-cabundle-configmap in storage-config

# Clean up previous test
oc delete isvc --all --force
oc delete cm global-ca-bundle
oc delete secret storage-config

cat <<EOF|oc apply -f -
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: VEhFQUNDRVNTS0VZ
  AWS_SECRET_ACCESS_KEY: VEhFUEFTU1dPUkQ=
kind: Secret
metadata:
  annotations:
    serving.kserve.io/s3-endpoint: minio.minio.svc:9000
    serving.kserve.io/s3-region: us-east-2
    serving.kserve.io/s3-useanoncredential: "false"
    serving.kserve.io/s3-usehttps: "1"
    serving.kserve.io/s3-cabundle-configmap: "local-cabundle"
  name: storage-config
  namespace: kserve-demo
type: Opaque
EOF

cat<<EOF | oc create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
secrets:
- name: storage-config
EOF


cat<<EOF | oc create -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    serving.knative.openshift.io/enablePassthrough: "true"
    sidecar.istio.io/inject: "true"
    sidecar.istio.io/rewriteAppHTTPProbers: "true"
  name: sklearn-iris-v2-rest
  namespace: kserve-demo
spec:
  predictor:
    serviceAccountName: sa
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      runtime: kserve-sklearnserver
      storageUri: s3://modelmesh-example-models/sklearn/mnist-svm.joblib
EOF

Checklist:

• Have you added unit/e2e tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

Release note:

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jooho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Jooho]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[spolti]

@Jooho is this ready to merge?

[Jooho]

@spolti I need to update the description. By the way, the latest version of odh is still 2.4 so it is a little bit different to test this feature. I am thinking of testing it with rhods instead of 2.5. (The installation method in 2.5 is much easier.)