Create Sagemaker pipeline schedules if specified #3271
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
github-actions
bot
added
internal
…zenml into feature/add-sagemaker-schedule
![hyperlint-ai[bot]](https://avatars.githubusercontent.com/in/718456?s=60&v=4){kind=small}
There was a problem hiding this comment.
The style guide flagged several spelling errors that seemed like false positives. We skipped posting inline suggestions for the following words:
- endhint
Note: We resolved prior Hyperlint review comments because:
We updated our inline suggestion AI.
We do this to avoid keeping outdated or irrelevant comments around. We'll leave a new review with current comments below.
| metadata.update( | ||
| { | ||
| "schedule_rule_name": execution["rule_name"], | ||
| "schedule_type": execution["schedule_type"], | ||
| "schedule_expression": execution["schedule_expr"], | ||
| "pipeline_name": execution["pipeline_name"], | ||
| } | ||
| ) | ||
|
|
||
| if next_execution := execution.get("next_execution"): | ||
| metadata["next_execution_time"] = next_execution.isoformat() |
There was a problem hiding this comment.
@schustmi ping on this
There was a problem hiding this comment.
This compute_metadata method gets called from prepare_or_run_pipeline and yielded as a result of it. This however only gets applied when there is already is a run (-> non-scheduled case), as otherwise there is no run to attach the metadata to.
For the scheduled case, there is SagemakerOrchestrator.get_pipeline_run_metadata which gets called at execution time. Ideally, you'd be able to return the metadata from this object.
Another chang would be to allow attaching metadata to schedules, which is actually where most of that metadata fits best IMO. However I would suggest doing that as part of the native schedule implementation, as right now it wouldn't lead to any visibility of that metadata anyway.
| events_client = session.boto_session.client("events") | ||
| rule_name = f"zenml-{deployment.pipeline_configuration.name}" | ||
|
|
||
| # Determine first execution time based on schedule type |
There was a problem hiding this comment.
The same code is duplicated above
|
|
||
| # Create IAM policy for EventBridge | ||
| iam_client = session.boto_session.client("iam") | ||
| role_name = self.config.execution_role.split("/")[ |
There was a problem hiding this comment.
Isn't the execution role the one that should be assumed by the pipeline during execution, and not used for something around the pipeline execution like this scheduling? This seems more like it should be tied to the role of the service connector if applicable or solved in some other way.
| 3. Set up the SageMaker pipeline as the target | ||
|
|
||
| {% hint style="info" %} | ||
| If you run the same pipeline with a schedule multiple times, the existing schedule will be updated with the new settings rather than creating a new schedule. This allows you to modify schedules by simply running the pipeline again with new schedule parameters. |
There was a problem hiding this comment.
I would not do this to keep it consistent with all other orchestrators.
|
@htahir1 A sagemaker pipeline has a |
| 3. Enable automatic execution based on the schedule | ||
|
|
||
| {% hint style="info" %} | ||
| If you run the same pipeline with a schedule multiple times, the existing schedule will **not** be updated with the new settings. Rather, ZenML will create a new sagemaker pipeline and attach a new schedule to it. The user must manually delete the old pipeline and their attached schedule using the AWS CLI or API. See details here: [SageMaker Pipeline Schedules](https://docs.aws.amazon.com/sagemaker/latest/dg/pipeline-eventbridge.html) |
There was a problem hiding this comment.
| If you run the same pipeline with a schedule multiple times, the existing schedule will **not** be updated with the new settings. Rather, ZenML will create a new sagemaker pipeline and attach a new schedule to it. The user must manually delete the old pipeline and their attached schedule using the AWS CLI or API. See details here: [SageMaker Pipeline Schedules](https://docs.aws.amazon.com/sagemaker/latest/dg/pipeline-eventbridge.html) | |
| If you run the same pipeline with a schedule multiple times, the existing schedule will **not** be updated with the new settings. Rather, ZenML will create a new SageMaker pipeline and attach a new schedule to it. The user must manually delete the old pipeline and their attached schedule using the AWS CLI or API. See details here: [SageMaker Pipeline Schedules](https://docs.aws.amazon.com/sagemaker/latest/dg/pipeline-eventbridge.html) |
Issues:
- Style Guide - (Spelling-error) Did you really mean 'sagemaker'?
Fix Explanation:
The term 'sagemaker' should be capitalized to 'SageMaker' as it refers to the AWS service name. This aligns with the proper noun usage and the style guide recommendation.
|
LLM Finetuning template updates in |
|
Classification template updates in |
|
E2E template updates in |
|
NLP template updates in |
Describe changes
Add schedule support for sagemaker. Tested manually.
Pre-requisites
Please ensure you have done the following:
developand the open PR is targetingdevelop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.Types of changes