Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support Passing in {{ target.user }} for BQ Query Comments #581

Open
3 tasks done
iamtodor opened this issue Feb 20, 2024 · 9 comments
Open
3 tasks done
Labels
pkg:dbt-bigquery Issue affects dbt-bigquery triage:awaiting-response Awaiting a response from the reporter type:enhancement New feature request

Comments

@iamtodor
Copy link

iamtodor commented Feb 20, 2024

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Hey folks!

I have a question in regards to Big query job comment(job-label): recently we upgraded to 1.4.9 and I would like to utilize this API https://docs.getdbt.com/reference/project-configs/query-comment#bigquery-include-query-comment-items-as-job-labels
I walked thru the doc and I was able to have the default one with the config:

query-comment:
  job-label: True

Our pipeline process looks like this: Airflow task calls dbt run from python operator. Obviously, we have plenty of tasks in the DAG. The business task is to gather information in regard to how many bytes are processed per hour per DAG. For this, I use the built-in region-us.INFORMATION_SCHEMA.JOBS from BigQuery. This is where labels come into play. As I mentioned previously python operator runs dbt task, and from the Python task I am able to find out the DAG name. Hence I am capable of putting the DAG name into dbt run command the way, for instance, we pass vars so I would like to ask/understand whether it's possible, and if so, then how?

I found that I also can add a custom comment in the way https://docs.getdbt.com/reference/project-configs/query-comment#append-a-custom-comment:

query-comment:
  comment: "run by {{ target.user }} in dbt"
  append: True

but dbt knows nothing about Airflow, while Airflow knows about dbt.

If I miss something I would be glad to clarify it. And, if I am moving towards the wrong direction I would be pleased to be corrected :)

I asked the same question in slack https://getdbt.slack.com/archives/C99SNSRTK/p1708089270625579 , but got no attention

Describe alternatives you've considered

Seems like there are no alternatives

Who will this benefit?

BigQuery users who would like to distinguish jobs per DAG

Are you interested in contributing this feature?

There is a chance, with all the context and pitfalls to be provided

Anything else?

Nope

@iamtodor iamtodor added type:enhancement New feature request triage:product In Product's queue labels Feb 20, 2024
@dbeatty10 dbeatty10 added the feature:cost-reduction Issues related to measuring and reducing execution cost label May 30, 2024
Copy link
Contributor

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

@github-actions github-actions bot added the Stale Mark an issue or PR as stale, to be closed label Nov 27, 2024
@iamtodor
Copy link
Author

Please don't make it stale

@github-actions github-actions bot removed the Stale Mark an issue or PR as stale, to be closed label Nov 28, 2024
@amychen1776
Copy link
Contributor

Hello - I'm going to close out this issue because this is neither a bug or feature request. Please ask these questions in our community related sites like Discourse or Slack.

As an aside, you might want to take a look at environmental variables through - that might work since you need to supply this via an yml file config.

@amychen1776 amychen1776 closed this as not planned Won't fix, can't repro, duplicate, stale Jan 6, 2025
@iamtodor
Copy link
Author

iamtodor commented Jan 6, 2025

@amychen1776 could you please let me know what I should do to make this issue a feature request?

@amychen1776
Copy link
Contributor

@iamtodor were you able to get it working with an env_var?

@iamtodor
Copy link
Author

iamtodor commented Jan 7, 2025

@amychen1776 nope, I wasnt

@amychen1776 amychen1776 changed the title Pass (modify/append) BigQuery job label as a CLI flag [Feature] Support Passing in {{ target.user }} for BQ Query Comments Jan 7, 2025
@amychen1776 amychen1776 removed triage:product In Product's queue feature:cost-reduction Issues related to measuring and reducing execution cost labels Jan 7, 2025
@amychen1776
Copy link
Contributor

I'll adjust the issue then

@amychen1776 amychen1776 reopened this Jan 7, 2025
@iamtodor
Copy link
Author

iamtodor commented Jan 7, 2025

@amychen1776 thank you!

@amychen1776
Copy link
Contributor

@iamtodor My coworker just flagged that there is a way around this using a macro which would be the way to get target.user https://docs.getdbt.com/reference/project-configs/query-comment#advanced-use-a-macro-to-generate-a-comment

@mikealfare mikealfare added the pkg:dbt-bigquery Issue affects dbt-bigquery label Jan 14, 2025
@mikealfare mikealfare transferred this issue from dbt-labs/dbt-bigquery Jan 14, 2025
mikealfare pushed a commit that referenced this issue Jan 14, 2025
* Fixes for incremental strategy py model.WIP

* doc string

* Remove extra comment

* Uncomment change schema test

* Update dbt/include/bigquery/macros/materializations/incremental.sql

Add python language exception for time_ingestion_partitioning

Co-authored-by: colin-rogers-dbt <[email protected]>

* Remove tox command flag for test python

* Adding the env vars in integration

* Switch to cluster execution for the python models

* Remove comment and add changie

* Skipping tests since dataproc is unstable. Restoring default to serverless

---------

Co-authored-by: colin-rogers-dbt <[email protected]>
@amychen1776 amychen1776 added the triage:awaiting-response Awaiting a response from the reporter label Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pkg:dbt-bigquery Issue affects dbt-bigquery triage:awaiting-response Awaiting a response from the reporter type:enhancement New feature request
Projects
None yet
Development

No branches or pull requests

4 participants