Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] pinot sql transformation to sr sql with date functionPinot dialect date function #55106

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

[Feature] pinot sql transformation to sr sql with date functionPinot dialect date function #55106

wants to merge 8 commits into from
+1,501 −84

Conversation

maggie-zhu
Copy link

@maggie-zhu maggie-zhu commented Jan 15, 2025 •
edited by Youngwb
Loading

Why I'm doing:

We have many pinot sqls in the prod environment when we try to migrate the pinot to starrocks. We hope the starrocks can support the sql transformation automatically.

What I'm doing:

To support the sql transformation from pinot sql to starrocks sql when use some date/string/aggregation functions.

Fixes #55276

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.4
    • 3.3
    • 3.2
    • 3.1
    • 3.0

@maggie-zhu maggie-zhu requested review from a team as code owners January 15, 2025 07:29
@maggie-zhu maggie-zhu force-pushed the pinotDialectDateFunction branch from e5a810e to 89b9dd5 Compare January 15, 2025 07:37
murphyatwork
murphyatwork reviewed Jan 16, 2025
View reviewed changes
...-core/src/main/java/com/starrocks/connector/parser/pinot/ComplexFunctionCallTransformer.java Outdated
if (args.length < 2 || args.length > 5) {
throw new SemanticException("The datetrunc function must include between 2 and 5 parameters, inclusive.");
}
// DATETRUNC(unit, timeValue) or DATETRUNC(unit, timeValue, inputTimeUnitStr) output is milliseconds --> unix_timestamp(date_trunc('day',event_timestamp)) * 1000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, too long

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have change it to be shorter

...-core/src/main/java/com/starrocks/connector/parser/pinot/ComplexFunctionCallTransformer.java Outdated

public class ComplexFunctionCallTransformer {
public static Expr transform(String functionName, Expr... args) {
if (functionName.equalsIgnoreCase("datetimeconvert")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you make it more extensible with separated class/function for each function ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have added some seperated functions for each function

fe/fe-core/src/main/java/com/starrocks/connector/parser/pinot/AstBuilder.java
Expr convertedFunctionCall = Pinot2SRFunctionCallTransformer.convert(functionName, arguments);

if (convertedFunctionCall != null) {
if (functionName.equalsIgnoreCase("fromdatetime")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make it a static ?

Youngwb
Youngwb reviewed Jan 17, 2025
View reviewed changes
...core/src/main/java/com/starrocks/connector/parser/pinot/Pinot2SRFunctionCallTransformer.java Outdated
import java.util.List;
import java.util.Map;

public class Pinot2SRFunctionCallTransformer {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better abstract a class FunctionCallTransformer, both Trino2SRFunctionCallTransformer and Pinot2SRFunctionCallTransformer can inherit from it, to avoid duplicate code.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have create a abstract base class

fe/fe-core/src/main/java/com/starrocks/sql/parser/SqlParser.java Outdated
statements.add(statement);
}
String sqlTrans = AstToSQLBuilder.toSQL(statements.get(0));
System.out.println("sqlTrans: " + sqlTrans);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not print here

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have delete it

@maggie-zhu maggie-zhu force-pushed the pinotDialectDateFunction branch 4 times, most recently from edf677c to 9dd9d64 Compare January 21, 2025 02:43
@maggie-zhu maggie-zhu force-pushed the pinotDialectDateFunction branch 3 times, most recently from 5b958d1 to 277c98a Compare January 21, 2025 06:01
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
1 Security Hotspot
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Youngwb
Youngwb previously approved these changes Jan 21, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

@github-actions GitHub Actions
Copy link

[FE Incremental Coverage Report]

✅ pass : 396 / 489 (80.98%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/connector/parser/pinot/AstBuilder.java 25 46 54.35% [46, 47, 77, 79, 80, 82, 88, 89, 90, 91, 93, 94, 95, 96, 97, 101, 102, 104, 113, 114, 115]
🔵 com/starrocks/connector/parser/FunctionCallTransformer.java 42 57 73.68% [44, 45, 52, 85, 94, 95, 96, 98, 99, 106, 136, 137, 138, 139, 140]
🔵 com/starrocks/connector/parser/BaseFunctionCallTransformer.java 30 39 76.92% [79, 80, 81, 82, 94, 96, 97, 98, 99]
🔵 com/starrocks/connector/parser/pinot/PinotParserUtils.java 118 146 80.82% [37, 111, 112, 113, 114, 115, 116, 117, 118, 120, 121, 123, 124, 125, 126, 127, 128, 129, 131, 139, 140, 172, 181, 193, 212, 214, 220, 226]
🔵 com/starrocks/connector/parser/pinot/ComplexFunctionCallTransformer.java 122 139 87.77% [54, 59, 127, 139, 143, 156, 157, 158, 159, 161, 162, 163, 184, 204, 281, 283, 284]
🔵 com/starrocks/connector/parser/pinot/Pinot2SRFunctionCallTransformer.java 42 45 93.33% [54, 79, 91]
🔵 com/starrocks/connector/parser/PlaceholderExpr.java 8 8 100.00% []
🔵 com/starrocks/connector/parser/BaseComplexFunctionCallTransformer.java 1 1 100.00% []
🔵 com/starrocks/connector/parser/trino/Trino2SRFunctionCallTransformer.java 3 3 100.00% []
🔵 com/starrocks/connector/parser/trino/AstBuilder.java 4 4 100.00% []
🔵 com/starrocks/connector/parser/trino/ComplexFunctionCallTransformer.java 1 1 100.00% []

@github-actions GitHub Actions
Copy link

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

@maggie-zhu maggie-zhu force-pushed the pinotDialectDateFunction branch from 277c98a to 0900713 Compare January 22, 2025 07:15
@maggie-zhu maggie-zhu force-pushed the pinotDialectDateFunction branch from 0900713 to 082ebbd Compare January 22, 2025 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Youngwb Youngwb Youngwb left review comments

@murphyatwork murphyatwork Awaiting requested review from murphyatwork

At least 2 approving reviews are required to merge this pull request.

Assignees

@maggie-zhu maggie-zhu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature] support pinot dialect
3 participants
@maggie-zhu @Youngwb @murphyatwork