Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add log workload into cli #13353

Merged
merged 24 commits into from
Jan 15, 2025
Merged

Add log workload into cli #13353

merged 24 commits into from
Jan 15, 2025

Conversation

maximyurchuk
Copy link
Collaborator

@maximyurchuk maximyurchuk commented Jan 14, 2025

#13372

New workload for emulation of log writing was added

First column in primary key is timestamp with specified ttl

After timestamp goes arbitrary number of int and string columns

Timestamp can be modified with normal distribution with specified standard deviation

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 09:44:58 UTC Pre-commit check linux-x86_64-relwithdebinfo for 225c962 has started.
2025-01-14 09:45:10 UTC Artifacts will be uploaded here
🔴 2025-01-14 09:46:59 UTC Graph compare failed, see the logs.

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 09:45:03 UTC Pre-commit check linux-x86_64-release-asan for 225c962 has started.
2025-01-14 09:45:15 UTC Artifacts will be uploaded here
🔴 2025-01-14 09:47:03 UTC Graph compare failed, see the logs.

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 10:14:34 UTC Pre-commit check linux-x86_64-release-asan for 5fbb255 has started.
2025-01-14 10:14:46 UTC Artifacts will be uploaded here
2025-01-14 10:17:37 UTC ya make is running...
🟡 2025-01-14 11:37:43 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11330 11282 0 15 9 24

2025-01-14 11:38:44 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-14 11:50:20 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
80 (only retried tests) 50 0 1 7 22

2025-01-14 11:50:28 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-14 11:58:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
52 (only retried tests) 29 0 1 0 22

🟢 2025-01-14 11:58:41 UTC Build successful.
🟢 2025-01-14 11:59:07 UTC ydbd size 3.6 GiB changed* by +3.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 607b445 merge: 5fbb255 diff diff %
ydbd size 3 895 272 368 Bytes 3 895 275 680 Bytes +3.2 KiB +0.000%
ydbd stripped size 1 360 912 368 Bytes 1 360 913 456 Bytes +1.1 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 10:16:45 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5fbb255 has started.
2025-01-14 10:16:57 UTC Artifacts will be uploaded here
2025-01-14 10:19:56 UTC ya make is running...
🟡 2025-01-14 11:06:26 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
18300 17005 0 4 1170 121

2025-01-14 11:07:58 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-14 11:29:35 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
183 (only retried tests) 64 0 0 0 119

🟢 2025-01-14 11:29:45 UTC Build successful.
🟢 2025-01-14 11:30:04 UTC ydbd size 2.1 GiB changed* by +2.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 607b445 merge: 5fbb255 diff diff %
ydbd size 2 247 416 072 Bytes 2 247 418 152 Bytes +2.0 KiB +0.000%
ydbd stripped size 472 861 904 Bytes 472 862 288 Bytes +384 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 12:20:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for fd8ae45 has started.
2025-01-14 12:21:00 UTC Artifacts will be uploaded here
2025-01-14 12:23:53 UTC ya make is running...
🟡 2025-01-14 13:07:27 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
18300 17010 0 1 1169 120

2025-01-14 13:08:55 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-14 13:18:36 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
177 (only retried tests) 61 0 0 0 116

🟢 2025-01-14 13:18:47 UTC Build successful.
🟢 2025-01-14 13:19:07 UTC ydbd size 2.1 GiB changed* by +62.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: ecc265c merge: fd8ae45 diff diff %
ydbd size 2 247 422 776 Bytes 2 247 486 720 Bytes +62.4 KiB +0.003%
ydbd stripped size 472 862 928 Bytes 472 872 048 Bytes +8.9 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 12:20:51 UTC Pre-commit check linux-x86_64-release-asan for fd8ae45 has started.
2025-01-14 12:21:03 UTC Artifacts will be uploaded here
2025-01-14 12:23:54 UTC ya make is running...
🟡 2025-01-14 13:33:15 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11330 11281 0 20 2 27

2025-01-14 13:34:33 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-14 13:46:26 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
78 (only retried tests) 51 0 1 0 26

2025-01-14 13:46:34 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-01-14 13:58:14 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
49 (only retried tests) 24 0 0 1 24

🟢 2025-01-14 13:58:21 UTC Build successful.
🟢 2025-01-14 13:58:49 UTC ydbd size 3.6 GiB changed* by +4.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 5316ae5 merge: fd8ae45 diff diff %
ydbd size 3 895 409 064 Bytes 3 895 413 224 Bytes +4.1 KiB +0.000%
ydbd stripped size 1 360 962 832 Bytes 1 360 964 240 Bytes +1.4 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 14:20:55 UTC Pre-commit check linux-x86_64-release-asan for 8bb4ea0 has started.
2025-01-14 14:21:24 UTC Artifacts will be uploaded here
2025-01-14 14:25:00 UTC ya make is running...
2025-01-14 14:32:11 UTC Check cancelled

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 14:26:26 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8bb4ea0 has started.
2025-01-14 14:26:35 UTC Artifacts will be uploaded here
2025-01-14 14:29:27 UTC ya make is running...
2025-01-14 14:31:53 UTC Check cancelled

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 14:32:52 UTC Pre-commit check linux-x86_64-release-asan for e380402 has started.
2025-01-14 14:33:20 UTC Artifacts will be uploaded here
2025-01-14 14:36:16 UTC ya make is running...
2025-01-14 14:58:26 UTC Check cancelled

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 14:33:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for e380402 has started.
2025-01-14 14:34:02 UTC Artifacts will be uploaded here
2025-01-14 14:36:51 UTC ya make is running...
2025-01-14 14:58:27 UTC Check cancelled

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 14:59:20 UTC Pre-commit check linux-x86_64-release-asan for 0e38c71 has started.
2025-01-14 15:00:58 UTC Artifacts will be uploaded here
2025-01-14 15:04:22 UTC ya make is running...
🟡 2025-01-14 16:35:17 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11330 11282 0 19 2 27

2025-01-14 16:36:31 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-14 16:48:18 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
79 (only retried tests) 55 0 0 0 24

🟢 2025-01-14 16:48:26 UTC Build successful.
🟢 2025-01-14 16:48:51 UTC ydbd size 3.6 GiB changed* by +22.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: df99e52 merge: 0e38c71 diff diff %
ydbd size 3 895 409 352 Bytes 3 895 432 248 Bytes +22.4 KiB +0.001%
ydbd stripped size 1 360 972 048 Bytes 1 360 977 744 Bytes +5.6 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 15:00:24 UTC Pre-commit check linux-x86_64-relwithdebinfo for 0e38c71 has started.
2025-01-14 15:01:20 UTC Artifacts will be uploaded here
2025-01-14 15:04:50 UTC ya make is running...
🟡 2025-01-14 16:24:53 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
18300 17001 0 5 1170 124

2025-01-14 16:26:32 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-14 16:38:46 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
187 (only retried tests) 66 0 0 0 121

🟢 2025-01-14 16:38:53 UTC Build successful.
🟢 2025-01-14 16:39:14 UTC ydbd size 2.1 GiB changed* by +18.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: df99e52 merge: 0e38c71 diff diff %
ydbd size 2 247 481 760 Bytes 2 247 500 544 Bytes +18.3 KiB +0.001%
ydbd stripped size 472 870 640 Bytes 472 873 232 Bytes +2.5 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

ydb/library/workload/log_writer/log_writer.cpp Outdated Show resolved Hide resolved
ydb/library/workload/log_writer/log_writer.cpp Outdated Show resolved Hide resolved
ydb/library/workload/log_writer/log_writer.cpp Outdated Show resolved Hide resolved
.DefaultValue((ui64)LogWriterWorkloadConstants::PARTITION_SIZE_MB).StoreResult(&PartitionSizeMb);
opts.AddLongOption("auto-partition", "Enable auto partitioning by load.")
.DefaultValue((ui64)LogWriterWorkloadConstants::PARTITIONS_BY_LOAD).StoreResult(&PartitionsByLoad);
opts.AddLongOption("len", "String len")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

str-len at least?
I'd call them str-length, int-columns, key-columns

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was taken from kv workload also..

.DefaultValue((ui64)LogWriterWorkloadConstants::STR_COLUMNS_CNT).StoreResult(&StrColumnsCnt);
opts.AddLongOption("key-cols", "Number of key columns")
.DefaultValue((ui64)LogWriterWorkloadConstants::KEY_COLUMNS_CNT).StoreResult(&KeyColumnsCnt);
opts.AddLongOption("ttl_minutes", "TTL for timestamp column")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's call it just --ttl? And leave "minutes" in option description: "TTL for timestamp column, in minutes"

I have an idea of making an universal time duration parser. So that in future you could use
--ttl 60 (minutes by default)
--ttl 24h
--ttl 3d e.t.c.

ydb/public/lib/ydb_cli/commands/ydb_workload.cpp Outdated Show resolved Hide resolved
ydb/public/lib/ydb_cli/commands/ydb_workload.cpp Outdated Show resolved Hide resolved
ydb/tests/workloads/log_writer/tests/test_workload.py Outdated Show resolved Hide resolved
ydb/library/workload/log_writer/registrar.cpp Outdated Show resolved Hide resolved
Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 17:40:43 UTC Pre-commit check linux-x86_64-relwithdebinfo for 0f844d7 has started.
2025-01-14 17:40:54 UTC Artifacts will be uploaded here
2025-01-14 17:43:43 UTC ya make is running...
🟡 2025-01-14 18:26:11 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
18300 16844 0 1 1397 58

2025-01-14 18:27:42 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-14 18:39:38 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
347 (only retried tests) 225 0 0 7 115

🟢 2025-01-14 18:39:45 UTC Build successful.
🟢 2025-01-14 18:40:03 UTC ydbd size 2.1 GiB changed* by +2.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f4f16ff merge: 0f844d7 diff diff %
ydbd size 2 247 489 248 Bytes 2 247 491 328 Bytes +2.0 KiB +0.000%
ydbd stripped size 472 871 760 Bytes 472 872 144 Bytes +384 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 14, 2025

2025-01-14 17:41:56 UTC Pre-commit check linux-x86_64-release-asan for 0f844d7 has started.
2025-01-14 17:42:07 UTC Artifacts will be uploaded here
2025-01-14 17:45:07 UTC ya make is running...
🟡 2025-01-14 18:42:48 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11330 11281 0 20 4 25

2025-01-14 18:43:48 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-14 18:55:26 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
82 (only retried tests) 56 0 1 2 23

2025-01-14 18:55:35 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-14 19:06:49 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
49 (only retried tests) 24 0 1 0 24

🟢 2025-01-14 19:06:56 UTC Build successful.
🟢 2025-01-14 19:07:23 UTC ydbd size 3.6 GiB changed* by +3.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f4f16ff merge: 0f844d7 diff diff %
ydbd size 3 895 403 088 Bytes 3 895 406 400 Bytes +3.2 KiB +0.000%
ydbd stripped size 1 360 972 400 Bytes 1 360 973 488 Bytes +1.1 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@maximyurchuk maximyurchuk changed the title wip log workload Add log workload into cli Jan 14, 2025
@maximyurchuk maximyurchuk marked this pull request as ready for review January 14, 2025 17:57
@maximyurchuk maximyurchuk requested a review from a team as a code owner January 14, 2025 17:57
@maximyurchuk maximyurchuk merged commit b7192f6 into ydb-platform:main Jan 15, 2025
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants