Add Control file module files and validation #2445

inv-jishnu · 2025-01-02T05:03:15Z

Description

This PR adds files related to control file and its validation. A control file is used to map column names specified in import source file to actual table columns.

Related issues and/or PRs

Please review and merge this PR once the following PRs are merged

Changes made

I have added files related to control file and its validation

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

Road map to merge remaining data loader core files. Current status

General
- Add ScalarDB Dao and related files - Add ScalarDB Dao and related files #2417
- TableMetadataService(partially replaced by ConsensusUtils): Add table metadata service #2434
Export
- Export options and validations: Add export options validator #2435
- ProducerTasks: 1 PR incoming
Import
- Dto classes and utilities: Add data chunk and task result enums and dtos #2442
- Import processor and task code: 2-3 PRs incoming
- Code for Import transaction batch and data chunk import: 1 PR Incoming
- ControlFile related Dtos: Add Control file module files and validation #2445
- Import logger: 1 PR incoming
- Import Manager: 1 PR incoming

Release notes

NA

…ort-task

…r/import-task

komamitsu · 2025-01-09T03:11:56Z

core/src/main/java/com/scalar/db/common/error/CoreError.java

+  DATA_LOADER_MISSING_COLUMN_MAPPING(
+      Category.USER_ERROR,
+      "0159",
+      "No mapping found for column '%s' in table '%s' in the control file. \\nControl file validation set at 'FULL'. All columns need to be mapped.",


Suggested change

"No mapping found for column '%s' in table '%s' in the control file. \\nControl file validation set at 'FULL'. All columns need to be mapped.",

"No mapping found for column '%s' in table '%s' in the control file. Control file validation set at 'FULL'. All columns need to be mapped.",

Ignore this comment if the newline is really necessary.

komamitsu · 2025-01-09T03:20:18Z

core/src/main/java/com/scalar/db/common/error/CoreError.java

+  DATA_LOADER_ERROR_CRUD_EXCEPTION(
+      Category.INTERNAL_ERROR,
+      "0047",
+      "Something went wrong while trying to save the data. Details %s",


Suggested change

"Something went wrong while trying to save the data. Details %s",

"Something went wrong while trying to save the data. Details: %s",

komamitsu · 2025-01-09T03:20:31Z

core/src/main/java/com/scalar/db/common/error/CoreError.java

+  DATA_LOADER_ERROR_SCAN(
+      Category.INTERNAL_ERROR,
+      "0048",
+      "Something went wrong while scanning. Are you sure you are running in the correct transaction mode? Details %s",


Suggested change

"Something went wrong while scanning. Are you sure you are running in the correct transaction mode? Details %s",

"Something went wrong while scanning. Are you sure you are running in the correct transaction mode? Details: %s",

komamitsu · 2025-01-09T03:23:37Z

...ore/src/main/java/com/scalar/db/dataloader/core/dataimport/controlfile/ControlFileTable.java

+  @JsonProperty("namespace")
+  private String namespace;
+
+  /** The name of the table in ScalarDB. */


namespace above is also just the name of a namespace. How about renaming this to table as well or the above namespace to namespaceName for consistency?

komamitsu · 2025-01-09T05:18:26Z

core/src/main/java/com/scalar/db/common/error/CoreError.java

+      ""),
+  DATA_LOADER_CONTROL_FILE_MISSING_DATA_MAPPINGS(
+      Category.USER_ERROR, "0160", "The control file is missing data mappings", "", ""),
+  DATA_LOADER__MISSING_NAMESPACE_OR_TABLE(


Suggested change

DATA_LOADER__MISSING_NAMESPACE_OR_TABLE(

DATA_LOADER_MISSING_NAMESPACE_OR_TABLE(

komamitsu · 2025-01-09T05:47:26Z

data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDBDao.java

+      String namespace,
+      String table,
+      Key partitionKey,
+      Key clusteringKey,


Suggested change

Key clusteringKey,

@Nullable Key clusteringKey,

Java is a poor language unable to represent nullable or non-nullable with type, and @Nullable annotation is very helpful for us.

komamitsu · 2025-01-09T05:50:00Z

data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDBDao.java

+      logger.info(String.format(GET_COMPLETED_MSG, loggingKey));
+      return result;
+    } catch (CrudException e) {
+      throw new ScalarDBDaoException("error GET " + loggingKey, e.getCause());


Suggested change

throw new ScalarDBDaoException("error GET " + loggingKey, e.getCause());

throw new ScalarDBDaoException("error GET " + loggingKey, e);

Just including e would be a bit better not to lose the context.

komamitsu · 2025-01-09T05:53:22Z

data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDBDao.java

+      logger.info(SCAN_START_MSG);
+      Scanner scanner = storage.scan(scan);
+      List<Result> allResults = scanner.all();
+      scanner.close();


It's possible this close isn't called if scanner.all() throws an exception. How about using try-with-resources statement for scanner?

komamitsu · 2025-01-09T05:57:50Z

...-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDBManager.java

+  /* Distributed storage for ScalarDB connection that is running in storage mode. */
+  @Nullable private final DistributedStorage storage;
+  /* Distributed Transaction manager for ScalarDB connection that is running in transaction mode */
+  private final DistributedTransactionManager transactionManager;


These 3 variables should have @Nullable annotation as well.

komamitsu · 2025-01-09T05:58:56Z

...-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/dao/ScalarDBManager.java

+  }
+
+  /** @return storage for ScalarDB connection that is running in storage mode */
+  public DistributedStorage getDistributedStorage() {


Suggested change

public DistributedStorage getDistributedStorage() {

@Nullable public DistributedStorage getDistributedStorage() {

inv-jishnu and others added 30 commits December 4, 2024 15:59

Util classes for data loader

753618b

Fix spotbug issue

8d39d02

Removed error message and added core error

bf94c49

Applied spotless

47be388

Fixed unit test failures

913eb1c

Merge branch 'master' into feat/data-loader/utils

1f204b8

Basic data import enum and exception

6cfa83a

Removed exception class for now

d381b2b

Added DECIMAL_FORMAT

67f2474

Path util class updated

14e3593

Feedback changes

a096d51

Merge branch 'master' into feat/data-loader/utils

dbf1940

Merge branch 'master' into feat/data-loader/utils

cd8add9

Changes

52890c8

Merge branch 'master' into feat/data-loader/import-data-1

5114639

Merge branch 'feat/data-loader/utils' into feat/data-loader/scaladb-dao

4f9cd75

Added ScalarDB Dao

1997eb8

Merge branch 'master' into feat/data-loader/scaladb-dao

91e6310

Remove unnecessary files

8a7338b

Initial commit [skip ci]

2b52eeb

Changes

e206073

Changes

26d3144

spotbugs exclude

b86487d

spotbugs exclude -2

818a2b4

Added a file [skip ci]

90c4105

Added unit test files [skip ci]

3d5d3e0

Spotbug fixes

6495202

Removed use of List.of to fix CI error

90abd9e

Merged changes from master after resolving conflict

ba2b3dd

Merge branch 'master' into feat/data-loader/metadata-service

b1b811b

inv-jishnu and others added 12 commits December 19, 2024 15:45

Applied spotless

30db988

Minor change in test

03324e1

Improve javadocs

ccb1ace

Add private constructor to TableMetadataUtil

a374f1a

Apply spotless fix

a65c9b5

Partial feedback changes

acedabe

Data chunk and task result enums and dtos

bf31a01

Spotless applied

57cd330

Changes

7a39564

Merge branch 'feat/data-loader/scaladb-dao' into feat/data-loader/imp…

093cb1d

…ort-task

Merge branch 'feat/data-loader/metadata-service' into feat/data-loade…

bfebd95

…r/import-task

Control file files

fd1c186

inv-jishnu marked this pull request as draft January 2, 2025 05:03

This was referenced Jan 2, 2025

Add dtos and other classes for task #2446

Draft

Add export tasks #2450

Draft

inv-jishnu self-assigned this Jan 7, 2025

ypeckstadt added the enhancement New feature or request label Jan 8, 2025

Merge branch 'master' into feat/data-loader/control-file

cebb543

ypeckstadt marked this pull request as ready for review January 8, 2025 23:47

ypeckstadt requested review from komamitsu, brfrn169, feeblefakie and Torch3333 January 8, 2025 23:47

komamitsu reviewed Jan 9, 2025

View reviewed changes

inv-jishnu marked this pull request as draft January 9, 2025 10:08

Changes -1

8ecb39c

This was referenced Jan 10, 2025

Add import processes #2462

Draft

Add import log classes and utils #2463

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Control file module files and validation #2445

Add Control file module files and validation #2445

inv-jishnu commented Jan 2, 2025 •

edited

Loading

komamitsu Jan 9, 2025

komamitsu Jan 9, 2025

komamitsu Jan 9, 2025

komamitsu Jan 9, 2025

komamitsu Jan 9, 2025

komamitsu Jan 9, 2025

komamitsu Jan 9, 2025

komamitsu Jan 9, 2025

komamitsu Jan 9, 2025

komamitsu Jan 9, 2025

	"No mapping found for column '%s' in table '%s' in the control file. \\nControl file validation set at 'FULL'. All columns need to be mapped.",
	"No mapping found for column '%s' in table '%s' in the control file. Control file validation set at 'FULL'. All columns need to be mapped.",

	"Something went wrong while trying to save the data. Details %s",
	"Something went wrong while trying to save the data. Details: %s",

	"Something went wrong while scanning. Are you sure you are running in the correct transaction mode? Details %s",
	"Something went wrong while scanning. Are you sure you are running in the correct transaction mode? Details: %s",

	DATA_LOADER__MISSING_NAMESPACE_OR_TABLE(
	DATA_LOADER_MISSING_NAMESPACE_OR_TABLE(

	throw new ScalarDBDaoException("error GET " + loggingKey, e.getCause());
	throw new ScalarDBDaoException("error GET " + loggingKey, e);

	public DistributedStorage getDistributedStorage() {
	@Nullable public DistributedStorage getDistributedStorage() {

Add Control file module files and validation #2445

Are you sure you want to change the base?

Add Control file module files and validation #2445

Conversation

inv-jishnu commented Jan 2, 2025 • edited Loading

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inv-jishnu commented Jan 2, 2025 •

edited

Loading