Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Control file module files and validation #2445

Draft
wants to merge 44 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
753618b
Util classes for data loader
inv-jishnu Dec 4, 2024
8d39d02
Fix spotbug issue
inv-jishnu Dec 4, 2024
bf94c49
Removed error message and added core error
inv-jishnu Dec 6, 2024
47be388
Applied spotless
inv-jishnu Dec 6, 2024
913eb1c
Fixed unit test failures
inv-jishnu Dec 6, 2024
1f204b8
Merge branch 'master' into feat/data-loader/utils
ypeckstadt Dec 11, 2024
6cfa83a
Basic data import enum and exception
inv-jishnu Dec 11, 2024
d381b2b
Removed exception class for now
inv-jishnu Dec 11, 2024
67f2474
Added DECIMAL_FORMAT
inv-jishnu Dec 12, 2024
14e3593
Path util class updated
inv-jishnu Dec 12, 2024
a096d51
Feedback changes
inv-jishnu Dec 13, 2024
dbf1940
Merge branch 'master' into feat/data-loader/utils
ypeckstadt Dec 13, 2024
cd8add9
Merge branch 'master' into feat/data-loader/utils
ypeckstadt Dec 16, 2024
52890c8
Changes
inv-jishnu Dec 16, 2024
5114639
Merge branch 'master' into feat/data-loader/import-data-1
inv-jishnu Dec 17, 2024
4f9cd75
Merge branch 'feat/data-loader/utils' into feat/data-loader/scaladb-dao
inv-jishnu Dec 17, 2024
1997eb8
Added ScalarDB Dao
inv-jishnu Dec 17, 2024
91e6310
Merge branch 'master' into feat/data-loader/scaladb-dao
inv-jishnu Dec 17, 2024
8a7338b
Remove unnecessary files
inv-jishnu Dec 17, 2024
2b52eeb
Initial commit [skip ci]
inv-jishnu Dec 17, 2024
e206073
Changes
inv-jishnu Dec 17, 2024
26d3144
Changes
inv-jishnu Dec 18, 2024
b86487d
spotbugs exclude
inv-jishnu Dec 18, 2024
818a2b4
spotbugs exclude -2
inv-jishnu Dec 18, 2024
90c4105
Added a file [skip ci]
inv-jishnu Dec 18, 2024
3d5d3e0
Added unit test files [skip ci]
inv-jishnu Dec 18, 2024
6495202
Spotbug fixes
inv-jishnu Dec 19, 2024
90abd9e
Removed use of List.of to fix CI error
inv-jishnu Dec 19, 2024
ba2b3dd
Merged changes from master after resolving conflict
inv-jishnu Dec 19, 2024
b1b811b
Merge branch 'master' into feat/data-loader/metadata-service
inv-jishnu Dec 19, 2024
30db988
Applied spotless
inv-jishnu Dec 19, 2024
03324e1
Minor change in test
inv-jishnu Dec 19, 2024
ccb1ace
Improve javadocs
ypeckstadt Dec 20, 2024
a374f1a
Add private constructor to TableMetadataUtil
ypeckstadt Dec 20, 2024
a65c9b5
Apply spotless fix
ypeckstadt Dec 20, 2024
acedabe
Partial feedback changes
inv-jishnu Dec 24, 2024
bf31a01
Data chunk and task result enums and dtos
inv-jishnu Dec 24, 2024
57cd330
Spotless applied
inv-jishnu Dec 24, 2024
7a39564
Changes
inv-jishnu Dec 26, 2024
093cb1d
Merge branch 'feat/data-loader/scaladb-dao' into feat/data-loader/imp…
inv-jishnu Dec 31, 2024
bfebd95
Merge branch 'feat/data-loader/metadata-service' into feat/data-loade…
inv-jishnu Dec 31, 2024
fd1c186
Control file files
inv-jishnu Dec 31, 2024
cebb543
Merge branch 'master' into feat/data-loader/control-file
ypeckstadt Jan 8, 2025
8ecb39c
Changes -1
inv-jishnu Jan 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions core/src/main/java/com/scalar/db/common/error/CoreError.java
Original file line number Diff line number Diff line change
Expand Up @@ -719,6 +719,50 @@ public enum CoreError implements ScalarDbError {
"",
""),

DATA_LOADER_DUPLICATE_DATA_MAPPINGS(
Category.USER_ERROR,
"0158",
"Duplicate data mappings found for table '%s' in the control file",
"",
""),
DATA_LOADER_MISSING_COLUMN_MAPPING(
Category.USER_ERROR,
"0159",
"No mapping found for column '%s' in table '%s' in the control file. \\nControl file validation set at 'FULL'. All columns need to be mapped.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"No mapping found for column '%s' in table '%s' in the control file. \\nControl file validation set at 'FULL'. All columns need to be mapped.",
"No mapping found for column '%s' in table '%s' in the control file. Control file validation set at 'FULL'. All columns need to be mapped.",

Ignore this comment if the newline is really necessary.

"",
""),
DATA_LOADER_CONTROL_FILE_MISSING_DATA_MAPPINGS(
Category.USER_ERROR, "0160", "The control file is missing data mappings", "", ""),
DATA_LOADER__MISSING_NAMESPACE_OR_TABLE(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
DATA_LOADER__MISSING_NAMESPACE_OR_TABLE(
DATA_LOADER_MISSING_NAMESPACE_OR_TABLE(

Category.USER_ERROR,
"0161",
"The provided namespace '%s' and/or table name '%s' is incorrect and could not be found",
"",
""),
DATA_LOADER_TARGET_COLUMN_NOT_FOUND(
Category.USER_ERROR,
"0162",
"The target column '%s' for source field '%s' could not be found in table '%s'",
"",
""),
DATA_LOADER_MISSING_PARTITION_KEY(
Category.USER_ERROR,
"0163",
"The required partition key '%s' is missing in the control file mapping for table '%s'",
"",
""),
DATA_LOADER_MISSING_CLUSTERING_KEY(
Category.USER_ERROR,
"0164",
"The required clustering key '%s' is missing in the control file mapping for table '%s'",
"",
""),
DATA_LOADER_MULTIPLE_MAPPINGS_FOR_COLUMN_FOUND(
Category.USER_ERROR,
"0165",
"Multiple data mappings found for column '%s' in table '%s'",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"Multiple data mappings found for column '%s' in table '%s'",
"Duplicated data mappings found for column '%s' in table '%s'",

"Duplicated" sounds a bit clear?

"",
""),
//
// Errors for the concurrency error category
//
Expand Down Expand Up @@ -969,6 +1013,18 @@ public enum CoreError implements ScalarDbError {
"Handling the before-preparation snapshot hook failed. Details: %s",
"",
""),
DATA_LOADER_ERROR_CRUD_EXCEPTION(
Category.INTERNAL_ERROR,
"0047",
"Something went wrong while trying to save the data. Details %s",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"Something went wrong while trying to save the data. Details %s",
"Something went wrong while trying to save the data. Details: %s",

"",
""),
DATA_LOADER_ERROR_SCAN(
Category.INTERNAL_ERROR,
"0048",
"Something went wrong while scanning. Are you sure you are running in the correct transaction mode? Details %s",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"Something went wrong while scanning. Are you sure you are running in the correct transaction mode? Details %s",
"Something went wrong while scanning. Are you sure you are running in the correct transaction mode? Details: %s",

"",
""),

//
// Errors for the unknown transaction status error category
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package com.scalar.db.dataloader.core.dataimport.controlfile;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import java.util.ArrayList;
import java.util.List;
import lombok.Getter;
import lombok.Setter;

/** Represents the control file */
@Getter
@Setter
public class ControlFile {

@JsonProperty("tables")
private final List<ControlFileTable> tables;

/** Class constructor */
public ControlFile() {
this.tables = new ArrayList<>();
}

@JsonCreator
public ControlFile(@JsonProperty("tables") List<ControlFileTable> tables) {
this.tables = tables;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
package com.scalar.db.dataloader.core.dataimport.controlfile;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import java.util.ArrayList;
import java.util.List;
import lombok.Getter;
import lombok.Setter;

/**
* Represents the configuration for a single table in the control file, including its namespace,
* table name, and field mappings. This class is used to define how data from a control file maps to
* a specific table in ScalarDB.
*/
@Getter
@Setter
public class ControlFileTable {

/** The namespace of the table in ScalarDB. */
@JsonProperty("namespace")
private String namespace;

/** The name of the table in ScalarDB. */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

namespace above is also just the name of a namespace. How about renaming this to table as well or the above namespace to namespaceName for consistency?

@JsonProperty("table_name")
private String tableName;

/**
* A list of mappings defining the correspondence between control file fields and table columns.
*/
@JsonProperty("mappings")
private final List<ControlFileTableFieldMapping> mappings;

/**
* Creates a new {@code ControlFileTable} instance with the specified namespace and table name.
* The mappings list is initialized as an empty list.
*
* @param namespace The namespace of the table in ScalarDB.
* @param tableName The name of the table in ScalarDB.
*/
public ControlFileTable(String namespace, String tableName) {
this.namespace = namespace;
this.tableName = tableName;
this.mappings = new ArrayList<>();
}

/**
* Constructs a {@code ControlFileTable} instance using data from a serialized JSON object. This
* constructor is used for deserialization of API requests or control files.
*
* @param namespace The namespace of the table in ScalarDB.
* @param tableName The name of the table in ScalarDB.
* @param mappings A list of mappings that define the relationship between control file fields and
* table columns.
*/
@JsonCreator
public ControlFileTable(
@JsonProperty("namespace") String namespace,
@JsonProperty("table_name") String tableName,
@JsonProperty("mappings") List<ControlFileTableFieldMapping> mappings) {
this.namespace = namespace;
this.tableName = tableName;
this.mappings = mappings;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
package com.scalar.db.dataloader.core.dataimport.controlfile;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.Getter;
import lombok.Setter;

/**
* Represents the mapping of a single field in the control file to a column in a ScalarDB table.
* This class defines how data from a specific field in the input source should be mapped to the
* corresponding column in the database.
*/
@Getter
@Setter
public class ControlFileTableFieldMapping {

/** The name of the field in the input source (e.g., JSON or CSV). */
@JsonProperty("source_field")
private String sourceField;

/** The name of the column in the ScalarDB table that the field maps to. */
@JsonProperty("target_column")
private String targetColumn;

/**
* Constructs a {@code ControlFileTableFieldMapping} instance using data from a serialized JSON
* object. This constructor is primarily used for deserialization of control file mappings.
*
* @param sourceField The name of the field in the input source (e.g., JSON or CSV).
* @param targetColumn The name of the corresponding column in the ScalarDB table.
*/
@JsonCreator
public ControlFileTableFieldMapping(
@JsonProperty("source_field") String sourceField,
@JsonProperty("target_column") String targetColumn) {
this.sourceField = sourceField;
this.targetColumn = targetColumn;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
package com.scalar.db.dataloader.core.dataimport.controlfile;

/** Represents the control file */
public class ControlFileValidationException extends Exception {

/**
* Class constructor
*
* @param message error message
*/
public ControlFileValidationException(String message) {
super(message);
}
}
Loading
Loading