Some options

japila-books · Jan 14, 2024 · bdfeeb6 · bdfeeb6
1 parent ae3de87
commit bdfeeb6
Show file tree

Hide file tree

Showing 3 changed files with 46 additions and 14 deletions.
diff --git a/docs/spark-connector/DeltaWriteOptions.md b/docs/spark-connector/DeltaWriteOptions.md
@@ -4,28 +4,30 @@
 
 `DeltaWriteOptions` is [DeltaWriteOptionsImpl](DeltaWriteOptionsImpl.md) and [DeltaOptionParser](DeltaOptionParser.md).
 
-## <span id="replaceWhere"> replaceWhere
+## replaceWhere { #replaceWhere }
 
 ```scala
 replaceWhere: Option[String]
 ```
 
 `replaceWhere` is the value of [replaceWhere](options.md#REPLACE_WHERE_OPTION) option.
 
+---
+
 `replaceWhere` is used when:
 
-* `WriteIntoDelta` command is [created](../commands/WriteIntoDelta.md#canOverwriteSchema) and [executed](../commands/WriteIntoDelta.md#run)
 * `CreateDeltaTableCommand` command is requested for a [Delta Operation](../commands/CreateDeltaTableCommand.md#getOperation) (for history purposes)
+* `WriteIntoDelta` command is [created](../commands/WriteIntoDelta.md#canOverwriteSchema), [executed](../commands/WriteIntoDelta.md#run) and requested to [write](../commands/WriteIntoDelta.md#write)
 
-## <span id="userMetadata"> userMetadata
+## userMetadata { #userMetadata }
 
 ```scala
 userMetadata: Option[String]
 ```
 
 `userMetadata` is the value of [userMetadata](options.md#USER_METADATA_OPTION) option.
 
-## <span id="optimizeWrite"> optimizeWrite
+## optimizeWrite { #optimizeWrite }
 
 ```scala
 optimizeWrite: Option[Boolean]

diff --git a/docs/spark-connector/DeltaWriteOptionsImpl.md b/docs/spark-connector/DeltaWriteOptionsImpl.md
@@ -21,7 +21,7 @@ canMergeSchema: Boolean
 * `WriteIntoDelta` is [created](../commands/WriteIntoDelta.md#canMergeSchema)
 * `DeltaSink` is [created](DeltaSink.md#canMergeSchema)
 
-## <span id="canOverwriteSchema"> canOverwriteSchema
+## canOverwriteSchema { #canOverwriteSchema }
 
 ```scala
 canOverwriteSchema: Boolean
@@ -35,14 +35,30 @@ canOverwriteSchema: Boolean
 * `WriteIntoDelta` is [created](../commands/WriteIntoDelta.md#canOverwriteSchema)
 * `DeltaSink` is [created](DeltaSink.md#canOverwriteSchema)
 
-## <span id="rearrangeOnly"> rearrangeOnly
+## rearrangeOnly { #rearrangeOnly }
 
 ```scala
 rearrangeOnly: Boolean
 ```
 
-`rearrangeOnly` is the value of [dataChange](options.md#DATA_CHANGE_OPTION) option.
+`rearrangeOnly` is the negation of the value of [dataChange](options.md#DATA_CHANGE_OPTION) option.
+
+---
 
 `rearrangeOnly` is used when:
 
 * `WriteIntoDelta` is requested to [write](../commands/WriteIntoDelta.md#write)
+
+## isDynamicPartitionOverwriteMode { #isDynamicPartitionOverwriteMode }
+
+```scala
+isDynamicPartitionOverwriteMode: Boolean
+```
+
+`isDynamicPartitionOverwriteMode`...FIXME
+
+---
+
+`isDynamicPartitionOverwriteMode` is used when:
+
+* `WriteIntoDelta` is requested to [write data out](../commands/WriteIntoDelta.md#write)
diff --git a/docs/spark-connector/options.md b/docs/spark-connector/options.md
@@ -4,9 +4,9 @@ title: Options
 
 # Options
 
-## <span id="checkpointLocation"> checkpointLocation
+## checkpointLocation { #checkpointLocation }
 
-Checkpoint directory for storing checkpoint data of streaming queries ([Spark Structured Streaming]({{ book.structured_streaming }}/configuration-properties/#spark.sql.streaming.checkpointLocation)).
+Checkpoint directory for streaming queries ([Spark Structured Streaming]({{ book.structured_streaming }}/configuration-properties/#spark.sql.streaming.checkpointLocation)).
 
 ## <span id="DATA_CHANGE_OPTION"><span id="dataChange"> dataChange
 
@@ -73,11 +73,13 @@ Enables schema migration (and allows automatic schema merging during a write ope
 
 Equivalent SQL Session configuration: [spark.databricks.delta.schema.autoMerge.enabled](../configuration-properties/DeltaSQLConf.md#DELTA_SCHEMA_AUTO_MIGRATE)
 
-## <span id="OPTIMIZE_WRITE_OPTION"><span id="optimizeWrite"> optimizeWrite
+## <span id="OPTIMIZE_WRITE_OPTION"> optimizeWrite { #optimizeWrite }
 
-Enables...FIXME
+`optimizeWrite` is a writer option.
 
-## <span id="OVERWRITE_SCHEMA_OPTION"><span id="overwriteSchema"> overwriteSchema
+!!! danger "Not used"
+
+## <span id="OVERWRITE_SCHEMA_OPTION"> overwriteSchema { #overwriteSchema }
 
 Enables overwriting schema or change partitioning of a delta table during an overwrite write operation
 
@@ -86,7 +88,17 @@ Use [DeltaOptions.canOverwriteSchema](DeltaWriteOptionsImpl.md#canOverwriteSchem
 !!! note
     The schema cannot be overwritten when using [replaceWhere](#REPLACE_WHERE_OPTION) option.
 
-## <span id="path"> path
+## <span id="PARTITION_OVERWRITE_MODE_OPTION"> partitionOverwriteMode { #partitionOverwriteMode }
+
+Mutually exclusive with [replaceWhere](#replaceWhere)
+
+Used when:
+
+* `DeltaDynamicPartitionOverwriteCommand` is executed
+* `DeltaWriteOptionsImpl` is requested to [isDynamicPartitionOverwriteMode](DeltaWriteOptionsImpl.md#isDynamicPartitionOverwriteMode) and for [partitionOverwriteModeInOptions](DeltaWriteOptionsImpl.md#partitionOverwriteModeInOptions)
+* `WriteIntoDeltaBuilder` is requested to [overwriteDynamicPartitions](../WriteIntoDeltaBuilder.md#overwriteDynamicPartitions)
+
+## path { #path }
 
 **(required)** Directory on a Hadoop DFS-compliant file system with an optional [time travel](../time-travel/index.md) identifier
 
@@ -121,10 +133,12 @@ Use [DeltaOptions.readChangeFeed](DeltaReadOptions.md#readChangeFeed) for the va
 
 ## <span id="REPLACE_WHERE_OPTION"> replaceWhere { #replaceWhere }
 
-Partition predicates (unless [replaceWhere.dataColumns.enabled](../configuration-properties/DeltaSQLConf.md#replaceWhere.dataColumns.enabled) is enabled to allow for arbitrary non-partition data predicates)
+Partition predicates to overwrite only the data that matches predicates over partition columns (unless [replaceWhere.dataColumns.enabled](../configuration-properties/DeltaSQLConf.md#replaceWhere.dataColumns.enabled) is enabled)
 
 Available as [DeltaWriteOptions.replaceWhere](DeltaWriteOptions.md#replaceWhere)
 
+Mutually exclusive with [partitionOverwriteMode](#partitionOverwriteMode)
+
 !!! example "Demo"
     Learn more in [Demo: replaceWhere](../demo/replaceWhere.md).