Skip to content

Commit 2d9968f

Browse files
prakharjain09allisonport-db
authored andcommitted
Allow every Delta txn other than those containing Metadata updates against blind appends
Now any transaction which has non-file actions goes through conflict detection against blind append unlike before. This may lead to failure of such transactions. This was intentionally done to stop METADATA / other non-file-action transactions to follow conflict detection. But as a side effect, Streaming Updates (which have SetTransaction action) also started following this flow and started failing. Instead of allowList, we want to move to blockList approach. The blockList approach is less safe from correctness perspective but more safe from rollout perspective. Added UT GitOrigin-RevId: d09aea4bacbe6de3d000cde58c84f1d9ee1bb638
1 parent 59d5ea2 commit 2d9968f

File tree

1 file changed

+6
-3
lines changed

1 file changed

+6
-3
lines changed

core/src/main/scala/org/apache/spark/sql/delta/ConflictChecker.scala

+6-3
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,11 @@ private[delta] class CurrentTransactionInfo(
5353
/** Final actions to commit - including the [[CommitInfo]] */
5454
lazy val finalActionsToCommit: Seq[Action] = actions ++ commitInfo
5555

56-
/** Whether this transaction wants to commit actions other than [[FileAction]] */
57-
val hasOnlyFileActions = actions.forall(_.isInstanceOf[FileAction])
56+
/** Whether this transaction wants to make any [[Metadata]] update */
57+
lazy val metadataChanged: Boolean = actions.exists {
58+
case _: Metadata => true
59+
case _ => false
60+
}
5861
}
5962

6063
/**
@@ -166,7 +169,7 @@ private[delta] class ConflictChecker(
166169
recordTime("checked-appends") {
167170
// Fail if new files have been added that the txn should have read.
168171
val addedFilesToCheckForConflicts = isolationLevel match {
169-
case WriteSerializable if currentTransactionInfo.hasOnlyFileActions =>
172+
case WriteSerializable if !currentTransactionInfo.metadataChanged =>
170173
winningCommitSummary.changedDataAddedFiles // don't conflict with blind appends
171174
case Serializable | WriteSerializable =>
172175
winningCommitSummary.changedDataAddedFiles ++ winningCommitSummary.blindAppendAddedFiles

0 commit comments

Comments
 (0)