You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- ddd6a8ee676b3dab729ea720182a022e0977567e [ES-1190931][DBRRM-1133] Add event to detect Delta tables... by Christos Stavrakakis <[email protected]>
- 8ae8d533e7b02a2756420c88bbf930ece746ec80 [LC-4832][Delta][CC] Uses metastoreID from createStagingT... by Lukas Rupprecht <[email protected]>
- f4fd1d10ab960eb88850531cc57ed8f5d1638b2f Revert "[ES-1167842] Revert logging of clusteringColumns ... by databricks-jenkins <[email protected]>
- 2a0c28897f11375da25facfa46a01e5afc2c0d7b [ES-1167842] Revert logging of clusteringColumns in 15.4 by Eric Liang <[email protected]>
- e7519ecfcda0aa8ebb2c2a29c12b128415f67274 [SC-170456] Record Unsupported Data Filtering Predicates by Supun Nakandala <[email protected]>
- 3a44c7c1655a4f6b918b0c2fdb5842a773b7a36b [LC-4852] Add Test Suite for Protocol Transition Improvem... by Andreas Chatzistergiou <[email protected]>
- a42d1c3bffc195ac82ac8c4a04dd67918cdd3ce6 [LC-4787][MAC] Followup to registering MAC hook for CRTAS... by Stella Wan <[email protected]>
- 923046452f1300fcd474521197386d8840253aa8 [Delta-OSS] Remove dropped columns when running REORG PURGE by Zihao Xu <[email protected]>
- e995fd6acc8557a9c3ff15315ba07982f58172f0 [ES-1192280][LC-4831][SC-171393] Fix DFP Null Safe Equali... by Thang Long Vu <[email protected]>
- 45e7f85c1e4f9d059325f434123145d1edd78194 [LC-4787][MAC] Register MAC hook for CRTAS and DataFrame ... by Stella Wan <[email protected]>
- 4d0b912a5027b934614f1ae81f950f3f605134c9 [ES-1178751] Increase replication level in MERGE source m... by Johan Lasperas <[email protected]>
- de0a9d0677593cda59fe569b3ebcea417aded562 [ES-1183111] Enforce correct parallelism to copy files wi... by Johan Lasperas <[email protected]>
- b7ea96b356a6bac91562ce7fbf2c82a90526b364 [DELTA-OSS-AUTO][SC-171316][SPARK] Add Integration Test i... by Mingkang Li <[email protected]>
- b9239b289727034c072a75833b138ca0fafe6893 [ES-1113859] Batch resolve DeltaMergeAction in ResolveMer... by Carmen Kwan <[email protected]>
- f1ca70446f0bc2d8a0cad1a608ef17a7b784eecc [AUTO][SC-170767][SPARK-48851][SQL] Change the value of `... by panbingkun <[email protected]>
- cadf44a02bf1c286d9aa9c704a1a22729aa0108d [ES-1053014][ES-1125181][BEHAVE-78] Fix VARCHAR/CHAR to s... by Dhruv Arya <[email protected]>
- 4396b20e0c4b2bc42c97d909a516f092d945653f [DELTA-OSS-AUTO][SC-171306][Spark] Only attempt parsing p... by Amogh Jahagirdar <[email protected]>
- a85044eb6fef3090abd90d143a3d97023f0e82a1 [ES-1183822][TEST-ONLY] Use DeltaSQLTestUtils in more Del... by Johan Lasperas <[email protected]>
- b0801357aef6e7d5c2bd8a0d336313142dbb6bff [DELTA-OSS-EXTERNAL][BEHAVE-87][Spark] Improve Delta Prot... by Andreas Chatzistergiou <[email protected]>
- ae9fb7f8f44b471f7e81e0b195e15e0d655d8fec [SC-171037] Annotate Delta Log Keys by Jiaheng Tang <[email protected]>
- d512153866293aa3388ea631426f6ce85fcfc0f8 [LC-3943] Put automatic protocol upgrade for collations f... by Stefan Kandic <[email protected]>
- fe6efbb1cfd45d778718b877a824f730a2dd8ef2 Check UniForm Hudi table must be UC by Annie Wang <[email protected]>
- 8a391f7155895b979866455055741a42808020a8 [LC-4634] Enable IcebergCompatV2 Delta Uniform By Alter T... by Zihao Xu <[email protected]>
- 5028821d836b75c283b730bbf807fdfa5d28cc21 [CC][LC-4664][Test-Only] Make TrackingCommitCoordinatorCl... by Dhruv Arya <[email protected]>
- d3fb4e4ae691f1ccf99274645cb3fea31d9a10ec [SC-166402] managed data skipping columns by Paul Loh <[email protected]>
- 8c98e895d4860387d216278168acd3977437d715 [LC-4668] Expose uncertainty in isDeltaTable resolution v... by Lars Kroll <[email protected]>
- 3376b76a114fac408e6363ac864921d7178d9804 [ES-1010221] Part 2 of Scala 2.13 migration for Delta tests by Jintian Liang <[email protected]>
- 7880534e5f814f768648699823731edb3eb86c77 [SC-163065] Make two copies of runQueriesWithOrderingFrom... by Thang Long Vu <[email protected]>
- 8a59d4b4946aef5cfbc75594bd05ce979c249cd4 [SQL][COLLATION] Add COLLATE tag to structured QPL by Uros Bojanic <[email protected]>
- 91d9489acc1b15d4bea86baf3b75ba6e5e0b9abf [LC-4638] Implement more efficient SerDe for the row_inde... by Christos Stavrakakis <[email protected]>
- 11d61890a1e26b8380e7cae9b1d750acf5a14f96 [ES-1177667][TEST-ONLY] Remove timeout source from withTe... by Johan Lasperas <[email protected]>
- bc90067254419651895419cd0228d3cd5cbe4d8d [ES-1109125] Report accurate type when checking Iceberg c... by Hao Jiang <[email protected]>
- 2bf266859083ecfc1d46240638b7c421c24285ad [SC-160780][FOLLOW-UP] Add SQLConf flag for #96237 change by Vitalii Li <[email protected]>
- 23f0eb9982898acf167f3d3eb4d1905133dff27e [LC-4740][CoordinatedCommits] Extend NewTransactionSuite ... by Jun <[email protected]>
- 66a20cfed0eaf369a8a2b35e709a23311e0a19f3 [LC-3943] Forbid automatic protocol upgrade when using co... by Stefan Kandic <[email protected]>
- 876af9aa7e6de42c501b89eae58111e2df7ba8cc [LC-4491] Shade Hudi into DBR by Annie Wang <[email protected]>
- 673ca7fe43e3110c67143642f3a77f4f5106bc23 [LC-4754][Delta][MC] Fix error message formatting in Delt... by Sumeet Varma <[email protected]>
- 321e72867fdc98d501963bccaf17f8577bc4c445 [LC-4192][Delta][MC] Use Coordinated Commits Properties f... by Sumeet Varma <[email protected]>
- 84fe62011e78b5528adb70cc06a1be906ce97b02 [LC-4263][Delta] Add usage logging to UCCommitCooridnator... by Yumingxuan Guo <[email protected]>
- 4cde1a20dddc7408beb251cfbf62baed7bc29940 [LC-4392][Auto-Clustering-Key-Selection] Add additional t... by parimarjannegi-db <[email protected]>
- 7b5a138dd7388ac59a08528c0e27c622cf0bb2f3 [LC-4549] Delta structured logging migration for remainin... by Jiaheng Tang <[email protected]>
- c0e90e6bd23cc181e900ee4633b2eb213a913f9d [LC-4131] Validate that CDF works with collations by Stefan Kandic <[email protected]>
- c472de831911ccd1cc0fd7b23e1fd417cecde195 [Delta][Copybara] Introduce `// EDGE-NO-INHERITANCE` tag by Paddy Xu <[email protected]>
- 78f75a2bb3a05ad37b59e42eb35f4b61cb1c7acf [DELTA-OSS-AUTO][SC-170469][Hudi] Flesh out tests and upd... by Annie Wang <[email protected]>
- 49d07ae43b3dbc09c2679e35a4c751b99f53c60a [ES-1151956] Fixes for "show tables extended" by rayman7718 <[email protected]>
- c35f0fdd59377faae33aa06f477fe5b066babe82 [ES-1159798][LC-4729][BEHAVE-74] Fix CDC Commit Timestamp... by Thang Long Vu <[email protected]>
- cfd343651f3c00a6f551f3e2ef5180b80f2dd47b [LC-4661] Deprecate tableVersion field in type widening m... by Johan Lasperas <[email protected]>
- 7014ad1ec51af3f55c2f9713b032c6c9d32ff755 [AUTO][SC-170122][SPARK-48697][LC-4703][SQL] Add collatio... by Stefan Kandic <[email protected]>
- 35628a58efa0c704bf2e2ddfcba833ff9b40f07a [Delta][CC] Add integration test for DROP/UNDROP. by Yumingxuan Guo <[email protected]>
- 3351fa6531e3e370ebe41f931c3946fa00206d5f [LC-4693][Uniform] Make refresh on UFI no-op when there's... by ChengJi-db <[email protected]>
- 10d743671c381ea09131f3519be7fc90c460db75 [LC-4143][Delta][CC] Adds asynchronous backfilling in UC ... by Lukas Rupprecht <[email protected]>
- d55d080248832d881d33e52d9b1d9efd66a197bf [LC-4698] add logging to UniForm Iceberg conversion num o... by Fred Storage Liu <[email protected]>
- f992112f04b14423f323d4f171040ec8cca1b3ad [LC-4499] All Delta Commands should have all operation me... by sabir-akhadov <[email protected]>
- 7d0c410dba226ae191f51280c382b272fba459b1 [LC-3778] Log duration of DeltaSource operations by Tom van Bussel <[email protected]>
- e5e29b59cab4218ed77903079eb2b08216e6f982 Import changes from OSS PR delta-io#3326 by Annie Wang <[email protected]>
- 4c08927c6eee0da4a9a00795b36a4b734f2c8a04 Import changes from OSS PR delta-io#3323 by Annie Wang <[email protected]>
- 1fd8ee89f3b7398db6f13486bd42ca2535023976 Import changes from OSS PR delta-io#3320 by Annie Wang <[email protected]>
- db210bb4f2bd0d682863c99ca4c385ed0d0d0118 [LC-4692] Regular delta min/max stats should always use U... by Stefan Kandic <[email protected]>
- e873cab62acf71139d89f4019f59fb5aeec81f14 Import changes from OSS PR delta-io#3309 by Annie Wang <[email protected]>
- 6d1aca32ebc1d9009e77ca26f4fd0c73ba8e7f6e Import changes from OSS PR delta-io#3310 by Annie Wang <[email protected]>
- 0070442d167c53884e083127eeaa7e69c71f28d9 Import changes from OSS PR delta-io#3311 by Annie Wang <[email protected]>
- 8b8df4f43b5a85d1626ea4a2ce612b34ea67544d [ES-1089258][ES-1043275][Delta] Creates unresolved tables... by Lukas Rupprecht <[email protected]>
- 602f7e429fd3c73d43c24ac7f231a9a10bfe8830 [LC-4043] Allow auto REFRESH on UniForm tables by Hao Jiang <[email protected]>
- 26755b6dba8caeca1b7c8797eb634abcb5532e35 [ES-1154731] Disable managed Hive Metastore related Conve... by Ming DAI <[email protected]>
- 568c954c44eafe752d6c614ca36dc3f2a0a651d4 [ES-1150026][Delta] Handle concurrent CREATE TABLE IF NOT... by Sumeet Varma <[email protected]>
- 195a5fbcd168a0d1e7e992dfdb226b3a9e21a6a3 [LC-4657][Delta][MC] Throw exception when additional list... by Sumeet Varma <[email protected]>
- 438d79963e76478a641e256ee53edca557024c99 [LC-4524][Delta][MC] Fetch unbackfilled deltas to detect ... by Sumeet Varma <[email protected]>
- 3c6800ac28f998e3183a54d4ff6cf058607cad16 [LC-4996][BEHAVE-60] Introduce stable type widening table... by Johan Lasperas <[email protected]>
- c0d24087192bcbbd80f42fa83e2229482c877d66 [ES-1154366][ES-1154367] Fix ConvertToDeltaSuite in serve... by Ming DAI <[email protected]>
- 6b7e4d0f9004d044fd88248c4b024d61e1d4b91d [Liquid] Improve error message when clustering column not... by Chirag Singh <[email protected]>
- 75e21adf355c43bbafeb6dffefe7627614ca6338 [LC-4242] History truncation/validation support for write... by Andreas Chatzistergiou <[email protected]>
- 71aec8cc7f136db13de264f661446526a9e7a2c2 [ES-1162704] Disable some Python Delta tests in serverless by Christos Stavrakakis <[email protected]>
- 96b6ce441e287e9b008261844017eb9d09ffe692 update delta-sharing client to version 1.1.0 by Jade Wang <[email protected]>
- 510e7d69ce180c869031affff075bc163077fa82 [SC-160780] Support Delta API for DML commands when table... by Vitalii Li <[email protected]>
- 562322d9ab5ecf7fb237fa0e950470d83a252ed1 [LC-4549] Delta structured logging migration by Jiaheng Tang <[email protected]>
- 850e368261d2cb89cc732139c8ee1574978366d5 [LC-4526] Support snapshot expiration for uniform iceberg... by ChengJi-db <[email protected]>
- 6e11dd7f11e79a429068a6daa59d4e201fa894c8 [ES-1151640] Break down ImplicitDMLCastingSuite by Lars Kroll <[email protected]>
- 837d86463cef6bbb1e7571f06431cedd67b76c7e [ES-1154733][ES-1154732][ES-1154381][ES-1154382][TEST-ONL... by Paddy Xu <[email protected]>
- 51bfb0c60bb2b2d64f01b7c5f96187e65b0ad4b1 [LC-4534] Add ThreadLocal trait for underlying execution ... by leonwind-db <[email protected]>
- ade2f1670859abf5cad4f4a3f0240566ec8c3a0f [LS-989][Delta] Add checkpointType, numColumns and numFla... by Sumeet Varma <[email protected]>
- 62c938e3eb626e04000e0acd4d37953a24d3c52d [LS-985][Delta] Add detailed mismatch reasons for Increme... by Sumeet Varma <[email protected]>
- bfda87427e6c9c019d83ee108e8e70a5e5950d8e [SPARK-48576][SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE by Uros Bojanic <[email protected]>
- 2ff6142a85ef692a055071fa722ac10caeaf685f [LC-4301] Fix the predicates used by metadata queries by Wei Luo <[email protected]>
- 6340541a0bbc4c9d71e04aff99a3cb21720dd975 [DELTA-OSS-AUTO][SC-168789][Spark] Optimize batching / in... by Adam Binford <[email protected]>
- 1c691ed0e5df46402a4ebe44b16b89b748f2915e [SC-162602] Add per-kdtree scan stats (only scanned files) by Eric Liang <[email protected]>
- 895a09ff141e810f87d2c5b353d3963d8ac00586 [SC-167647] map lookup for createPhysicalSchema by andrewxue-db <[email protected]>
- d89308d177f5e269e7cb6bdfbe6c0978bd9d1cd2 [Delta OSS] Fix compile error on Spark master due to Pars... by Jiaheng Tang <[email protected]>
- c7ba28b7e35319db1a41d5252506d82b39933ec2 [STATS-356] Improve recomputeDeltaFileStats logic with ME... by Pat Sukprasert <[email protected]>
- 750eeac292c99c0003b6011038b99e2ee92083db [LC-4499] All Delta Commands should have all operation me... by sabir-akhadov <[email protected]>
- ebe3e3e296c593fa6e5020e8127b410cd9d4c4c6 [LC-3942] Add historical schema read compatibility checks... by Nikola Mandic <[email protected]>
- 20dbe80816413dcfa565777fdd9878f4f00c1af2 [SC-167257][Auto-Cluster-Key-Selection] Show operational ... by Supun Nakandala <[email protected]>
- 2460f7604cba40783e44a43cbeb4bc39956427d8 [ES-1155687][LC-812] Make DPO track read files by Christos Stavrakakis <[email protected]>
- 59e3a4f57aaa7a7298e978ace615ff04b4507418 [LC-4484] Match RangeBloomFilterMightContain expressions ... by Tom van Bussel <[email protected]>
- 01c09b8556e57b2fa94c3e6f850b2ae426364976 [ES-1077008][LC-4411][BEHAVE-32] Add a Delta config to en... by Paddy Xu <[email protected]>
- 7bd2395d454a1f2383d5b58f16e7de643cacae8b [ES-1154374][TEST-ONLY] Fix INSERT with delta view tests ... by Johan Lasperas <[email protected]>
- d8f90b3560e6fe8d95bf757538660a995cf185a0 [LC-4323] Delta metadata queries should be QPL tagged wit... by sabir-akhadov <[email protected]>
- e673c6972eb230234b8206309dcffc135af6387e [LC-4132][SQL] Disable changing collation of clustering c... by Nikola Mandic <[email protected]>
- 9c34481f90d4e526a0f5c3671e5e59aae969c44c [LC-3259][Delta][MC][Pt. 3] Creates separate CommitStore ... by Lukas Rupprecht <[email protected]>
- 0249b90ffe247bc689586891e90f81bd0568ad6d [LC-4564][LC-4565][Liquid OSS] Support show tblproperties... by Jiaheng Tang <[email protected]>
- 24a50e9f98a540ba72c9a7a585ddad0a8196ee92 [LC-4548] Support Spark Structured Logging in Delta by Jiaheng Tang <[email protected]>
(And 402 more changes)
GitOrigin-RevId: ddd6a8ee676b3dab729ea720182a022e0977567e
Copy file name to clipboardexpand all lines: PROTOCOL.md
+2-4
Original file line number
Diff line number
Diff line change
@@ -477,9 +477,7 @@ The following is an example `remove` action.
477
477
```
478
478
479
479
### Add CDC File
480
-
The `cdc` action is used to add a [file](#change-data-files) containing only the data that was changed as part of the transaction. The `cdc` action is allowed to add a [Data File](#data-files) that is also added by an `add` action, when it does not contain any copied rows and the `_change_type` column is filled for all rows.
481
-
482
-
When change data readers encounter a `cdc` action in a particular Delta table version, they must read the changes made in that version exclusively using the `cdc` files. If a version has no `cdc` action, then the data in `add` and `remove` actions are read as inserted and deleted rows, respectively.
480
+
The `cdc` action is used to add a [file](#change-data-files) containing only the data that was changed as part of the transaction. When change data readers encounter a `cdc` action in a particular Delta table version, they must read the changes made in that version exclusively using the `cdc` files. If a version has no `cdc` action, then the data in `add` and `remove` actions are read as inserted and deleted rows, respectively.
483
481
484
482
The schema of the `cdc` action is as follows:
485
483
@@ -525,7 +523,7 @@ Specifically, to read the row-level changes made in a version, the following str
525
523
526
524
##### Note for non-change data readers
527
525
528
-
In a table with Change Data Feed enabled, the data Parquet files referenced by `add` and `remove` actions are allowed to contain an extra column `_change_type`. This column is not present in the table's schema. When accessing these files, readers should disregard this column and only process columns defined within the table's schema.
526
+
In a table with Change Data Feed enabled, the data Parquet files referenced by `add` and `remove` actions are allowed to contain an extra column `_change_type`. This column is not present in the table's schema and will consistently have a `null` value. When accessing these files, readers should disregard this column and only process columns defined within the table's schema.
529
527
530
528
### Transaction Identifiers
531
529
Incremental processing systems (e.g., streaming systems) that track progress using their own application-specific versions need to record what progress has been made, in order to avoid duplicating data in the face of failures and retries during a write.
0 commit comments