You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- d17584edb7d55dba9d45245d26912d531e8fc172 WIP by zhipeng.mao <[email protected]>
- 11d61890a1e26b8380e7cae9b1d750acf5a14f96 [ES-1177667][TEST-ONLY] Remove timeout source from withTe... by Johan Lasperas <[email protected]>
- bc90067254419651895419cd0228d3cd5cbe4d8d [ES-1109125] Report accurate type when checking Iceberg c... by Hao Jiang <[email protected]>
- 2bf266859083ecfc1d46240638b7c421c24285ad [SC-160780][FOLLOW-UP] Add SQLConf flag for #96237 change by Vitalii Li <[email protected]>
- 23f0eb9982898acf167f3d3eb4d1905133dff27e [LC-4740][CoordinatedCommits] Extend NewTransactionSuite ... by Jun <[email protected]>
- 66a20cfed0eaf369a8a2b35e709a23311e0a19f3 [LC-3943] Forbid automatic protocol upgrade when using co... by Stefan Kandic <[email protected]>
- 876af9aa7e6de42c501b89eae58111e2df7ba8cc [LC-4491] Shade Hudi into DBR by Annie Wang <[email protected]>
- 673ca7fe43e3110c67143642f3a77f4f5106bc23 [LC-4754][Delta][MC] Fix error message formatting in Delt... by Sumeet Varma <[email protected]>
- 321e72867fdc98d501963bccaf17f8577bc4c445 [LC-4192][Delta][MC] Use Coordinated Commits Properties f... by Sumeet Varma <[email protected]>
- 84fe62011e78b5528adb70cc06a1be906ce97b02 [LC-4263][Delta] Add usage logging to UCCommitCooridnator... by Yumingxuan Guo <[email protected]>
- 4cde1a20dddc7408beb251cfbf62baed7bc29940 [LC-4392][Auto-Clustering-Key-Selection] Add additional t... by parimarjannegi-db <[email protected]>
- 7b5a138dd7388ac59a08528c0e27c622cf0bb2f3 [LC-4549] Delta structured logging migration for remainin... by Jiaheng Tang <[email protected]>
- c0e90e6bd23cc181e900ee4633b2eb213a913f9d [LC-4131] Validate that CDF works with collations by Stefan Kandic <[email protected]>
- c472de831911ccd1cc0fd7b23e1fd417cecde195 [Delta][Copybara] Introduce `// EDGE-NO-INHERITANCE` tag by Paddy Xu <[email protected]>
- 78f75a2bb3a05ad37b59e42eb35f4b61cb1c7acf [DELTA-OSS-AUTO][SC-170469][Hudi] Flesh out tests and upd... by Annie Wang <[email protected]>
- 49d07ae43b3dbc09c2679e35a4c751b99f53c60a [ES-1151956] Fixes for "show tables extended" by rayman7718 <[email protected]>
- c35f0fdd59377faae33aa06f477fe5b066babe82 [ES-1159798][LC-4729][BEHAVE-74] Fix CDC Commit Timestamp... by Thang Long Vu <[email protected]>
- cfd343651f3c00a6f551f3e2ef5180b80f2dd47b [LC-4661] Deprecate tableVersion field in type widening m... by Johan Lasperas <[email protected]>
- 7014ad1ec51af3f55c2f9713b032c6c9d32ff755 [AUTO][SC-170122][SPARK-48697][LC-4703][SQL] Add collatio... by Stefan Kandic <[email protected]>
- 35628a58efa0c704bf2e2ddfcba833ff9b40f07a [Delta][CC] Add integration test for DROP/UNDROP. by Yumingxuan Guo <[email protected]>
- 3351fa6531e3e370ebe41f931c3946fa00206d5f [LC-4693][Uniform] Make refresh on UFI no-op when there's... by ChengJi-db <[email protected]>
- 10d743671c381ea09131f3519be7fc90c460db75 [LC-4143][Delta][CC] Adds asynchronous backfilling in UC ... by Lukas Rupprecht <[email protected]>
- d55d080248832d881d33e52d9b1d9efd66a197bf [LC-4698] add logging to UniForm Iceberg conversion num o... by Fred Storage Liu <[email protected]>
- f992112f04b14423f323d4f171040ec8cca1b3ad [LC-4499] All Delta Commands should have all operation me... by sabir-akhadov <[email protected]>
- 7d0c410dba226ae191f51280c382b272fba459b1 [LC-3778] Log duration of DeltaSource operations by Tom van Bussel <[email protected]>
- e5e29b59cab4218ed77903079eb2b08216e6f982 Import changes from OSS PR delta-io#3326 by Annie Wang <[email protected]>
- 4c08927c6eee0da4a9a00795b36a4b734f2c8a04 Import changes from OSS PR delta-io#3323 by Annie Wang <[email protected]>
- 1fd8ee89f3b7398db6f13486bd42ca2535023976 Import changes from OSS PR delta-io#3320 by Annie Wang <[email protected]>
- db210bb4f2bd0d682863c99ca4c385ed0d0d0118 [LC-4692] Regular delta min/max stats should always use U... by Stefan Kandic <[email protected]>
- e873cab62acf71139d89f4019f59fb5aeec81f14 Import changes from OSS PR delta-io#3309 by Annie Wang <[email protected]>
- 6d1aca32ebc1d9009e77ca26f4fd0c73ba8e7f6e Import changes from OSS PR delta-io#3310 by Annie Wang <[email protected]>
- 0070442d167c53884e083127eeaa7e69c71f28d9 Import changes from OSS PR delta-io#3311 by Annie Wang <[email protected]>
- 8b8df4f43b5a85d1626ea4a2ce612b34ea67544d [ES-1089258][ES-1043275][Delta] Creates unresolved tables... by Lukas Rupprecht <[email protected]>
- 602f7e429fd3c73d43c24ac7f231a9a10bfe8830 [LC-4043] Allow auto REFRESH on UniForm tables by Hao Jiang <[email protected]>
- 26755b6dba8caeca1b7c8797eb634abcb5532e35 [ES-1154731] Disable managed Hive Metastore related Conve... by Ming DAI <[email protected]>
- 568c954c44eafe752d6c614ca36dc3f2a0a651d4 [ES-1150026][Delta] Handle concurrent CREATE TABLE IF NOT... by Sumeet Varma <[email protected]>
- 195a5fbcd168a0d1e7e992dfdb226b3a9e21a6a3 [LC-4657][Delta][MC] Throw exception when additional list... by Sumeet Varma <[email protected]>
- 438d79963e76478a641e256ee53edca557024c99 [LC-4524][Delta][MC] Fetch unbackfilled deltas to detect ... by Sumeet Varma <[email protected]>
- 3c6800ac28f998e3183a54d4ff6cf058607cad16 [LC-4996][BEHAVE-60] Introduce stable type widening table... by Johan Lasperas <[email protected]>
- c0d24087192bcbbd80f42fa83e2229482c877d66 [ES-1154366][ES-1154367] Fix ConvertToDeltaSuite in serve... by Ming DAI <[email protected]>
- 6b7e4d0f9004d044fd88248c4b024d61e1d4b91d [Liquid] Improve error message when clustering column not... by Chirag Singh <[email protected]>
- 75e21adf355c43bbafeb6dffefe7627614ca6338 [LC-4242] History truncation/validation support for write... by Andreas Chatzistergiou <[email protected]>
- 71aec8cc7f136db13de264f661446526a9e7a2c2 [ES-1162704] Disable some Python Delta tests in serverless by Christos Stavrakakis <[email protected]>
- 96b6ce441e287e9b008261844017eb9d09ffe692 update delta-sharing client to version 1.1.0 by Jade Wang <[email protected]>
- 510e7d69ce180c869031affff075bc163077fa82 [SC-160780] Support Delta API for DML commands when table... by Vitalii Li <[email protected]>
- 562322d9ab5ecf7fb237fa0e950470d83a252ed1 [LC-4549] Delta structured logging migration by Jiaheng Tang <[email protected]>
- 850e368261d2cb89cc732139c8ee1574978366d5 [LC-4526] Support snapshot expiration for uniform iceberg... by ChengJi-db <[email protected]>
- 6e11dd7f11e79a429068a6daa59d4e201fa894c8 [ES-1151640] Break down ImplicitDMLCastingSuite by Lars Kroll <[email protected]>
- 837d86463cef6bbb1e7571f06431cedd67b76c7e [ES-1154733][ES-1154732][ES-1154381][ES-1154382][TEST-ONL... by Paddy Xu <[email protected]>
- 51bfb0c60bb2b2d64f01b7c5f96187e65b0ad4b1 [LC-4534] Add ThreadLocal trait for underlying execution ... by leonwind-db <[email protected]>
- ade2f1670859abf5cad4f4a3f0240566ec8c3a0f [LS-989][Delta] Add checkpointType, numColumns and numFla... by Sumeet Varma <[email protected]>
- 62c938e3eb626e04000e0acd4d37953a24d3c52d [LS-985][Delta] Add detailed mismatch reasons for Increme... by Sumeet Varma <[email protected]>
- bfda87427e6c9c019d83ee108e8e70a5e5950d8e [SPARK-48576][SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE by Uros Bojanic <[email protected]>
- 2ff6142a85ef692a055071fa722ac10caeaf685f [LC-4301] Fix the predicates used by metadata queries by Wei Luo <[email protected]>
- 6340541a0bbc4c9d71e04aff99a3cb21720dd975 [DELTA-OSS-AUTO][SC-168789][Spark] Optimize batching / in... by Adam Binford <[email protected]>
- 1c691ed0e5df46402a4ebe44b16b89b748f2915e [SC-162602] Add per-kdtree scan stats (only scanned files) by Eric Liang <[email protected]>
- 895a09ff141e810f87d2c5b353d3963d8ac00586 [SC-167647] map lookup for createPhysicalSchema by andrewxue-db <[email protected]>
- d89308d177f5e269e7cb6bdfbe6c0978bd9d1cd2 [Delta OSS] Fix compile error on Spark master due to Pars... by Jiaheng Tang <[email protected]>
- c7ba28b7e35319db1a41d5252506d82b39933ec2 [STATS-356] Improve recomputeDeltaFileStats logic with ME... by Pat Sukprasert <[email protected]>
- 750eeac292c99c0003b6011038b99e2ee92083db [LC-4499] All Delta Commands should have all operation me... by sabir-akhadov <[email protected]>
- ebe3e3e296c593fa6e5020e8127b410cd9d4c4c6 [LC-3942] Add historical schema read compatibility checks... by Nikola Mandic <[email protected]>
- 20dbe80816413dcfa565777fdd9878f4f00c1af2 [SC-167257][Auto-Cluster-Key-Selection] Show operational ... by Supun Nakandala <[email protected]>
- 2460f7604cba40783e44a43cbeb4bc39956427d8 [ES-1155687][LC-812] Make DPO track read files by Christos Stavrakakis <[email protected]>
- 59e3a4f57aaa7a7298e978ace615ff04b4507418 [LC-4484] Match RangeBloomFilterMightContain expressions ... by Tom van Bussel <[email protected]>
- 01c09b8556e57b2fa94c3e6f850b2ae426364976 [ES-1077008][LC-4411][BEHAVE-32] Add a Delta config to en... by Paddy Xu <[email protected]>
- 7bd2395d454a1f2383d5b58f16e7de643cacae8b [ES-1154374][TEST-ONLY] Fix INSERT with delta view tests ... by Johan Lasperas <[email protected]>
- d8f90b3560e6fe8d95bf757538660a995cf185a0 [LC-4323] Delta metadata queries should be QPL tagged wit... by sabir-akhadov <[email protected]>
- e673c6972eb230234b8206309dcffc135af6387e [LC-4132][SQL] Disable changing collation of clustering c... by Nikola Mandic <[email protected]>
- a34f0d7e23c1dcf55310be6bf4148bd4ad3839e1 [ES-1154398][TEST-ONLY][WARMFIX] IdentityColumnAdmissionS... by Carmen Kwan <[email protected]>
- 9c34481f90d4e526a0f5c3671e5e59aae969c44c [LC-3259][Delta][MC][Pt. 3] Creates separate CommitStore ... by Lukas Rupprecht <[email protected]>
- 0249b90ffe247bc689586891e90f81bd0568ad6d [LC-4564][LC-4565][Liquid OSS] Support show tblproperties... by Jiaheng Tang <[email protected]>
- 24a50e9f98a540ba72c9a7a585ddad0a8196ee92 [LC-4548] Support Spark Structured Logging in Delta by Jiaheng Tang <[email protected]>
- f4b3d9042b1ffebc8ed78a3fe2634e6672213cc0 [ES-1154365][ES-1154357] Disable test cases not supported... by Chirag Singh <[email protected]>
- a39c357817393943cc2997bbc9f020b8e02c14a8 [Liquid][LC-3844] Do not attempt lazy clustering on files... by Chirag Singh <[email protected]>
- b8ff879f8471592e7d08312be088152c575ecb3f [Liquid][LC-4335][LC-4314] Execute eager clustering batch... by Chirag Singh <[email protected]>
- 1bce3b75784df6960e1c8261dea5c97f86f0e3bd [Delta] Rename managed commit to coordinated commits by Dhruv Arya <[email protected]>
- f727c3e6717ed8f235560055ade0848a2333c6dd [DELTA-OSS-AUTO][SC-167757][SPARK][MINOR] Code cleanup (r... by Jacek Laskowski <[email protected]>
- edf995cf281c10917d747620a4edc8ab37bf032d [Liquid][LC-2605] Change verifyClusteringColumns to not p... by Chirag Singh <[email protected]>
- 04129352860bfd2973d2717a69b793a4ae880217 [BEHAVE-43][SC-168642][Spark] Support dropping the CHECK ... by Tom van Bussel <[email protected]>
- 863b8c7f7af9d720949a9b098f89f8bc7e3a1279 [STATS-368] Make statistics on load work with tables with... by Satya Valluri <[email protected]>
- 75de39d97097df2b75596073f2df19d83fda4ac0 [SC-168457] Also track auto table and clustering column s... by Eric Liang <[email protected]>
- 635eb6b63cbc3809b51ceaf023334bcbe5158ecd [LC-4146][MAC][Liquid-PO Integration] Handle large tables... by Stella Wan <[email protected]>
- 493c1415599a7910da34b6a37e3023e6cf95ce25 [DELTA-OSS-AUTO][SC-167314] Update flink mima checks by Qianru Lao <[email protected]>
- c8ce5d5360467dc9c01e851a7ed539f3d0792096 [ES-1115071][LC-4301][Liquid] Make metadata query optimiz... by Wei Luo <[email protected]>
- 261d280360847c040db11e4e1ac31dd9cfa58e69 [LC-4507] Run delete and insert steps in parallel during ... by Christos Stavrakakis <[email protected]>
- aae17ff4806845f0f50e0fbd61808ed4e67b07f9 [DBRRM-1031][SC-168577][ES-1145536] Fix Kryo serializatio... by Josh Rosen <[email protected]>
- 320d96e453fb6de07460ac524e6ed6843b667d37 [DELTA-OSS-AUTO][SC-168636]Improve documentation in PROTO... by Andreas Chatzistergiou <[email protected]>
- 9f7979357c735e5b0be9158eccf16d5fc19b76a6 [LS-986][Delta] Handle multi-byte UTF-8 characters while ... by Sumeet Varma <[email protected]>
- c447349ec894c1a54ac4118dc4d470dc12c684df [ES-1151654] Improve the running time of DeletionVectorsS... by Andreas Chatzistergiou <[email protected]>
- 307cd1089a3389a6a9a0aa099529255761e7e107 [AUTOSTATS][ES-1147462] fallback to old style column sele... by mohamedzait <[email protected]>
- af36e130f5b2c6c005cdaa7bc14d214c408e650e [ES-1133698] Fix the issue of not using the correct refre... by Lin Zhou <[email protected]>
- d61bff57c8b4966d1e79c5e17520e0c03d1eb7ab [SC-162599] Include number of incremental transactions in... by Ming DAI <[email protected]>
- 1daf0511dcbc95a9564b8ba5f5a9aca24dd81ebb [LC-4523] Add tracing for slow Delta operations by Christos Stavrakakis <[email protected]>
- dc119c92f185e943490c790e26e541a4646496db [LC-4494][DELTA][MC] Add usage logs related to managed co... by Prakhar Jain <[email protected]>
- 8758faef118a5c2c1cc9794fb724ba7f996390b0 [AUTOSTATS][ES-1147462] do not use auto stats when Row ID... by mohamedzait <[email protected]>
- 9f83c9c342c2e47a6c44d694b27d0cbee66f5ba2 [LS-984][Delta] Add usage logs for exhaustive reasons beh... by Sumeet Varma <[email protected]>
- 6e6297437231b7a460a0cc0d23de616cf171a2ee [LC-4489][MC] Make naming of manage commit consistent by Dhruv Arya <[email protected]>
- 4641410e78e3ca719544ef77054c83df2ba72e7b [LC-3022][Liquid OSS] Support create table from external ... by Jiaheng Tang <[email protected]>
- 8ee182fff5c5a309adde44799404313f2a03f545 [LC-4485][Delta][MC] Remove deprecated FileNames.deltaFil... by Sumeet Varma <[email protected]>
- db6a256db943247ac12b78d48de9d83d7aa0a504 [SC-139180] Fetch globalTempDatabase name directly withou... by Weitao Wen <[email protected]>
(And 374 more changes)
GitOrigin-RevId: d17584edb7d55dba9d45245d26912d531e8fc172
Copy file name to clipboardexpand all lines: PROTOCOL.md
+2-4
Original file line number
Diff line number
Diff line change
@@ -477,9 +477,7 @@ The following is an example `remove` action.
477
477
```
478
478
479
479
### Add CDC File
480
-
The `cdc` action is used to add a [file](#change-data-files) containing only the data that was changed as part of the transaction. The `cdc` action is allowed to add a [Data File](#data-files) that is also added by an `add` action, when it does not contain any copied rows and the `_change_type` column is filled for all rows.
481
-
482
-
When change data readers encounter a `cdc` action in a particular Delta table version, they must read the changes made in that version exclusively using the `cdc` files. If a version has no `cdc` action, then the data in `add` and `remove` actions are read as inserted and deleted rows, respectively.
480
+
The `cdc` action is used to add a [file](#change-data-files) containing only the data that was changed as part of the transaction. When change data readers encounter a `cdc` action in a particular Delta table version, they must read the changes made in that version exclusively using the `cdc` files. If a version has no `cdc` action, then the data in `add` and `remove` actions are read as inserted and deleted rows, respectively.
483
481
484
482
The schema of the `cdc` action is as follows:
485
483
@@ -525,7 +523,7 @@ Specifically, to read the row-level changes made in a version, the following str
525
523
526
524
##### Note for non-change data readers
527
525
528
-
In a table with Change Data Feed enabled, the data Parquet files referenced by `add` and `remove` actions are allowed to contain an extra column `_change_type`. This column is not present in the table's schema. When accessing these files, readers should disregard this column and only process columns defined within the table's schema.
526
+
In a table with Change Data Feed enabled, the data Parquet files referenced by `add` and `remove` actions are allowed to contain an extra column `_change_type`. This column is not present in the table's schema and will consistently have a `null` value. When accessing these files, readers should disregard this column and only process columns defined within the table's schema.
529
527
530
528
### Transaction Identifiers
531
529
Incremental processing systems (e.g., streaming systems) that track progress using their own application-specific versions need to record what progress has been made, in order to avoid duplicating data in the face of failures and retries during a write.
0 commit comments