Skip to content

Commit f4fb28b

Browse files
committed
Squashed commits:
- d17584edb7d55dba9d45245d26912d531e8fc172 WIP by zhipeng.mao <[email protected]> - 11d61890a1e26b8380e7cae9b1d750acf5a14f96 [ES-1177667][TEST-ONLY] Remove timeout source from withTe... by Johan Lasperas <[email protected]> - bc90067254419651895419cd0228d3cd5cbe4d8d [ES-1109125] Report accurate type when checking Iceberg c... by Hao Jiang <[email protected]> - 2bf266859083ecfc1d46240638b7c421c24285ad [SC-160780][FOLLOW-UP] Add SQLConf flag for #96237 change by Vitalii Li <[email protected]> - 23f0eb9982898acf167f3d3eb4d1905133dff27e [LC-4740][CoordinatedCommits] Extend NewTransactionSuite ... by Jun <[email protected]> - 66a20cfed0eaf369a8a2b35e709a23311e0a19f3 [LC-3943] Forbid automatic protocol upgrade when using co... by Stefan Kandic <[email protected]> - 876af9aa7e6de42c501b89eae58111e2df7ba8cc [LC-4491] Shade Hudi into DBR by Annie Wang <[email protected]> - 673ca7fe43e3110c67143642f3a77f4f5106bc23 [LC-4754][Delta][MC] Fix error message formatting in Delt... by Sumeet Varma <[email protected]> - 321e72867fdc98d501963bccaf17f8577bc4c445 [LC-4192][Delta][MC] Use Coordinated Commits Properties f... by Sumeet Varma <[email protected]> - 84fe62011e78b5528adb70cc06a1be906ce97b02 [LC-4263][Delta] Add usage logging to UCCommitCooridnator... by Yumingxuan Guo <[email protected]> - 4cde1a20dddc7408beb251cfbf62baed7bc29940 [LC-4392][Auto-Clustering-Key-Selection] Add additional t... by parimarjannegi-db <[email protected]> - 7b5a138dd7388ac59a08528c0e27c622cf0bb2f3 [LC-4549] Delta structured logging migration for remainin... by Jiaheng Tang <[email protected]> - c0e90e6bd23cc181e900ee4633b2eb213a913f9d [LC-4131] Validate that CDF works with collations by Stefan Kandic <[email protected]> - c472de831911ccd1cc0fd7b23e1fd417cecde195 [Delta][Copybara] Introduce `// EDGE-NO-INHERITANCE` tag by Paddy Xu <[email protected]> - 78f75a2bb3a05ad37b59e42eb35f4b61cb1c7acf [DELTA-OSS-AUTO][SC-170469][Hudi] Flesh out tests and upd... by Annie Wang <[email protected]> - 49d07ae43b3dbc09c2679e35a4c751b99f53c60a [ES-1151956] Fixes for "show tables extended" by rayman7718 <[email protected]> - c35f0fdd59377faae33aa06f477fe5b066babe82 [ES-1159798][LC-4729][BEHAVE-74] Fix CDC Commit Timestamp... by Thang Long Vu <[email protected]> - cfd343651f3c00a6f551f3e2ef5180b80f2dd47b [LC-4661] Deprecate tableVersion field in type widening m... by Johan Lasperas <[email protected]> - 7014ad1ec51af3f55c2f9713b032c6c9d32ff755 [AUTO][SC-170122][SPARK-48697][LC-4703][SQL] Add collatio... by Stefan Kandic <[email protected]> - 35628a58efa0c704bf2e2ddfcba833ff9b40f07a [Delta][CC] Add integration test for DROP/UNDROP. by Yumingxuan Guo <[email protected]> - 3351fa6531e3e370ebe41f931c3946fa00206d5f [LC-4693][Uniform] Make refresh on UFI no-op when there's... by ChengJi-db <[email protected]> - 10d743671c381ea09131f3519be7fc90c460db75 [LC-4143][Delta][CC] Adds asynchronous backfilling in UC ... by Lukas Rupprecht <[email protected]> - d55d080248832d881d33e52d9b1d9efd66a197bf [LC-4698] add logging to UniForm Iceberg conversion num o... by Fred Storage Liu <[email protected]> - f992112f04b14423f323d4f171040ec8cca1b3ad [LC-4499] All Delta Commands should have all operation me... by sabir-akhadov <[email protected]> - 7d0c410dba226ae191f51280c382b272fba459b1 [LC-3778] Log duration of DeltaSource operations by Tom van Bussel <[email protected]> - e5e29b59cab4218ed77903079eb2b08216e6f982 Import changes from OSS PR delta-io#3326 by Annie Wang <[email protected]> - 4c08927c6eee0da4a9a00795b36a4b734f2c8a04 Import changes from OSS PR delta-io#3323 by Annie Wang <[email protected]> - 1fd8ee89f3b7398db6f13486bd42ca2535023976 Import changes from OSS PR delta-io#3320 by Annie Wang <[email protected]> - db210bb4f2bd0d682863c99ca4c385ed0d0d0118 [LC-4692] Regular delta min/max stats should always use U... by Stefan Kandic <[email protected]> - e873cab62acf71139d89f4019f59fb5aeec81f14 Import changes from OSS PR delta-io#3309 by Annie Wang <[email protected]> - 6d1aca32ebc1d9009e77ca26f4fd0c73ba8e7f6e Import changes from OSS PR delta-io#3310 by Annie Wang <[email protected]> - 0070442d167c53884e083127eeaa7e69c71f28d9 Import changes from OSS PR delta-io#3311 by Annie Wang <[email protected]> - 8b8df4f43b5a85d1626ea4a2ce612b34ea67544d [ES-1089258][ES-1043275][Delta] Creates unresolved tables... by Lukas Rupprecht <[email protected]> - 602f7e429fd3c73d43c24ac7f231a9a10bfe8830 [LC-4043] Allow auto REFRESH on UniForm tables by Hao Jiang <[email protected]> - 26755b6dba8caeca1b7c8797eb634abcb5532e35 [ES-1154731] Disable managed Hive Metastore related Conve... by Ming DAI <[email protected]> - 568c954c44eafe752d6c614ca36dc3f2a0a651d4 [ES-1150026][Delta] Handle concurrent CREATE TABLE IF NOT... by Sumeet Varma <[email protected]> - 195a5fbcd168a0d1e7e992dfdb226b3a9e21a6a3 [LC-4657][Delta][MC] Throw exception when additional list... by Sumeet Varma <[email protected]> - 438d79963e76478a641e256ee53edca557024c99 [LC-4524][Delta][MC] Fetch unbackfilled deltas to detect ... by Sumeet Varma <[email protected]> - 3c6800ac28f998e3183a54d4ff6cf058607cad16 [LC-4996][BEHAVE-60] Introduce stable type widening table... by Johan Lasperas <[email protected]> - c0d24087192bcbbd80f42fa83e2229482c877d66 [ES-1154366][ES-1154367] Fix ConvertToDeltaSuite in serve... by Ming DAI <[email protected]> - 6b7e4d0f9004d044fd88248c4b024d61e1d4b91d [Liquid] Improve error message when clustering column not... by Chirag Singh <[email protected]> - 75e21adf355c43bbafeb6dffefe7627614ca6338 [LC-4242] History truncation/validation support for write... by Andreas Chatzistergiou <[email protected]> - 71aec8cc7f136db13de264f661446526a9e7a2c2 [ES-1162704] Disable some Python Delta tests in serverless by Christos Stavrakakis <[email protected]> - 96b6ce441e287e9b008261844017eb9d09ffe692 update delta-sharing client to version 1.1.0 by Jade Wang <[email protected]> - 510e7d69ce180c869031affff075bc163077fa82 [SC-160780] Support Delta API for DML commands when table... by Vitalii Li <[email protected]> - 562322d9ab5ecf7fb237fa0e950470d83a252ed1 [LC-4549] Delta structured logging migration by Jiaheng Tang <[email protected]> - 850e368261d2cb89cc732139c8ee1574978366d5 [LC-4526] Support snapshot expiration for uniform iceberg... by ChengJi-db <[email protected]> - 6e11dd7f11e79a429068a6daa59d4e201fa894c8 [ES-1151640] Break down ImplicitDMLCastingSuite by Lars Kroll <[email protected]> - 837d86463cef6bbb1e7571f06431cedd67b76c7e [ES-1154733][ES-1154732][ES-1154381][ES-1154382][TEST-ONL... by Paddy Xu <[email protected]> - 51bfb0c60bb2b2d64f01b7c5f96187e65b0ad4b1 [LC-4534] Add ThreadLocal trait for underlying execution ... by leonwind-db <[email protected]> - ade2f1670859abf5cad4f4a3f0240566ec8c3a0f [LS-989][Delta] Add checkpointType, numColumns and numFla... by Sumeet Varma <[email protected]> - 62c938e3eb626e04000e0acd4d37953a24d3c52d [LS-985][Delta] Add detailed mismatch reasons for Increme... by Sumeet Varma <[email protected]> - bfda87427e6c9c019d83ee108e8e70a5e5950d8e [SPARK-48576][SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE by Uros Bojanic <[email protected]> - 2ff6142a85ef692a055071fa722ac10caeaf685f [LC-4301] Fix the predicates used by metadata queries by Wei Luo <[email protected]> - 6340541a0bbc4c9d71e04aff99a3cb21720dd975 [DELTA-OSS-AUTO][SC-168789][Spark] Optimize batching / in... by Adam Binford <[email protected]> - 1c691ed0e5df46402a4ebe44b16b89b748f2915e [SC-162602] Add per-kdtree scan stats (only scanned files) by Eric Liang <[email protected]> - 895a09ff141e810f87d2c5b353d3963d8ac00586 [SC-167647] map lookup for createPhysicalSchema by andrewxue-db <[email protected]> - d89308d177f5e269e7cb6bdfbe6c0978bd9d1cd2 [Delta OSS] Fix compile error on Spark master due to Pars... by Jiaheng Tang <[email protected]> - c7ba28b7e35319db1a41d5252506d82b39933ec2 [STATS-356] Improve recomputeDeltaFileStats logic with ME... by Pat Sukprasert <[email protected]> - 750eeac292c99c0003b6011038b99e2ee92083db [LC-4499] All Delta Commands should have all operation me... by sabir-akhadov <[email protected]> - ebe3e3e296c593fa6e5020e8127b410cd9d4c4c6 [LC-3942] Add historical schema read compatibility checks... by Nikola Mandic <[email protected]> - 20dbe80816413dcfa565777fdd9878f4f00c1af2 [SC-167257][Auto-Cluster-Key-Selection] Show operational ... by Supun Nakandala <[email protected]> - 2460f7604cba40783e44a43cbeb4bc39956427d8 [ES-1155687][LC-812] Make DPO track read files by Christos Stavrakakis <[email protected]> - 59e3a4f57aaa7a7298e978ace615ff04b4507418 [LC-4484] Match RangeBloomFilterMightContain expressions ... by Tom van Bussel <[email protected]> - 01c09b8556e57b2fa94c3e6f850b2ae426364976 [ES-1077008][LC-4411][BEHAVE-32] Add a Delta config to en... by Paddy Xu <[email protected]> - 7bd2395d454a1f2383d5b58f16e7de643cacae8b [ES-1154374][TEST-ONLY] Fix INSERT with delta view tests ... by Johan Lasperas <[email protected]> - d8f90b3560e6fe8d95bf757538660a995cf185a0 [LC-4323] Delta metadata queries should be QPL tagged wit... by sabir-akhadov <[email protected]> - e673c6972eb230234b8206309dcffc135af6387e [LC-4132][SQL] Disable changing collation of clustering c... by Nikola Mandic <[email protected]> - a34f0d7e23c1dcf55310be6bf4148bd4ad3839e1 [ES-1154398][TEST-ONLY][WARMFIX] IdentityColumnAdmissionS... by Carmen Kwan <[email protected]> - 9c34481f90d4e526a0f5c3671e5e59aae969c44c [LC-3259][Delta][MC][Pt. 3] Creates separate CommitStore ... by Lukas Rupprecht <[email protected]> - 0249b90ffe247bc689586891e90f81bd0568ad6d [LC-4564][LC-4565][Liquid OSS] Support show tblproperties... by Jiaheng Tang <[email protected]> - 24a50e9f98a540ba72c9a7a585ddad0a8196ee92 [LC-4548] Support Spark Structured Logging in Delta by Jiaheng Tang <[email protected]> - f4b3d9042b1ffebc8ed78a3fe2634e6672213cc0 [ES-1154365][ES-1154357] Disable test cases not supported... by Chirag Singh <[email protected]> - a39c357817393943cc2997bbc9f020b8e02c14a8 [Liquid][LC-3844] Do not attempt lazy clustering on files... by Chirag Singh <[email protected]> - b8ff879f8471592e7d08312be088152c575ecb3f [Liquid][LC-4335][LC-4314] Execute eager clustering batch... by Chirag Singh <[email protected]> - 1bce3b75784df6960e1c8261dea5c97f86f0e3bd [Delta] Rename managed commit to coordinated commits by Dhruv Arya <[email protected]> - f727c3e6717ed8f235560055ade0848a2333c6dd [DELTA-OSS-AUTO][SC-167757][SPARK][MINOR] Code cleanup (r... by Jacek Laskowski <[email protected]> - edf995cf281c10917d747620a4edc8ab37bf032d [Liquid][LC-2605] Change verifyClusteringColumns to not p... by Chirag Singh <[email protected]> - 04129352860bfd2973d2717a69b793a4ae880217 [BEHAVE-43][SC-168642][Spark] Support dropping the CHECK ... by Tom van Bussel <[email protected]> - 863b8c7f7af9d720949a9b098f89f8bc7e3a1279 [STATS-368] Make statistics on load work with tables with... by Satya Valluri <[email protected]> - 75de39d97097df2b75596073f2df19d83fda4ac0 [SC-168457] Also track auto table and clustering column s... by Eric Liang <[email protected]> - 635eb6b63cbc3809b51ceaf023334bcbe5158ecd [LC-4146][MAC][Liquid-PO Integration] Handle large tables... by Stella Wan <[email protected]> - 493c1415599a7910da34b6a37e3023e6cf95ce25 [DELTA-OSS-AUTO][SC-167314] Update flink mima checks by Qianru Lao <[email protected]> - c8ce5d5360467dc9c01e851a7ed539f3d0792096 [ES-1115071][LC-4301][Liquid] Make metadata query optimiz... by Wei Luo <[email protected]> - 261d280360847c040db11e4e1ac31dd9cfa58e69 [LC-4507] Run delete and insert steps in parallel during ... by Christos Stavrakakis <[email protected]> - aae17ff4806845f0f50e0fbd61808ed4e67b07f9 [DBRRM-1031][SC-168577][ES-1145536] Fix Kryo serializatio... by Josh Rosen <[email protected]> - 320d96e453fb6de07460ac524e6ed6843b667d37 [DELTA-OSS-AUTO][SC-168636]Improve documentation in PROTO... by Andreas Chatzistergiou <[email protected]> - 9f7979357c735e5b0be9158eccf16d5fc19b76a6 [LS-986][Delta] Handle multi-byte UTF-8 characters while ... by Sumeet Varma <[email protected]> - c447349ec894c1a54ac4118dc4d470dc12c684df [ES-1151654] Improve the running time of DeletionVectorsS... by Andreas Chatzistergiou <[email protected]> - 307cd1089a3389a6a9a0aa099529255761e7e107 [AUTOSTATS][ES-1147462] fallback to old style column sele... by mohamedzait <[email protected]> - af36e130f5b2c6c005cdaa7bc14d214c408e650e [ES-1133698] Fix the issue of not using the correct refre... by Lin Zhou <[email protected]> - d61bff57c8b4966d1e79c5e17520e0c03d1eb7ab [SC-162599] Include number of incremental transactions in... by Ming DAI <[email protected]> - 1daf0511dcbc95a9564b8ba5f5a9aca24dd81ebb [LC-4523] Add tracing for slow Delta operations by Christos Stavrakakis <[email protected]> - dc119c92f185e943490c790e26e541a4646496db [LC-4494][DELTA][MC] Add usage logs related to managed co... by Prakhar Jain <[email protected]> - 8758faef118a5c2c1cc9794fb724ba7f996390b0 [AUTOSTATS][ES-1147462] do not use auto stats when Row ID... by mohamedzait <[email protected]> - 9f83c9c342c2e47a6c44d694b27d0cbee66f5ba2 [LS-984][Delta] Add usage logs for exhaustive reasons beh... by Sumeet Varma <[email protected]> - 6e6297437231b7a460a0cc0d23de616cf171a2ee [LC-4489][MC] Make naming of manage commit consistent by Dhruv Arya <[email protected]> - 4641410e78e3ca719544ef77054c83df2ba72e7b [LC-3022][Liquid OSS] Support create table from external ... by Jiaheng Tang <[email protected]> - 8ee182fff5c5a309adde44799404313f2a03f545 [LC-4485][Delta][MC] Remove deprecated FileNames.deltaFil... by Sumeet Varma <[email protected]> - db6a256db943247ac12b78d48de9d83d7aa0a504 [SC-139180] Fetch globalTempDatabase name directly withou... by Weitao Wen <[email protected]> (And 374 more changes) GitOrigin-RevId: d17584edb7d55dba9d45245d26912d531e8fc172
1 parent c422db8 commit f4fb28b

File tree

92 files changed

+2792
-5797
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

92 files changed

+2792
-5797
lines changed

PROTOCOL.md

+2-4
Original file line numberDiff line numberDiff line change
@@ -477,9 +477,7 @@ The following is an example `remove` action.
477477
```
478478

479479
### Add CDC File
480-
The `cdc` action is used to add a [file](#change-data-files) containing only the data that was changed as part of the transaction. The `cdc` action is allowed to add a [Data File](#data-files) that is also added by an `add` action, when it does not contain any copied rows and the `_change_type` column is filled for all rows.
481-
482-
When change data readers encounter a `cdc` action in a particular Delta table version, they must read the changes made in that version exclusively using the `cdc` files. If a version has no `cdc` action, then the data in `add` and `remove` actions are read as inserted and deleted rows, respectively.
480+
The `cdc` action is used to add a [file](#change-data-files) containing only the data that was changed as part of the transaction. When change data readers encounter a `cdc` action in a particular Delta table version, they must read the changes made in that version exclusively using the `cdc` files. If a version has no `cdc` action, then the data in `add` and `remove` actions are read as inserted and deleted rows, respectively.
483481

484482
The schema of the `cdc` action is as follows:
485483

@@ -525,7 +523,7 @@ Specifically, to read the row-level changes made in a version, the following str
525523

526524
##### Note for non-change data readers
527525

528-
In a table with Change Data Feed enabled, the data Parquet files referenced by `add` and `remove` actions are allowed to contain an extra column `_change_type`. This column is not present in the table's schema. When accessing these files, readers should disregard this column and only process columns defined within the table's schema.
526+
In a table with Change Data Feed enabled, the data Parquet files referenced by `add` and `remove` actions are allowed to contain an extra column `_change_type`. This column is not present in the table's schema and will consistently have a `null` value. When accessing these files, readers should disregard this column and only process columns defined within the table's schema.
529527

530528
### Transaction Identifiers
531529
Incremental processing systems (e.g., streaming systems) that track progress using their own application-specific versions need to record what progress has been made, in order to avoid duplicating data in the face of failures and retries during a write.

benchmarks/build.sbt

+1-1
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
*/
1616

1717
name := "benchmarks"
18-
scalaVersion := "2.12.18"
18+
scalaVersion := "2.12.17"
1919

2020
lazy val root = (project in file("."))
2121
.settings(

build.sbt

-110
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,6 @@
1717
// scalastyle:off line.size.limit
1818

1919
import java.nio.file.Files
20-
21-
import sbt.internal.inc.Analysis
22-
import sbtprotoc.ProtocPlugin.autoImport._
23-
24-
import xsbti.compile.CompileAnalysis
25-
2620
import Checkstyle._
2721
import Mima._
2822
import Unidoc._
@@ -45,8 +39,6 @@ val LATEST_RELEASED_SPARK_VERSION = "3.5.0"
4539
val SPARK_MASTER_VERSION = "4.0.0-SNAPSHOT"
4640
val sparkVersion = settingKey[String]("Spark version")
4741
spark / sparkVersion := getSparkVersion()
48-
connectCommon / sparkVersion := getSparkVersion()
49-
connectServer / sparkVersion := getSparkVersion()
5042

5143
// Dependent library versions
5244
val defaultSparkVersion = LATEST_RELEASED_SPARK_VERSION
@@ -66,9 +58,6 @@ val hadoopVersionForHive2 = "2.7.2"
6658
val hive2Version = "2.3.3"
6759
val tezVersionForHive2 = "0.8.4"
6860

69-
val protoVersion = "3.25.1"
70-
val grpcVersion = "1.62.2"
71-
7261
scalaVersion := default_scala_version.value
7362

7463
// crossScalaVersions must be set to Nil on the root project
@@ -198,105 +187,6 @@ def crossSparkSettings(): Seq[Setting[_]] = getSparkVersion() match {
198187
)
199188
}
200189

201-
def runTaskOnlyOnSparkMaster[T](
202-
task: sbt.TaskKey[T],
203-
taskName: String,
204-
projectName: String,
205-
emptyValue: => T): Def.Initialize[Task[T]] = {
206-
if (getSparkVersion() == SPARK_MASTER_VERSION) {
207-
Def.task(task.value)
208-
} else {
209-
Def.task {
210-
// scalastyle:off println
211-
println(s"Project $projectName: Skipping `$taskName` as Spark version " +
212-
s"${getSparkVersion()} does not equal $SPARK_MASTER_VERSION.")
213-
// scalastyle:on println
214-
emptyValue
215-
}
216-
}
217-
}
218-
219-
lazy val connectCommon = (project in file("spark-connect/common"))
220-
.settings(
221-
name := "delta-connect-common",
222-
commonSettings,
223-
crossSparkSettings(),
224-
releaseSettings,
225-
Compile / compile := runTaskOnlyOnSparkMaster(
226-
task = Compile / compile,
227-
taskName = "compile",
228-
projectName = "delta-connect-common",
229-
emptyValue = Analysis.empty.asInstanceOf[CompileAnalysis]
230-
).value,
231-
Test / test := runTaskOnlyOnSparkMaster(
232-
task = Test / test,
233-
taskName = "test",
234-
projectName = "delta-connect-common",
235-
emptyValue = ()).value,
236-
publish := runTaskOnlyOnSparkMaster(
237-
task = publish,
238-
taskName = "publish",
239-
projectName = "delta-connect-common",
240-
emptyValue = ()).value,
241-
libraryDependencies ++= Seq(
242-
"io.grpc" % "protoc-gen-grpc-java" % grpcVersion asProtocPlugin(),
243-
"io.grpc" % "grpc-protobuf" % grpcVersion,
244-
"io.grpc" % "grpc-stub" % grpcVersion,
245-
"com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf",
246-
"javax.annotation" % "javax.annotation-api" % "1.3.2",
247-
248-
"org.apache.spark" %% "spark-connect-common" % sparkVersion.value % "provided",
249-
),
250-
PB.protocVersion := protoVersion,
251-
Compile / PB.targets := Seq(
252-
PB.gens.java -> (Compile / sourceManaged).value,
253-
PB.gens.plugin("grpc-java") -> (Compile / sourceManaged).value
254-
),
255-
)
256-
257-
lazy val connectServer = (project in file("spark-connect/server"))
258-
.dependsOn(connectCommon % "compile->compile;test->test;provided->provided")
259-
.dependsOn(spark % "compile->compile;test->test;provided->provided")
260-
.settings(
261-
name := "delta-connect-server",
262-
commonSettings,
263-
releaseSettings,
264-
Compile / compile := runTaskOnlyOnSparkMaster(
265-
task = Compile / compile,
266-
taskName = "compile",
267-
projectName = "delta-connect-server",
268-
emptyValue = Analysis.empty.asInstanceOf[CompileAnalysis]
269-
).value,
270-
Test / test := runTaskOnlyOnSparkMaster(
271-
task = Test / test,
272-
taskName = "test",
273-
projectName = "delta-connect-server",
274-
emptyValue = ()
275-
).value,
276-
publish := runTaskOnlyOnSparkMaster(
277-
task = publish,
278-
taskName = "publish",
279-
projectName = "delta-connect-server",
280-
emptyValue = ()
281-
).value,
282-
crossSparkSettings(),
283-
libraryDependencies ++= Seq(
284-
"com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf",
285-
286-
"org.apache.spark" %% "spark-hive" % sparkVersion.value % "provided",
287-
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "provided",
288-
"org.apache.spark" %% "spark-core" % sparkVersion.value % "provided",
289-
"org.apache.spark" %% "spark-catalyst" % sparkVersion.value % "provided",
290-
"org.apache.spark" %% "spark-connect" % sparkVersion.value % "provided",
291-
292-
"org.apache.spark" %% "spark-catalyst" % sparkVersion.value % "test" classifier "tests",
293-
"org.apache.spark" %% "spark-core" % sparkVersion.value % "test" classifier "tests",
294-
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "test" classifier "tests",
295-
"org.apache.spark" %% "spark-hive" % sparkVersion.value % "test" classifier "tests",
296-
"org.apache.spark" %% "spark-connect" % sparkVersion.value % "test" classifier "tests",
297-
),
298-
)
299-
300190
lazy val spark = (project in file("spark"))
301191
.dependsOn(storage)
302192
.enablePlugins(Antlr4Plugin)

dev/check-delta-connect-codegen-python.py

-83
This file was deleted.

dev/delta-connect-gen-protos.sh

-104
This file was deleted.

dev/pyproject.toml

-23
This file was deleted.

dev/requirements.txt

-9
This file was deleted.

dev/tox.ini

+1-1
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,6 @@
1717
[pycodestyle]
1818
ignore=E226,E241,E305,E402,E722,E731,E741,W503,W504
1919
max-line-length=100
20-
exclude=cloudpickle.py,heapq3.py,shared.py,python/docs/conf.py,work/*/*.py,python/.eggs/*,dist/*,*python/delta/connect*
20+
exclude=cloudpickle.py,heapq3.py,shared.py,python/docs/conf.py,work/*/*.py,python/.eggs/*,dist/*
2121
[pydocstyle]
2222
ignore=D100,D101,D102,D103,D104,D105,D106,D107,D200,D201,D202,D203,D204,D205,D206,D207,D208,D209,D210,D211,D212,D213,D214,D215,D300,D301,D302,D400,D401,D402,D403,D404,D405,D406,D407,D408,D409,D410,D411,D412,D413,D414,D415,D417

0 commit comments

Comments
 (0)