Skip to content

Commit a442563

Browse files
cstavrxzhseh
authored andcommitted
Squashed commits:
- ddd6a8ee676b3dab729ea720182a022e0977567e [ES-1190931][DBRRM-1133] Add event to detect Delta tables... by Christos Stavrakakis <[email protected]> - 8ae8d533e7b02a2756420c88bbf930ece746ec80 [LC-4832][Delta][CC] Uses metastoreID from createStagingT... by Lukas Rupprecht <[email protected]> - f4fd1d10ab960eb88850531cc57ed8f5d1638b2f Revert "[ES-1167842] Revert logging of clusteringColumns ... by databricks-jenkins <[email protected]> - 2a0c28897f11375da25facfa46a01e5afc2c0d7b [ES-1167842] Revert logging of clusteringColumns in 15.4 by Eric Liang <[email protected]> - e7519ecfcda0aa8ebb2c2a29c12b128415f67274 [SC-170456] Record Unsupported Data Filtering Predicates by Supun Nakandala <[email protected]> - 3a44c7c1655a4f6b918b0c2fdb5842a773b7a36b [LC-4852] Add Test Suite for Protocol Transition Improvem... by Andreas Chatzistergiou <[email protected]> - a42d1c3bffc195ac82ac8c4a04dd67918cdd3ce6 [LC-4787][MAC] Followup to registering MAC hook for CRTAS... by Stella Wan <[email protected]> - 923046452f1300fcd474521197386d8840253aa8 [Delta-OSS] Remove dropped columns when running REORG PURGE by Zihao Xu <[email protected]> - e995fd6acc8557a9c3ff15315ba07982f58172f0 [ES-1192280][LC-4831][SC-171393] Fix DFP Null Safe Equali... by Thang Long Vu <[email protected]> - 45e7f85c1e4f9d059325f434123145d1edd78194 [LC-4787][MAC] Register MAC hook for CRTAS and DataFrame ... by Stella Wan <[email protected]> - 4d0b912a5027b934614f1ae81f950f3f605134c9 [ES-1178751] Increase replication level in MERGE source m... by Johan Lasperas <[email protected]> - de0a9d0677593cda59fe569b3ebcea417aded562 [ES-1183111] Enforce correct parallelism to copy files wi... by Johan Lasperas <[email protected]> - b7ea96b356a6bac91562ce7fbf2c82a90526b364 [DELTA-OSS-AUTO][SC-171316][SPARK] Add Integration Test i... by Mingkang Li <[email protected]> - b9239b289727034c072a75833b138ca0fafe6893 [ES-1113859] Batch resolve DeltaMergeAction in ResolveMer... by Carmen Kwan <[email protected]> - f1ca70446f0bc2d8a0cad1a608ef17a7b784eecc [AUTO][SC-170767][SPARK-48851][SQL] Change the value of `... by panbingkun <[email protected]> - cadf44a02bf1c286d9aa9c704a1a22729aa0108d [ES-1053014][ES-1125181][BEHAVE-78] Fix VARCHAR/CHAR to s... by Dhruv Arya <[email protected]> - 4396b20e0c4b2bc42c97d909a516f092d945653f [DELTA-OSS-AUTO][SC-171306][Spark] Only attempt parsing p... by Amogh Jahagirdar <[email protected]> - a85044eb6fef3090abd90d143a3d97023f0e82a1 [ES-1183822][TEST-ONLY] Use DeltaSQLTestUtils in more Del... by Johan Lasperas <[email protected]> - b0801357aef6e7d5c2bd8a0d336313142dbb6bff [DELTA-OSS-EXTERNAL][BEHAVE-87][Spark] Improve Delta Prot... by Andreas Chatzistergiou <[email protected]> - ae9fb7f8f44b471f7e81e0b195e15e0d655d8fec [SC-171037] Annotate Delta Log Keys by Jiaheng Tang <[email protected]> - d512153866293aa3388ea631426f6ce85fcfc0f8 [LC-3943] Put automatic protocol upgrade for collations f... by Stefan Kandic <[email protected]> - fe6efbb1cfd45d778718b877a824f730a2dd8ef2 Check UniForm Hudi table must be UC by Annie Wang <[email protected]> - 8a391f7155895b979866455055741a42808020a8 [LC-4634] Enable IcebergCompatV2 Delta Uniform By Alter T... by Zihao Xu <[email protected]> - 5028821d836b75c283b730bbf807fdfa5d28cc21 [CC][LC-4664][Test-Only] Make TrackingCommitCoordinatorCl... by Dhruv Arya <[email protected]> - d3fb4e4ae691f1ccf99274645cb3fea31d9a10ec [SC-166402] managed data skipping columns by Paul Loh <[email protected]> - 8c98e895d4860387d216278168acd3977437d715 [LC-4668] Expose uncertainty in isDeltaTable resolution v... by Lars Kroll <[email protected]> - 3376b76a114fac408e6363ac864921d7178d9804 [ES-1010221] Part 2 of Scala 2.13 migration for Delta tests by Jintian Liang <[email protected]> - 7880534e5f814f768648699823731edb3eb86c77 [SC-163065] Make two copies of runQueriesWithOrderingFrom... by Thang Long Vu <[email protected]> - 8a59d4b4946aef5cfbc75594bd05ce979c249cd4 [SQL][COLLATION] Add COLLATE tag to structured QPL by Uros Bojanic <[email protected]> - 91d9489acc1b15d4bea86baf3b75ba6e5e0b9abf [LC-4638] Implement more efficient SerDe for the row_inde... by Christos Stavrakakis <[email protected]> - 11d61890a1e26b8380e7cae9b1d750acf5a14f96 [ES-1177667][TEST-ONLY] Remove timeout source from withTe... by Johan Lasperas <[email protected]> - bc90067254419651895419cd0228d3cd5cbe4d8d [ES-1109125] Report accurate type when checking Iceberg c... by Hao Jiang <[email protected]> - 2bf266859083ecfc1d46240638b7c421c24285ad [SC-160780][FOLLOW-UP] Add SQLConf flag for #96237 change by Vitalii Li <[email protected]> - 23f0eb9982898acf167f3d3eb4d1905133dff27e [LC-4740][CoordinatedCommits] Extend NewTransactionSuite ... by Jun <[email protected]> - 66a20cfed0eaf369a8a2b35e709a23311e0a19f3 [LC-3943] Forbid automatic protocol upgrade when using co... by Stefan Kandic <[email protected]> - 876af9aa7e6de42c501b89eae58111e2df7ba8cc [LC-4491] Shade Hudi into DBR by Annie Wang <[email protected]> - 673ca7fe43e3110c67143642f3a77f4f5106bc23 [LC-4754][Delta][MC] Fix error message formatting in Delt... by Sumeet Varma <[email protected]> - 321e72867fdc98d501963bccaf17f8577bc4c445 [LC-4192][Delta][MC] Use Coordinated Commits Properties f... by Sumeet Varma <[email protected]> - 84fe62011e78b5528adb70cc06a1be906ce97b02 [LC-4263][Delta] Add usage logging to UCCommitCooridnator... by Yumingxuan Guo <[email protected]> - 4cde1a20dddc7408beb251cfbf62baed7bc29940 [LC-4392][Auto-Clustering-Key-Selection] Add additional t... by parimarjannegi-db <[email protected]> - 7b5a138dd7388ac59a08528c0e27c622cf0bb2f3 [LC-4549] Delta structured logging migration for remainin... by Jiaheng Tang <[email protected]> - c0e90e6bd23cc181e900ee4633b2eb213a913f9d [LC-4131] Validate that CDF works with collations by Stefan Kandic <[email protected]> - c472de831911ccd1cc0fd7b23e1fd417cecde195 [Delta][Copybara] Introduce `// EDGE-NO-INHERITANCE` tag by Paddy Xu <[email protected]> - 78f75a2bb3a05ad37b59e42eb35f4b61cb1c7acf [DELTA-OSS-AUTO][SC-170469][Hudi] Flesh out tests and upd... by Annie Wang <[email protected]> - 49d07ae43b3dbc09c2679e35a4c751b99f53c60a [ES-1151956] Fixes for "show tables extended" by rayman7718 <[email protected]> - c35f0fdd59377faae33aa06f477fe5b066babe82 [ES-1159798][LC-4729][BEHAVE-74] Fix CDC Commit Timestamp... by Thang Long Vu <[email protected]> - cfd343651f3c00a6f551f3e2ef5180b80f2dd47b [LC-4661] Deprecate tableVersion field in type widening m... by Johan Lasperas <[email protected]> - 7014ad1ec51af3f55c2f9713b032c6c9d32ff755 [AUTO][SC-170122][SPARK-48697][LC-4703][SQL] Add collatio... by Stefan Kandic <[email protected]> - 35628a58efa0c704bf2e2ddfcba833ff9b40f07a [Delta][CC] Add integration test for DROP/UNDROP. by Yumingxuan Guo <[email protected]> - 3351fa6531e3e370ebe41f931c3946fa00206d5f [LC-4693][Uniform] Make refresh on UFI no-op when there's... by ChengJi-db <[email protected]> - 10d743671c381ea09131f3519be7fc90c460db75 [LC-4143][Delta][CC] Adds asynchronous backfilling in UC ... by Lukas Rupprecht <[email protected]> - d55d080248832d881d33e52d9b1d9efd66a197bf [LC-4698] add logging to UniForm Iceberg conversion num o... by Fred Storage Liu <[email protected]> - f992112f04b14423f323d4f171040ec8cca1b3ad [LC-4499] All Delta Commands should have all operation me... by sabir-akhadov <[email protected]> - 7d0c410dba226ae191f51280c382b272fba459b1 [LC-3778] Log duration of DeltaSource operations by Tom van Bussel <[email protected]> - e5e29b59cab4218ed77903079eb2b08216e6f982 Import changes from OSS PR delta-io#3326 by Annie Wang <[email protected]> - 4c08927c6eee0da4a9a00795b36a4b734f2c8a04 Import changes from OSS PR delta-io#3323 by Annie Wang <[email protected]> - 1fd8ee89f3b7398db6f13486bd42ca2535023976 Import changes from OSS PR delta-io#3320 by Annie Wang <[email protected]> - db210bb4f2bd0d682863c99ca4c385ed0d0d0118 [LC-4692] Regular delta min/max stats should always use U... by Stefan Kandic <[email protected]> - e873cab62acf71139d89f4019f59fb5aeec81f14 Import changes from OSS PR delta-io#3309 by Annie Wang <[email protected]> - 6d1aca32ebc1d9009e77ca26f4fd0c73ba8e7f6e Import changes from OSS PR delta-io#3310 by Annie Wang <[email protected]> - 0070442d167c53884e083127eeaa7e69c71f28d9 Import changes from OSS PR delta-io#3311 by Annie Wang <[email protected]> - 8b8df4f43b5a85d1626ea4a2ce612b34ea67544d [ES-1089258][ES-1043275][Delta] Creates unresolved tables... by Lukas Rupprecht <[email protected]> - 602f7e429fd3c73d43c24ac7f231a9a10bfe8830 [LC-4043] Allow auto REFRESH on UniForm tables by Hao Jiang <[email protected]> - 26755b6dba8caeca1b7c8797eb634abcb5532e35 [ES-1154731] Disable managed Hive Metastore related Conve... by Ming DAI <[email protected]> - 568c954c44eafe752d6c614ca36dc3f2a0a651d4 [ES-1150026][Delta] Handle concurrent CREATE TABLE IF NOT... by Sumeet Varma <[email protected]> - 195a5fbcd168a0d1e7e992dfdb226b3a9e21a6a3 [LC-4657][Delta][MC] Throw exception when additional list... by Sumeet Varma <[email protected]> - 438d79963e76478a641e256ee53edca557024c99 [LC-4524][Delta][MC] Fetch unbackfilled deltas to detect ... by Sumeet Varma <[email protected]> - 3c6800ac28f998e3183a54d4ff6cf058607cad16 [LC-4996][BEHAVE-60] Introduce stable type widening table... by Johan Lasperas <[email protected]> - c0d24087192bcbbd80f42fa83e2229482c877d66 [ES-1154366][ES-1154367] Fix ConvertToDeltaSuite in serve... by Ming DAI <[email protected]> - 6b7e4d0f9004d044fd88248c4b024d61e1d4b91d [Liquid] Improve error message when clustering column not... by Chirag Singh <[email protected]> - 75e21adf355c43bbafeb6dffefe7627614ca6338 [LC-4242] History truncation/validation support for write... by Andreas Chatzistergiou <[email protected]> - 71aec8cc7f136db13de264f661446526a9e7a2c2 [ES-1162704] Disable some Python Delta tests in serverless by Christos Stavrakakis <[email protected]> - 96b6ce441e287e9b008261844017eb9d09ffe692 update delta-sharing client to version 1.1.0 by Jade Wang <[email protected]> - 510e7d69ce180c869031affff075bc163077fa82 [SC-160780] Support Delta API for DML commands when table... by Vitalii Li <[email protected]> - 562322d9ab5ecf7fb237fa0e950470d83a252ed1 [LC-4549] Delta structured logging migration by Jiaheng Tang <[email protected]> - 850e368261d2cb89cc732139c8ee1574978366d5 [LC-4526] Support snapshot expiration for uniform iceberg... by ChengJi-db <[email protected]> - 6e11dd7f11e79a429068a6daa59d4e201fa894c8 [ES-1151640] Break down ImplicitDMLCastingSuite by Lars Kroll <[email protected]> - 837d86463cef6bbb1e7571f06431cedd67b76c7e [ES-1154733][ES-1154732][ES-1154381][ES-1154382][TEST-ONL... by Paddy Xu <[email protected]> - 51bfb0c60bb2b2d64f01b7c5f96187e65b0ad4b1 [LC-4534] Add ThreadLocal trait for underlying execution ... by leonwind-db <[email protected]> - ade2f1670859abf5cad4f4a3f0240566ec8c3a0f [LS-989][Delta] Add checkpointType, numColumns and numFla... by Sumeet Varma <[email protected]> - 62c938e3eb626e04000e0acd4d37953a24d3c52d [LS-985][Delta] Add detailed mismatch reasons for Increme... by Sumeet Varma <[email protected]> - bfda87427e6c9c019d83ee108e8e70a5e5950d8e [SPARK-48576][SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE by Uros Bojanic <[email protected]> - 2ff6142a85ef692a055071fa722ac10caeaf685f [LC-4301] Fix the predicates used by metadata queries by Wei Luo <[email protected]> - 6340541a0bbc4c9d71e04aff99a3cb21720dd975 [DELTA-OSS-AUTO][SC-168789][Spark] Optimize batching / in... by Adam Binford <[email protected]> - 1c691ed0e5df46402a4ebe44b16b89b748f2915e [SC-162602] Add per-kdtree scan stats (only scanned files) by Eric Liang <[email protected]> - 895a09ff141e810f87d2c5b353d3963d8ac00586 [SC-167647] map lookup for createPhysicalSchema by andrewxue-db <[email protected]> - d89308d177f5e269e7cb6bdfbe6c0978bd9d1cd2 [Delta OSS] Fix compile error on Spark master due to Pars... by Jiaheng Tang <[email protected]> - c7ba28b7e35319db1a41d5252506d82b39933ec2 [STATS-356] Improve recomputeDeltaFileStats logic with ME... by Pat Sukprasert <[email protected]> - 750eeac292c99c0003b6011038b99e2ee92083db [LC-4499] All Delta Commands should have all operation me... by sabir-akhadov <[email protected]> - ebe3e3e296c593fa6e5020e8127b410cd9d4c4c6 [LC-3942] Add historical schema read compatibility checks... by Nikola Mandic <[email protected]> - 20dbe80816413dcfa565777fdd9878f4f00c1af2 [SC-167257][Auto-Cluster-Key-Selection] Show operational ... by Supun Nakandala <[email protected]> - 2460f7604cba40783e44a43cbeb4bc39956427d8 [ES-1155687][LC-812] Make DPO track read files by Christos Stavrakakis <[email protected]> - 59e3a4f57aaa7a7298e978ace615ff04b4507418 [LC-4484] Match RangeBloomFilterMightContain expressions ... by Tom van Bussel <[email protected]> - 01c09b8556e57b2fa94c3e6f850b2ae426364976 [ES-1077008][LC-4411][BEHAVE-32] Add a Delta config to en... by Paddy Xu <[email protected]> - 7bd2395d454a1f2383d5b58f16e7de643cacae8b [ES-1154374][TEST-ONLY] Fix INSERT with delta view tests ... by Johan Lasperas <[email protected]> - d8f90b3560e6fe8d95bf757538660a995cf185a0 [LC-4323] Delta metadata queries should be QPL tagged wit... by sabir-akhadov <[email protected]> - e673c6972eb230234b8206309dcffc135af6387e [LC-4132][SQL] Disable changing collation of clustering c... by Nikola Mandic <[email protected]> - 9c34481f90d4e526a0f5c3671e5e59aae969c44c [LC-3259][Delta][MC][Pt. 3] Creates separate CommitStore ... by Lukas Rupprecht <[email protected]> - 0249b90ffe247bc689586891e90f81bd0568ad6d [LC-4564][LC-4565][Liquid OSS] Support show tblproperties... by Jiaheng Tang <[email protected]> - 24a50e9f98a540ba72c9a7a585ddad0a8196ee92 [LC-4548] Support Spark Structured Logging in Delta by Jiaheng Tang <[email protected]> (And 402 more changes) GitOrigin-RevId: ddd6a8ee676b3dab729ea720182a022e0977567e
1 parent 6e3e4ef commit a442563

File tree

64 files changed

+238
-6789
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+238
-6789
lines changed

PROTOCOL.md

+2-4
Original file line numberDiff line numberDiff line change
@@ -477,9 +477,7 @@ The following is an example `remove` action.
477477
```
478478

479479
### Add CDC File
480-
The `cdc` action is used to add a [file](#change-data-files) containing only the data that was changed as part of the transaction. The `cdc` action is allowed to add a [Data File](#data-files) that is also added by an `add` action, when it does not contain any copied rows and the `_change_type` column is filled for all rows.
481-
482-
When change data readers encounter a `cdc` action in a particular Delta table version, they must read the changes made in that version exclusively using the `cdc` files. If a version has no `cdc` action, then the data in `add` and `remove` actions are read as inserted and deleted rows, respectively.
480+
The `cdc` action is used to add a [file](#change-data-files) containing only the data that was changed as part of the transaction. When change data readers encounter a `cdc` action in a particular Delta table version, they must read the changes made in that version exclusively using the `cdc` files. If a version has no `cdc` action, then the data in `add` and `remove` actions are read as inserted and deleted rows, respectively.
483481

484482
The schema of the `cdc` action is as follows:
485483

@@ -525,7 +523,7 @@ Specifically, to read the row-level changes made in a version, the following str
525523

526524
##### Note for non-change data readers
527525

528-
In a table with Change Data Feed enabled, the data Parquet files referenced by `add` and `remove` actions are allowed to contain an extra column `_change_type`. This column is not present in the table's schema. When accessing these files, readers should disregard this column and only process columns defined within the table's schema.
526+
In a table with Change Data Feed enabled, the data Parquet files referenced by `add` and `remove` actions are allowed to contain an extra column `_change_type`. This column is not present in the table's schema and will consistently have a `null` value. When accessing these files, readers should disregard this column and only process columns defined within the table's schema.
529527

530528
### Transaction Identifiers
531529
Incremental processing systems (e.g., streaming systems) that track progress using their own application-specific versions need to record what progress has been made, in order to avoid duplicating data in the face of failures and retries during a write.

benchmarks/build.sbt

+1-1
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
*/
1616

1717
name := "benchmarks"
18-
scalaVersion := "2.12.18"
18+
scalaVersion := "2.12.17"
1919

2020
lazy val root = (project in file("."))
2121
.settings(

build.sbt

-110
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,6 @@
1717
// scalastyle:off line.size.limit
1818

1919
import java.nio.file.Files
20-
21-
import sbt.internal.inc.Analysis
22-
import sbtprotoc.ProtocPlugin.autoImport._
23-
24-
import xsbti.compile.CompileAnalysis
25-
2620
import Checkstyle._
2721
import Mima._
2822
import Unidoc._
@@ -45,8 +39,6 @@ val LATEST_RELEASED_SPARK_VERSION = "3.5.0"
4539
val SPARK_MASTER_VERSION = "4.0.0-SNAPSHOT"
4640
val sparkVersion = settingKey[String]("Spark version")
4741
spark / sparkVersion := getSparkVersion()
48-
connectCommon / sparkVersion := getSparkVersion()
49-
connectServer / sparkVersion := getSparkVersion()
5042

5143
// Dependent library versions
5244
val defaultSparkVersion = LATEST_RELEASED_SPARK_VERSION
@@ -66,9 +58,6 @@ val hadoopVersionForHive2 = "2.7.2"
6658
val hive2Version = "2.3.3"
6759
val tezVersionForHive2 = "0.8.4"
6860

69-
val protoVersion = "3.25.1"
70-
val grpcVersion = "1.62.2"
71-
7261
scalaVersion := default_scala_version.value
7362

7463
// crossScalaVersions must be set to Nil on the root project
@@ -198,105 +187,6 @@ def crossSparkSettings(): Seq[Setting[_]] = getSparkVersion() match {
198187
)
199188
}
200189

201-
def runTaskOnlyOnSparkMaster[T](
202-
task: sbt.TaskKey[T],
203-
taskName: String,
204-
projectName: String,
205-
emptyValue: => T): Def.Initialize[Task[T]] = {
206-
if (getSparkVersion() == SPARK_MASTER_VERSION) {
207-
Def.task(task.value)
208-
} else {
209-
Def.task {
210-
// scalastyle:off println
211-
println(s"Project $projectName: Skipping `$taskName` as Spark version " +
212-
s"${getSparkVersion()} does not equal $SPARK_MASTER_VERSION.")
213-
// scalastyle:on println
214-
emptyValue
215-
}
216-
}
217-
}
218-
219-
lazy val connectCommon = (project in file("spark-connect/common"))
220-
.settings(
221-
name := "delta-connect-common",
222-
commonSettings,
223-
crossSparkSettings(),
224-
releaseSettings,
225-
Compile / compile := runTaskOnlyOnSparkMaster(
226-
task = Compile / compile,
227-
taskName = "compile",
228-
projectName = "delta-connect-common",
229-
emptyValue = Analysis.empty.asInstanceOf[CompileAnalysis]
230-
).value,
231-
Test / test := runTaskOnlyOnSparkMaster(
232-
task = Test / test,
233-
taskName = "test",
234-
projectName = "delta-connect-common",
235-
emptyValue = ()).value,
236-
publish := runTaskOnlyOnSparkMaster(
237-
task = publish,
238-
taskName = "publish",
239-
projectName = "delta-connect-common",
240-
emptyValue = ()).value,
241-
libraryDependencies ++= Seq(
242-
"io.grpc" % "protoc-gen-grpc-java" % grpcVersion asProtocPlugin(),
243-
"io.grpc" % "grpc-protobuf" % grpcVersion,
244-
"io.grpc" % "grpc-stub" % grpcVersion,
245-
"com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf",
246-
"javax.annotation" % "javax.annotation-api" % "1.3.2",
247-
248-
"org.apache.spark" %% "spark-connect-common" % sparkVersion.value % "provided",
249-
),
250-
PB.protocVersion := protoVersion,
251-
Compile / PB.targets := Seq(
252-
PB.gens.java -> (Compile / sourceManaged).value,
253-
PB.gens.plugin("grpc-java") -> (Compile / sourceManaged).value
254-
),
255-
)
256-
257-
lazy val connectServer = (project in file("spark-connect/server"))
258-
.dependsOn(connectCommon % "compile->compile;test->test;provided->provided")
259-
.dependsOn(spark % "compile->compile;test->test;provided->provided")
260-
.settings(
261-
name := "delta-connect-server",
262-
commonSettings,
263-
releaseSettings,
264-
Compile / compile := runTaskOnlyOnSparkMaster(
265-
task = Compile / compile,
266-
taskName = "compile",
267-
projectName = "delta-connect-server",
268-
emptyValue = Analysis.empty.asInstanceOf[CompileAnalysis]
269-
).value,
270-
Test / test := runTaskOnlyOnSparkMaster(
271-
task = Test / test,
272-
taskName = "test",
273-
projectName = "delta-connect-server",
274-
emptyValue = ()
275-
).value,
276-
publish := runTaskOnlyOnSparkMaster(
277-
task = publish,
278-
taskName = "publish",
279-
projectName = "delta-connect-server",
280-
emptyValue = ()
281-
).value,
282-
crossSparkSettings(),
283-
libraryDependencies ++= Seq(
284-
"com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf",
285-
286-
"org.apache.spark" %% "spark-hive" % sparkVersion.value % "provided",
287-
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "provided",
288-
"org.apache.spark" %% "spark-core" % sparkVersion.value % "provided",
289-
"org.apache.spark" %% "spark-catalyst" % sparkVersion.value % "provided",
290-
"org.apache.spark" %% "spark-connect" % sparkVersion.value % "provided",
291-
292-
"org.apache.spark" %% "spark-catalyst" % sparkVersion.value % "test" classifier "tests",
293-
"org.apache.spark" %% "spark-core" % sparkVersion.value % "test" classifier "tests",
294-
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "test" classifier "tests",
295-
"org.apache.spark" %% "spark-hive" % sparkVersion.value % "test" classifier "tests",
296-
"org.apache.spark" %% "spark-connect" % sparkVersion.value % "test" classifier "tests",
297-
),
298-
)
299-
300190
lazy val spark = (project in file("spark"))
301191
.dependsOn(storage)
302192
.enablePlugins(Antlr4Plugin)

dev/check-delta-connect-codegen-python.py

-83
This file was deleted.

dev/delta-connect-gen-protos.sh

-104
This file was deleted.

dev/pyproject.toml

-23
This file was deleted.

dev/requirements.txt

-9
This file was deleted.

dev/tox.ini

+1-1
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,6 @@
1717
[pycodestyle]
1818
ignore=E226,E241,E305,E402,E722,E731,E741,W503,W504
1919
max-line-length=100
20-
exclude=cloudpickle.py,heapq3.py,shared.py,python/docs/conf.py,work/*/*.py,python/.eggs/*,dist/*,*python/delta/connect*
20+
exclude=cloudpickle.py,heapq3.py,shared.py,python/docs/conf.py,work/*/*.py,python/.eggs/*,dist/*
2121
[pydocstyle]
2222
ignore=D100,D101,D102,D103,D104,D105,D106,D107,D200,D201,D202,D203,D204,D205,D206,D207,D208,D209,D210,D211,D212,D213,D214,D215,D300,D301,D302,D400,D401,D402,D403,D404,D405,D406,D407,D408,D409,D410,D411,D412,D413,D414,D415,D417

0 commit comments

Comments
 (0)