Skip to content

Commit 44c01e9

Browse files
committed
Squashed commits:
- d17584edb7d55dba9d45245d26912d531e8fc172 WIP by zhipeng.mao <[email protected]> - 11d61890a1e26b8380e7cae9b1d750acf5a14f96 [ES-1177667][TEST-ONLY] Remove timeout source from withTe... by Johan Lasperas <[email protected]> - bc90067254419651895419cd0228d3cd5cbe4d8d [ES-1109125] Report accurate type when checking Iceberg c... by Hao Jiang <[email protected]> - 2bf266859083ecfc1d46240638b7c421c24285ad [SC-160780][FOLLOW-UP] Add SQLConf flag for #96237 change by Vitalii Li <[email protected]> - 23f0eb9982898acf167f3d3eb4d1905133dff27e [LC-4740][CoordinatedCommits] Extend NewTransactionSuite ... by Jun <[email protected]> - 66a20cfed0eaf369a8a2b35e709a23311e0a19f3 [LC-3943] Forbid automatic protocol upgrade when using co... by Stefan Kandic <[email protected]> - 876af9aa7e6de42c501b89eae58111e2df7ba8cc [LC-4491] Shade Hudi into DBR by Annie Wang <[email protected]> - 673ca7fe43e3110c67143642f3a77f4f5106bc23 [LC-4754][Delta][MC] Fix error message formatting in Delt... by Sumeet Varma <[email protected]> - 321e72867fdc98d501963bccaf17f8577bc4c445 [LC-4192][Delta][MC] Use Coordinated Commits Properties f... by Sumeet Varma <[email protected]> - 84fe62011e78b5528adb70cc06a1be906ce97b02 [LC-4263][Delta] Add usage logging to UCCommitCooridnator... by Yumingxuan Guo <[email protected]> - 4cde1a20dddc7408beb251cfbf62baed7bc29940 [LC-4392][Auto-Clustering-Key-Selection] Add additional t... by parimarjannegi-db <[email protected]> - 7b5a138dd7388ac59a08528c0e27c622cf0bb2f3 [LC-4549] Delta structured logging migration for remainin... by Jiaheng Tang <[email protected]> - c0e90e6bd23cc181e900ee4633b2eb213a913f9d [LC-4131] Validate that CDF works with collations by Stefan Kandic <[email protected]> - c472de831911ccd1cc0fd7b23e1fd417cecde195 [Delta][Copybara] Introduce `// EDGE-NO-INHERITANCE` tag by Paddy Xu <[email protected]> - 78f75a2bb3a05ad37b59e42eb35f4b61cb1c7acf [DELTA-OSS-AUTO][SC-170469][Hudi] Flesh out tests and upd... by Annie Wang <[email protected]> - 49d07ae43b3dbc09c2679e35a4c751b99f53c60a [ES-1151956] Fixes for "show tables extended" by rayman7718 <[email protected]> - c35f0fdd59377faae33aa06f477fe5b066babe82 [ES-1159798][LC-4729][BEHAVE-74] Fix CDC Commit Timestamp... by Thang Long Vu <[email protected]> - cfd343651f3c00a6f551f3e2ef5180b80f2dd47b [LC-4661] Deprecate tableVersion field in type widening m... by Johan Lasperas <[email protected]> - 7014ad1ec51af3f55c2f9713b032c6c9d32ff755 [AUTO][SC-170122][SPARK-48697][LC-4703][SQL] Add collatio... by Stefan Kandic <[email protected]> - 35628a58efa0c704bf2e2ddfcba833ff9b40f07a [Delta][CC] Add integration test for DROP/UNDROP. by Yumingxuan Guo <[email protected]> - 3351fa6531e3e370ebe41f931c3946fa00206d5f [LC-4693][Uniform] Make refresh on UFI no-op when there's... by ChengJi-db <[email protected]> - 10d743671c381ea09131f3519be7fc90c460db75 [LC-4143][Delta][CC] Adds asynchronous backfilling in UC ... by Lukas Rupprecht <[email protected]> - d55d080248832d881d33e52d9b1d9efd66a197bf [LC-4698] add logging to UniForm Iceberg conversion num o... by Fred Storage Liu <[email protected]> - f992112f04b14423f323d4f171040ec8cca1b3ad [LC-4499] All Delta Commands should have all operation me... by sabir-akhadov <[email protected]> - 7d0c410dba226ae191f51280c382b272fba459b1 [LC-3778] Log duration of DeltaSource operations by Tom van Bussel <[email protected]> - e5e29b59cab4218ed77903079eb2b08216e6f982 Import changes from OSS PR delta-io#3326 by Annie Wang <[email protected]> - 4c08927c6eee0da4a9a00795b36a4b734f2c8a04 Import changes from OSS PR delta-io#3323 by Annie Wang <[email protected]> - 1fd8ee89f3b7398db6f13486bd42ca2535023976 Import changes from OSS PR delta-io#3320 by Annie Wang <[email protected]> - db210bb4f2bd0d682863c99ca4c385ed0d0d0118 [LC-4692] Regular delta min/max stats should always use U... by Stefan Kandic <[email protected]> - e873cab62acf71139d89f4019f59fb5aeec81f14 Import changes from OSS PR delta-io#3309 by Annie Wang <[email protected]> - 6d1aca32ebc1d9009e77ca26f4fd0c73ba8e7f6e Import changes from OSS PR delta-io#3310 by Annie Wang <[email protected]> - 0070442d167c53884e083127eeaa7e69c71f28d9 Import changes from OSS PR delta-io#3311 by Annie Wang <[email protected]> - 8b8df4f43b5a85d1626ea4a2ce612b34ea67544d [ES-1089258][ES-1043275][Delta] Creates unresolved tables... by Lukas Rupprecht <[email protected]> - 602f7e429fd3c73d43c24ac7f231a9a10bfe8830 [LC-4043] Allow auto REFRESH on UniForm tables by Hao Jiang <[email protected]> - 26755b6dba8caeca1b7c8797eb634abcb5532e35 [ES-1154731] Disable managed Hive Metastore related Conve... by Ming DAI <[email protected]> - 568c954c44eafe752d6c614ca36dc3f2a0a651d4 [ES-1150026][Delta] Handle concurrent CREATE TABLE IF NOT... by Sumeet Varma <[email protected]> - 195a5fbcd168a0d1e7e992dfdb226b3a9e21a6a3 [LC-4657][Delta][MC] Throw exception when additional list... by Sumeet Varma <[email protected]> - 438d79963e76478a641e256ee53edca557024c99 [LC-4524][Delta][MC] Fetch unbackfilled deltas to detect ... by Sumeet Varma <[email protected]> - 3c6800ac28f998e3183a54d4ff6cf058607cad16 [LC-4996][BEHAVE-60] Introduce stable type widening table... by Johan Lasperas <[email protected]> - c0d24087192bcbbd80f42fa83e2229482c877d66 [ES-1154366][ES-1154367] Fix ConvertToDeltaSuite in serve... by Ming DAI <[email protected]> - 6b7e4d0f9004d044fd88248c4b024d61e1d4b91d [Liquid] Improve error message when clustering column not... by Chirag Singh <[email protected]> - 75e21adf355c43bbafeb6dffefe7627614ca6338 [LC-4242] History truncation/validation support for write... by Andreas Chatzistergiou <[email protected]> - 71aec8cc7f136db13de264f661446526a9e7a2c2 [ES-1162704] Disable some Python Delta tests in serverless by Christos Stavrakakis <[email protected]> - 96b6ce441e287e9b008261844017eb9d09ffe692 update delta-sharing client to version 1.1.0 by Jade Wang <[email protected]> - 510e7d69ce180c869031affff075bc163077fa82 [SC-160780] Support Delta API for DML commands when table... by Vitalii Li <[email protected]> - 562322d9ab5ecf7fb237fa0e950470d83a252ed1 [LC-4549] Delta structured logging migration by Jiaheng Tang <[email protected]> - 850e368261d2cb89cc732139c8ee1574978366d5 [LC-4526] Support snapshot expiration for uniform iceberg... by ChengJi-db <[email protected]> - 6e11dd7f11e79a429068a6daa59d4e201fa894c8 [ES-1151640] Break down ImplicitDMLCastingSuite by Lars Kroll <[email protected]> - 837d86463cef6bbb1e7571f06431cedd67b76c7e [ES-1154733][ES-1154732][ES-1154381][ES-1154382][TEST-ONL... by Paddy Xu <[email protected]> - 51bfb0c60bb2b2d64f01b7c5f96187e65b0ad4b1 [LC-4534] Add ThreadLocal trait for underlying execution ... by leonwind-db <[email protected]> - ade2f1670859abf5cad4f4a3f0240566ec8c3a0f [LS-989][Delta] Add checkpointType, numColumns and numFla... by Sumeet Varma <[email protected]> - 62c938e3eb626e04000e0acd4d37953a24d3c52d [LS-985][Delta] Add detailed mismatch reasons for Increme... by Sumeet Varma <[email protected]> - bfda87427e6c9c019d83ee108e8e70a5e5950d8e [SPARK-48576][SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE by Uros Bojanic <[email protected]> - 2ff6142a85ef692a055071fa722ac10caeaf685f [LC-4301] Fix the predicates used by metadata queries by Wei Luo <[email protected]> - 6340541a0bbc4c9d71e04aff99a3cb21720dd975 [DELTA-OSS-AUTO][SC-168789][Spark] Optimize batching / in... by Adam Binford <[email protected]> - 1c691ed0e5df46402a4ebe44b16b89b748f2915e [SC-162602] Add per-kdtree scan stats (only scanned files) by Eric Liang <[email protected]> - 895a09ff141e810f87d2c5b353d3963d8ac00586 [SC-167647] map lookup for createPhysicalSchema by andrewxue-db <[email protected]> - d89308d177f5e269e7cb6bdfbe6c0978bd9d1cd2 [Delta OSS] Fix compile error on Spark master due to Pars... by Jiaheng Tang <[email protected]> - c7ba28b7e35319db1a41d5252506d82b39933ec2 [STATS-356] Improve recomputeDeltaFileStats logic with ME... by Pat Sukprasert <[email protected]> - 750eeac292c99c0003b6011038b99e2ee92083db [LC-4499] All Delta Commands should have all operation me... by sabir-akhadov <[email protected]> - ebe3e3e296c593fa6e5020e8127b410cd9d4c4c6 [LC-3942] Add historical schema read compatibility checks... by Nikola Mandic <[email protected]> - 20dbe80816413dcfa565777fdd9878f4f00c1af2 [SC-167257][Auto-Cluster-Key-Selection] Show operational ... by Supun Nakandala <[email protected]> - 2460f7604cba40783e44a43cbeb4bc39956427d8 [ES-1155687][LC-812] Make DPO track read files by Christos Stavrakakis <[email protected]> - 59e3a4f57aaa7a7298e978ace615ff04b4507418 [LC-4484] Match RangeBloomFilterMightContain expressions ... by Tom van Bussel <[email protected]> - 01c09b8556e57b2fa94c3e6f850b2ae426364976 [ES-1077008][LC-4411][BEHAVE-32] Add a Delta config to en... by Paddy Xu <[email protected]> - 7bd2395d454a1f2383d5b58f16e7de643cacae8b [ES-1154374][TEST-ONLY] Fix INSERT with delta view tests ... by Johan Lasperas <[email protected]> - d8f90b3560e6fe8d95bf757538660a995cf185a0 [LC-4323] Delta metadata queries should be QPL tagged wit... by sabir-akhadov <[email protected]> - e673c6972eb230234b8206309dcffc135af6387e [LC-4132][SQL] Disable changing collation of clustering c... by Nikola Mandic <[email protected]> - a34f0d7e23c1dcf55310be6bf4148bd4ad3839e1 [ES-1154398][TEST-ONLY][WARMFIX] IdentityColumnAdmissionS... by Carmen Kwan <[email protected]> - 9c34481f90d4e526a0f5c3671e5e59aae969c44c [LC-3259][Delta][MC][Pt. 3] Creates separate CommitStore ... by Lukas Rupprecht <[email protected]> - 0249b90ffe247bc689586891e90f81bd0568ad6d [LC-4564][LC-4565][Liquid OSS] Support show tblproperties... by Jiaheng Tang <[email protected]> - 24a50e9f98a540ba72c9a7a585ddad0a8196ee92 [LC-4548] Support Spark Structured Logging in Delta by Jiaheng Tang <[email protected]> - f4b3d9042b1ffebc8ed78a3fe2634e6672213cc0 [ES-1154365][ES-1154357] Disable test cases not supported... by Chirag Singh <[email protected]> - a39c357817393943cc2997bbc9f020b8e02c14a8 [Liquid][LC-3844] Do not attempt lazy clustering on files... by Chirag Singh <[email protected]> - b8ff879f8471592e7d08312be088152c575ecb3f [Liquid][LC-4335][LC-4314] Execute eager clustering batch... by Chirag Singh <[email protected]> - 1bce3b75784df6960e1c8261dea5c97f86f0e3bd [Delta] Rename managed commit to coordinated commits by Dhruv Arya <[email protected]> - f727c3e6717ed8f235560055ade0848a2333c6dd [DELTA-OSS-AUTO][SC-167757][SPARK][MINOR] Code cleanup (r... by Jacek Laskowski <[email protected]> - edf995cf281c10917d747620a4edc8ab37bf032d [Liquid][LC-2605] Change verifyClusteringColumns to not p... by Chirag Singh <[email protected]> - 04129352860bfd2973d2717a69b793a4ae880217 [BEHAVE-43][SC-168642][Spark] Support dropping the CHECK ... by Tom van Bussel <[email protected]> - 863b8c7f7af9d720949a9b098f89f8bc7e3a1279 [STATS-368] Make statistics on load work with tables with... by Satya Valluri <[email protected]> - 75de39d97097df2b75596073f2df19d83fda4ac0 [SC-168457] Also track auto table and clustering column s... by Eric Liang <[email protected]> - 635eb6b63cbc3809b51ceaf023334bcbe5158ecd [LC-4146][MAC][Liquid-PO Integration] Handle large tables... by Stella Wan <[email protected]> - 493c1415599a7910da34b6a37e3023e6cf95ce25 [DELTA-OSS-AUTO][SC-167314] Update flink mima checks by Qianru Lao <[email protected]> - c8ce5d5360467dc9c01e851a7ed539f3d0792096 [ES-1115071][LC-4301][Liquid] Make metadata query optimiz... by Wei Luo <[email protected]> - 261d280360847c040db11e4e1ac31dd9cfa58e69 [LC-4507] Run delete and insert steps in parallel during ... by Christos Stavrakakis <[email protected]> - aae17ff4806845f0f50e0fbd61808ed4e67b07f9 [DBRRM-1031][SC-168577][ES-1145536] Fix Kryo serializatio... by Josh Rosen <[email protected]> - 320d96e453fb6de07460ac524e6ed6843b667d37 [DELTA-OSS-AUTO][SC-168636]Improve documentation in PROTO... by Andreas Chatzistergiou <[email protected]> - 9f7979357c735e5b0be9158eccf16d5fc19b76a6 [LS-986][Delta] Handle multi-byte UTF-8 characters while ... by Sumeet Varma <[email protected]> - c447349ec894c1a54ac4118dc4d470dc12c684df [ES-1151654] Improve the running time of DeletionVectorsS... by Andreas Chatzistergiou <[email protected]> - 307cd1089a3389a6a9a0aa099529255761e7e107 [AUTOSTATS][ES-1147462] fallback to old style column sele... by mohamedzait <[email protected]> - af36e130f5b2c6c005cdaa7bc14d214c408e650e [ES-1133698] Fix the issue of not using the correct refre... by Lin Zhou <[email protected]> - d61bff57c8b4966d1e79c5e17520e0c03d1eb7ab [SC-162599] Include number of incremental transactions in... by Ming DAI <[email protected]> - 1daf0511dcbc95a9564b8ba5f5a9aca24dd81ebb [LC-4523] Add tracing for slow Delta operations by Christos Stavrakakis <[email protected]> - dc119c92f185e943490c790e26e541a4646496db [LC-4494][DELTA][MC] Add usage logs related to managed co... by Prakhar Jain <[email protected]> - 8758faef118a5c2c1cc9794fb724ba7f996390b0 [AUTOSTATS][ES-1147462] do not use auto stats when Row ID... by mohamedzait <[email protected]> - 9f83c9c342c2e47a6c44d694b27d0cbee66f5ba2 [LS-984][Delta] Add usage logs for exhaustive reasons beh... by Sumeet Varma <[email protected]> - 6e6297437231b7a460a0cc0d23de616cf171a2ee [LC-4489][MC] Make naming of manage commit consistent by Dhruv Arya <[email protected]> - 4641410e78e3ca719544ef77054c83df2ba72e7b [LC-3022][Liquid OSS] Support create table from external ... by Jiaheng Tang <[email protected]> - 8ee182fff5c5a309adde44799404313f2a03f545 [LC-4485][Delta][MC] Remove deprecated FileNames.deltaFil... by Sumeet Varma <[email protected]> - db6a256db943247ac12b78d48de9d83d7aa0a504 [SC-139180] Fetch globalTempDatabase name directly withou... by Weitao Wen <[email protected]> (And 4593 more changes) GitOrigin-RevId: d17584edb7d55dba9d45245d26912d531e8fc172
0 parents  commit 44c01e9

File tree

1,011 files changed

+288666
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

1,011 files changed

+288666
-0
lines changed

.gitattributes

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
*.bat text eol=crlf
2+
*.cmd text eol=crlf
3+
*.bin binary

.gitignore

+131
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
*#*#
2+
*.#*
3+
*.iml
4+
*.ipr
5+
*.iws
6+
*.pyc
7+
*.pyo
8+
*.swp
9+
*~
10+
.DS_Store
11+
.ammonite
12+
.bloop
13+
.bsp
14+
.cache
15+
.classpath
16+
.ensime
17+
.ensime_cache/
18+
.ensime_lucene
19+
.generated-mima*
20+
.idea/
21+
.idea_modules/
22+
.metals
23+
.project
24+
.pydevproject
25+
.scala_dependencies
26+
.settings
27+
/lib/
28+
R-unit-tests.log
29+
R/unit-tests.out
30+
R/cran-check.out
31+
R/pkg/vignettes/sparkr-vignettes.html
32+
R/pkg/tests/fulltests/Rplots.pdf
33+
build/*.jar
34+
build/apache-maven*
35+
build/scala*
36+
build/zinc*
37+
cache
38+
checkpoint
39+
conf/*.cmd
40+
conf/*.conf
41+
conf/*.properties
42+
conf/*.sh
43+
conf/*.xml
44+
conf/java-opts
45+
dependency-reduced-pom.xml
46+
derby.log
47+
dev/create-release/*final
48+
dev/create-release/*txt
49+
dev/pr-deps/
50+
dist/
51+
docs/_site
52+
docs/api
53+
sql/docs
54+
sql/site
55+
lib_managed/
56+
lint-r-report.log
57+
log/
58+
logs/
59+
metals.sbt
60+
out/
61+
project/boot/
62+
project/build/target/
63+
project/plugins/lib_managed/
64+
project/plugins/project/build.properties
65+
project/plugins/src_managed/
66+
project/plugins/target/
67+
python/lib/pyspark.zip
68+
python/deps
69+
docs/python/_static/
70+
docs/python/_templates/
71+
docs/python/_build/
72+
python/test_coverage/coverage_data
73+
python/test_coverage/htmlcov
74+
python/pyspark/python
75+
reports/
76+
scalastyle-on-compile.generated.xml
77+
scalastyle-output.xml
78+
scalastyle.txt
79+
spark-*-bin-*.tgz
80+
spark-tests.log
81+
src_managed/
82+
streaming-tests.log
83+
target/
84+
unit-tests.log
85+
work/
86+
docs/.jekyll-metadata
87+
88+
# For Hive
89+
TempStatsStore/
90+
metastore/
91+
metastore_db/
92+
sql/hive-thriftserver/test_warehouses
93+
warehouse/
94+
spark-warehouse/
95+
96+
# For R session data
97+
.RData
98+
.RHistory
99+
.Rhistory
100+
*.Rproj
101+
*.Rproj.*
102+
103+
.Rproj.user
104+
105+
**/src/main/resources/js
106+
107+
# For SBT
108+
.jvmopts
109+
sbt-launch-*.jar
110+
111+
# For Python linting
112+
pep8*.py
113+
pycodestyle*.py
114+
115+
# For IDE settings
116+
.vscode
117+
118+
# For Terraform
119+
**/.terraform/*
120+
*.tfstate
121+
*.tfstate.*
122+
crash.log
123+
crash.*.log
124+
*.tfvars
125+
*.tfvars.json
126+
override.tf
127+
override.tf.json
128+
*_override.tf
129+
*_override.tf.json
130+
.terraformrc
131+
.terraform.rc

.sbtopts

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
-J-Xmx4G

CODE_OF_CONDUCT.md

+79
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Delta Lake Code of Conduct
2+
3+
## Our Pledge
4+
5+
In the interest of fostering an open and welcoming environment, we as
6+
contributors and maintainers pledge to making participation in our project and
7+
our community a harassment-free experience for everyone, regardless of age, body
8+
size, disability, ethnicity, sex characteristics, gender identity and expression,
9+
level of experience, education, socio-economic status, nationality, personal
10+
appearance, race, religion, or sexual identity and orientation.
11+
12+
## Our Standards
13+
14+
Examples of behavior that contributes to creating a positive environment
15+
include:
16+
17+
* Using welcoming and inclusive language
18+
* Being respectful of differing viewpoints and experiences
19+
* Gracefully accepting constructive criticism
20+
* Focusing on what is best for the community
21+
* Showing empathy towards other community members
22+
23+
Examples of unacceptable behavior by participants include:
24+
shipit
25+
* The use of sexualized language or imagery and unwelcome sexual attention or
26+
advances
27+
* Trolling, insulting/derogatory comments, and personal or political attacks
28+
* Public or private harassment
29+
* Publishing others' private information, such as a physical or electronic
30+
address, without explicit permission
31+
* Other conduct which could reasonably be considered inappropriate in a
32+
professional setting
33+
shipit
34+
## Our Responsibilities
35+
36+
Project maintainers are responsible for clarifying the standards of acceptable
37+
behavior and are expected to take appropriate and fair corrective action in
38+
response to any instances of unacceptable behavior.
39+
40+
Project maintainers have the right and responsibility to remove, edit, or
41+
reject comments, commits, code, wiki edits, issues, and other contributions
42+
that are not aligned to this Code of Conduct, or to ban temporarily or
43+
permanently any contributor for other behaviors that they deem inappropriate,
44+
threatening, offensive, or harmful.
45+
46+
## Scope
47+
48+
This Code of Conduct applies both within project spaces and in public spaces
49+
when an individual is representing the project or its community. Examples of
50+
representing a project or community include using an official project e-mail
51+
address, posting via an official social media account, or acting as an appointed
52+
representative at an online or offline event. Representation of a project may be
53+
further defined and clarified by project maintainers.
54+
55+
## Enforcement
56+
57+
Instances of abusive, harassing, or otherwise unacceptable behavior may be
58+
reported by contacting the Technical Steering Committee defined [here](https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md#governance). All
59+
complaints will be reviewed and investigated and will result in a response that
60+
is deemed necessary and appropriate to the circumstances. The project team is
61+
obligated to maintain confidentiality with regard to the reporter of an incident.
62+
Further details of specific enforcement policies may be posted separately.
63+
64+
Project maintainers who do not follow or enforce the Code of Conduct in good
65+
faith may face temporary or permanent repercussions as determined by other
66+
members of the project's leadership.
67+
68+
## Attribution
69+
70+
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
71+
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
72+
73+
[homepage]: https://www.contributor-covenant.org
74+
75+
For answers to common questions about this code of conduct, see
76+
https://www.contributor-covenant.org/faq
77+
78+
## Linux Foundation Code of Conduct
79+
Your use is additionally subject to the [Linux Foundation Code of Conduct](https://lfprojects.org/policies/code-of-conduct/)

CONTRIBUTING.md

+75
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
We happily welcome contributions to Delta Lake. We use [GitHub Issues](/../../issues/) to track community reported issues and [GitHub Pull Requests ](/../../pulls/) for accepting changes.
2+
3+
# Governance
4+
Delta Lake is an independent open-source project and not controlled by any single company. To emphasize this we joined the [Delta Lake Project](https://community.linuxfoundation.org/delta-lake/) in 2019, which is a sub-project of the Linux Foundation Projects. Within the project, we make decisions based on [these rules](https://delta.io/pdfs/delta-charter.pdf).
5+
6+
Delta Lake is supported by a wide set of developers from over 50 organizations across multiple repositories. Since 2019, more than 190 developers have contributed to Delta Lake! The Delta Lake community is growing by leaps and bounds with more than 6000 members in the [Delta Users slack](https://go.delta.io/slack)).
7+
8+
For more information, please refer to the [founding technical charter](https://delta.io/pdfs/delta-charter.pdf).
9+
10+
# Communication
11+
- Before starting work on a major feature, please reach out to us via [GitHub](https://github.com/delta-io/delta/issues), [Slack](https://go.delta.io/slack), [email](https://groups.google.com/g/delta-users), etc. We will make sure no one else is already working on it and ask you to open a GitHub issue.
12+
- A "major feature" is defined as any change that is > 100 LOC altered (not including tests), or changes any user-facing behavior.
13+
- We will use the GitHub issue to discuss the feature and come to agreement.
14+
- This is to prevent your time being wasted, as well as ours.
15+
- The GitHub review process for major features is also important so that organizations with commit access can come to agreement on design.
16+
- If it is appropriate to write a design document, the document must be hosted either in the GitHub tracking issue, or linked to from the issue and hosted in a world-readable location. Examples of design documents include [sample 1](https://docs.google.com/document/d/16S7xoAmXpSax7W1OWYYHo5nZ71t5NvrQ-F79pZF6yb8), [sample 2](https://docs.google.com/document/d/1MJhmW_H7doGWY2oty-I78vciziPzBy_nzuuB-Wv5XQ8), and [sample 3](https://docs.google.com/document/d/19CU4eJuBXOwW7FC58uSqyCbcLTsgvQ5P1zoPOPgUSpI).
17+
- Specifically, if the goal is to add a new extension, please read the extension policy.
18+
- Small patches and bug fixes don't need prior communication. If you have identified a bug and have ways to solve it, please create an [issue](https://github.com/delta-io/delta/issues) or create a [pull request](https://github.com/delta-io/delta/pulls).
19+
- If you have an example code that explains a use case or a feature, create a pull request to post under [examples](https://github.com/delta-io/delta/tree/master/examples).
20+
21+
22+
# Coding style
23+
We generally follow the [Apache Spark Scala Style Guide](https://spark.apache.org/contributing.html).
24+
25+
# Sign your work
26+
The sign-off is a simple line at the end of the explanation for the patch. Your signature certifies that you wrote the patch or otherwise have the right to pass it on as an open-source patch. The rules are pretty simple: if you can certify the below (from developercertificate.org):
27+
28+
```
29+
Developer Certificate of Origin
30+
Version 1.1
31+
32+
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
33+
1 Letterman Drive
34+
Suite D4700
35+
San Francisco, CA, 94129
36+
37+
Everyone is permitted to copy and distribute verbatim copies of this
38+
license document, but changing it is not allowed.
39+
40+
41+
Developer's Certificate of Origin 1.1
42+
43+
By making a contribution to this project, I certify that:
44+
45+
(a) The contribution was created in whole or in part by me and I
46+
have the right to submit it under the open source license
47+
indicated in the file; or
48+
49+
(b) The contribution is based upon previous work that, to the best
50+
of my knowledge, is covered under an appropriate open source
51+
license and I have the right under that license to submit that
52+
work with modifications, whether created in whole or in part
53+
by me, under the same open source license (unless I am
54+
permitted to submit under a different license), as indicated
55+
in the file; or
56+
57+
(c) The contribution was provided directly to me by some other
58+
person who certified (a), (b) or (c) and I have not modified
59+
it.
60+
61+
(d) I understand and agree that this project and the contribution
62+
are public and that a record of the contribution (including all
63+
personal information I submit with it, including my sign-off) is
64+
maintained indefinitely and may be redistributed consistent with
65+
this project or the open source license(s) involved.
66+
```
67+
68+
Then you just add a line to every git commit message:
69+
70+
```
71+
Signed-off-by: Jane Smith <[email protected]>
72+
Use your real name (sorry, no pseudonyms or anonymous contributions.)
73+
```
74+
75+
If you set your `user.name` and `user.email` git configs, you can sign your commit automatically with `git commit -s`.

Dockerfile

+60
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#
2+
# Copyright (2021) The Delta Lake Project Authors.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
FROM ubuntu:focal-20221019
17+
18+
ENV DEBIAN_FRONTEND noninteractive
19+
ENV DEBCONF_NONINTERACTIVE_SEEN true
20+
21+
RUN apt-get update
22+
RUN apt-get install -y software-properties-common
23+
RUN apt-get install -y curl
24+
RUN apt-get install -y wget
25+
RUN apt-get install -y openjdk-8-jdk
26+
RUN apt-get install -y python3.8
27+
RUN apt-get install -y python3-pip
28+
RUN apt-get install -y git
29+
30+
# Upgrade pip. This is needed to use prebuilt wheels for packages cffi (dep of cryptography) and
31+
# cryptography. Otherwise, building wheels for these packages fails.
32+
RUN pip3 install --upgrade pip
33+
34+
RUN pip3 install pyspark==3.5.0
35+
36+
RUN pip3 install mypy==0.982
37+
38+
RUN pip3 install pydocstyle==3.0.0
39+
40+
RUN pip3 install pandas==1.0.5
41+
42+
RUN pip3 install pyarrow==8.0.0
43+
44+
RUN pip3 install numpy==1.20.3
45+
46+
RUN pip3 install importlib_metadata==3.10.0
47+
48+
RUN pip3 install cryptography==37.0.4
49+
50+
# We must install cryptography before twine. Else, twine will pull a newer version of
51+
# cryptography that requires a newer version of Rust and may break tests.
52+
RUN pip3 install twine==4.0.1
53+
54+
RUN pip3 install wheel==0.33.4
55+
56+
RUN pip3 install setuptools==41.0.1
57+
58+
# Do not add any non-deterministic changes (e.g., copy from files
59+
# from repo) in this Dockerfile, so that the docker image
60+
# generated from this can be reused across builds.

0 commit comments

Comments
 (0)