Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spark] Add Delta Connect Server Library #3136

Merged
merged 61 commits into from
Jun 3, 2024
Merged
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
c20918d
Add Delta Connect
longvu-db May 21, 2024
9d7f871
Fix merge conflicts
longvu-db May 21, 2024
c1c2f0f
Add generated python files
longvu-db May 21, 2024
08d70fa
Add yaml file
longvu-db May 21, 2024
4956d87
Update
longvu-db May 21, 2024
730d087
add lint ignore
longvu-db May 21, 2024
ba8a4e9
Remove unnecessary Delta repo
longvu-db May 21, 2024
05bcf0c
Rename connect to spark-connect
longvu-db May 21, 2024
7570e84
Update
longvu-db May 22, 2024
5dc923e
Update script
longvu-db May 22, 2024
2364564
Add the necessary mypy ignore stuff
longvu-db May 22, 2024
f4d4538
Remove debug
longvu-db May 22, 2024
84703f2
Remove ,
longvu-db May 22, 2024
1d09b50
Remove unnecessary
longvu-db May 22, 2024
e5f9bca
Undo debug
longvu-db May 22, 2024
a52ec9e
Add Delta Relation
longvu-db May 22, 2024
1095a0e
Try this
longvu-db May 23, 2024
afdb49c
Address comments
longvu-db May 23, 2024
d2b4d1b
Add mypy-protobuf
longvu-db May 24, 2024
d11295b
Swap places
longvu-db May 24, 2024
c3434b8
Try adding buf
longvu-db May 24, 2024
411a5c1
Add python3
longvu-db May 25, 2024
c359273
Address TD's comments
longvu-db May 25, 2024
adbf9ec
Try another way to install buf
longvu-db May 26, 2024
a515f4d
move directory again
longvu-db May 26, 2024
0807676
Try again
longvu-db May 26, 2024
53c1e64
Try again with PATH buf
longvu-db May 27, 2024
8cfc005
Remove Scala tests
longvu-db May 27, 2024
2db4ca9
Add black
longvu-db May 27, 2024
8728810
Disable other checks
longvu-db May 27, 2024
99dc990
Move back
longvu-db May 27, 2024
13a886a
Remove unnecessary comment
longvu-db May 27, 2024
c6a48f3
Add Delta Connect Server Library
longvu-db May 22, 2024
8e71c21
SBT
longvu-db May 23, 2024
44ec48d
Update
longvu-db May 23, 2024
a9f9009
Update
longvu-db May 23, 2024
55d961d
Update
longvu-db May 24, 2024
fbff246
Polish build.sbt
longvu-db May 24, 2024
80f25c6
Move to Spark Master only
longvu-db May 24, 2024
a2776f9
Update
longvu-db May 26, 2024
df0e6fd
Fix build.sbt
longvu-db May 26, 2024
7b3590c
Add back missing files
longvu-db May 26, 2024
215c69a
Fix copyright
longvu-db May 26, 2024
b8f5153
add back delta connect proto
longvu-db May 26, 2024
206d9f7
Update
longvu-db May 26, 2024
5286bbf
Move again
longvu-db May 27, 2024
64dc7b2
Enable again
longvu-db May 27, 2024
137af5f
Update
longvu-db May 27, 2024
e55e837
Remove Clone
longvu-db May 28, 2024
eec6765
Update
longvu-db May 28, 2024
22c5375
Resolve Merge conflicts
longvu-db May 29, 2024
abdba0f
Untab
longvu-db May 29, 2024
e2d5c8f
Resolve Andreas' comments
longvu-db May 29, 2024
e1a4fda
Add publish constraints
longvu-db May 29, 2024
5266b4b
Do not compile, test, or publish Spark Connect common/server projects…
scottsand-db May 29, 2024
f29a7f6
Polish
longvu-db May 30, 2024
9920d94
Nit
longvu-db May 30, 2024
c1c33de
Nit
longvu-db May 30, 2024
1de93c6
Add .
longvu-db May 30, 2024
2bfabcc
Remove dead variable
longvu-db May 30, 2024
5fb9047
Reorganize the imports
longvu-db Jun 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address TD's comments
longvu-db committed May 25, 2024
commit c3592731ada76fec1a04cba514f48b850686544c
20 changes: 11 additions & 9 deletions dev/check-delta-connect-codegen-python.py
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@
# limitations under the License.
#

# Utility for checking whether generated the Delta Connect Python protobuf codes are out of sync.
# Utility for checking whether generated the Delta Connect Python protobuf codes are in sync.
# usage: ./dev/check-delta-connect-codegen-python.py

import os
@@ -25,8 +25,8 @@
import tempfile
import subprocess

# Location of your Spark git development area
DELTA_HOME = os.environ.get("DELTA_HOME", os.getcwd())
# Location of your Delta git development area
DELTA_HOME = os.environ.get("DELTA_HOME", os.path.abspath(os.path.join(__file__, os.pardir, os.pardir)))


def fail(msg):
@@ -43,11 +43,13 @@ def run_cmd(cmd):


def check_connect_protos():
print(f"Start checking the generated codes in {DELTA_HOME}/python/delta/connect/proto/.")
generated_python_proto_codes_path = os.path.join(DELTA_HOME, "python", "delta", "connect", "proto")
print(f"Start checking the generated codes in {generated_python_proto_codes_path}")
generate_python_proto_codes_file = os.path.join(DELTA_HOME, "dev", "delta-connect-gen-protos.sh")
with tempfile.TemporaryDirectory() as tmp:
run_cmd(f"{DELTA_HOME}/dev/delta-connect-gen-protos.sh {tmp}")
run_cmd([generate_python_proto_codes_file, tmp])
result = filecmp.dircmp(
f"{DELTA_HOME}/python/delta/connect/proto/",
generated_python_proto_codes_path,
tmp,
ignore=["__init__.py", "__pycache__"],
)
@@ -70,11 +72,11 @@ def check_connect_protos():
success = False

if success:
print(f"Finish checking the generated codes in {DELTA_HOME}/python/delta/connect/proto/: SUCCESS")
print(f"Finish checking the generated codes in {generated_python_proto_codes_path}: SUCCESS")
else:
fail(
f"Generated files for {DELTA_HOME}/python/delta/connect/proto/ are out of sync! "
"Please run ./dev/delta-connect-gen-protos.sh"
f"Generated files for {generated_python_proto_codes_path} are out of sync! " +
f"Please run {generate_python_proto_codes_file}."
)


Loading