Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conditional call GetInfo CLI_ODBC_KEYWORDS to restore compatible with lower version Kyuubi and HS2 #6016

Closed
wants to merge 1 commit into from

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Jan 25, 2024

🔍 Description

Issue References 🔗

Currently, the Kyuubi Beeline is not compatible with HS2 2.3 and lower versions of Kyuubi, because the server can not recognize TGetInfoType.CLI_ODBC_KEYWORDS and then corrupts the socket.

Describe Your Solution 🔧

Conditional call GetInfo CLI_ODBC_KEYWORDS, when protocol version lower than TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V11, just throw an SQLFeatureNotSupportedException

Types of changes 🔖

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Use beeline to connect a HS2 2.3

Behavior Without This Pull Request ⚰️

root@hadoop-master1:/opt/kyuubi-dev# bin/beeline -u 'jdbc:hive2://localhost:10000/default'
Warn: Not find kyuubi environment file /etc/kyuubi/conf/kyuubi-env.sh, using default ones...
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Apache Hive (version 2.3.9)
Driver: Kyuubi Project Hive JDBC Client (version 1.8.0)
Beeline version 1.8.0 by Apache Kyuubi
0: jdbc:hive2://localhost:10000/default> select 1;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)

And the HS2 server-side error message is

24/01/25 03:50:53 [37217e6b-e8a9-4069-9973-10471df52e6d HiveServer2-Handler-Pool: Thread-37]: INFO conf.HiveConf: Using the default value passed in for log id: 37217e6b-e8a9-4069-9973-10471df52e6d
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: ERROR server.TThreadPoolServer: Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228) ~[hive-exec-2.3.9.jar:2.3.9]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[hive-exec-2.3.9.jar:2.3.9]
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-2.3.9.jar:2.3.9]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-2.3.9.jar:2.3.9]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_392]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_392]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_392]
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: INFO thrift.ThriftCLIService: Session disconnected without closing properly.
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: INFO thrift.ThriftCLIService: Closing the session: SessionHandle [37217e6b-e8a9-4069-9973-10471df52e6d]

Behavior With This Pull Request 🎉

root@hadoop-master1:/opt/kyuubi-dev# bin/beeline -u 'jdbc:hive2://localhost:10000/default'
Warn: Not find kyuubi environment file /etc/kyuubi/conf/kyuubi-env.sh, using default ones...
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Apache Hive (version 2.3.9)
Driver: Kyuubi Project Hive JDBC Client (version 1.9.0-SNAPSHOT)
Beeline version 1.9.0-SNAPSHOT by Apache Kyuubi
0: jdbc:hive2://localhost:10000/default> select 1;
+------+
| _c0  |
+------+
| 1    |
+------+
1 row selected (2.403 seconds)
0: jdbc:hive2://localhost:10000/default>

Checklist 📝

Be nice. Be informative.

@pan3793
Copy link
Member Author

pan3793 commented Jan 25, 2024

cc @zhouyifan279

@zhouyifan279
Copy link
Contributor

LGTM

@pan3793 pan3793 requested a review from turboFei January 25, 2024 03:59
@codecov-commenter
Copy link

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (ac388d1) 61.18% compared to head (413c7a4) 61.07%.
Report is 2 commits behind head on master.

Files Patch % Lines
...pache/kyuubi/jdbc/hive/KyuubiDatabaseMetaData.java 40.00% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #6016      +/-   ##
============================================
- Coverage     61.18%   61.07%   -0.11%     
  Complexity       23       23              
============================================
  Files           623      623              
  Lines         37060    37064       +4     
  Branches       5024     5024              
============================================
- Hits          22674    22638      -36     
- Misses        11950    11985      +35     
- Partials       2436     2441       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pan3793 pan3793 self-assigned this Jan 25, 2024
@pan3793 pan3793 added this to the v1.8.1 milestone Jan 25, 2024
@pan3793 pan3793 closed this in f531a37 Jan 29, 2024
pan3793 added a commit that referenced this pull request Jan 29, 2024
…compatible with lower version Kyuubi and HS2

# 🔍 Description
## Issue References 🔗

Currently, the Kyuubi Beeline is not compatible with HS2 2.3 and lower versions of Kyuubi, because the server can not recognize `TGetInfoType.CLI_ODBC_KEYWORDS` and then corrupts the socket.

## Describe Your Solution 🔧

Conditional call GetInfo CLI_ODBC_KEYWORDS, when protocol version lower than TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V11, just throw an SQLFeatureNotSupportedException

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Use beeline to connect a HS2 2.3

#### Behavior Without This Pull Request ⚰️
```
roothadoop-master1:/opt/kyuubi-dev# bin/beeline -u 'jdbc:hive2://localhost:10000/default'
Warn: Not find kyuubi environment file /etc/kyuubi/conf/kyuubi-env.sh, using default ones...
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Apache Hive (version 2.3.9)
Driver: Kyuubi Project Hive JDBC Client (version 1.8.0)
Beeline version 1.8.0 by Apache Kyuubi
0: jdbc:hive2://localhost:10000/default> select 1;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
```

And the HS2 server-side error message is
```
24/01/25 03:50:53 [37217e6b-e8a9-4069-9973-10471df52e6d HiveServer2-Handler-Pool: Thread-37]: INFO conf.HiveConf: Using the default value passed in for log id: 37217e6b-e8a9-4069-9973-10471df52e6d
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: ERROR server.TThreadPoolServer: Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228) ~[hive-exec-2.3.9.jar:2.3.9]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[hive-exec-2.3.9.jar:2.3.9]
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-2.3.9.jar:2.3.9]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-2.3.9.jar:2.3.9]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_392]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_392]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_392]
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: INFO thrift.ThriftCLIService: Session disconnected without closing properly.
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: INFO thrift.ThriftCLIService: Closing the session: SessionHandle [37217e6b-e8a9-4069-9973-10471df52e6d]
```

#### Behavior With This Pull Request 🎉

```
roothadoop-master1:/opt/kyuubi-dev# bin/beeline -u 'jdbc:hive2://localhost:10000/default'
Warn: Not find kyuubi environment file /etc/kyuubi/conf/kyuubi-env.sh, using default ones...
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Apache Hive (version 2.3.9)
Driver: Kyuubi Project Hive JDBC Client (version 1.9.0-SNAPSHOT)
Beeline version 1.9.0-SNAPSHOT by Apache Kyuubi
0: jdbc:hive2://localhost:10000/default> select 1;
+------+
| _c0  |
+------+
| 1    |
+------+
1 row selected (2.403 seconds)
0: jdbc:hive2://localhost:10000/default>
```

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6016 from pan3793/jdbc-comp.

Closes #6016

413c7a4 [Cheng Pan] Conditional call GetInfo CLI_ODBC_KEYWORDS to restore compatible with lower version Kyuubi and HS2

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit f531a37)
Signed-off-by: Cheng Pan <[email protected]>
@pan3793
Copy link
Member Author

pan3793 commented Jan 29, 2024

Merged to master/1.8

zhaohehuhu pushed a commit to zhaohehuhu/incubator-kyuubi that referenced this pull request Feb 5, 2024
…store compatible with lower version Kyuubi and HS2

# 🔍 Description
## Issue References 🔗

Currently, the Kyuubi Beeline is not compatible with HS2 2.3 and lower versions of Kyuubi, because the server can not recognize `TGetInfoType.CLI_ODBC_KEYWORDS` and then corrupts the socket.

## Describe Your Solution 🔧

Conditional call GetInfo CLI_ODBC_KEYWORDS, when protocol version lower than TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V11, just throw an SQLFeatureNotSupportedException

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Use beeline to connect a HS2 2.3

#### Behavior Without This Pull Request ⚰️
```
roothadoop-master1:/opt/kyuubi-dev# bin/beeline -u 'jdbc:hive2://localhost:10000/default'
Warn: Not find kyuubi environment file /etc/kyuubi/conf/kyuubi-env.sh, using default ones...
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Apache Hive (version 2.3.9)
Driver: Kyuubi Project Hive JDBC Client (version 1.8.0)
Beeline version 1.8.0 by Apache Kyuubi
0: jdbc:hive2://localhost:10000/default> select 1;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
```

And the HS2 server-side error message is
```
24/01/25 03:50:53 [37217e6b-e8a9-4069-9973-10471df52e6d HiveServer2-Handler-Pool: Thread-37]: INFO conf.HiveConf: Using the default value passed in for log id: 37217e6b-e8a9-4069-9973-10471df52e6d
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: ERROR server.TThreadPoolServer: Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228) ~[hive-exec-2.3.9.jar:2.3.9]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[hive-exec-2.3.9.jar:2.3.9]
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-2.3.9.jar:2.3.9]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-2.3.9.jar:2.3.9]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_392]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_392]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_392]
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: INFO thrift.ThriftCLIService: Session disconnected without closing properly.
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: INFO thrift.ThriftCLIService: Closing the session: SessionHandle [37217e6b-e8a9-4069-9973-10471df52e6d]
```

#### Behavior With This Pull Request 🎉

```
roothadoop-master1:/opt/kyuubi-dev# bin/beeline -u 'jdbc:hive2://localhost:10000/default'
Warn: Not find kyuubi environment file /etc/kyuubi/conf/kyuubi-env.sh, using default ones...
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Apache Hive (version 2.3.9)
Driver: Kyuubi Project Hive JDBC Client (version 1.9.0-SNAPSHOT)
Beeline version 1.9.0-SNAPSHOT by Apache Kyuubi
0: jdbc:hive2://localhost:10000/default> select 1;
+------+
| _c0  |
+------+
| 1    |
+------+
1 row selected (2.403 seconds)
0: jdbc:hive2://localhost:10000/default>
```

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes apache#6016 from pan3793/jdbc-comp.

Closes apache#6016

413c7a4 [Cheng Pan] Conditional call GetInfo CLI_ODBC_KEYWORDS to restore compatible with lower version Kyuubi and HS2

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
zhaohehuhu pushed a commit to zhaohehuhu/incubator-kyuubi that referenced this pull request Mar 21, 2024
…store compatible with lower version Kyuubi and HS2

# 🔍 Description
## Issue References 🔗

Currently, the Kyuubi Beeline is not compatible with HS2 2.3 and lower versions of Kyuubi, because the server can not recognize `TGetInfoType.CLI_ODBC_KEYWORDS` and then corrupts the socket.

## Describe Your Solution 🔧

Conditional call GetInfo CLI_ODBC_KEYWORDS, when protocol version lower than TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V11, just throw an SQLFeatureNotSupportedException

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Use beeline to connect a HS2 2.3

#### Behavior Without This Pull Request ⚰️
```
roothadoop-master1:/opt/kyuubi-dev# bin/beeline -u 'jdbc:hive2://localhost:10000/default'
Warn: Not find kyuubi environment file /etc/kyuubi/conf/kyuubi-env.sh, using default ones...
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Apache Hive (version 2.3.9)
Driver: Kyuubi Project Hive JDBC Client (version 1.8.0)
Beeline version 1.8.0 by Apache Kyuubi
0: jdbc:hive2://localhost:10000/default> select 1;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
```

And the HS2 server-side error message is
```
24/01/25 03:50:53 [37217e6b-e8a9-4069-9973-10471df52e6d HiveServer2-Handler-Pool: Thread-37]: INFO conf.HiveConf: Using the default value passed in for log id: 37217e6b-e8a9-4069-9973-10471df52e6d
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: ERROR server.TThreadPoolServer: Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228) ~[hive-exec-2.3.9.jar:2.3.9]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[hive-exec-2.3.9.jar:2.3.9]
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-2.3.9.jar:2.3.9]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-2.3.9.jar:2.3.9]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_392]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_392]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_392]
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: INFO thrift.ThriftCLIService: Session disconnected without closing properly.
24/01/25 03:50:53 [HiveServer2-Handler-Pool: Thread-37]: INFO thrift.ThriftCLIService: Closing the session: SessionHandle [37217e6b-e8a9-4069-9973-10471df52e6d]
```

#### Behavior With This Pull Request 🎉

```
roothadoop-master1:/opt/kyuubi-dev# bin/beeline -u 'jdbc:hive2://localhost:10000/default'
Warn: Not find kyuubi environment file /etc/kyuubi/conf/kyuubi-env.sh, using default ones...
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Apache Hive (version 2.3.9)
Driver: Kyuubi Project Hive JDBC Client (version 1.9.0-SNAPSHOT)
Beeline version 1.9.0-SNAPSHOT by Apache Kyuubi
0: jdbc:hive2://localhost:10000/default> select 1;
+------+
| _c0  |
+------+
| 1    |
+------+
1 row selected (2.403 seconds)
0: jdbc:hive2://localhost:10000/default>
```

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes apache#6016 from pan3793/jdbc-comp.

Closes apache#6016

413c7a4 [Cheng Pan] Conditional call GetInfo CLI_ODBC_KEYWORDS to restore compatible with lower version Kyuubi and HS2

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants