Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request for latest version release (for rustls Support) #1743

Open
Femure opened this issue Mar 10, 2025 · 0 comments
Open

Request for latest version release (for rustls Support) #1743

Femure opened this issue Mar 10, 2025 · 0 comments

Comments

@Femure
Copy link

Femure commented Mar 10, 2025

Hi, I'm wondering when the latest version of tokenizers will be published on crates.io?
I noticed that the version available here on GitHub is still a development version, which isn't yet available as a stable release on crates.io.

The reason I'm asking is that I would like to use rustls instead of openssl, but rustls support is only available in the latest version. Unfortunately, when I try to use the current GitHub version directly, I run into compilation issues, such as:

error[E0432]: unresolved import `onig`
 --> tokenizers/src/utils/onig.rs:3:5
  |
3 | use onig::Regex;
  |     ^^^^ help: a similar path exists: `super::onig`

error[E0599]: the method `unwrap` exists for enum `Result<Replace, Box<dyn Error + Send + Sync>>`, but its trait bounds were not satisfied
 --> tokenizers/src/normalizers/replace.rs:56:56
  |
56 |         Self::new(self.pattern.clone(), &self.content).unwrap()
   |                                                        ^^^^^^

error[E0599]: the method `unwrap` exists for enum `Result<SysRegex, Box<dyn Error + Send + Sync>>`, but its trait bounds were not satisfied
 --> tokenizers/src/pre_tokenizers/byte_level.rs:46:6
  |
46 |     ).unwrap();

error[E0599]: the method `get` exists for struct `Lazy<SysRegex>`, but its trait bounds were not satisfied
 --> tokenizers/src/pre_tokenizers/byte_level.rs:40:1
  |
40 | lazy_static! {

error[E0433]: failed to resolve: use of undeclared crate or module `onig`
 --> tokenizers/src/utils/onig.rs:12:60
  |
12 |     pub fn find_iter<'r, 't>(&'r self, inside: &'t str) -> onig::FindMatches<'r, 't> {
   |                                                            ^^^^ use of undeclared crate or module `onig`

Because of these compilation errors, I can't use the latest dev version in my project.
Would it be possible to release an updated, stable version on crates.io soon?
Or, alternatively, do you have any guidance on working around these issues with the current state of the repo?

@Femure Femure changed the title Crates.io Release Request for Latest tokenizers (for rustls Support) Request for latest version release (for rustls Support) Mar 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant