You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I'm wondering when the latest version of tokenizers will be published on crates.io?
I noticed that the version available here on GitHub is still a development version, which isn't yet available as a stable release on crates.io.
The reason I'm asking is that I would like to use rustls instead of openssl, but rustls support is only available in the latest version. Unfortunately, when I try to use the current GitHub version directly, I run into compilation issues, such as:
error[E0432]: unresolved import `onig`
--> tokenizers/src/utils/onig.rs:3:5
|
3 | use onig::Regex;
| ^^^^ help: a similar path exists: `super::onig`
error[E0599]: the method `unwrap` exists for enum `Result<Replace, Box<dyn Error + Send + Sync>>`, but its trait bounds were not satisfied
--> tokenizers/src/normalizers/replace.rs:56:56
|
56 | Self::new(self.pattern.clone(), &self.content).unwrap()
| ^^^^^^
error[E0599]: the method `unwrap` exists for enum `Result<SysRegex, Box<dyn Error + Send + Sync>>`, but its trait bounds were not satisfied
--> tokenizers/src/pre_tokenizers/byte_level.rs:46:6
|
46 | ).unwrap();
error[E0599]: the method `get` exists for struct `Lazy<SysRegex>`, but its trait bounds were not satisfied
--> tokenizers/src/pre_tokenizers/byte_level.rs:40:1
|
40 | lazy_static! {
error[E0433]: failed to resolve: use of undeclared crate or module `onig`
--> tokenizers/src/utils/onig.rs:12:60
|
12 | pub fn find_iter<'r, 't>(&'r self, inside: &'t str) -> onig::FindMatches<'r, 't> {
| ^^^^ use of undeclared crate or module `onig`
Because of these compilation errors, I can't use the latest dev version in my project.
Would it be possible to release an updated, stable version on crates.io soon?
Or, alternatively, do you have any guidance on working around these issues with the current state of the repo?
The text was updated successfully, but these errors were encountered:
Femure
changed the title
Crates.io Release Request for Latest tokenizers (for rustls Support)
Request for latest version release (for rustls Support)
Mar 10, 2025
Hi, I'm wondering when the latest version of tokenizers will be published on crates.io?
I noticed that the version available here on GitHub is still a development version, which isn't yet available as a stable release on crates.io.
The reason I'm asking is that I would like to use rustls instead of openssl, but rustls support is only available in the latest version. Unfortunately, when I try to use the current GitHub version directly, I run into compilation issues, such as:
Because of these compilation errors, I can't use the latest dev version in my project.
Would it be possible to release an updated, stable version on crates.io soon?
Or, alternatively, do you have any guidance on working around these issues with the current state of the repo?
The text was updated successfully, but these errors were encountered: