Thread safe?

[drupol]

Hello,

I have been reviewing the documentation and would like to confirm whether encoding text using Tokenizer.from_file(str(tokenizer_path)).encode(text) is thread-safe.

Below is a simplified version of my implementation for context:

@final
class HuggingFaceTokenizer:
    _tokenizer = None

    def __init__(self):
        tokenizer_path = Path("foobar", "tokenizers", "tokenizer.json")
        self._tokenizer: Tokenizer = Tokenizer.from_file(str(tokenizer_path))
    
    @override
    def tokens_count(self, text: str | None) -> int:
        """Count the number of tokens in the provided text."""
        return 0 if not text else len(self._tokenizer.encode(text))

My concern is whether the _tokenizer.encode() method is inherently safe to use across multiple threads without additional synchronization mechanisms (e.g., locks).

If thread safety is not guaranteed, I am considering implementing a thread-safe mechanism as follows:

@final
class HuggingFaceTokenizer:
    _tokenizer: Tokenizer | None = None
    _lock = threading.Lock()

    def __init__(self):
        tokenizer_path = Path("foobar", "tokenizers", "tokenizer.json")
        with self._lock:
            if not self._tokenizer:
                self._tokenizer = Tokenizer.from_file(str(tokenizer_path))

    @override
    def tokens_count(self, text: str | None) -> int:
        """Count the number of tokens in the provided text."""
        if not text:
            return 0

        with self._lock:
            return len(self._tokenizer.encode(text))

Could you please clarify:

1. Is the encode method inherently thread-safe?
2. If not, is the approach above (using a lock) sufficient to ensure thread safety?
3. Are there any recommended best practices for safely using Tokenizer in multithreaded environments?

Thank you in advance!