Speculative decoding: Base implementation

[Andrei-Aksionov]

Hey there 👋

This PR includes base implementation of a speculative decoding, the one that was proposed in Fast Inference from Transformers via Speculative Decoding.

The focus was on implementing a working solution without focusing on optimizations.
For instance, this doesn't support batched inference.
It should be added in next PR/PRs.

---

TODO:

• 1. Add tests
• 2. Check on a GPU that all tensors are placed correctly
• [❌] 3. Provide initial numbers of tokens/second. Will be a baseline for future optimizations
  Don't have access to all Llama models. Other models have different vocab sizes between different model sizes.
  For the method to properly work, we need:
  a) significant size difference (> 10x)
  b) the same vocabulary for both models
  c) target model has to accept draft tokens most of the time

[Andrei-Aksionov]

Basically, it's ready for a review.

Tests are failing because there is an error with loading a pythia-14m model (is used for tests) and all tokenizers. Something is wrong on HF side. Will rerun tests tomorrow.

As for numbers, Ideally I wanted to test Llama 1b and 8b models, but don't have access to 8b model repo 😞.
Qwen has lots of different sizes, but their vocabs differs between different sizes, so that doesn't fit the bill.

[Andrei-Aksionov]

Oddly enough, these tests pass locally, in a Studio and in a fork.
I know that HF token for this repo was initially provided by Carlos, then Sebastian used his one.
Now @lantiga or @t-vi your turn :)

[t-vi]

Hi Andrei, thank you. I'll take a look.

[t-vi]

So the HF key is working again (thank you @Borda ), but there are two failures left.

[Borda]

So the HF key is working again (thank you @Borda ), but there are two failures left.

looking into it and will debug it in #1940