About regression borders

[xumwen]

Congratulations and thank you for your amazing work and for releasing the code. It's truly valuable and inspiring.
I have a question regarding the regression tasks in your code. I'm curious about how you define the std_borders variable. It appears that these borders are directly loaded from the pretrained checkpoint. I'm having some trouble understanding the nature of these borders.
[-93.08673 -26.132122 -22.632263 ... 23.695412 27.15525 86.942566]
Could you please clarify whether they are defined based on some prior knowledge of the data, a specific statistical distribution, or if they are learned parameters during the model training? Any insights into this aspect would be greatly appreciated.
Thank you very much for your time.

[noahho]

The define the bucket boundaries for our regression prediction distribution. These borders are meta-learned across datasets, meaning they are optimized during pretraining to create a well-distributed set of buckets that generalize effectively.

When the model is later fine-tuned or applied to a specific dataset, these bucket boundaries are adjusted using z-normalization (i.e., rescaling based on the dataset’s mean and standard deviation). std_borders and mean_borders refer to the standardization paramaters for the fitted dataset. This ensures that the buckets align well with the distribution of the target variable in the given dataset.