Discrepancy Between Custom and Default reg:pseudohubererror Loss Function in XGBoost

[saima1704]

I am trying to create a custom loss function for the reg:pseudohubererror objective in XGBoost. However, my custom implementation does not match the results produced by the default objective. I have also posted this question on Stack Overflow-in-xgboost to reach a broader audience.

Any guidance or insights on where I might have gone wrong would be greatly appreciated! Thank you in advance for your support.

# Define custom Pseudo-Huber loss function
pseudo_huber_loss <- function(preds, dtrain) {
  
  labels <- getinfo(dtrain, "label")
  
 
  d <- labels - preds
  
  
  delta <- 1
  
 
  scale <- 1 + (d / delta)^2
  scale_sqrt <- sqrt(scale)
  
  grad <- d / scale_sqrt
  
  hess <- (1 / scale) / scale_sqrt
  
  return(list(grad = grad, hess = hess))
}
set.seed(42)
X_train <- matrix(rnorm(100 * 10), ncol = 10)
y_train <- rnorm(100)

dtrain <- xgb.DMatrix(data = X_train, label = y_train)


params_default <- list(
  objective = "reg:pseudohubererror",  # Default Pseudo-Huber loss
  eta = 0.0121,                        # Learning rate
  max_depth = 6,                       # Maximum depth of trees
  eval_metric = "rmse"                 # Evaluation metric
)


xgb_model_default <- xgb.train(
  params = params_default,
  data = dtrain,
  nrounds = 100,
  watchlist = list(train = dtrain),
  verbose = 1
)

y_pred_default <- predict(xgb_model_default, X_train)
print("Default Model Predictions:")
print(head(y_pred_default))

params_custom <- list(
  objective = pseudo_huber_loss,       # Custom Pseudo-Huber loss function
  eta = 0.0121,                        # Same learning rate as default
  max_depth = 6,                       # Same max depth
  eval_metric = "rmse"                 # Same evaluation metric
)


xgb_model_custom <- xgb.train(
  params = params_custom,
  data = dtrain,
  nrounds = 100,
  watchlist = list(train = dtrain),
  verbose = 1
)


y_pred_custom <- predict(xgb_model_custom, X_train)
print("Custom Model Predictions:")
print(head(y_pred_custom))
print(head(y_pred_default))

[trivialfis]

The objective is defined here:

xgboost/src/objective/regression_obj.cu

Line 284 in 9ab7567

float z = predt(i, j) - labels(i, j);

If you can verify that it's the same with your objective, then please try to add base_score=0.5 to the training parameters to disable the intercept estimation in XBGoost.

[trivialfis]

Feel free to reopen if you have any more questions.