Spark35_glue5_upgrade

[vidhyamanisankar]

SPP-12455
Update the imputation's engine and ratio_calculator

• Split big the chained transaction into small transactions
• Truncate the logical plan using localCheckpoint(eager=True) while calling the ratio_calculator, imputation_helper in the loop.
• Use broadcast join on smaller df
• Secondary ordering by reference in mean_of_ratios. This does not impact the calculated.forward/backward links, since it will only apply to contributors with equal growth ratios, but keeps the selection of rows for trimming deterministic.

Synopsis

Upgrading to use glue 5, spark 3.5.2, and python 3.11 for glue jobs. AWS

Checklist

• Documentation created/updated
• Tests created/updated

Description

Add a more detailed description of the pr if necessary (can reference release
notes if included).

[BenLatham]

Looks good to me, however I don't fully understand broadcasts, so it would be helpful to discuss those to ensure I've not missed anything.

[BenLatham]

Is it worth maintaining backwards compatibility testing for python 3.7, or could we drop 3.7 support?

[vidhyamanisankar]

I agree. Spark 3.5.x officially supports Python 3.8 and later. Support for Python 3.7 was deprecated in Spark 3.4.0.

[vidhyamanisankar]

However pyproject.toml has the pyspark dependency versions ranging '>=3.1.1 <3.5.3' and spark 3.1.1 supported python version - https://archive.apache.org/dist/spark/docs/3.1.1/ . so at-least we need to support python3.7.

[mwirikia]

comment from ID

This should be kept as a draft for now IMO…we can’t merge into main until MDQ have confirmed they have resource to test and we can deploy and test as part of a release.