Skip to content

Commit

Permalink
Merge branch 'main' into benchmarking_script
Browse files Browse the repository at this point in the history
  • Loading branch information
Jack-Khuu authored Dec 23, 2024
2 parents b9de90b + 68b8087 commit ad5c82a
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 7 deletions.
10 changes: 5 additions & 5 deletions docs/quantization.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,22 +142,22 @@ To use linear:a8wxdq and embedding:wx, you must set up the torchao experimental

From the torchchat root directory, run
```
sh torchchat/utils/scripts/build_torchao_ops.sh
bash torchchat/utils/scripts/build_torchao_ops.sh
```

This should take about 10 seconds to complete.

Note: if you want to use the new kernels in the AOTI and C++ runners, you must pass the flag link_torchao_ops when running the scripts the build the runners.

```
sh torchchat/utils/scripts/build_native.sh aoti link_torchao_ops
bash torchchat/utils/scripts/build_native.sh aoti link_torchao_ops
```

```
sh torchchat/utils/scripts/build_native.sh et link_torchao_ops
bash torchchat/utils/scripts/build_native.sh et link_torchao_ops
```

Note before running `sh torchchat/utils/scripts/build_native.sh et link_torchao_ops`, you must first install executorch with `sh torchchat/utils/scripts/install_et.sh` if you have not done so already.
Note before running `bash torchchat/utils/scripts/build_native.sh et link_torchao_ops`, you must first install executorch with `bash torchchat/utils/scripts/install_et.sh` if you have not done so already.

### Examples

Expand Down Expand Up @@ -212,7 +212,7 @@ Currently, torchchat can only run them on Eager mode.

From the torchchat root directory, run
```
sh torchchat/utils/scripts/build_torchao_ops.sh mps
bash torchchat/utils/scripts/build_torchao_ops.sh mps
```

### Examples
Expand Down
4 changes: 2 additions & 2 deletions torchchat/utils/quantize.py
Original file line number Diff line number Diff line change
Expand Up @@ -934,15 +934,15 @@ def quantized_model(self) -> nn.Module:
torch.ops.load_library(libs[0])
print("Loaded torchao cpu ops.")
except Exception as e:
print("Unabled to load torchao cpu ops library. Slow fallback kernels will be used.")
print("Unable to load torchao cpu ops library. Slow fallback kernels will be used.")

try:
libname = "libtorchao_ops_mps_aten.dylib"
libpath = f"{torchao_build_path}/cmake-out/lib/{libname}"
torch.ops.load_library(libpath)
print("Loaded torchao mps ops.")
except Exception as e:
print("Unabled to load torchao mps ops library.")
print("Unable to load torchao mps ops library.")

except Exception as e:
print("Unabled to import torchao experimental quant_api with error: ", e)
2 changes: 2 additions & 0 deletions torchchat/utils/scripts/updown.py
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,8 @@ def updown_processor(
lines = file.readlines()
print_flag = False

# Use bash; set it to fail on the first failing command
output("#! /bin/bash", replace_list=None, suppress_list=None)
output("set -eou pipefail", replace_list=None, suppress_list=None)

if create_sections:
Expand Down

0 comments on commit ad5c82a

Please sign in to comment.