Skip to content

Commit c5fb7ad

Browse files
authored
Merge pull request #217 from pavan-kumar-s/master
For CI/CD of tutorials webpage
2 parents fc0993d + dbb6b46 commit c5fb7ad

File tree

233 files changed

+87075
-32295
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

233 files changed

+87075
-32295
lines changed

.github/workflows/main.yml

+49-30
Original file line numberDiff line numberDiff line change
@@ -13,40 +13,59 @@ jobs:
1313
- name: Checkout Source Repo
1414
uses: actions/checkout@v2
1515
with:
16-
repository: 'openCOBRA/COBRA.tutorials'
16+
repository: '${{ github.repository_owner }}/COBRA.tutorials'
1717
token: ${{ secrets.GITHUB_TOKEN }}
1818
fetch-depth: 0
1919

20-
- name: Get All Changed Files
21-
id: files
22-
uses: jitterbit/get-changed-files@v1
23-
24-
- name: Find changed files
25-
id: getfile
20+
- name: Get the repository's owner name
2621
run: |
27-
files=$(echo "${{ steps.files.outputs.all }}" | tr ' ' '\n' | grep -E '\.mlx$')
28-
echo "::set-output name=file::$files"
29-
30-
31-
- name: Give execute permission to the script
32-
run: chmod +x build.sh
33-
34-
- name: Give execute permission to the other script
35-
run: chmod +x setup.sh
36-
37-
38-
- name: Sync with Destination Repo
22+
echo "REPO_OWNER=${{ github.repository_owner }}" >> $GITHUB_ENV
23+
24+
- name: Clone the destination repository
3925
run: |
40-
counter=0
41-
for file in ${{ steps.getfile.outputs.file }}; do
42-
if [ $counter -eq 0 ]
43-
then
44-
# This is the first iteration, handle the file differently
45-
./setup.sh opencobra/cobratoolbox $file
46-
./build.sh opencobra/cobratoolbox $file
47-
else
48-
./build.sh openCOBRA/cobratoolbox $file
26+
rm -rf cobratoolbox
27+
echo "Cloning the destination repository: [email protected]:opencobra/cobratoolbox.git"
28+
git clone --depth 1 --branch gh-pages ssh://[email protected]/$REPO_OWNER/cobratoolbox.git
29+
30+
- name: Get Changed mlx Files and sinc with destination repository
31+
id: getFile
32+
run: |
33+
changed_files=$(git diff --name-only HEAD~1 HEAD | grep '\.mlx' | tr '\n' ' ')
34+
# Setup virtual frame buffer
35+
for file in $changed_files; do
36+
if [[ $file != "" ]]; then
37+
echo "Processing: $file"
38+
ABSOLUTE_FILE_PATH=$(realpath "$file")
39+
HTML_FILE_PATH=$(echo "$ABSOLUTE_FILE_PATH" | sed 's/.mlx/.html/g')
40+
PDF_FILE_PATH=$(echo "$ABSOLUTE_FILE_PATH" | sed 's/.mlx/.pdf/g')
41+
M_FILE_PATH=$(echo "$ABSOLUTE_FILE_PATH" | sed 's/.mlx/.m/g')
42+
/usr/local/MATLAB/R2024a/bin/matlab -batch "matlab.internal.liveeditor.openAndConvert('$ABSOLUTE_FILE_PATH', '$HTML_FILE_PATH')"
43+
/usr/local/MATLAB/R2024a/bin/matlab -batch "matlab.internal.liveeditor.openAndConvert('$ABSOLUTE_FILE_PATH', '$PDF_FILE_PATH')"
44+
/usr/local/MATLAB/R2024a/bin/matlab -batch "matlab.internal.liveeditor.openAndConvert('$ABSOLUTE_FILE_PATH', '$M_FILE_PATH')"
45+
cd cobratoolbox
46+
TARGET_DIR="stable/tutorials/$(dirname "$file")"
47+
mkdir -p "$TARGET_DIR"
48+
echo "Copying the HTML, PDF, mlx and .m files to the target directory..."
49+
cp "$HTML_FILE_PATH" "$TARGET_DIR/"
50+
cd ../
4951
fi
50-
counter=$((counter+1))
5152
done
52-
if: steps.getfile.outputs.file != ''
53+
54+
- name: Pushing the changes to both COBRA.tutorials and cobratoolbox repos
55+
run: |
56+
# Set up git config
57+
echo "Setting up git config..."
58+
git config --global user.name "github-actions[bot]"
59+
git config --global user.email "github-actions[bot]@users.noreply.github.com"
60+
# Add, commit, and push the changes
61+
cd cobratoolbox
62+
git pull
63+
git add .
64+
git commit -m "Sync files from source repo" || echo "No changes to commit"
65+
git push
66+
cd ..
67+
rm -rf cobratoolbox
68+
git add .
69+
git commit -m "created .HTML, .pdf and .m files"
70+
git push origin master
71+
echo "Script execution completed."

.github/workflows/readme.md

+63-50
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
## Continuous Integration of Tutorials:
22
### High level Overview
3-
The whole idea of the continuous integration of the tutorials is that whenever a user contributes a tutorial in the format of a .mlx file on the [tutorials repo](https://github.com/opencobra/COBRA.tutorials) it should be converted to html and then rendered accordingly on the Cobratoolbox website in the [tutorials section](https://opencobra.github.io/cobratoolbox/stable/tutorials/index.html). This involves using MATLAB on a self-hosted server (King server) to generate the html file. This html is then pushed to the websites codebase repository which is the [gh-pages branch](https://github.com/opencobra/cobratoolbox/tree/gh-pages) of the main cobratoolbox repository.
3+
The whole idea of the continuous integration of the tutorials is that whenever a user contributes a tutorial in the format of a .mlx file on the [tutorials repo](https://github.com/opencobra/COBRA.tutorials) it should be converted to html and then rendered accordingly on the Cobratoolbox website in the tutorials section. This involves using MATLAB on a self-hosted server (King server) to generate the html file. This html is then pushed to the website's codebase repository which is the [./stable](https://github.com/opencobra/cobratoolbox/tree/gh-pages/stable) folder of the gh-pages branch of the main cobratoolbox repository.
44

5-
GitHub actions is used to detect when a push (specifically .mlx push) is made to the tutorials repo. Then once the .mlx has been converted it is pushed to the gh-branch of the main repo. Again, GitHub actions can detect this push and configures the website to incorporate the extra tutorial.
5+
GitHub actions is used to detect when a push (specifically .mlx push) is made to the tutorials repo. Then once the .mlx has been converted it is pushed to the stable folder of the gh-pages branch. Again, GitHub actions can detect this push and configures the website to incorporate the extra tutorial.
66

77
### Detailed Documentation
88
**Step 1: Pushing MLX files to the tutorials repository:**
9-
To understand GitHub actions you need to look for the github workflow folder where you will find a .yml which contains all the details about the github action. The worflows can be found by navigating to ./.github/workflows/ . In the tutorials repo you will find a ‘[main.yml](https://github.com/opencobra/COBRA.tutorials/blob/master/.github/workflows/main.yml)’ file.
9+
To understand GitHub actions you need to look for the github workflow folder where you will find a .yml which contains all the details about the github action. The worflows can be found by navigating to ./.github/workflows/. In the tutorials repo you will find a ‘[main.yml](https://github.com/opencobra/COBRA.tutorials/blob/master/.github/workflows/main.yml)’ file.
1010

1111
**What does main.yml do?**
1212
Here is an explanation of each section of the .yml file. Pictures of the sections are added and an explanation is given beneath the picture.
@@ -30,70 +30,83 @@ jobs:
3030
- name: Checkout Source Repo
3131
uses: actions/checkout@v2
3232
with:
33-
repository: 'openCOBRA/COBRA.tutorials'
33+
repository: '${{ github.repository_owner }}/COBRA.tutorials'
3434
token: ${{ secrets.GITHUB_TOKEN }}
3535
fetch-depth: 0
3636
```
3737

3838

3939
- Next, we have a series of ‘jobs’ to compute.
40-
- The ‘runs-on’ parameter indicates where these jobs are computed. Here I specify it runs on ‘self-hosted’ because we need Matlab on King to run the .mlx to html. Generally, I would avoid using a self-hosted server but since Matlab is not an opensource programming language it needs to be ran a computer which has Matlab installed with a license.
40+
- The ‘runs-on’ parameter indicates where these jobs are computed. Here, it runs on ‘self-hosted’ because we need Matlab on King to run the .mlx to html. Generally, we should avoid using a self-hosted server but since Matlab is not an opensource programming language it needs to be ran a computer which has Matlab installed with a license.
4141
- There are several steps to do in the jobs section. Here the first step is to checkout the source repo i.e. get all the details about the repo and the pushes made to the repo.
42-
42+
4343
```
44-
- name: Get All Changed Files
45-
id: files
46-
uses: jitterbit/get-changed-files@v1
47-
48-
- name: Find changed files
49-
id: getfile
44+
- name: Get the repository's owner name
5045
run: |
51-
files=$(echo "${{ steps.files.outputs.all }}" | tr ' ' '\n' | grep -E '\.mlx$')
52-
echo "::set-output name=file::$files"
46+
echo "REPO_OWNER=${{ github.repository_owner }}" >> $GITHUB_ENV
5347
```
54-
55-
56-
- Here we have two more steps. The first step in this picture is used to find all the files that have been changed based on the most recent push.
57-
- The next step is then used to find all the .mlx files that were pushed to the repository.
48+
- Here we are getting the repository's owner name and storing it in the variable, REPO_OWNER. This variable will be used in further jobs.
5849

5950
```
60-
- name: Give execute permission to the script
61-
run: chmod +x build.sh
62-
63-
- name: Give execute permission to the other script
64-
run: chmod +x setup.sh
51+
- name: Clone the destination repository
52+
run: |
53+
rm -rf cobratoolbox
54+
echo "Cloning the destination repository: [email protected]:opencobra/cobratoolbox.git"
55+
git clone --depth 1 --branch gh-pages ssh://[email protected]/$REPO_OWNER/cobratoolbox.git
6556
```
57+
- Here cobratoolbox repo (only gh-pages branch) is cloned to push the generated .html pages in further steps
6658

59+
```
60+
- name: Get Changed mlx Files and sinc with destination repository
61+
id: getFile
62+
run: |
63+
changed_files=$(git diff --name-only HEAD~1 HEAD | grep '\.mlx' | tr '\n' ' ')
64+
# Setup virtual frame buffer
65+
for file in $changed_files; do
66+
if [[ $file != "" ]]; then
67+
echo "Processing: $file"
68+
ABSOLUTE_FILE_PATH=$(realpath "$file")
69+
HTML_FILE_PATH=$(echo "$ABSOLUTE_FILE_PATH" | sed 's/.mlx/.html/g')
70+
PDF_FILE_PATH=$(echo "$ABSOLUTE_FILE_PATH" | sed 's/.mlx/.pdf/g')
71+
M_FILE_PATH=$(echo "$ABSOLUTE_FILE_PATH" | sed 's/.mlx/.m/g')
72+
/usr/local/MATLAB/R2024a/bin/matlab -batch "matlab.internal.liveeditor.openAndConvert('$ABSOLUTE_FILE_PATH', '$HTML_FILE_PATH')"
73+
/usr/local/MATLAB/R2024a/bin/matlab -batch "matlab.internal.liveeditor.openAndConvert('$ABSOLUTE_FILE_PATH', '$PDF_FILE_PATH')"
74+
/usr/local/MATLAB/R2024a/bin/matlab -batch "matlab.internal.liveeditor.openAndConvert('$ABSOLUTE_FILE_PATH', '$M_FILE_PATH')"
75+
cd cobratoolbox
76+
TARGET_DIR="stable/tutorials/$(dirname "$file")"
77+
mkdir -p "$TARGET_DIR"
78+
echo "Copying the HTML, PDF, mlx and .m files to the target directory..."
79+
cp "$HTML_FILE_PATH" "$TARGET_DIR/"
80+
cd ../
81+
fi
82+
done
83+
```
84+
- Here we actually do the conversions and copy the .HTML file to cobratoolbox repository. The fourth line in this picture (changed_files=$(git diff --name-only HEAD~1 HEAD | grep '\.mlx' | tr '\n' ' ')) is used to find all the .mlx files that have been changed based on the most recent push.
85+
- Then for each modified .mlx file, an HTML file, a pdf file, and a Matlab code file are created and stored in the same directory where the .mlx file is located. We are using MATLAB R2024a here, if the king server has some other version, then this step has to be modified accordingly. Further, HTML file alone is copied to the cobratoolbox repo.
6786

68-
The chmod command just makes the .sh files executable.
69-
70-
**Quick note on setup.sh and build.sh:**
71-
72-
• The setup.sh script automates the process of synchronizing .mlx files to the ghpages branch of the cobratoolbox GitHub repository. It requires three inputs: the repository identifier in owner/name format, a token for authentication, and the file path of the .mlx files to be synchronized. Upon execution, the script clones the cobratoolbox repository, configures git for automated operations, and targets aspecific directory within stable/tutorials/ to update. It clears this directory and copies the new .mlx files into it, ensuring that any changes are committed and pushed. This operation keeps the gh-pages branch of the cobratoolbox repository consistently updated with the latest .mlx files for documentation or tutorials.
73-
74-
• The build.sh script is designed for converting .mlx files to .html format and synchronizing them with the gh-pages branch of the cobratoolbox repository,. It takes three arguments: the repository identifier, a token for authentication, and the path of the .mlx file to be converted. Initially, the script converts the .mlx file to .html using MATLAB commands, assuming MATLAB is installed and accessible in the PATH. It then clones the target repository, sets up git with predefined user details, and switches to the gh-pages branch. The script creates a target directory within stable/tutorials/, copies the converted .html file into this directory, and finalizes by committing and pushing the changes.
75-
76-
• Both files can be found on the tutorial’s repository. Here are the links to [setup.sh](https://github.com/opencobra/COBRA.tutorials/blob/master/setup.sh) and [build.sh](https://github.com/opencobra/COBRA.tutorials/blob/master/build.sh)
7787

7888
```
79-
- name: Sync with Destination Repo
80-
run: |
81-
counter=0
82-
for file in ${{ steps.getfile.outputs.file }}; do
83-
if [ $counter -eq 0 ]
84-
then
85-
# This is the first iteration, handle the file differently
86-
./setup.sh opencobra/cobratoolbox ${{ secrets.DEST_REPO_TOKEN }} $file
87-
./build.sh opencobra/cobratoolbox ${{ secrets.DEST_REPO_TOKEN }} $file
88-
else
89-
./build.sh openCOBRA/cobratoolbox ${{ secrets.DEST_REPO_TOKEN }}
90-
fi
91-
counter=$((counter+1))
92-
done
93-
if: steps.getfile.outputs.file != ''
89+
- name: Pushing the changes to both COBRA.tutorials and cobratoolbox repos
90+
run: |
91+
# Set up git config
92+
echo "Setting up git config..."
93+
git config --global user.name "github-actions[bot]"
94+
git config --global user.email "github-actions[bot]@users.noreply.github.com"
95+
# Add, commit, and push the changes
96+
cd cobratoolbox
97+
git pull
98+
git add .
99+
git commit -m "Sync files from source repo" || echo "No changes to commit"
100+
git push
101+
cd ..
102+
rm -rf cobratoolbox
103+
git add .
104+
git commit -m "created .HTML, .pdf and .m files"
105+
git push origin master
106+
echo "Script execution completed."
94107
```
95108

96-
Here is the code to run the setup.sh and build.sh. We loop through all the .mlx files that were pushed. If it is the first file we are looking at we also run setup.sh to create the folder locations in the cobratoolbox – ghpages branch repository. Then afterwards build,sh is ran to convert the file to html and push to the created folder location
109+
The converted files are further pushed to the cobratoolbox and COBRA.tutorial repo. Note that only .html pages are pushed to cobratoolbox repository and all the other formats are stored in COBRA.tutorial repo
97110

98111
### Configuring the King Server
99112

@@ -102,6 +115,6 @@ Go to this page of the repo to create a new self-hosted runner:
102115
![image](https://github.com/opencobra/cobratoolbox/assets/68754265/05535af0-9ccf-4c38-9e79-512f738cc0f0)
103116

104117

105-
By pressing the green new runner button, you are given easy instructions on how to set it up. You should have access to a terminal on King for this. To run the self-hosted runner nagivate to the folder you created it in and run ./run.sh to run the self-hosted runner.
118+
By pressing the green new runner button, we are given easy instructions on how to set it up. We should have access to a terminal on King for this. To run the self-hosted runner nagivate to the folder we created it in and run ./run.sh to run the self-hosted runner.
106119

107-
You also need to make sure you have Matlab downloaded and working on the king server also. In the ‘[build.sh](https://github.com/opencobra/COBRA.tutorials/blob/master/build.sh)’ file the location of matlab is currently in my directory but you can add Matlab to another location and change the link to the location in the build.sh file.
120+
We also need to make sure you have Matlab downloaded and working on the king server also.

README.rst

+6-6
Original file line numberDiff line numberDiff line change
@@ -102,25 +102,25 @@ Submit your changes and open a pull request
102102

103103
.. |icon_analysis| raw:: html
104104

105-
<img src="https://king.nuigalway.ie/cobratoolbox/img/icon_analysis.png" height="14px">
105+
<img src="https://github.com/opencobra/cobratoolbox/tree/gh-pages/stable/_static/img/analysis.png" height="14px">
106106

107107
.. |icon_base| raw:: html
108108

109-
<img src="https://king.nuigalway.ie/cobratoolbox/img/icon_base.png" height="14px">
109+
<img src="https://github.com/opencobra/cobratoolbox/tree/gh-pages/stable/_static/img/base.png" height="14px">
110110

111111
.. |icon_dataIntegration| raw:: html
112112

113-
<img src="https://king.nuigalway.ie/cobratoolbox/img/icon_di.png" height="14px">
113+
<img src="https://github.com/opencobra/cobratoolbox/tree/gh-pages/stable/_static/img/dataIntegration.png" height="14px">
114114

115115
.. |icon_design| raw:: html
116116

117-
<img src="https://king.nuigalway.ie/cobratoolbox/img/icon_design.png" height="14px">
117+
<img src="https://github.com/opencobra/cobratoolbox/tree/gh-pages/stable/_static/img/design.png" height="14px">
118118

119119
.. |icon_reconstruction| raw:: html
120120

121-
<img src="https://king.nuigalway.ie/cobratoolbox/img/icon_reconstruction.png" height="14px">
121+
<img src="https://github.com/opencobra/cobratoolbox/tree/gh-pages/stable/_static/img/reconstruction.png" height="14px">
122122

123123
.. |icon_visualization| raw:: html
124124

125-
<img src="https://king.nuigalway.ie/cobratoolbox/img/icon_visualization.png" height="14px">
125+
<img src="https://github.com/opencobra/cobratoolbox/tree/gh-pages/stable/_static/img/visualization.png" height="14px">
126126

0 commit comments

Comments
 (0)