Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

install from google colab #2253

Closed
khouakhi opened this issue Jun 15, 2020 · 8 comments
Closed

install from google colab #2253

khouakhi opened this issue Jun 15, 2020 · 8 comments
Labels
stale No movement on issue.

Comments

@khouakhi
Copy link

I tried to install cdms2 in conda from Colab and I get

Executing transaction: ...working... failed
ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::bzip2-1.0.8-h516909a_2'.
NotADirectoryError(20, 'Not a directory')
Attempting to roll back.
Rolling back transaction: ...working... done

details below:

`!wget -c https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
!chmod +x Miniconda3-latest-Linux-x86_64.sh
!bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local

!conda install -c conda-forge cdms2`

Solving environment: ...working... done

Package Plan

environment location: /usr/bin

added / updated specs:
- cdms2
- python=3.6

The following NEW packages will be INSTALLED:

_libgcc_mutex:      0.1-conda_forge              conda-forge
_openmp_mutex:      4.5-0_gnu                    conda-forge
attrs:              19.3.0-py_0                  conda-forge
brotlipy:           0.7.0-py36h8c4c3a4_1000      conda-forge
bzip2:              1.0.8-h516909a_2             conda-forge
ca-certificates:    2020.4.5.2-hecda079_0        conda-forge
cdat_info:          8.2-py_7                     conda-forge
cdms2:              3.1.4-py36h25070a4_10        conda-forge
cdtime:             3.1.3-py36h76a73ab_3         conda-forge
certifi:            2020.4.5.2-py36h9f0ad1d_0    conda-forge
cffi:               1.14.0-py36hd463f26_0        conda-forge
chardet:            3.0.4-py36h9f0ad1d_1006      conda-forge
cryptography:       2.9.2-py36h45558ae_0         conda-forge
curl:               7.69.1-h33f0ec9_0            conda-forge
decorator:          4.4.2-py_0                   conda-forge
distarray:          2.12.2-py_1                  conda-forge
esmf:               8.0.1-nompi_hbeb3ca6_0       conda-forge
esmpy:              8.0.1-nompi_py36hbd5f7c1_0   conda-forge
future:             0.18.2-py36h9f0ad1d_1        conda-forge
g2clib:             1.6.0-hf3f1b0b_9             conda-forge
hdf4:               4.2.13-hf30be14_1003         conda-forge
hdf5:               1.10.6-nompi_h3c11f04_100    conda-forge
idna:               2.9-py_1                     conda-forge
importlib-metadata: 1.6.1-py36h9f0ad1d_0         conda-forge
importlib_metadata: 1.6.1-0                      conda-forge
ipython_genutils:   0.2.0-py_1                   conda-forge
jasper:             1.900.1-h07fcdf6_1006        conda-forge
jpeg:               9d-h516909a_0                conda-forge
jsonschema:         3.2.0-py36h9f0ad1d_1         conda-forge
jupyter_core:       4.6.3-py36h9f0ad1d_1         conda-forge
krb5:               1.17.1-h2fd8d38_0            conda-forge
lazy-object-proxy:  1.5.0-py36h8c4c3a4_0         conda-forge
ld_impl_linux-64:   2.34-h53a641e_5              conda-forge
libblas:            3.8.0-11_openblas            conda-forge
libcblas:           3.8.0-11_openblas            conda-forge
libcdms:            3.1.2-h054cd8a_112           conda-forge
libcf:              1.0.3-py36ha507d3b_108       conda-forge
libcurl:            7.69.1-hf7181ac_0            conda-forge
libdrs:             3.1.2-hc2e2db3_112           conda-forge
libdrs_f:           3.1.2-hae7e664_110           conda-forge
libedit:            3.1.20191231-h46ee950_0      conda-forge
libffi:             3.2.1-he1b5a44_1007          conda-forge
libgcc-ng:          9.2.0-h24d8f2e_2             conda-forge
libgfortran-ng:     7.5.0-hdf63c60_6             conda-forge
libgomp:            9.2.0-h24d8f2e_2             conda-forge
liblapack:          3.8.0-11_openblas            conda-forge
libnetcdf:          4.7.4-nompi_h84807e1_104     conda-forge
libopenblas:        0.3.6-h6e990d7_6             conda-forge
libpng:             1.6.37-hed695b0_1            conda-forge
libssh2:            1.9.0-hab1572f_2             conda-forge
libstdcxx-ng:       9.2.0-hdf63c60_2             conda-forge
libtiff:            4.1.0-hc7e4089_6             conda-forge
libuuid:            2.32.1-h14c3975_1000         conda-forge
libwebp-base:       1.1.0-h516909a_3             conda-forge
lz4-c:              1.9.2-he1b5a44_1             conda-forge
nbformat:           5.0.6-py_0                   conda-forge
ncurses:            6.1-hf484d3e_1002            conda-forge
netcdf-fortran:     4.5.2-nompi_hfef6a68_106     conda-forge
numpy:              1.18.5-py36h7314795_0        conda-forge
openblas:           0.3.6-h6e990d7_6             conda-forge
openssl:            1.1.1g-h516909a_0            conda-forge
pip:                20.1.1-py_1                  conda-forge
pycparser:          2.20-py_0                    conda-forge
pyopenssl:          19.1.0-py_1                  conda-forge
pyrsistent:         0.16.0-py36h8c4c3a4_0        conda-forge
pysocks:            1.7.1-py36h9f0ad1d_1         conda-forge
python:             3.6.10-h8356626_1011_cpython conda-forge
python_abi:         3.6-1_cp36m                  conda-forge
readline:           8.0-hf8c457e_0               conda-forge
requests:           2.23.0-pyh8c360ce_2          conda-forge
setuptools:         47.1.1-py36h9f0ad1d_0        conda-forge
six:                1.15.0-pyh9f0ad1d_0          conda-forge
sqlite:             3.30.1-hcee41ef_0            conda-forge
tk:                 8.6.10-hed695b0_0            conda-forge
traitlets:          4.3.3-py36h9f0ad1d_1         conda-forge
urllib3:            1.25.9-py_0                  conda-forge
wheel:              0.34.2-py_1                  conda-forge
xz:                 5.2.5-h516909a_0             conda-forge
zipp:               3.1.0-py_0                   conda-forge
zlib:               1.2.11-h516909a_1006         conda-forge
zstd:               1.4.4-h6597ccf_3             conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... failed
ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::bzip2-1.0.8-h516909a_2'.
NotADirectoryError(20, 'Not a directory')
Attempting to roll back.

Rolling back transaction: ...working... done

NotADirectoryError(20, 'Not a directory')
@muryanto1
Copy link
Member

muryanto1 commented Jun 15, 2020 •
edited
Loading

@khouakhi Is it possible for you to use Miniconda 4.7.12?

wget https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh -O Miniconda3-4.7.12.1-Linux-x86_64.sh
bash ./Miniconda3-4.7.12.1-Linux-x86_64.sh -b -p miniconda3
source miniconda3/etc/profile.d/conda.sh
conda activate base

# Please pick up an unofficial release of cdat 8.2 on Linux
conda create -n cdat82 -c cdat/label/v82 -c conda-forge "cdat=8.2" "python=3.7" "numpy=1.17" mesalib="18.3.1" jupyter notebook ipywidgets
conda activate cdat82

# Note that I ran into problem when using latest miniconda3 on macos, I am not sure about # Linux, but we know for sure Miniconda 4.7.12 works with cdat 82 on both Linux and MacOS.

@khouakhi
Copy link
Author

I managed to install cdms2 and cdutil but I could not import the modules
Colab notebook here

!wget https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh
!chmod +x Miniconda3-4.7.12.1-Linux-x86_64.sh
!bash Miniconda3-4.7.12.1-Linux-x86_64.sh -b -f -p /usr/local
import sys
sys.path.append('/usr/local/lib/python3.6/site-packages/')

!conda install -q -y --prefix /usr/local cdms2 cdutil -c conda-forge

`The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
attrs-19.3.0               |             py_0          35 KB  conda-forge
bzip2-1.0.8                |       h516909a_2         396 KB  conda-forge
ca-certificates-2020.4.5.2 |       hecda079_0         147 KB  conda-forge
cdat_info-8.2              |             py_7          16 KB  conda-forge
cdms2-3.1.4                |   py37ha6f5e91_3         677 KB  conda-forge
cdtime-3.1.3               |   py37h006a570_3          55 KB  conda-forge
cdutil-8.2                 |             py_2         451 KB  conda-forge
certifi-2020.4.5.2         |   py37hc8dfbb8_0         152 KB  conda-forge
conda-4.8.3                |   py37hc8dfbb8_1         3.0 MB  conda-forge
curl-7.69.1                |       h33f0ec9_0         137 KB  conda-forge
decorator-4.4.2            |             py_0          11 KB  conda-forge
distarray-2.12.2           |             py_1          10 KB  conda-forge
esmf-8.0.1                 | nompi_hbeb3ca6_0        25.2 MB  conda-forge
esmpy-8.0.1                |nompi_py37h777d1d2_0         169 KB  conda-forge
expat-2.2.9                |       he1b5a44_2         191 KB  conda-forge
future-0.18.2              |   py37hc8dfbb8_1         712 KB  conda-forge
g2clib-1.6.0               |       hf3f1b0b_9          56 KB  conda-forge
genutil-8.2                |   py37hc1659b7_3         132 KB  conda-forge
hdf4-4.2.13                |    hf30be14_1003         964 KB  conda-forge
hdf5-1.10.6                |nompi_h3c11f04_100         3.0 MB  conda-forge
importlib-metadata-1.6.1   |   py37hc8dfbb8_0          44 KB  conda-forge
importlib_metadata-1.6.1   |                0           3 KB  conda-forge
ipython_genutils-0.2.0     |             py_1          21 KB  conda-forge
jasper-1.900.1             |    h07fcdf6_1006         286 KB  conda-forge
jpeg-9d                    |       h516909a_0         266 KB  conda-forge
jsonschema-3.2.0           |   py37hc8dfbb8_1          89 KB  conda-forge
jupyter_core-4.6.3         |   py37hc8dfbb8_1          71 KB  conda-forge
krb5-1.17.1                |       h173b8e3_0         1.3 MB
lazy-object-proxy-1.5.0    |   py37h8f50634_0          28 KB  conda-forge
libblas-3.8.0              |      11_openblas          10 KB  conda-forge
libcblas-3.8.0             |      11_openblas          10 KB  conda-forge
libcdms-3.1.2              |     h054cd8a_112         1.5 MB  conda-forge
libcf-1.0.3                | py37hda0e254_108         254 KB  conda-forge
libcurl-7.69.1             |       hf7181ac_0         573 KB  conda-forge
libdrs-3.1.2               |     hc2e2db3_112         526 KB  conda-forge
libdrs_f-3.1.2             |     hae7e664_110         534 KB  conda-forge
libgfortran-ng-7.5.0       |       hdf63c60_6         1.7 MB  conda-forge
liblapack-3.8.0            |      11_openblas          10 KB  conda-forge
libnetcdf-4.7.4            |nompi_h84807e1_104         1.3 MB  conda-forge
libopenblas-0.3.6          |       h6e990d7_6         7.7 MB  conda-forge
libpng-1.6.37              |       hed695b0_1         308 KB  conda-forge
libssh2-1.9.0              |       hab1572f_2         298 KB  conda-forge
libtiff-4.1.0              |       hc7e4089_6         668 KB  conda-forge
libuuid-2.32.1             |    h14c3975_1000          26 KB  conda-forge
libwebp-base-1.1.0         |       h516909a_3         845 KB  conda-forge
lz4-c-1.8.3                |    he1b5a44_1001         187 KB  conda-forge
nbformat-5.0.6             |             py_0          99 KB  conda-forge
netcdf-fortran-4.5.2       |nompi_hfef6a68_106         1.2 MB  conda-forge
numpy-1.18.5               |   py37h8960a57_0         5.1 MB  conda-forge
openblas-0.3.6             |       h6e990d7_6         8.2 MB  conda-forge
openssl-1.1.1g             |       h516909a_0         2.1 MB  conda-forge
pyrsistent-0.16.0          |   py37h8f50634_0          89 KB  conda-forge
python_abi-3.7             |          1_cp37m           4 KB  conda-forge
traitlets-4.3.3            |   py37hc8dfbb8_1         133 KB  conda-forge
udunits2-2.2.27.6          |    h4e0c4b3_1001         152 KB  conda-forge
zipp-3.1.0                 |             py_0          10 KB  conda-forge
zstd-1.4.4                 |       h3b9ef0a_2         982 KB  conda-forge
------------------------------------------------------------
                                       Total:        71.9 MB

The following NEW packages will be INSTALLED:

attrs conda-forge/noarch::attrs-19.3.0-py_0
bzip2 conda-forge/linux-64::bzip2-1.0.8-h516909a_2
cdat_info conda-forge/noarch::cdat_info-8.2-py_7
cdms2 conda-forge/linux-64::cdms2-3.1.4-py37ha6f5e91_3
cdtime conda-forge/linux-64::cdtime-3.1.3-py37h006a570_3
cdutil conda-forge/noarch::cdutil-8.2-py_2
curl conda-forge/linux-64::curl-7.69.1-h33f0ec9_0
decorator conda-forge/noarch::decorator-4.4.2-py_0
distarray conda-forge/noarch::distarray-2.12.2-py_1
esmf conda-forge/linux-64::esmf-8.0.1-nompi_hbeb3ca6_0
esmpy conda-forge/linux-64::esmpy-8.0.1-nompi_py37h777d1d2_0
expat conda-forge/linux-64::expat-2.2.9-he1b5a44_2
future conda-forge/linux-64::future-0.18.2-py37hc8dfbb8_1
g2clib conda-forge/linux-64::g2clib-1.6.0-hf3f1b0b_9
genutil conda-forge/linux-64::genutil-8.2-py37hc1659b7_3
hdf4 conda-forge/linux-64::hdf4-4.2.13-hf30be14_1003
hdf5 conda-forge/linux-64::hdf5-1.10.6-nompi_h3c11f04_100
importlib-metadata conda-forge/linux-64::importlib-metadata-1.6.1-py37hc8dfbb8_0
importlib_metadata conda-forge/noarch::importlib_metadata-1.6.1-0
ipython_genutils conda-forge/noarch::ipython_genutils-0.2.0-py_1
jasper conda-forge/linux-64::jasper-1.900.1-h07fcdf6_1006
jpeg conda-forge/linux-64::jpeg-9d-h516909a_0
jsonschema conda-forge/linux-64::jsonschema-3.2.0-py37hc8dfbb8_1
jupyter_core conda-forge/linux-64::jupyter_core-4.6.3-py37hc8dfbb8_1
krb5 pkgs/main/linux-64::krb5-1.17.1-h173b8e3_0
lazy-object-proxy conda-forge/linux-64::lazy-object-proxy-1.5.0-py37h8f50634_0
libblas conda-forge/linux-64::libblas-3.8.0-11_openblas
libcblas conda-forge/linux-64::libcblas-3.8.0-11_openblas
libcdms conda-forge/linux-64::libcdms-3.1.2-h054cd8a_112
libcf conda-forge/linux-64::libcf-1.0.3-py37hda0e254_108
libcurl conda-forge/linux-64::libcurl-7.69.1-hf7181ac_0
libdrs conda-forge/linux-64::libdrs-3.1.2-hc2e2db3_112
libdrs_f conda-forge/linux-64::libdrs_f-3.1.2-hae7e664_110
libgfortran-ng conda-forge/linux-64::libgfortran-ng-7.5.0-hdf63c60_6
liblapack conda-forge/linux-64::liblapack-3.8.0-11_openblas
libnetcdf conda-forge/linux-64::libnetcdf-4.7.4-nompi_h84807e1_104
libopenblas conda-forge/linux-64::libopenblas-0.3.6-h6e990d7_6
libpng conda-forge/linux-64::libpng-1.6.37-hed695b0_1
libssh2 conda-forge/linux-64::libssh2-1.9.0-hab1572f_2
libtiff conda-forge/linux-64::libtiff-4.1.0-hc7e4089_6
libuuid conda-forge/linux-64::libuuid-2.32.1-h14c3975_1000
libwebp-base conda-forge/linux-64::libwebp-base-1.1.0-h516909a_3
lz4-c conda-forge/linux-64::lz4-c-1.8.3-he1b5a44_1001
nbformat conda-forge/noarch::nbformat-5.0.6-py_0
netcdf-fortran conda-forge/linux-64::netcdf-fortran-4.5.2-nompi_hfef6a68_106
numpy conda-forge/linux-64::numpy-1.18.5-py37h8960a57_0
openblas conda-forge/linux-64::openblas-0.3.6-h6e990d7_6
pyrsistent conda-forge/linux-64::pyrsistent-0.16.0-py37h8f50634_0
python_abi conda-forge/linux-64::python_abi-3.7-1_cp37m
traitlets conda-forge/linux-64::traitlets-4.3.3-py37hc8dfbb8_1
udunits2 conda-forge/linux-64::udunits2-2.2.27.6-h4e0c4b3_1001
zipp conda-forge/noarch::zipp-3.1.0-py_0
zstd conda-forge/linux-64::zstd-1.4.4-h3b9ef0a_2

The following packages will be UPDATED:

ca-certificates pkgs/main::ca-certificates-2019.10.16~ --> conda-forge::ca-certificates-2020.4.5.2-hecda079_0
certifi pkgs/main::certifi-2019.9.11-py37_0 --> conda-forge::certifi-2020.4.5.2-py37hc8dfbb8_0
conda pkgs/main::conda-4.7.12-py37_0 --> conda-forge::conda-4.8.3-py37hc8dfbb8_1
openssl pkgs/main::openssl-1.1.1d-h7b6447c_3 --> conda-forge::openssl-1.1.1g-h516909a_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

import cdms2

@muryanto1
Copy link
Member

I cannot tell why you cannot import cdms2.
Would you be willing to try creating a separate environment just like I showed above?

@khouakhi
Copy link
Author

Installed CDAT and 'import cv2' works but

import cdms2

ImportError Traceback (most recent call last)
in ()
----> 1 import cdms2

1 frames
/usr/local/lib/python3.7/site-packages/cdms2/dataset.py in ()
6 from .error import CDMSError
7 import sys
----> 8 from . import Cdunif
9 import numpy
10 from . import cdmsNode

ImportError: cannot import name 'Cdunif'

@Xunius Xunius mentioned this issue Jun 24, 2020
Open
@jasonb5
Copy link
Contributor

jasonb5 commented Jun 29, 2020

@khouakhi The mixing of these two environments, colab and conda, is not something we support. It can create undesirable behavior. In your colab environment, cdms2 works well when running using the python it was installed with, using it any other way than this is not supported. The following demonstrates cdms2 working correctly using the python it was installed with.

%%bash
# Numpy was missing
conda install -c conda-forge "numpy=1.17"
/usr/local/bin/python -c "import cdms2; print(cdms2);
f = cdms2.open('http://aims3.llnl.gov/thredds/dodsC/user_pub_work/CMIP6/CMIP/E3SM-Project/E3SM-1-1/piControl/r1i1p1f1/Amon/cl/gr/v20191029/cl_Amon_E3SM-1-1_piControl_r1i1p1f1_gr_185001-187412.nc');
print(f.variables)"

At this time the only recommendation we can make to continue using colab is the following example.

%%bash
cat << EOF > test.py
import cdms2
f = cdms2.open('http://aims3.llnl.gov/thredds/dodsC/user_pub_work/CMIP6/CMIP/E3SM-Project/E3SM-1-1/piControl/r1i1p1f1/Amon/cl/gr/v20191029/cl_Amon_E3SM-1-1_piControl_r1i1p1f1_gr_185001-187412.nc')
print(f.variables)
EOF

/usr/local/bin/python test.py

@khouakhi
Copy link
Author

@jasonb5 I added numpy as suggested and used cat << EOF > test.py
not I understand your demo

get miniconda

!wget -c https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
!chmod +x Miniconda3-latest-Linux-x86_64.sh
!bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local

conda install cdms2 with numpy 1.17

!conda install -c conda-forge "numpy=1.17" cdms2

import cdms2

cat << EOF > test.py
import cdms2
/bin/bash: warning: here-document at line 0 delimited by end-of-file (wanted `EOF')

@jasonb5
Copy link
Contributor

jasonb5 commented Jul 8, 2020

@khouakhi Looks like you might be missing the ending line with EOF

cat << EOF > test.py
import cdms2
EOF

@jasonb5 jasonb5 added the stale No movement on issue. label Aug 4, 2020
@jasonb5
Copy link
Contributor

jasonb5 commented Aug 4, 2020

@khouakhi Please reopen if you still need assistance.

@jasonb5 jasonb5 closed this as completed Aug 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale No movement on issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jasonb5 @khouakhi @muryanto1