assume we are working with Ubuntu 16.04 xenial and Docker 17.09.1-ce:<\/p>\n
\r\n$ lsb_release -a\r\nNo LSB modules are available.\r\nDistributor ID:\tUbuntu\r\nDescription:\tUbuntu 16.04.3 LTS\r\nRelease:\t16.04\r\nCodename:\txenial\r\n\r\n$ docker --version\r\nDocker version 17.09.1-ce, build 19e2cf6\r\n<\/pre>\n1. go to ngc.nvidia.com and get NGC tensorflow image link. Then make docker login to NGC registry:<\/p>\n
\r\ndocker login nvcr.io\r\n\r\nUsername: $oauthtoken\r\nPassword: <Your Key>\r\n<\/pre>\nWe are prompted here to type username which is «$oauthtoken» and password which is an API key. API key is generated at the NGC user account, in the «Configuration» section.
\nNow we get NGC tensorflow image. Go to «Registry» section, choose nvidia\/tensorflow image section, scroll down and copy the needed (latest) tag pull command. Currently its 17.12 tag and the command is the following:<\/p>\n\r\n$ docker pull nvcr.io\/nvidia\/tensorflow:17.12\r\n<\/pre>\nNow we have the image in the local images registry:<\/p>\n
\r\n$ docker images\r\nREPOSITORY TAG IMAGE ID CREATED SIZE\r\nhello-world latest f2a91732366c 5 weeks ago 1.85kB\r\nnvcr.io\/nvidia\/tensorflow 17.12 19afd620fc8e 5 weeks ago 2.88GB\r\nubuntu latest 20c44cd7596f 5 weeks ago 123MB\r\n<\/pre>\nNow we will make our own image based on NVIDIA one.
\n1. getminiconda<\/code> installer:<\/p>\n\r\nwget https:\/\/repo.continuum.io\/miniconda\/Miniconda3-4.2.12-Linux-x86_64.sh\r\n<\/pre>\nI am downloading the specific 4.2 version as a last one with
python=3.5<\/code> used by default.<\/p>\n2. now create
Dockerfile<\/code> for the first stage:<\/p>\n\r\nFROM nvcr.io\/nvidia\/tensorflow:17.12\r\nMAINTAINER your_name_here <your_email_here>\r\nENV LANG=C.UTF-8 LC_ALL=C.UTF-8\r\nENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:\/usr\/lib\/x86_64-linux-gnu\r\n\r\nRUN apt-get update --fix-missing && apt-get install -y wget bzip2 ca-certificates \\\r\n libglib2.0-0 libxext6 libsm6 libxrender1 \\\r\n git mercurial subversion\r\n\r\nCOPY .\/Miniconda3-4.2.12-Linux-x86_64.sh \/root\/miniconda.sh\r\n\r\nRUN echo 'export PATH=\/opt\/conda\/bin:$PATH' > \/etc\/profile.d\/conda.sh && \\\r\n \/bin\/bash \/root\/miniconda.sh -b -p \/opt\/conda && \\\r\n rm \/root\/miniconda.sh\r\n\r\nENV PATH=\/opt\/conda\/bin:$PATH\r\n\r\nRUN apt-get install -y libgtk2.0 && \\\r\n rm -rf \/var\/lib\/apt\/lists\/*\r\n\r\nRUN conda update --all -y && \\\r\n conda install numpy=1.11.0 pandas matplotlib scikit-learn seaborn scipy tqdm jupyter ipython -y && \\\r\n conda install -c conda-forge ipywidgets netcdf4 basemap graphviz -y && \\\r\n conda install -c jaikumarm hyperopt\r\n\r\nRUN conda config --add channels conda-forge && \\\r\n conda config --add channels intel && \\\r\n conda config --add channels menpo && \\\r\n conda install -c menpo opencv3=3.1.0 -y\r\n\r\nRUN conda clean -y -all && \\\r\n rm -rf \/opt\/conda\/pkgs\/*\r\n\r\nCOPY .\/jupyter_notebook_config.py \/root\/.jupyter\/\r\n\r\nRUN conda install h5py -y\r\n\r\nVOLUME [ "\/app", "\/data" ]\r\nEXPOSE 8888\r\nWORKDIR \/app\r\nENTRYPOINT [ "\/usr\/local\/bin\/nvidia_entrypoint.sh" ]\r\nCMD ["jupyter", "notebook", "--ip='*'", "--port=8888", "--no-browser", "--allow-root", "--debug"]\r\n<\/pre>\n
jupyter_notebook_config.py<\/code> is a file that will configure jupyter notebook parameters on start. Right now its content is the following:<\/p>\n\r\nc.NotebookApp.ip = '*'\r\nc.NotebookApp.open_browser = False\r\n\r\n## Hashed password to use for web authentication.\r\n# \r\n# To generate, type in a python\/IPython shell:\r\n# \r\n# from notebook.auth import passwd; passwd()\r\n# \r\n# The string should be of the form type:salt:hashed-password.\r\nc.NotebookApp.password = 'sha1:your_password_hash_here'\r\n\r\nc.NotebookApp.port = 8888\r\n<\/pre>\nNote here that we are installing
numpy=1.11.0<\/code>. It is tensorflow 1.4 requirement.
\nAlso note here that we are installingopencv3=3.1.0<\/code>. It is the version that will not conflict withmatplotlib<\/code>,numpy=1.11.0<\/code> and others in terms of dependencies. The repository foropencv3<\/code> ismenpo<\/code>: it is the one containing opencv3 withcontrib<\/code> modules.<\/p>\n3. From this step we will compose docker image of the stage 1 (i`ve named it «
nvcr.io\/mk\/py35_nv_tf:int01<\/code>«).
\nRemember that we have to do it in the directory whereDockerfile<\/code>,jupyter_notebook_config.py<\/code> andMiniconda3-4.2.12-Linux-x86_64.sh<\/code> are located.<\/p>\n\r\n$ docker build -t nvcr.io\/mk\/py35_nv_tf:int01 .\r\n<\/pre>\nwe may check now if the docker container can be run here:<\/p>\n
\r\n$ nvidia-docker run -it --rm nvcr.io\/mk\/py35_nv_tf:int01 \/bin\/bash\r\n<\/pre>\n4. Now we will launch docker container using this image and build TF inside it. We are using this approach because of NVIDIA docker runtime usage that distribute some necessary libraries in
nvidia-docker run\/exec ...<\/code> case only.
\nParticularlylibcuda.so.1<\/code> of the proper version:
\n(inside container)<\/p>\n\r\n# ll \/usr\/lib\/x86_64-linux-gnu | grep libcuda.so\r\nlrwxrwxrwx 1 root root 17 Dec 25 21:41 libcuda.so -> libcuda.so.387.34\r\nlrwxrwxrwx 1 root root 17 Dec 25 21:41 libcuda.so.1 -> libcuda.so.387.34\r\n-rw-r--r-- 1 root root 11008248 Nov 21 10:18 libcuda.so.387.34\r\n<\/pre>\nSo we build TF inside the container, and then commit all changes made to the image of the stage 2.
\nlaunch container:<\/p>\n\r\n$ nvidia-docker run -it --rm nvcr.io\/mk\/py35_nv_tf:int01 \/bin\/bash\r\n<\/pre>\nand inside this container:<\/p>\n
\r\n# cd \/opt\/tensorflow\/\r\n# export TF_UNOFFICIAL_SETTING=1\r\n# yes "" | .\/configure\r\n# bazel build -c opt --config=cuda tensorflow\/tools\/pip_package:build_pip_package\r\n<\/pre>\nthis will take a long — about 20min.
\n(still inside container)<\/p>\n\r\n# bazel-bin\/tensorflow\/tools\/pip_package\/build_pip_package \/tmp\/tf_pip\r\n# pip install --upgrade \/tmp\/tf_pip\/tensorflow-*.whl\r\n# rm -rf \/tmp\/tf_pip\/tensorflow-*.whl\r\n# bazel clean --expunge\r\n\r\n# pip install keras==2.0.8\r\n<\/pre>\nand now one more step: i am going to make Keras download VGG16 head:<\/p>\n
\r\n# ipython\r\n<\/pre>\n\r\nimport keras\r\nfrom keras.applications import VGG16\r\nconv_base1 = VGG16(weights='imagenet', include_top=False, input_shape=(100, 100, 3))\r\n<\/pre>\nhere Keras will download the file
vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5<\/code> to directory\/root\/.keras\/models\/<\/code><\/p>\nAfter this we are going to detach from the container using the
<Ctrl>+<Q> -> <Ctrl>+<P><\/code> keys sequence.<\/p>\nAnd now commit all changes from the container to the new image:<\/p>\n
\r\n$ docker commit 87bcd85ec902 nvcr.io\/mk\/py35_nv_tf\r\n<\/pre>\n87bcd85ec902 here is the ID of docker container where we have built TF.
\nafter this we may stop and remove this container:<\/p>\n\r\n$ docker stop 87bcd85ec902\r\n$ docker rm 87bcd85ec902\r\n<\/pre>\n5. some polish. Create the new Dockerfile containing the following:<\/p>\n
\r\nFROM nvcr.io\/mk\/py35_nv_tf:int02\r\n\r\nMAINTAINER YOUR NAME HERE <your@email.here>\r\n\r\nVOLUME [ "\/app", "\/data" ]\r\nEXPOSE 8888\r\n\r\nWORKDIR \/app\r\n\r\nENTRYPOINT [ "\/usr\/local\/bin\/nvidia_entrypoint.sh" ]\r\n\r\nCMD ["jupyter", "notebook", "--ip='*'", "--port=8888", "--no-browser", "--allow-root", "--debug"]\r\n<\/pre>\nrename
nvcr.io\/mk\/py35_nv_tf:latest<\/code> tonvcr.io\/mk\/py35_nv_tf:int02<\/code> and build the new image<\/p>\n\r\n$ docker tag nvcr.io\/mk\/py35_nv_tf:latest nvcr.io\/mk\/py35_nv_tf:int02\r\n$ docker build -t nvcr.io\/mk\/py35_nv_tf .\r\n<\/pre>\nafter these 5 steps we are able to use Python 3.5 with NVIDIA-optimized Tensorflow 1.4 build for Python 3.5. And there is an opencv3 in the image as well.
\nThe container may be run with the following (jupyter notebook server will start automatically):<\/p>\n\r\n$ nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 -it --rm --mount type=bind,source=$HOME\/py35docker_app\/,target=\/app -p 8888:8888 nvcr.io\/mk\/py35_nv_tf\r\n<\/pre>\nor the following:<\/p>\n
\r\n$ nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 -it --rm --mount type=bind,source=$HOME\/py35docker_app\/,target=\/app -p 8888:8888 nvcr.io\/mk\/py35_nv_tf \/bin\/bash\r\n<\/pre>\nin this case we will get command line of the container.<\/p>\n","protected":false},"excerpt":{"rendered":"
assume we are working with Ubuntu 16.04 xenial and Docker 17.09.1-ce: 1. go to ngc.nvidia.com and get NGC tensorflow image link. Then make docker login to NGC registry: We are prompted here to type username which is «$oauthtoken» and password which is an API key. API key is generated at the NGC user account, in… \u0427\u0438\u0442\u0430\u0442\u044c \u0434\u0430\u043b\u0435\u0435 NVIDIA NGC tensorflow image py27 to py35 with OpenCV (opencv3 for python)<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3,4],"tags":[],"_links":{"self":[{"href":"http:\/\/mbrdancer.ru\/index.php?rest_route=\/wp\/v2\/posts\/59"}],"collection":[{"href":"http:\/\/mbrdancer.ru\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/mbrdancer.ru\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/mbrdancer.ru\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/mbrdancer.ru\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=59"}],"version-history":[{"count":30,"href":"http:\/\/mbrdancer.ru\/index.php?rest_route=\/wp\/v2\/posts\/59\/revisions"}],"predecessor-version":[{"id":92,"href":"http:\/\/mbrdancer.ru\/index.php?rest_route=\/wp\/v2\/posts\/59\/revisions\/92"}],"wp:attachment":[{"href":"http:\/\/mbrdancer.ru\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=59"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/mbrdancer.ru\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=59"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/mbrdancer.ru\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=59"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}