Docker构buildDockerfile,内存更多

如何使用更多的内存Docker docker build Dockerfile?

这是一个不同的问题, 当docker构buildDockerfile时,允许更多的内存

在本地安装软件时,有足够的内存来成功构build和安装marian工具

但是,当使用Dockerfile https://github.com/marian-nmt/marian/blob/master/scripts/docker/Dockerfile.cpu构buildDocker镜像时,会出现多个内存耗尽错误

 virtual memory exhausted: Cannot allocate memory 

[OUT]:

 Step : RUN cmake $MARIANPATH && make -j ---> Running in 4867d166d17a -- The C compiler identification is GNU 5.4.0 -- The CXX compiler identification is GNU 5.4.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done CUDA_TOOLKIT_ROOT_DIR not found or specified -- Cannot find CUDA libraries. Compiling without them. -- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Boost version: 1.58.0 -- Found the following Boost libraries: -- system -- filesystem -- program_options -- timer -- iostreams -- python -- thread -- chrono -- regex -- date_time -- atomic -- Found Python -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.12", minimum required is "2.7") -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.8") -- Found Git: /usr/bin/git (found version "2.7.4") -- Git version: 5abc774 -- Found SparseHash: /usr/include -- Configuring done -- Generating done -- Build files have been written to: /marian/build Scanning dependencies of target fast_align Scanning dependencies of target extract_lex Scanning dependencies of target libcnpy Scanning dependencies of target atools Scanning dependencies of target libyaml-cpp-amun [ 2%] Building CXX object src/amun/3rd_party/extract_lex/CMakeFiles/extract_lex.dir/exception.cpp.o [ 2%] Building CXX object src/amun/3rd_party/extract_lex/CMakeFiles/extract_lex.dir/utils.cpp.o [ 3%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/fast_align.dir/src/ttables.cc.o [ 6%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/fast_align.dir/src/fast_align.cc.o [ 7%] Building CXX object src/amun/3rd_party/extract_lex/CMakeFiles/extract_lex.dir/extract-lex-main.cpp.o [ 10%] Building CXX object src/amun/3rd_party/CMakeFiles/libcnpy.dir/cnpy/cnpy.cpp.o [ 10%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/atools.dir/src/atools.cc.o [ 10%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/atools.dir/src/alignment_io.cc.o [ 11%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/regex_yaml.cpp.o [ 12%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scanner.cpp.o [ 14%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/convert.cpp.o [ 15%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/stream.cpp.o [ 16%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/memory.cpp.o [ 17%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/node.cpp.o [ 19%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/directives.cpp.o [ 20%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/null.cpp.o [ 21%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitfromevents.cpp.o [ 23%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/singledocparser.cpp.o [ 24%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitterstate.cpp.o [ 25%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scantag.cpp.o [ 26%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/simplekey.cpp.o [ 28%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/binary.cpp.o [ 29%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/parser.cpp.o [ 30%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/nodeevents.cpp.o [ 32%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emit.cpp.o [ 33%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/exp.cpp.o [ 35%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/nodebuilder.cpp.o [ 38%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scanscalar.cpp.o [ 39%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitter.cpp.o [ 41%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/tag.cpp.o [ 42%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scantoken.cpp.o [ 43%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/ostream_wrapper.cpp.o Scanning dependencies of target libcommon [ 37%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/node_data.cpp.o [ 44%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/contrib/graphbuilder.cpp.o [ 46%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitterutils.cpp.o [ 34%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/contrib/graphbuilderadapter.cpp.o [ 47%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/parse.cpp.o [ 48%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/scorer.cpp.o [ 50%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/git_version.cpp.o [ 51%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/base_matrix.cpp.o [ 52%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/sentence.cpp.o [ 53%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/config.cpp.o [ 55%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/utils.cpp.o [ 56%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/vocab.cpp.o [ 65%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/filter.cpp.o [ 66%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/god.cpp.o [ 67%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/exception.cpp.o [ 69%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/search.cpp.o [ 69%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/hypothesis.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/translation_task.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/printer.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/sentences.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/history.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/logging.cpp.o [ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/processor/bpe.cpp.o [ 71%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/output_collector.cpp.o [ 73%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/types.cpp.o [ 74%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/loader.cpp.o virtual memory exhausted: Cannot allocate memory virtual memory exhausted: Cannot allocate memory virtual memory exhausted: Cannot allocate memory 

最终导致:

 src/amun/CMakeFiles/libcommon.dir/build.make:254: recipe for target 'src/amun/CMakeFiles/libcommon.dir/common/loader.cpp.o' failed virtual memory exhausted: Cannot allocate memory make[2]: *** [src/amun/CMakeFiles/cpumode.dir/cpu/decoder/encoder_decoder.cpp.o] Error 1 src/amun/CMakeFiles/cpumode.dir/build.make:110: recipe for target 'src/amun/CMakeFiles/cpumode.dir/cpu/decoder/encoder_decoder.cpp.o' failed [ 79%] Built target libcnpy virtual memory exhausted: Cannot allocate memory src/amun/CMakeFiles/libcommon.dir/build.make:326: recipe for target 'src/amun/CMakeFiles/libcommon.dir/common/printer.cpp.o' failed make[2]: *** [src/amun/CMakeFiles/libcommon.dir/common/printer.cpp.o] Error 1 CMakeFiles/Makefile2:340: recipe for target 'src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/all' failed make[1]: *** [src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/all] Error 2 CMakeFiles/Makefile2:182: recipe for target 'src/amun/CMakeFiles/libcommon.dir/all' failed make[1]: *** [src/amun/CMakeFiles/libcommon.dir/all] Error 2 make[1]: *** [src/amun/CMakeFiles/cpumode.dir/all] Error 2 CMakeFiles/Makefile2:110: recipe for target 'src/amun/CMakeFiles/cpumode.dir/all' failed make: *** [all] Error 2 

看着docker build --help ,有选项来控制内存使用情况:

 $ docker build --help Usage: docker build [OPTIONS] PATH | URL | - Build an image from a Dockerfile Options: --build-arg list Set build-time variables (default []) --cache-from stringSlice Images to consider as cache sources --cgroup-parent string Optional parent cgroup for the container --compress Compress the build context using gzip --cpu-period int Limit the CPU CFS (Completely Fair Scheduler) period --cpu-quota int Limit the CPU CFS (Completely Fair Scheduler) quota -c, --cpu-shares int CPU shares (relative weight) --cpuset-cpus string CPUs in which to allow execution (0-3, 0,1) --cpuset-mems string MEMs in which to allow execution (0-3, 0,1) --disable-content-trust Skip image verification (default true) -f, --file string Name of the Dockerfile (Default is 'PATH/Dockerfile') --force-rm Always remove intermediate containers --help Print usage --isolation string Container isolation technology --label list Set metadata for an image (default []) -m, --memory string Memory limit --memory-swap string Swap limit equal to memory plus swap: '-1' to enable unlimited swap --network string Set the networking mode for the RUN instructions during build (default "default") --no-cache Do not use cache when building the image --pull Always attempt to pull a newer version of the image -q, --quiet Suppress the build output and print image ID on success --rm Remove intermediate containers after a successful build (default true) --security-opt stringSlice Security options --shm-size string Size of /dev/shm, default value is 64MB -t, --tag list Name and optionally a tag in the 'name:tag' format (default []) --ulimit ulimit Ulimit options (default []) 

但是我找不出正确的语法,把-m选项放在哪里-_- |||

这不是在Dockerfile之前:

 # Before Docker file. $ docker build -m 4g Dockerfile.cpu -t ibot-cpu . "docker build" requires exactly 1 argument(s). See 'docker build --help'. Usage: docker build [OPTIONS] PATH | URL | - Build an image from a Dockerfile 

这不是在-t之前的Dockerfile之后

 # Before -t $ docker build Dockerfile.cpu -m 4g -t ibot-cpu . "docker build" requires exactly 1 argument(s). See 'docker build --help'. Usage: docker build [OPTIONS] PATH | URL | - Build an image from a Dockerfile 

这不是在本地path之前

 # Before local path $ docker build Dockerfile.cpu -t ibot-cpu -m 4g . "docker build" requires exactly 1 argument(s). See 'docker build --help'. Usage: docker build [OPTIONS] PATH | URL | - Build an image from a Dockerfile 

这也不是在最后的本地path之后…

 # At the end... $ docker build Dockerfile.cpu -t ibot-cpu . -m 4g "docker build" requires exactly 1 argument(s). See 'docker build --help'. Usage: docker build [OPTIONS] PATH | URL | - Build an image from a Dockerfile 

如何使用更多的内存Docker docker build Dockerfile?

我的docker版本:

 docker version Client: Version: 17.03.1-ce API version: 1.27 Go version: go1.7.5 Git commit: c6d412e Built: Tue Mar 28 00:40:02 2017 OS/Arch: darwin/amd64 Server: Version: 17.04.0-ce API version: 1.28 (minimum version 1.12) Go version: go1.7.5 Git commit: 4845c56 Built: Wed Apr 5 18:45:47 2017 OS/Arch: linux/amd64 Experimental: false 

这不是关于秩序的事情。 Dockerfile必须用-f指定

 docker build -f Dockerfile.cpu -t ibot-cpu -m 4g . 

正如你所看到的帮助。

但是,考虑到默认情况下,docker不会限制容器内存。 它可以采取无空洞的记忆。

我可以看到你在OSX上,它通过Linux VM运行Docker。 configuration最大内存点击任务栏中的鲸鱼图标。 默认是2G。

有关更多信息,请参阅我的其他答案: 如何分配更多的内存到docker集装箱

    Interesting Posts