Julia不能在Docker容器中加载大数据
我有一个加载8 GB数据的Julia程序。 它在我的本地机器工作正常。
但是当我在Docker容器中尝试它时，它不会加载数据并给出总线错误。 它在docker容器中使用20 MB这样的小数据工作正常。
Dockerfile
FROM ubuntu:16.04 WORKDIR /julia RUN apt-get -y update RUN apt-get -y install unzip RUN apt-get -y install cmake RUN apt-get -y install clang RUN apt-get -y install wget RUN cd /tmp/ RUN wget "https://julialang.s3.amazonaws.com/bin/linux/x64/0.5/julia-0.5.0-linux-x86_64.tar.gz" RUN tar -xzvf julia-0.5.0-linux-x86_64.tar.gz RUN mv julia-3c9d75391c/ ~/julia ENV PATH="/root/julia/bin:${PATH}" RUN julia --eval 'Pkg.add("JSON")' RUN julia --eval 'Pkg.add("HttpServer")' RUN julia --eval 'Pkg.add("URIParser")' RUN julia --eval 'Pkg.clone("https://github.com/deep-compute/AdaGram.jl.git")' RUN julia --eval 'Pkg.build("AdaGram")' CMD ["julia", "/tmp/adagram_server.jl", "80", "/julia/full.embed"] 泊坞窗，compose.yml
version: "3.1" services: julia: image: ramidavalapati/julia:v-1 volumes: - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed ports: - 8080:80 command: ["sleep", "1h"]
接下来我在做
sudo docker-compose up sudo docker exec -it $(sudo docker-compose ps -q julia) bash root@3748d5958f77:/julia# julia julia> using AdaGram julia> AdaGram.load_model("/julia/full.embed")
错误
signal (7): Bus error while loading no file, in expression starting on line 0 macro expansion at ./cartesian.jl:62 [inlined] macro expansion at ./multidimensional.jl:429 [inlined] _unsafe_batchsetindex! at ./multidimensional.jl:421 _setindex! at ./multidimensional.jl:370 [inlined] setindex! at ./abstractarray.jl:832 [inlined] #9 at /root/.julia/v0.5/AdaGram/src/AdaGram.jl:64 #600 at ./multi.jl:1030 run_work_thunk at ./multi.jl:1001 run_work_thunk at ./multi.jl:1010 [inlined] #597 at ./event.jl:68 unknown function (ip: 0x7fe1822db16f) jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:189 [inlined] jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1942 jl_apply at /home/centos/buildbot/slave/package_tarball64/build/src/julia.h:1392 [inlined] start_task at /home/centos/buildbot/slave/package_tarball64/build/src/task.c:253 unknown function (ip: 0xffffffffffffffff) Allocations: 9661042 (Pool: 9659980; Big: 1062); GC: 16 Bus error (core dumped)
docker版本
Client: Version: 17.09.0-ce API version: 1.32 Go version: go1.8.3 Git commit: afdb6d4 Built: Tue Sep 26 22:42:18 2017 OS/Arch: linux/amd64 Server: Version: 17.09.0-ce API version: 1.32 (minimum version 1.12) Go version: go1.8.3 Git commit: afdb6d4 Built: Tue Sep 26 22:40:56 2017 OS/Arch: linux/amd64 Experimental: false
docker信息
Containers: 24 Running: 0 Paused: 0 Stopped: 24 Images: 24 Server Version: 17.09.0-ce Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: active NodeID: jlkmigmtyjhz6yvi3zuvkobu7 Is Manager: true ClusterID: rqt03ulgvvnym235m1qm8vd17 Managers: 4 Nodes: 15 Orchestration: Task History Retention Limit: 5 Raft: Snapshot Interval: 10000 Number of Old Snapshots to Retain: 0 Heartbeat Tick: 1 Election Tick: 3 Dispatcher: Heartbeat Period: 5 seconds CA Configuration: Expiry Duration: 3 months Force Rotate: 0 Autolock Managers: false Root Rotation In Progress: false Node Address: XXXX Manager Addresses: XXXX:2377 XXXX:2377 XXXX:2377 XXXX:2377 Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0 runc version: 3f2f8b84a77f73d38244dd690525642a72156c64 init version: 949e6fa Security Options: apparmor seccomp Profile: default Kernel Version: 4.10.0-35-generic Operating System: Ubuntu 16.04.3 LTS OSType: linux Architecture: x86_64 CPUs: 12 Total Memory: 251.8GiB Name: ram ID: 3OGG:275C:Q3IW:O4HX:DPLP:DPI3:5TIT:AG5J:EDMK:7NK3:L4UZ:BTQH Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Username: ramidavalapati Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support
在此先感谢您提供任何帮助
链接到40MBfull.embed文件
问题是，docker容器中没有足够的共享内存（默认是64 MB）。
问题是通过在运行--shm-size提供选项--shm-size来解决的。
docker – 撰写文件
version: "3.1" services: julia: image: ramidavalapati/julia:v-1 shm_size: 1g volumes: - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed ports: - 8080:80 command: ["sleep", "1h"]
如果我们想在swarm mode工作，我们需要在卷段中引用共享内存。
version: "3.3" services: julia: image: ramidavalapati/julia:v-1 volumes: - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed - /dev/shm:/dev/shm ports: - 8080:80 command: ["sleep", "1h"]
这里容器使用它所运行的主机的共享内存。