Docker creates a mirror and uploads it to the cloud server

The relationship between docker container and image

• mirror image:
  • equivalent to a class
  • Unmodifiable content
• container:
  • For instances of mirrored classes, libraries can be updated in the environment
  • Containers can be saved as a new image
  • Then according to the saved new image, the container of the new image can be instantiated

1. Basic image related file creation

1.1 Create dockerfile

• document content

  # linux基础系统
  FROM nvidia/cuda:11.7.1-runtime-ubuntu22.04
  # 修改镜像源
  RUN sed -i s/archive.ubuntu.com/mirrors.aliyun.com/g /etc/apt/sources.list
  RUN sed -i s/security.ubuntu.com/mirrors.aliyun.com/g /etc/apt/sources.list
  RUN sed -i s/ports.ubuntu.com/mirrors.aliyun.com/g /etc/apt/sources.list
  # 安装基本环境库  vim unzip git wget python3 python3-venv python3-pip build-essential libgl-dev libglib2.0-0 wget
  RUN apt update && apt-get -y install vim unzip git wget python3 python3-venv python3-pip build-essential libgl-dev libglib2.0-0 wget
  # pip 代理
  RUN pip config set global.index-url https://mirrors.cloud.tencent.com/pypi/simple
  RUN pip config set global.trusted-host mirrors.cloud.tencent.com
  # 创建自己目录
  RUN mkdir /home/data  && chmod 777 /home/data
  # 将/home/data作为根目录
  WORKDIR /home/data
  
  # 本地文件复制上传到容器
  # 文件上传文件
  # copy load_file_path docker_file_path
  COPY upload_server_api.py /home/data/upload_server_api.py
  COPY upload_server_webui.py /home/data/upload_server_webui.py
  COPY do.sh  /home/data/do.sh
  
  # 赋予.sh权限
  RUN chmod 777 /home/data/do.sh
  # 执行.sh文件
  RUN /home/data/do.sh
  # 暴漏端口，连接的端口
  EXPOSE 7860
  #暴漏 upload_server_api.py端口
  EXPOSE 5566
  #暴漏 upload_server_webui.py端口
  EXPOSE 5560
  ENTRYPOINT ["python3","-m","http.server","2023"]
  

1.2. Create do.sh file

• document content

  #!/bin/bash
  # 进入根目录
  cd /home/data
  # 创建python虚拟环境，虚拟环境名为venv
  python3 -m venv venv
  # 虚拟环境安装相关库
  /home/data/venv/bin/python -m pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/ bottle
  /home/data/venv/bin/python -m pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/ flask
  /home/data/venv/bin/python -m pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/ cgai-io
  /home/data/venv/bin/python -m pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/ cgai-socket
  /home/data/venv/bin/python -m pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/ gradio
  # 后台运行upload_server_api.py,upload_server_webui.py文件
  nohup /home/data/venv/bin/python -u /home/data/upload_server_api.py >/dev/null 2>&1 &
  nohup /home/data/venv/bin/python -u /home/data/upload_server_webui.py >/dev/null 2>&1 &
  

1.3 Create upload_server_api.py file

• document content

  import os
  from bottle import route,request,static_file,run
  
  # 文件上传
  @route('/upload',method='POST')
  def do_upload():
      upload = request.files.get('upload')
      _dir = request.forms.get('dir')
      _dir = _dir.replace('\\','/')
      if _dir.endswith('/'):
          _dir = _dir[:-1]
      save_path = f'{
          
          _dir}/{
          
          upload.filename}'
      print('save_path:',save_path)
      upload.save(save_path,overwrite=True)
      return f'保存文件到:{
          
          save_path}'
  
  # 查看目录文件
  @route('/check',method='GET')
  def check():
      _dir = request.query.dir
      files = os.listdir(_dir)
      s = ''
      for i in files:
          s += i + '\n'
      return s
  if __name__=='__main__':
      run(host='0.0.0.0',port=5566)    
  

1.4 Create upload_server_webui.py file

• document content

  import os
  import socket
  import gradio as gr
  from cgai_io import copyfile,mvfile
  
  def get_ip():
      s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
      s.connect(("8.8.8.8",80))
      a = s.getsockname()      
      return a[0]
  def upload(save_dir,temp_file):
      file_path = temp_file.name
      file_name = os.path.basename(file_path)
      name,t = os.path.splitext(file_name)
      fs = name.split('_')
      fs.remove(fs[-1])
      oname = '_'.join(fs) + t
      save_path = os.path.join(save_dir,oname)
      mvfile(file_path,save_path)
      return f'上传到路径: {
          
          save_path}'
  
  def check(save_dir):
      files = os.listdir(save_dir)
      s = ''
      for i in files:
          s += i + '\n'
      return s
  
  with gr.Blocks() as demo:
      gr.Markdown(f"当前连接ip:  {
          
          get_ip()}")
      with gr.Column():
          save_dir = gr.Text(label='上传路径',value='/home')
          local_file = gr.File(label='上传文件(注意：1.文件名称不要有下划线_,否则上传后的文件名称可能有误) 2.文件大小超过4G请使用客户端上传')
      with gr.Row():
          upload_btn = gr.Button("上传")
          check_btn = gr.Button('检测')
      tip = gr.Text('',label='')
      upload_btn.click(fn=upload, inputs=[save_dir,local_file], outputs=tip)
      check_btn.click(fn=check, inputs=[save_dir], outputs=tip)
  
  demo.launch(server_name='0.0.0.0',server_port=5560)
  

1.5 File storage location

• Stored in the same folder, mine is stored in test_dockerthe folder
  

2. Create a mirror operation

2.1 Create a mirror image

• Related parameters

  parameter	meaning
  - f	dockerfile文件路径
  - t	新镜像标签名
  .	指明docker context为当前目录，context上下文路径，是指 docker 在构建镜像，有时候想要使用到本机的文件（比如复制），docker build 命令得知这个路径后，会将路径下的所有内容打包

• Enter test_dockerthe folder directory

  • 4 files are inside
    

• Create a mirror under test_dockerthe folder directory

  docker build -f kohya_ss_dockerfile -t wang0724/gpu_python3_ubuntu22:v1 .
  

  

• view mirror

  sudo docker image ls
  

  

2.3 Create container

• Instantiate a container based on an image

  # sudo docker run -itd --name=容器名称--runtime=nvidia --gpus all  -p 6161:7860  镜像名称 
  sudo docker run -itd --name=kohya_test --runtime=nvidia --gpus all  -p 6161:7860  wang0724/gpu_python3_ubuntu22:v1
  

  • The creation is successful, and an id is obtained, indicating that the creation is successful
    
• View running containers

  sudo docker ps 
  

  

2.2 Access to environmental containers

• into the container

  # sudo docker exec -it 容器id /bin/bash
  sudo docker exec -it 1283aa83664a /bin/bash
  

• View the files in the root directory
  • do.sh upload_server_api.py upload_server_webui.py, do.sh中创建的venv虚拟环境the files are in

    ls
    

    

2.3 Activate the python virtual environment

• Enter the python virtual environmentvenvFile Directory
• Activate the python virtual environment

  source venv/bin/activate
  

  
• After entering the virtual environment, you can install the python library you need

2.4 File transfer steps

• Purpose of file transfer=：
  • There are only basic files in the container now, you need to upload your own required files
• Precautions：
  • Need to exit the container to transfer files
    

2.4.1 Exit the container

• Related commands

  exit
  

2.4.2 File transfer

• 本地文件复制到容器路径

  # sudo docker cp load_file_path 容器id:docker_file_path
  sudo docker cp  kohya_ss 1283aa83664a:/home/data
  

• 容器路径复制到本地文件

  # sudo docker cp 容器id:docker_file_path load_file_path
  sudo docker cp  1283aa83664a:/home/data a.txt
  

2.4.3 Access to the container

• into the container

  # sudo docker exec -it 容器id /bin/bash
  sudo docker exec -it 1283aa83664a /bin/bash
  

2.4.4 View files

• kohya_sshas been successfully transferred to the container

  ls
  

  

3. Save the container as a new image

• Save the above container as a new image

  # docker commit -m="描述信息" -a="作者" 容器id 目标镜像名： [TAG]
  # sudo docker commit -m="test"  -a="w" 1283aa83664a os-harbo.cloudos:11443/home/test:v1
  sudo docker commit -m="image create" -a="wang" 1283aa83664a wang0724/gpu_python3_ubuntu22:v1
  

• View mirror list

  sudo docker image ls
  

  

4. Create a new image based on the image

4.1. Make a mirror image

• Create based on an existing image [that is, copy one]

  parameter	meaning
  old	container name
  v1	tag
  os-harbor-svc.default.svc.cloudos:0000/env	project path

  -- old:v1 已有镜像名称
  -- new:v2 新创建镜像名称
  sudo docker tag old:image1 os-harbor-svc.cloudos:0000/env/new:v2
  

• Check the mirror image again and find that there is one more
  

4.2. Upload image to sdg

• Log in to the warehouse

  sudo docker login  https://os-harbor-svc.cloudos:0000
  

• Submit a mirror

  sudo docker push os-harbor-svc.cloudos:0000/env/new:v2
  

• pull image

  sudo docker pull os-harbor-svc.cloudos:0000/env/new:v2