Image re-illumination practice study notes based on deep learning (1)

The practice of image re-illumination based on deep learning is one of the practical series of courses on "AI Continent". The project realization task is image re-illumination, which is a typical image enhancement task in the field of artificial intelligence computer vision image processing.

1. Why do you want to open such a project course

• First, lead small partners who are just starting to learn artificial intelligence to start with image processing tasks, familiarize themselves with the knowledge related to the project content, and master the ideas and methods of learning, including literature research, paper reading, and overcoming knowledge blind spots;
• Second, start from the zero-introduction AI practice, from the past paper talk, leap over to apply what you have learned, and learn in practice, including linux development environment, python/shell programming language, deep learning framework pytorch, Relighting code architecture and development management, code Specific practices, etc.

2. What kind of project is this

• This project is derived from "AIM 2020 Relighting Challenge Track 1 one-to-one relighting", which is a subtask of an image processing international challenge. The full name of AIM is Advances in Image Manipulation. The AIM competition is held once a year, and major universities, research institutes and companies participate in the competition. Similar competitions include PIRM (Perceptual Image Restoration and Manipulation), CLIC (Challenge on Learned Image Compression), and NTIRE (New Trends in Image Restoration and Enhancement).

• Official website : https://data.vision.ee.ethz.ch/cvl/aim20/

• Competition task : One-to-one Relighting: The goal is to convert the input image from a set of predefined lighting settings (ie, North, 6500K) to another set of predefined lighting settings (East, 4500K). The resolution of the image is 1024×1024, including input and output, only the input image is provided.

• Competition evaluation indicators : PSNR and SSIM indicators are used to evaluate the results during the ranking period and provide self-reported running time and implementation details.
  For the final ranking, the average perception score (MPS) is used, which is defined as the average of the standardized SSIM and LPIPS scores. Their average value in the entire test set for each submission is 0.5·(S + (1-L)), where S Is the SSIM score and L is the LPIPS score.
  LPIPS: Learned Perceptual Image Patch Similarity, image similarity measured by depth features, CVPR 2018.
  

• Competition steps :
  First, register to participate and be familiar with the requirements of the competition questions, such as tasks, evaluation indicators, divided into several stages, etc.;
  second, download the data, and analyze the data in combination with the competition questions, and clean the data if necessary;
  third, Develop the model, solve the task of the competition, and evaluate, take feedback, optimize the model, and iterate like this;
  fourth, download the test data, submit the results, which may include test code and test results, and then evaluate by the organizer;
  fifth, prepare the paper at the same time , Submit to the workshop of the competition, and then the organizer will review the manuscript;
  sixth, wait for the organizer to release the evaluation results, and get the competition ranking; wait for the review results of the paper, if you need to modify it, you must modify it; according to the situation, participate in the workshop to share your own methods and experience.
  Seventh, the end.

• Competition difficulties : For all AI competitions, "a thorough understanding of tasks and data, development of models, and continuous optimization" are undoubtedly common difficulties. In fact, the competition is still a relatively complex activity. Each of the above steps requires great care in order to be solid and successful in the competition.
  For the Relighting question of this competition, the evaluation index is undoubtedly the most critical link. All the participating partners know that it was MOS at the beginning and MPS at the end, so it is necessary to endure the changes of the competition rules by the organizer. Negative impact.
  What problem does the project solve : The task of image re-illumination is generally to give an input picture and output a picture after re-illumination. Re-illumination means that the lighting effect of the scene in the output picture is different. Image re-illumination is equivalent to regenerating a picture, but the size of the pixel value in the input image is changed through the artificial intelligence algorithm, so that the brightness of the pixel changes regularly, and the overall look is that the lighting of the image scene has changed.
  

• What is the effect of the project: Let’s take a look at the test results of the model selected for this project. The left is the input and the right is the model output. Looking at a few more comparisons, we can find: Third, from the scene lighting color temperature, it basically happened More consistent changes, for example, the color temperature of the input picture scene is colder, and the color temperature of the output picture scene is warmer.
  

3. How to start learning this project step by step

• Analyze project tasks and be familiar with project data (the second live broadcast content)
  project summary
• Get started with AI and know the
  project knowledge (third live broadcast content)
• Learning algorithm, grasping the soul of AI (fourth live broadcast content)
  project paper analysis
• Practice the code, look at the world from another perspective (the fifth live broadcast content)
  Project code analysis (key and most detailed part)

4. Where can I get more such learning resources?

Link to the first live broadcast of the course: https://www.bilibili.com/video/BV15o4y1o7od/