1. Background introduction of model migration, model debugging and tuning
Problems that may be encountered during model training\reasoning:
• Wrong code writing, difficult to locate the problem;
• Wrong model structure;
• Wrong weight update;
• Wrong loss function design;
• Calculation overflow at half precision
;
• Overfitting, underfitting;
• Gradient disappearance/explosion;
• Activation value saturation;
• Improper setting of learning rate;
• Improper setting of batch size and period (Epoch);
• Slow model training/reasoning;
• …
2. Introduction to MindSpore
Key Features
1. Automatic Parallelism: Segment dynamic whole graphs, perceive cluster topology, and achieve minimum communication overhead. Data parallelism and model parallelism: 2. Second-order optimization:
Use second-order calculations to correct the gradient update direction and find the optimal descent path for training gradients , thus speeding up the training convergence process.
3. Combination of dynamic and static graphs: the unified automatic differentiation engine supports dynamic and static graphs, and one line of code completes the mode switching, taking into account model development and execution efficiency.
4. Full-scenario deployment collaboration: ultra-lightweight technology, model compression, adaptive model generation loT ultra-lightweight runtime, based on heterogeneous hybrid parallelism.
5. Full-stack collaborative acceleration: Through deep graph optimization such as multi-level parallelism and graph-computing fusion, as well as cross-layer memory multiplexing technology, the ultimate performance of Yiteng is exerted; 6. Migration and debugging tools: coverage ecological migration
, precision tuning, performance Optimize the visualization of the whole scene, automate script migration, lower the development threshold, and improve debugging efficiency
7. AI+ scientific computing, scene application innovation, expand the boundaries of MindSpore
8. Safe and reliable, the first enterprise-level AI trusted computing framework (Trusted AI)
3. Introduction to MindInsight
Summary of this chapter
• Common debugging and tuning problems: overfitting, underfitting, etc.; • Key features
of MindSpore:
– Automatic parallelism
; – Second-
order optimization
; ; – Accuracy debugger; – Performance debugger; – Visualization of the training process; – Model interpretability;