Machine learning based on the C # - micro benchmarks and activation

In this chapter we will learn the following:

l What is a micro-benchmarks

l How to apply it to the code

l What is the activation function

l how to draw and benchmark activation function

Each developer needs to have a good benchmarking tool. Quality criteria everywhere; you can hear every day, the 10% reduction that a 25% increase you remember the old saying goes, when you hear a number is thrown, at 98.4% of cases this figure is false of. By the way, this figure is I made. When you hear things like, let that person prove it, what will you get the results we do not need qualitative;? We need to be able to prove continuous replication and quantitative results. Repeatable results are very important, not only consistency, but also for the credibility and accuracy. This is the micro-benchmarks into play.

We will use BenchmarkDotNet library, where you can find: https: //github.com/dotnet/BenchmarkDotNet. I think it is one of the most irreplaceable framework that you can use, I think it's important as unit testing and integration testing

In order to demonstrate the value of this tool, we will draw several activation function and compare their runs. As part of this, we will consider warm-up, left and RyuJIT, cold start and more aspects of program execution. Finally, we will get a set of quantitative results prove accurate measure function. If the version 2.0, we see something run more slowly, we can re-run the benchmark and compared.

I strongly recommend to integrate it into your continuous integration / build process continued, to compare the reference data in each version.

In this chapter, we will have two samples. The first function is to activate the viewer; it will draw each activation function, so that we can see how it looks. It can be found in the SharpNEAT Colin Green, which is open source. This package is absolutely incredible. I created it on the basis of a new ui and advanced versions to meet my needs, it is the most flexible tool that can currently be found. The first sample application with the latest SharpNEAT package, you can find it in https://github.com/colgreen/sharpneat.

Using visual drawing methods

The following is a global minimum and a local map, which is drawn by a custom version of SharpNEAT.

As mentioned earlier, we will draw and test several activation function. Everywhere we hear the word activation function, but do we really know what it means? Let us start from a quick explanation.

An activation function is used to determine whether a neuron is activated. Some people like to use to replace the fired activated. In any case, it ultimately determines something is on or off, is triggered or not, is to be activated or not.

Let us first look at a view of a single activation functions:

It's like when we draw together all the logic steep activation function approximation and activate Swish like when plotted as a function alone, because there are many types of activation function, which is:

At this point, you might be thinking, why should we care about is what kind of plot it? Good question. We care about them, because once you get into neural networks or other areas, you will often use these. It is very convenient, you can know whether the value of the activation function of neurons in your state on or off, and it will maintain the required value or to what extent. No doubt you will encounter in the learning developer's career as a machine and / or using an activation function, and understand the difference between TanH LeakyReLU activation function is very important.

Draw all functions

All drawing are done activation function in a function, the function name PlotAllFunctions:

private void PlotAllFunctions()
{
    // 首先，从母版窗格中清除所有旧的GraphPane
    collection MasterPane master = zed.MasterPane;
    master.PaneList.Clear();
    // 显示母版窗格标题，并设置
    outer margin to 10 points
    master.Title.IsVisible = true;
    master.Margin.All = 10;
    // 在主窗格上绘制多个函数.
    PlotOnMasterPane(Functions.LogisticApproximantSteep,"Logistic Steep (Approximant)");
    PlotOnMasterPane(Functions.LogisticFunctionSteep,"Logistic Steep (Function)");
    PlotOnMasterPane(Functions.SoftSign, "Soft Sign");
    PlotOnMasterPane(Functions.PolynomialApproximant,"Polynomial Approximant");
    PlotOnMasterPane(Functions.QuadraticSigmoid,"Quadratic Sigmoid");
    PlotOnMasterPane(Functions.ReLU, "ReLU");
    PlotOnMasterPane(Functions.LeakyReLU, "Leaky ReLU");
    PlotOnMasterPane(Functions.LeakyReLUShifted,"Leaky ReLU (Shifted)");
    PlotOnMasterPane(Functions.SReLU, "S-Shaped ReLU");
    PlotOnMasterPane(Functions.SReLUShifted,"S-Shaped ReLU (Shifted)");
    PlotOnMasterPane(Functions.ArcTan, "ArcTan");
    PlotOnMasterPane(Functions.TanH, "TanH");
    PlotOnMasterPane(Functions.ArcSinH, "ArcSinH");
    PlotOnMasterPane(Functions.ScaledELU,"Scaled Exponential Linear Unit");
    // 重新设置GraphPanes的轴范围。
    zed.AxisChange();
    // 使用默认窗格布局布局GraphPanes.
    using (Graphics g = this.CreateGraphics())
    {
        master.SetLayout(g, PaneLayout.SquareColPreferred);
    } 

主绘制函数

在后台，Plot函数负责执行和绘制每个函数:

private void Plot(Func<double, double> fn, string fnName,Color graphColor, GraphPane gpane = null)
{
    const double xmin = -2.0;
    const double xmax = 2.0;
    const int resolution = 2000;
    zed.IsShowPointValues = true;
    zed.PointValueFormat = "e";
    var pane = gpane ?? zed.GraphPane;
    pane.XAxis.MajorGrid.IsVisible = true;
    pane.YAxis.MajorGrid.IsVisible = true;
    pane.Title.Text = fnName;
    pane.YAxis.Title.Text = string.Empty;
    pane.XAxis.Title.Text = string.Empty;
    double[] xarr = new double[resolution];
    double[] yarr = new double[resolution];
    double incr = (xmax - xmin) / resolution;
    double x = xmin;
    for(int i=0; i < resolution; i++, x+=incr)
    {
        xarr[i] = x;
        yarr[i] = fn(x);
    }
    PointPairList list1 = new PointPairList(xarr, yarr);
    LineItem li=pane.AddCurve(string.Empty,list1,graphColor,SymbolType.None);
    li.Symbol.Fill = new Fill(Color.White);
    pane.Chart.Fill = new Fill(Color.White,
    Color.LightGoldenrodYellow, 45.0F);
}

这是执行我们传入的激活函数的地方，它的值用于y轴标绘值。著名的ZedGraph开源绘图包用于所有图形绘制。一旦执行了每个函数，就会生成相应的图。

确定基准点

BenchmarkDotNet生成了几个报告，其中一个是HTML报告，类似于在这里看到的:

Excel报告提供了运行程序时使用的每个参数的详细信息，是最广泛的信息来源。在很多情况下，这些参数中的大多数都使用默认值，超出了我们的需要，但至少我们可以选择删除我们需要删除的内容:

我们将在下一节中描述其中的一些参数，当我们回顾创建之前看到的内容的源代码时:

static void Main(string[] args)
{
    var config = ManualConfig.Create(DefaultConfig.Instance);
    // 建立一个结果导出器。
    // 请注意。默认情况下，结果文件将位于.BenchmarkDotNet.Artifactsresults目录
    config.Add(new CsvExporter
　　　　　　　　　　(CsvSeparator.CurrentCulture,
　　　　　　　　　  new BenchmarkDotNet.Reports.SummaryStyle
　　　　　　　　 　　{
    　　　　　　　　　　PrintUnitsInHeader = true,
    　　　　　　　　　　PrintUnitsInContent = false,
    　　　　　　　　　　TimeUnit = TimeUnit.Microsecond,
    　　　　　　　　　　SizeUnit = BenchmarkDotNet.Columns.SizeUnit.KB
　　　　　　　　 　　}
　　　　　　　　　　)
　　);
    // 遗留JITter 测试.
    config.Add(new Job(EnvMode.LegacyJitX64,EnvMode.Clr, RunMode.Short)
    　　{
        　　Env = { Runtime = Runtime.Clr, Platform = Platform.X64 },
        　　Run = { LaunchCount = 1, WarmupCount = 1,
        　　TargetCount = 1, RunStrategy =
　　　　　　　BenchmarkDotNet.Engines.RunStrategy.Throughput },
　　　　　　　Accuracy = { RemoveOutliers = true }
　　　　}.WithGcAllowVeryLargeObjects(true)
　　);
　　// RyuJIT测试。
　　config.Add(new Job(EnvMode.RyuJitX64, EnvMode.Clr,RunMode.Short)
　　　　{
　　　　　　Env = { Runtime = Runtime.Clr, Platform = Platform.X64 },
　　　　　　Run = { LaunchCount = 1, WarmupCount = 1,
　　　　　　TargetCount = 1, RunStrategy =
　　　　　　BenchmarkDotNet.Engines.RunStrategy.Throughput },
　　　　　　Accuracy = { RemoveOutliers = true }
　　　　}.WithGcAllowVeryLargeObjects(true)
　　);
　　// 取消注释以允许对未优化的程序集进行基准测试。
　　//config.Add(JitOptimizationsValidator.DontFailOnError);
　　// 运行基准测试。
　　var summary = BenchmarkRunner.Run<FunctionBenchmarks>(config);
}

让我们进一步分析这段代码:

首先，我们将创建一个手动配置对象，其中包含用于基准测试的配置参数:

var config = ManualConfig.Create(DefaultConfig.Instance);

接下来，我们将设置一个导出器来保存用于导出结果的参数。我们将使用微秒计时和千字节大小将结果导出到.csv文件:

config.Add(new CsvExporter
　　　　　　　　(CsvSeparator.CurrentCulture,
　　　　　　　　 new BenchmarkDotNet.Reports.SummaryStyle
　　　　　　　　 {
　　　　　　　　　　 PrintUnitsInHeader = true,
　　　　　　　　　　 PrintUnitsInContent = false,
　　　　　　　　　　 TimeUnit = TimeUnit.Microsecond,
　　　　　　　　　　 SizeUnit = BenchmarkDotNet.Columns.SizeUnit.KB
　　　　　　　　 }
　　　　　　　　)
);

接下来，我们将创建一个基准作业，它将处理x64体系结构上LegacyJitX64的度量。您可以随意更改此参数和任何其他参数，以进行实验，或者包含测试场景所需或需要的任何结果。在我们的例子中，我们将使用x64平台;启动计数、预热计数和目标计数为1;以及吞吐量的运行策略。我们也会对RyuJIT做同样的事情，但是我们不会在这里显示代码:

config.Add(new Job(EnvMode.LegacyJitX64, EnvMode.Clr,RunMode.Short)
　　　　　　 {
　　　　　　 　 Env = { Runtime = Runtime.Clr, Platform = Platform.X64 },
　　　　　　　　Run = { LaunchCount = 1, WarmupCount = 1, TargetCount = 1,
　　　　　　　　RunStrategy = Throughput },
　　　　　　　　Accuracy = { RemoveOutliers = true }
　　　　　　}.WithGcAllowVeryLargeObjects(true)
);

最后，我们将运行BenchmarkRunner来执行我们的测试:

var summary = BenchmarkRunner.Run<FunctionBenchmarks>(config);

BenchmarkDotNet将作为DOS命令行应用程序运行，下面是执行上述代码的一个例子:

让我们来看一个被绘制的激活函数的例子:

[Benchmark]
public double LogisticFunctionSteepDouble()
{
　　double a = 0.0;
　　for(int i=0; i<__loops; i++)
　　{
　　　　a = Functions.LogisticFunctionSteep(_x[i % _x.Length]);
　　}
　　return a;
}

此处使用了[Benchmark]属性。这向BenchmarkDotNet表明，这将是一个需要进行基准测试的测试。在内部调用如下函数:

对于logisticfunction陡峭函数，其实现与大多数激活函数一样简单(假设你知道公式)。在这种情况下不是绘制激活函数，而是对其进行基准测试。

你将注意到函数接受并返回double。我们也通过使用和返回浮点变量对相同的函数进行基准测试，因此我们使用double对函数之间的差异进行基准测试和浮动。因此，人们可以看到，有时性能影响比他们想象的要大:

总结

在本章中，我们学习了如何将微基准测试应用到代码中。我们还了解了如何绘制和基准测试激活函数，以及如何使用微基准测试。现在，您有了一个最强大的基准测试库，可以将其添加到所有代码中。在下一章中，我们将深入探讨直观的深度学习，并向您展示c#开发人员可以使用的最强大的机器学习测试框架之一。