EF Core is born, He Sheng

Table of contents

1. Since IEnumerable is born, how can IQueryable be born?

Second, the delayed execution of IQueryable

Third, the reuse of lQueryable

Four, EF Core pagination query

Five, the underlying operation of IQueryable

1. Scenario 1: The method needs to return query results

2. Scenario 2: Multiple IQueryable traversal nesting

1. Since IEnumerable is born, how can IQueryable be born?

We already know that you can use methods such as Where in LINQ to select rows for ordinary collections. For example, the following C# code can extract the data greater than 10 in the int array:

int[] nums = { 3,5,933,2,69,69,11 };
IEnumerable<int> items = nums.where(n => n > 10):

Right-click on the Where method, click the [Go to Definition] button, and you can see that the extension method in the Enumerable class of the Where method called here is declared as follows;

IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source,Func<TSource，bool> predicate);

We can also call methods such as Where on the DbSet type of EF Core to filter data. For example, the following code can filter out books whose price is higher than 1.1 yuan;

IQueryable<Book> books ctx.Books.Where(b => b.Price > 1.1 );

Looking at the declaration of the Where method called here, we will find that it is an extension method defined in the Queryable class. The declaration of the method is as follows;

IQueryable<TSource> Where<TSource>(this IQueryable<TSource> sourceExpression<Func<TSource, bool>> predicate);

This Where method is an extension method of IQueryable<TSource> type, and the return value is IQueryabl<TSource> type. IQueryable is actually an interface that inherits the IEnumerable interface, as shown below;

public interface IOuervable<out T> : IEnumerable<T>， IEnumerable, IQueryablel{}

The IQueryable interface is inherited from the IEnumerable interface; the Where method in the Queryable class is no different from the Where method in the Enumerable class except that the parameters and return value types are IQueryable. Then why did Microsoft introduce an IQueryable interface and a new Where method

For ordinary collections, the Where method will filter each piece of data in memory, and if EF Core also filters all data in memory, we need to add all the data in a database table to In the memory, the method is processed one by one through conditional judgment. If the amount of data is not large, there will be performance problems. Therefore, the where implementation in EF Core must have a mechanism of "converting Where conditions into SOL statements", so that data filtering can be performed on the database server. The process of using SOL statements to complete data filtering on the database server is called "server evaluation". The process of adding data to the memory of the application program first, and then performing data filtering in memory is called "client evaluation". Obviously, for In most cases, the performance of "client evaluation" is relatively low, and we should try to avoid "client evaluation".

The Where and other methods defined in the Enumerable class for ordinary collections are "client evaluation", so Microsoft created the IQueryable type, and defined the Where and other methods similar to the Enumerable class in Queryable and other classes. The Where method defined in Queryable supports converting LINQ queries into SQL statements. Therefore, when using EF Core, in order to avoid "client evaluation", we should try to call the IQueryable version of the method instead of directly calling the IEnumerable version of the method.

Give an example. Use the following code to get books whose price is higher than 1.1 yuan.

IQueryable<Book> books = ctx.Books.Where(b => b.Price > 1.1);
foreach (var b in books.Where(b => b.price > 1.1))
{
Console.WriteLine($"Id={b.Id),Title={b.Title}"); 
}

The SQL statement generated by the above code is as follows

SELECT [t].[Id]，[t].[AuthorName]，[t].[Price]，[t].[PubTime]， [t].[Title) FROM[T_Books] AS [t] WHERE [t].[Price] >1.1000000000000001E0

Here is the "server-side evaluation" performed by EF Core using SQL statements on the database server. Because the books variable is of type IQueryable<Book>, the Where method of the IQueryable version is called here.

Next, we slightly change the code, change the type of the books variable from IQueryable<Book> to IEnumerable<Book>, and do not make any changes to other codes, as shown in the code.

IEnumerablc<Book> books = ctx.Books;
foreach (var b in books,where(b => b.Price > 1.1))
{
Console.WriteLine($"Id={b.Id},Title={b.Title}"); 
}

Let's look at the corresponding generated SQL statement, as follows:

SELECT[t].[Id]，[t].[AuthorName]，[t].[Price]，[t].[PubTime], [t].[Title] FR[T_Books] AS [t]

This time, the program loads all the data in the T_Books table into the application memory, and then filters the data in the memory, which becomes "client evaluation". Because books is of type IEnumerable <Book>, the Where method of the IEnumerable version is called here.

Not only the Where method is defined in the Queryable class, but also Select, OrderBy, GroupBy, Min and other methods are defined. The usage of these methods is almost the same as that of the method with the same name defined in the Enumerable class. The only difference is that they are both "server-side evaluation" versions.

In short, when using EF Core, we should try our best to avoid "client evaluation". If you can use IQueryable<>m,
don't use IEnumerable<T> directly.

Second, the delayed execution of IQueryable

IQueryable not only brings the "server-side evaluation" function, but also provides the ability to delay execution. This section will introduce the delayed execution feature of IQueryable.

Write code to perform data queries.

IQueryable<Book> books = ctx.Books.Where(b => b.Price > 1.1); 
Console.WriteLine(books)

This code only queries books whose price is greater than 11 yuan, but there is no traversal output for the return value. We have enabled the SQL statement executed by the log output code for TestDbContext, and the log output result of the above program.

It can be seen from the output of the log results that the above code did not execute the SQL statement, but we clearly executed the Where method to filter and query the data.

Modify the code and traverse the query results, as shown below.

Console.writeLine("1,Where 之前");
IQueryable<Book> books = ctx.Books,Where(b->b,Price>1.1); 
Console.WriteLine("2， 遍历 IQueryable之前");
foreach (var b in books）
{
Console.WriteLine(b.Title + ":" + b.PubTime); 
}
Console.WriteLine("3，遍历IQueryable之后");

Observe the log output of the above program. Please carefully observe the output sequence of the SQL statement, "2. before traversing IQueryable" and "3. after traversing IQueryable" in the screenshot of the above output results. According to the code in C#, the code called Where is executed before "2. Before traversing IQueryabl", but in the execution result, the SQL statement is executed after "2. Before traversing IQueryable". Why?

In fact, IQueryable just represents "a query that can be executed in the database server", it is not executed immediately but "can be executed". This can actually be seen from the English meaning of the IQueryable type name. "IQueryabla" means "queryable", which can be queried, but the query is not executed, and the execution of the query is delayed.

So when will IQueryable execute the query. A principle is: For the IQueryable interface, the query will not be executed when the "non-immediate execution" method is called, and the query will be executed immediately when the "immediate execution" method is called. In addition to traversing IQueryable operations, there are immediate execution methods such as ToArray, ToList, Min, Max, and Count; methods such as GroupE OrderBy, Include, Skip, and Take are non-immediate execution methods. The simple way to judge whether a method is an immediate execution method is: if the return value type of a method is IQueryable type, this method is generally a non-immediate execution method, otherwise this method is an immediate execution method.

Why does EF Core implement such a complicated mechanism as "IQueryable delayed execution"? Because we can use IQueryable to stitch together complex query conditions before executing the query. For example, the following code defines a method that is used to query matching books based on the given keyword searchWords; if the searchAll parameter is true, then the book title or author name that contains the given searchWords will match, otherwise Only match the book title; if the orderByPrice parameter is true, the query results will be sorted by price, otherwise they will be sorted naturally; the upperPrice parameter represents the price limit, as shown in the code below

void QueryBooks(string searchWords, bool searchAll, bool orderByPrice, double upperPrice
{
using TestDbContext ctx = new TestDbContext(); 
IQueryable<Book> books = ctx.Books.Where(b => b.Price <= upperPrice); 
if (searchAll) //匹配书名或作者名 
{
books = books.Where(b => b.Title.Contains(searchWords)II 
b.AuthorName.Contains(searchWords));
} 
else //只匹配书名 
{ 
books = books.Where(b => b.Title.Contains(searchWords));
}
if (orderlyPrice) //按照价格排序 
{
books = books.OrderBy(b => b.Price); 
}
foreach (Book b in books) 
{
Console.WriteLine($"ib.Id),{b.Titlo},(b.Price},(b.AuthorName}");
}
}

We further filter the IQueryable<Book> object returned by ctx.Books.Where(b=>b.Price<=upperPrice) using Where, OrderBy and other methods according to the parameters passed by the user, only when using foreach to traverse books execute query

We write the following code to call the QueryBooks method:

QueryBooks("爱"，true，true，30);

View the log output of the program of the SQL statement executed by the above code;

It can be seen that we did not execute the SQL statement during the splicing of the IQueryable, and only executed the SQL statement when traversing the IQueryable at the end, and this SQL statement merged the two Where filter conditions we set into one Where condition, The SQL statement also includes the Order By statement we set.

Let's try calling the QueryBooks method again

QueryBooks("爱"，false，false，18);

View the log output of the program for the SQL statement executed by the above code.

It can be seen that due to the different parameters passed, the IQueryable we spliced is different, so the SQL statement generated when the query is finally executed is also different.

If you do not use EF Core but use SQL statements to implement the logic of "executing different SQL statements according to different parameters", we need to manually splice SQL statements. This process is very troublesome, and EF Core turns "dynamic splicing to generate query logic" into very simple.

In short, IOueryable represents a logic for querying data in the database, and this query is a delayed query. We can call the non-immediate execution method to add query logic to IQueryable, and the SQL statement is actually generated to execute the query when the immediate execution method is executed.

Third, the reuse of lQueryable

IQueryable is a logic to be queried, so it can be reused, as shown in the following code.

IQueryable<Book> books = ctx.Books.Where(b => b.Price >=8);
Console.WriteLine(books.Count());
Console.WriteLine(books.Max(b=> b.Price));
foreach(Book b in books.Where(b => b.PubTime.Year >2000))
{
Console.WriteLine(b.Title)i 
}

The above code first creates an IQueryable object to obtain books whose price is greater than or equal to 8 yuan, then calls the Count method to execute the IQueryable object to obtain the number of data items that meet the conditions, and then calls the Max method to execute the IQueryable object to obtain the highest price that meets the conditions. Finally, call the Where method for the books variable to further filter and obtain books published after 2000.

The above code will generate the following SQL statement:

SELECT COUNT(*) FROM [T_Books] AS [t] WHERE [t].[Price] >= 8.0E0;
SELECTMAX([t].[Price]) FROM [T_Books] AS [t] WHERE [t].[Price] >= 8.0E0;
SELECT[t].[Id]，[t].[AuthorName]，[t].[Price]，[t].[PubTime], [t].[Title] FROM
[T_Books] AS [t]
WHERE([t].[Price] >= 8.0EO) AND (DATEPART(year, [t].[PubTime]) > 2000);

It can be seen that since Count, Max, and foreach are all executed immediately, the three operations on IQueryable each execute corresponding query logic. IQueryable allows us to reuse the previously generated query logic, which will be used in the paging query described below.

Four, EF Core pagination query

If there is a lot of data in the database table, when displaying the query results to the front end, we usually need to display the query results in pages.

When realizing the paging display effect, the program needs to implement the method of obtaining data from the database table by paging. For example, each page displays 10 pieces of data. If we want to display the data on the third page (page number starts from 1), we need to obtain the data from the 20th page. The first 10 pieces of data.

We know that the Skip(n) method can be used to realize "skip n pieces of data", and the Take(n) method can be used to realize "taking up to n pieces of data". These two methods can be combined to obtain data in pages, such as Skip(3 ).Take(8) means "Get up to 8 pieces of data starting from the third one". These two methods are also supported in EF Core.

When implementing paging, in order to display the page number bar, we need to know the total number of data that meets the conditions. You can use the multiplexing of IQueryable to realize the two query operations of data paging query and obtaining the total number of data that meet the conditions.

The following encapsulates a method to output the content of the nth page (page number starts from 1) whose title does not contain "Zhang San", and output the total number of pages. Each page displays up to 5 pieces of data, as shown in the following code.

void OutputPage(int pageIndex, int pageSize)
{
using TestDbContext ctx = new TestDbContext(); 
IQueryable<Book> books =ctx.Books.Where(b =>lb.Title.Contains("张三")); 
long count = books.LongCount(); 
long pageCount = (long)Math.Ceiling(count *1.0 /pageSize); //页数//总条数
Console.WriteLine("页数:"+ pageCount);
var pagedBooks = books.Skip((pageIndex - 1) * pageSize).Take(pageSize) 
foreach (var b in pagedBooks)
{
Console.WriteLine(b.Id + "," + b.Title);
}
}

The pageIndex parameter of the OutputPage method represents the page number, and the pageSize parameter represents the page size. In the OutputPage method, we first create the query rule books, and then use the LongCount method to obtain the total number of data items that meet the conditions. The total number of data pages can be calculated by using countx1÷pageSize. Considering that the last page may not be full, we use the Ceiling method to obtain the total number of pages of integer type. Since the serial number of pageIndex starts from I, we use the Skip method to skip (pageIndex-1)x pageSize pieces of data, and then obtain the most pageSize pieces of data to obtain the correct paging data.

We use the code test to output the data of the first page and the second page.

OutputPage(1,5);
Console.WriteLine("******"); 
OutputPage(2，5)

Five, the underlying operation of IQueryable

There are two ways of reading database query results, DataReader and DataTable, in ADO.NET. If there are many pieces of data in the query result, DataTable will load all the data from the database server to the client memory at one time, and DataReader will read data from the database server in batches. The advantage of DataReader is that the memory usage of the client is small. The disadvantage is that if the process of traversing and reading data and processing it is slow, it will cause the program to occupy the database connection for a long time, thereby reducing the concurrent connection capability of the database server; the advantage of DataTable is that the data It is quickly loaded into the client's memory, so it will not occupy the database connection for a long time. The disadvantage is that if the amount of data is large, the memory usage of the client will be relatively large.

When IQueryable traverses and reads data, does it use a method similar to DataReader or DataTable? We insert hundreds of thousands of data into the T_Books table, and then use code to traverse IQueryable.

IQueryable<Book> books = ctx.Books.Where(b=>b.Id>2);
foreach (var b in books) 
{
Console.WriteLine(b.Id +"," + b.Title):
}

During the traversal execution, if we shut down the SQL Server server or disconnect the server from the network, the program will go wrong, which means that IQueryable reads the query results in a way similar to DataReader. In fact, the traversal of the IQueryable p part is to call DataReader to read data. Therefore, it needs to occupy a database connection while traversing the IQueryable.

If you need to read all the data into the client memory at one time, you can use methods such as ToArray, ToArrayAsync, ToList, and ToListAsync of IQueryable. As shown in the following code, read the first 500,000 pieces of data, then use ToListAsync to read the query results into the memory at one time, and then traverse the output data.

var books = await ctx.Books.Take(500000).ToListAsyne();
foreach (var b in books)
{
Console.WriteLine(b.Id + "," + b.Title):
}

In the process of traversing the data, if we close the SQL Server server or disconnect the server network, the program can run normally, which means that the ToListAsync method has loaded the query results into the client memory.

Unless the process of traversing IQueryable and data processing is time-consuming, it is generally not necessary to read the query results into memory at one time. However, in the following scenarios, it is necessary to read the query results into memory at one time.

1. Scenario 1: The method needs to return query results

If the method needs to return query results and destroy the context in the method, the method cannot return IQueryable. For example, implement the method shown in the following code to query books with Id>5, and then return the query results.

IQueryable<Book> QueryBooks()
{
using TestDbContext ctx = new TestDbContext(); 
return ctx.Books,Where(b=>b.Id>5);
}

Then call this method in your code.

foreach(var b in QueryBooks())
{
Console.WriteLine(b.Title)
}

After the above code runs, the program will throw a "Cannot access a disposed context instance" exception, because the TestDbContext object is destroyed in the QueryBooks method, and the context needs to load data from the database when traversing the IQueryable, so the program reports an error. If in the QueryBooks method, use ToList and other methods to load the data into the memory at one time.

IEnumerable<Book> QueryBooks()
{
using TestDbContext ctx = new TestDbContext(); 
return ctx.Books.Where(b => b.Id > 5).ToArray();
}

2. Scenario 2: Multiple IQueryable traversal nesting

When traversing an IQueryable, we may need to traverse another IQueryable at the same time. The bottom layer of IQueryable uses DataReader to read query results from the database server, and many databases do not support multiple DataReaders to execute at the same time.

We use the SQL Server database to implement two IQueryable traversal together, as shown in the following code.

var books = ctx.Books.Where(b=>b.Id>1);
foreach (var b in books)
{
Console.WriteLine (b.Id + "," + b.Title); 
foreach(var a in etx.Authors) 
{
Console.WriteLine(a,Id);
}
}

When the above program is executed, it will report an error "There is already an open DataReader associated with this Connection which must be closed first." This error is because two foreach loops are traversing IQueryable, resulting in two DataReaders being executed at the same time.

Although you can enable "allow multiple DataReader execution" by setting utipleActiveResultSets=true in the connection string, only SQLServer supports the MultipleActiveResultSets option, and other databases may not support it. Therefore, the author suggests to solve it by "loading the data into the memory at one time" to transform one of the loops. For example, just change var books=ctx.Books.Where(b=>b.Id>1) to var books = ctx .Books.Where
(b=>b.Id>1).ToList() will do.

To sum up, when doing daily development, we can directly traverse IQueryable. However, if the method needs to return query results or nested execution of multiple queries, it is necessary to consider the method of loading data into memory at one time. Of course, the data for one-time query should not be too much, so as not to cause excessive memory consumption.