1. Database development
1.1 Data and data management
What is information?
Information refers to the response to the way of being or the state of motion of the real world.
What is data?
Data refers to physical symbols stored on a certain medium that can be recognized;
the concept of data has been greatly broadened in the field of data processing, including not only data in the form of text composed of characters, but also other types of data (such as audio , video, etc.)
the relationship between information and data:
information and data are interdependent, data is the carrier of information, and information is the connotation of data;
data processing refers to the process of mutual conversion between data and information;
from the perspective of data processing , information is a kind of data that has been processed into a specific form.
Data processing
The core of data processing is data management technology, in which data management technology refers to the technology of classifying, organizing, encoding, storing, retrieving and maintaining data.
The development of data management technology has gone through several stages: manual management stage, file system stage, database system stage.
1.2 Database system
Definition: Database system (DBS) refers to the system composition after the database is introduced into the computer.
Composition: database, operating system, database management system, database management application system, database administrator (DBA), user
1.3 Database system model
The database management system stores and manages data according to the data model.
The data model should meet three requirements:
first, it can more realistically simulate the real world;
second, it is easy for people to understand;
third, it is easy to realize on the computer.
Data structure, data operation and integrity constraints are the three elements that constitute the data model. Among them, the integrity constraint of the data refers to entering the database in order to put the data that does not conform to the specification.
When a user inserts, modifies, or deletes data, the DBMS automatically monitors the data according to certain constraints, so that data that does not meet the specifications cannot enter the database, so as to ensure that the data stored in the database is correct, effective, and compatible.
Database management system data model:
The data models used by the database management system mainly include: hierarchical model, network model and relational model.
The hierarchical model is a typical tree structure, and its characteristics are:
1. There is one and only one node without a parent node, and this node is called the root node;
2. Other nodes have one and only one parent node;
3. The same parent The child nodes of a node are called sibling nodes;
4. Nodes without child nodes are called leaf nodes.
The network model constitutes a more complex network structure than the hierarchical structure model, and its characteristics are:
1. More than one node is allowed to have no parent node;
2. A node can have multiple parent nodes;
The logical structure of relational model data is a two-dimensional table, one row is an object member, and each column is an attribute of the object. Its characteristics are:
1. The components in each column can be the same type of data
2. The order of the columns can be arbitrary;
3. The order of the rows can be arbitrary;
4. The components in the table are the smallest data that cannot be divided item, that is, subtables are not allowed in the table.
Relational databases use a relational model as the way data is organized. Relational database is recognized as the most promising database management system because of its strict mathematical theory, simple and flexible use, and strong data independence. It has become the dominant database management system at present.
1.4 Common databases
Common databases are:
ORACLE, MySQL, SQL server, Access, Sybase, SQLite
1, ORACLE:
ORACLE is a database developed by Oracle Corporation, and is a relational database management system suitable for large, medium and microcomputers. It uses SQL language as its database language.
2. MySQL
MySQL is an open source small-scale relational database management system. The developer is the Swedish MySQL AB company. At present, MySQL is widely used in small and medium-sized websites on the Internet. It has small size, fast speed, low cost, and open source code specialty.
3. SQL server
is a real client/server architecture, a database software produced by Microsoft, with a graphical interface, which makes system management and database management more intuitive/simple; it has good scalability and can span from running Windows 95 /98 type computers to large multi-processors running Windows 2000 and other platforms.
4. Access
Access is a relational database management system released by Microsoft. It combines the two characteristics of MicrosoftJet Database Engine and graphical user interface, and is one of the system programs of Microsoft Office. It is a desktop database, which is only suitable for applications with a small amount of data. It is very good and efficient when dealing with a small amount of data and databases accessed by a single computer, but it cannot access more than 4 clients at the same time. The Access database has a certain limit. If the data reaches about 100M, it is easy to cause the server IIS to freeze, or consume the memory of the server and cause the server to crash.
5. Sybase
Sybase database products mainly have three versions, one is the version running under the UNIX operating system, the other is the version running under the Novell Netware environment, and the third is the version running under the Windows NT environment. Features of Sybase database:
1. Database based on client/server architecture
2. It is a truly open database and easy to transplant;
3. It is a high-performance database.
1.5 SQLite database
1.5.1 Introduction to SQLite database
SQLite is an open source, embedded relational database. The first version was born in May 2000, and the current highest version is SQLite3.
Download address: https://www.sqlite.org/download.html
Installation method (under Ubuntu)
sudo apt-get install sqlite3
Installation method (combined with C++ under windows)
Download sqlite-amalgamation-3410200.zip (header file after decompression), sqlite-dll-win64-x64-3410200.zip and sqlite-dll-win64-x64-3410200.zip Three files, and then unzip them into the same folder Sqlite3, see the figure below for details (I do not include the header file decompressed from the first file here) and then
generate the sqlite.lib file from the sqlite3.def file. The main process is : First open the "VS20xx Developer Command Prompt", and then enter in the command line in turn:
cd D:/Sqlite3
LIB /DEF:sqlite3.def /MACHINE:X64
You can get the sqlite.lib file, and then you can operate step by step according to the way of configuring the library.
SQLite features:
1. Zero configuration;
2. Portable;
3. Compact;
4. Simple;
5. Flexible;
6. Reliable
; 7. Easy to use.
1.5.2 SQL Statement Basics
SQL is an acronym for Structured Query Language, and SQL is a language specially used to communicate with databases. Currently SQL has become the most widely used database language. SQL has been adopted by many commercial database management system products, and different database management systems have made certain adaptations and extensions to the SQL specification in their practice. Therefore, the SQL language between different database management systems cannot be fully used with each other.
SQLite data type:
general data uses a fixed static data type, while SQLite uses a dynamic data type, which will be automatically judged according to the stored value. SQLite has the following five basic data types: 1. integer: signed integer (up to 64 bits); 2. real: floating-point type represented by 8 bytes; 3. text: character type, supporting multiple encodings (such as UTF-8, UTF-16), unlimited size; 4, blob: any type of data, unlimited size. BLOB (binary large object) binary large object, using binary to save data. 5. Null: Indicates a null value.
SQL statement for the database file:
1. Create and open the database: sqlite3 .db prompt: when the .db file does not exist, sqlite will create and open the database file; when the .db file exists, sqlite will open the database file 2. Exit the database command : .quit or .exit
SQL statement format:
All SQL statements end with a semicolon, and SQL statements are not case-sensitive. Two minus signs "–" represent comments. The core operations of relational data mainly include: 1. Create, modify and delete tables; 2. Add, modify, and delete rows; 3. Look up tables.
1. Create a table: create statement
//语法：create table 表名称(列名称1 数据类型，列名称2 数据类型，列名称3 数据类型，...)；
//例子：创建一表格，该表包含3列，列名分别是："id"、"name"、"addr"
//在终端下输入：
sqlite> create tabel persons(id integer, name text, addr text);
2. Create a table: Create statement (setting primary key)
When designing a table with sqlite, each table can manually set the primary key through the primary key. Each table can only have one primary key, and the column data set as the primary key cannot be repeated.
//语法：create table 表名称(
列名称1 数据类型 primary key，
列名称2 数据类型，
列名称3 数据类型，...)；
//例子：创建一表格，该表包含3列，列名分别是："id"、"name"、"addr"
//在终端下输入：
sqlite> create tabel persons(id integer primary key, name text, addr text);
3. Modify the table
Add columns to the existing table and modify the table name.
//语法： alter table 表名 add 列名 数据类型
sqlite> alter table persons add sex text;
//语法： alter table 表名 rename 表名
sqlite> alter table persons rename stu;
4. Delete table
is used to delete the table (the structure, attributes and index of the table will also be deleted)
//语法： drop table 表名；
sqlite> drop table persons;
5. Insert a new row: the insert into statement (all assignment)
assigns values to all columns in a row.
//语法：insert into 表名 values(列值1，列值2，列值3，列值4，...);
//注意，当列值为字符串时要加上' '号
sqlite> create table persons (id integer, name text, addr text);
sqlite> insert into persons values(1,'lucky','beijing');
6. Modify the data in the table: the update statement
uses where to find one or more rows according to the matching conditions, and modify the number of columns in the corresponding row in the table according to the search results (which column to modify is specified by the column name).
//语法：update 表名 set 列1 = 值1 [,列2 = 值2,...][匹配条件]
//匹配：where子句
// where子句用于规定匹配的条件。
//操作符 描述
// = 等于
// <> 不等于
// > 大于
// < 小于
// >= 大于等于
// <= 小于等于
//匹配条件语法： where 列名 操作符 列值
sqlite> update persons set id=2, addr = 'tianjin' where name = 'peter';
7. Delete the data in the table: the delete statement
uses where to find one or more rows according to the matching conditions, and deletes the rows found in the table according to the search results
//语法：delete from 表名 [匹配条件]
//注意：当表中由多列、多行符合匹配条件时会删除相应的多行
sqlite> delete from person where name = 'peter';
8. Query: The select statement (basic)
is used to select data from the table, and the result is stored in a result table (called the result set)
//语法1：select * from 表名 [匹配条件]；
//语法2：select 列名1[,列名2，...] from 表名 [匹配条件];
//提示：星号(*)是选取所有列的通配符
sqlite> select * from persons;
sqlite> select name from person where id = 1;
8. Syntax of matching conditions: (improved)
The database provides a wealth of operators with where clauses to achieve a variety of matching methods.
Such as: (1) in operator (2) and operator (3) or operator (4) between and operator (5) like operator (6) not operator
// in允许我们在where子句中规定多个值
// 匹配条件语法：where 列名 in (列值1，列值2，...)
// 例：1、select * from 表名 where 列名 in (值1，值2，...);
// 2、select 列名1[,列名2,...] from 表名 where 列名 in(列值1,列值2，...);
sqlite> select * from persons where id in (1,2);
sqlite> select name from persons where id in (1,2);
// and 可在where子语句中把两个或多个条件结合起来(多个条件之间是与的关系)
// 匹配条件语法：where 列1 = 值1 [and 列2 = 值2 and...]
// 例： 1、select * from 表名 where 列1 = 值1 [and 列2 = 值2 and ...];
// 2、select 列名1[,列名2,...] from 表名 where 列1 = 值1 [and 列2 = 值2 and ...];
sqlite> select * from persons where id = 1 and addr = 'beijing';
sqlite> select name from persons where id = 1 and addr = 'beijing';
//or可在where子语句中把两个或多个条件结合起来(多个条件之间是或的关系)
// 匹配条件语法：where 列1 = 值1 [or 列2 = 值2 or...]
// 例： 1、select * from 表名 where 列1 = 值1 [or 列2 = 值2 or ...];
// 2、select 列名1[,列名2,...] from 表名 where 列1 = 值1 [or 列2 = 值2 or ...];
sqlite> select * from persons where id = 1 or addr = 'beijing';
sqlite> select name from persons where id = 1 or addr = 'beijing';
//操作符between A and B会选介于A、B之间的数据范围。这些值可以是数值、文本或日期
//注意：不同的数据库对between A and B操作符的处理方式是有差异的（有的包含A,B，有的不包含A,B,有的只包含一边）
// 匹配条件语法：where 列名 between A and B
// 例： 1、select * from 表名 where 列名 between A and B;
// 2、select 列名1[,列名2,...] from 表名 where 列名 between A and B;
// 注：匹配字符串时会以ascii顺序匹配
sqlite> select * from persons where id between 1 and 3;
sqlite> select name from persons where id between 1 and 3;
//like 用于模糊查找
// 匹配条件语法：where 列名 like 列值
// 若列值为数字，相当于列名=列值
// 若列值为字符串，可以用通配符“%”代表缺少的字符（一个或多个）
sqlite> select * from persons where id like 3;
sqlite> select * from persons where addr like "%jing%";
//not 可取出原结果集的补集
//匹配条件语法：where 列名 not in 列值 等
//例： 1、where 列名 not in (列值1,列值2,...);
// 2、where not(列1 = 值1 [and 列2 = 值2 and...])
// 3、where not(列1 = 值1 [or 列2 = 值2 or...])
// 4、where 列名 not between A and B
// 5、where 列名 not like 列值
// order by 语句根据指定的列对结果集进行排序(默认按照升序对结果集进行排序，可使用desc关键字按照降序对结果集进行排序)
// 例：升序
// select * from 表名 order by 列名
// 降序
// select * from 表名
Nine, copy the table and modify the table structure
//1、复制一张表
create table tbl2 as select * from tbl1;
//2、复制一张表的部分内容
create table tbl2 as select * from tbl1 where id =104;
//3、修改表的结构
//第一步：创建新表
create table ntbl(id interger,name text,addr text);
//第二步：导入数据(如果有主键，要注意数据不要重复)
insert into ntbl(id,name,addr) select id,name,addr from tbl1;
//第三步：修改表名
alter table ntbl rename tbl1;
10. Transactions
Transactions can be started using the BEGIN TRANSACTION command or a simple BEGIN command. Such transactions usually continue to execute until the next COMMIT or ROLLBACK command is encountered. However, transactions are also rolled back when the database is shut down or an error occurs. Following is the simple syntax to start a transaction:
In SQLite, by default, each SQL statement is its own transaction. begin: start a transaction, and all subsequent operations can be canceled commit: confirm all commands after begin rollback: cancel all operations after begin Examples: sqlite->begin; sqlite->delete from persons
1.5.3 SQL statement advanced
1. Functions and aggregation
functions: SQL statements support the use of functions to process data. Functions are generally executed on data, which provides convenience for data conversion and processing.
Commonly used text processing functions are: length() returns the length of the string lower() converts the string to lowercase upper() converts the string to uppercase Syntax: select function name (column name) from table name; commonly used aggregate functions : Use aggregate functions to retrieve data for analysis and report generation avg() returns the average value of a column count() returns the number of rows in a column max() returns the maximum value of a column min() returns the minimum value of a column The value sum() returns the sum of the values of a certain column. 2. Data grouping group by grouping data so that a subset of table contents can be summarized, often used in conjunction with aggregate functions. For example, query the number of people in each class, the average number, and so on. Use: select column name 1[, column name 2,...] from table name group by column name
sqlite>select class,count(*) from persons group by class;
3. Filter grouping
In addition to using group by to group data, you can also use having to realize which groups are included and which groups are excluded.
Use: select function name (column name 1[), column name 2,...] from table name group by column name having function name limit value
//查看班级平均分大于90的班级
sqlite>select class,avg(score) from persons group by class having avg(score) >= 90
4. Constraint management Rules for how to insert or
process database data
Common constraint classification: primary key, unique constraint
, check constraint ) Never update the primary key, otherwise the primary key should not contain dynamically changing data, such as timestamp, creation time column, modification time column, etc. The primary key should be automatically generated by the computer (to ensure uniqueness) Syntax: create table table name ( column name 1 data type primary key, column name 2 data type, column name 3 data type,...);
Unique constraint:
used to ensure that the data in a column (or a group of columns) is unique, similar to the primary key, but different from the primary key. A
table can contain multiple unique constraints, but only one primary key
unique constraint column can be modified or updated
when creating a table , set the syntax by unique
: create table table name (column name 1 data type unique[, column name 2 data type unique,...]);
Check constraints:
Used to ensure that the data in a column (or a set of columns) satisfies a set of specified conditions.
Specify the range, check the maximum or minimum range, and implement
the create table table name (column name data type check (judgment statement)) through check;
5.
The concept of junction table (multi-table operation): When saving data, all data is often not stored in one table, but stored in multiple tables. The junction table is to query data from multiple tables. It is not conducive to decomposing data in a table, and it is easy to cause the same data to appear multiple times, wasting storage space; it is more intuitive to use a junction table to view each data, which makes it easier to process data.
For example, students' annual test scores and personal information of students are basically fixed (including student number, name, address, etc.); putting all the information in the same table will inevitably cause duplication of basic information such as student number.
Advantages of sub-tables:
separate storage of student information and grades, saving space, simple processing, and higher efficiency, especially when processing large amounts of data;
using a relational database to store data, the design of each table is very important, good table design , can simplify data processing, improve efficiency, and improve the robustness of the database.
Use join:
all tables to be joined by select statement and how they are key
Commonly used statements:
select column name 1, column name 2, ... from table 1, table 2... where judgment statement;
select * from tbl1, tbl2 where tbl1.id = tbl2.id
sqlite> select name, addr, score, year from persons, grade where persons.id = grade.id and name = 'lucy';
6. Views (virtual tables)
reuse SQL statements;
simplify complex SQL operations (such as the previous multi-table query)
sqlite> select name, addr, score, year from persons, grade where persons.id = grade.id and name = 'lucy';
Use a view to wrap the entire query into a virtual table named PersonGrade, which simplifies the SQL statement of the query:
sqlite> create view PersonGrade as select name, addr, score, year from persons, grade where persons.id = grade.id and name = 'lucy';
sqlite> select * from PersonGrade;
Delete a view: Syntax: drop view view name; summary: 1. The view does not contain data, so each time the view is used, the query must actually be executed; 2. The view is equivalent to the SQL statement after as when creating the view. 3. Retrieve the view from the returned result information (view), which is the same as the table
7. Trigger
concept: SQLite’s trigger is the callback function of the database, which will automatically execute the call when the specified database event occurs; 1. Only when the delete, insert or update operation is executed, it will trigger and execute the specified One or more SQL statements. (For example, triggers can be set to automatically delete lucy-related information in the grade table when deleting lucy information in the persons table) 2. Triggers are often used to ensure data consistency, and whenever a table is updated or deleted, it will be recorded Write to the log table Create a trigger: Syntax: create trigger trigger name [before/after] [insert|update|delete] on table name begin statement; end; For example: description: when executing: delete from persons where id = 1; When a statement is made, an event is triggered, and the SQL statement between begin and end (that is, the callback function) is executed. Note: old.id is equivalent to persons.id, but persons.id cannot be written here, and old.id represents the id of the deleted row ( id represents the associated column of the two tables)
create trigger tg_delete after delete on persons begin delete from grade where id = old.id;
8. SQLite log operations and index operations that improve query efficiency
are written to logs: sqlite> create table log (time text, data text);
Note here that two functions time('now') and date('now') are used when inserting.
For example: sqlite> insert into log values(time('now'),date('now'));
often used in conjunction with triggers, use To record the operation change time.
Query optimization-index: A large amount of data is often stored in the database, and the default method of ordinary query is to call sequential scan. For example, such a query: select * from table where id = 10000; if there is no index, the entire table must be traversed until the row with ID equal to 10000 is found. In order to improve the efficiency of the query, you can use the index;
what is the index? An index is a structure that sorts the values of one or more columns in a database table. Using an index, you can quickly access specific information in the data table; an index is a proper sort, optimized by a certain algorithm, so that the number of searches is much less Many; Disadvantages: Index data may take up a lot of storage space, so not all data is suitable for indexing; indexing improves the performance of retrieval operations, but reduces the performance of data insertion, modification, and deletion; creating an index: Syntax: create index index name on table name (column name); view index: .indices delete index: drop index index name; enter in the terminal (id column creates index and sorts): create index Note: 1. On the column as the primary key; 2 , Create indexes on the columns that often need to be sorted; 3, Create indexes on the columns that are often used in the WHERE clause to speed up the judgment of conditions.
create index id_index on persons(id);
Avoid using indexes: 1. The amount of data in the table is not large; 2. Most of the operations on the table are not queries; 3. A large number of NULLs appear
1.6 SQLite C Programming
1.6.1 Open and close database functions
int sqlite3_open(char *db_name, sqlite3 **db);
功能：
打开数据库
参数：
db_name: 数据库文件名，若文件名包含ASCII码表范围之外的字符，则其必须是utf-8编码
sqlite3: 数据库标识，此结构体为数据库操作句柄。通过此句柄可对数据库文件进行相应操作。
返回值：
成功返回SQLITE_OK,失败返回非SQLITE_OK
int sqlite3_close(sqlite3 *db);
功能：
关闭数据库、释放打开数据库时申请的资源
参数：
db: 数据库的标识
返回值：
成功返回SQLITE_OK
失败返回非SQLITE_OK
注意：
sqlite3使用了两个库:pthread、dl，故链接时应加上-lpthread和-ldl
1.6.2 The method of executing SQL statement in sqlite3 (callback method)
sqlite3_exec函数：
int sqlite3_exec(sqlite3 *db,const char *sql,exechandller_t callback,void *arg,char **errmsg);
功能：
执行sql执向的SQL语句，若结果集不为空，函数会调用函数指针callback所指向的函数。
参数：
db: 数据库的标识
sql: SQL语句(一条或多条)，以‘;’结尾
callback: 是回调函数指针，当这条语句执行之后，sqlite3会去调用你提供的这个函数
arg： 当执行sqlite_exec的时候传递给回调函数的参数；
errmsg： 存放错误信息的地址，执行失败后可以查阅这个指针；
打印错误信息方法：printf("%s\n",errmsg);
返回：
回调函数指针：
typedef int (*exechandller_t)(void *para,int n_column,char ** column_value,char **column_name);
功能：
此函数由用户定义，当sqlite3_exec函数执行sql语句后，结果集不为空时，sqlite3_exec函数会自动调用此函数，每次调用此函数时会把结果集的一行信息传递给此函数。
参数：
para: sqlite3_exec传给此函数的参数，para为任意数据类型的地址；
n_column: 结果集的列数；
column_value: 指针数据的地址，其存放一行信息中各个列值的首地址；
column_name:指针数据的地址，其存放一行信息中各个列值对应列名的首地址
返回值：
若为非0值，则通知sqlite3_exec终止回调
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include "sqlite3.h"
int loadinfo(void *data,int col, char **col_val,char **name){
static int row = 1;
printf("row= %d\n",row++);
for(int i = 0; i < col; i++)
{
printf("%s = %s \n",name[i], val[i]);
}i
return SQLITE_OK;
}
int main(int argc, char ** argv){
sqlite3 *db = NULL;
int ret;
char *err = NULL;
char *sql;
int arg = 33;
ret = sqlite3_open("stu.db", &db);
if(ret != SQLITE_OK){
printf("open error\n");
return -1;
}
sql = "insert into info values(120,'zz','ts')";
sqlite3_exec(db, sql, loadinfo,&arg,&err);
if(err){
printf("err:%s\n",err);
}
sqlite3_close(db);
return 0;
}
1.6.3 The method of executing SQL statement in sqlite3 (non-callback method)
sqlite3_get_table函数
int sqlite3_get_table(sqlite3 *db,const char *sql,char ***resultp,int *nrow,int *ncolumn,char **errmsg);
功能：
执行sql指向的sql语句，函数将结果集相关的数据的地址保存在函数的参数中。
参数：
db: 数据库的标识；
sql: SQL语句(一条或多条)，以’;‘结尾；
resultp: 指针数据的地址，其记录了结果集的数据。内存布局:先依次存放各列的列名，然后是每一行各列的值
nrow: 结果集的行数(不包含列名)
ncolumn: 结果集的列数
errmsg: 错误信息
sqlite3_free_table函数：
void sqlite3_free_table(char (** resultp);
功能：
释放sqlite3_get_table分配的内容
参数：
结果集数据的首地址
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include "sqlite3.h"
int main(int argc, char ** argv){
sqlite3 *db = NULL;
int ret;
char *err = NULL;
char *sql;
char **result;
int row,col;
ret = sqlite3_open("stu.db", &db);
if(ret != SQLITE_OK){
printf("open error\n");
return -1;
}
sql = "insert into info values(120,'zz','ts')";
sqlite3_get_table(db, sql, &result,&row,&col,&err);
if(err){
printf("err:%s\n",err);
sqlite3_close(db);
return -1;
}
//处理读取到的数据
for(int i = 0; i < row * col; i++){
printf("result[%d] = %s\n", i , result[i]);
}
sqlite3_free_table(result);
sqlite3_close(db);
return 0;
}