The strange malformed error of sqlite3 (1)

The strange malformed error of sqlite3 (1)

phenomenon

• Field equipment generates and inserts large-scale data, and when equipment abnormalities pull out the database for inspection, report malformederrors
• sqlite3 version 3.8.6
• Database file size 210 MB

Positioning problem

• Data size

  select count(*) from DataSheet;
  /*结果*/
  950131
  

• Primary key

  primary key(ctype,id,DataTime)
  

• Recurrence ( SQLite Expert )

  select * from DataSheet where id='000537719140' order by DataTime;  /*malformed*/
  select * from DataSheet where ctype=5002 and id='000537719140' order by DataTime; /*OK*/
  

  As mentioned above, the only difference between the two SQL statements is whether the oad condition is included in the where clause. Why is it so? First perform the following SQL comparison:

  select * from DataSheet where id='000537719140';  /*OK*/
  select * from DataSheet order by DataTime;     /*malformed, 全表扫描*/
  

  • When there are ctype, id, and DataTime in the where clause, the index (primary key) is hit. At this time, the eligible indexes are sorted, so the problem does not recurorder by DataTime
  • When the where clause does not exactly match ctype, id, and DataTime , the index (primary key) is missed, order by DataTimethe results that meet the conditions are sorted, and the full table scan is actually performed, and the problem recurs

• Test code: positioning line

  char * malformed = "select rowid, ctype, id, StartTime, DataTime from DataSheet;"; /*malformed*/
  char * okayQuery = "select rowid, ctype, id, DataTime from DataSheet;"; /*OK*/
  char * sql = okayQuery;
  sqlite3_prepare_v2(db, sql, strlen(sql), &stmt, NULL);
  ncols = sqlite3_column_count(stmt);
  int rowno = 0;
  do {
        
        
      rc = sqlite3_step(stmt);
      switch (rc) {
        
        
      case SQLITE_ROW:	break;
      case SQLITE_DONE:	rowno = -100;	break;
      default:			fprintf(stderr, "rowno = %d, sqlite3 error %d\n", rowno, rc);  break;
      }
      if (rowno >= 0) {
        
         rowno++; }
  }
  while (rowno >= 0);
  sqlite3_finalize(stmt);
  

  • When sql = okayQuery, query a total 950131(rowno:950130)of records, and then SQLITE_DONEend
  • When sql = malformed, 950300report an error when fetching the first record11 (SQLITE_CORRUPT)
    • When the query is normal, only 950131(rowno:950130)records can be traversed
    • When the query is abnormal, an error occurred when traversing to 950300(rowno:950301)a record!

• Test code: positioning time

  char * malformed = "select rowid, ctype, id, DataTime, StartTime from DataSheet;";
  char * okayQuery = "select rowid, ctype, id, DataTime from DataSheet;";
  char * sql = malformed;
  sqlite3_prepare_v2(db, sql, strlen(sql), &stmt, NULL);
  ncols = sqlite3_column_count(stmt);
  int rowno = 0;
  do {
        
        
    rc = sqlite3_step(stmt);
    switch (rc) {
        
        
    case SQLITE_ROW:
      if (rowno >= 950129 && rowno <= 950301/* 950300 */) {
        
        
        rowid	 = sqlite3_column_int(stmt, 0);
        ctype  = (char*)sqlite3_column_text(stmt, 1);
        id = (char*)sqlite3_column_text(stmt, 2);
        colltime = (char *)sqlite3_column_text(stmt, 3);
        fprintf(stderr, "rowno: %d, rowid: %i, ctype: %s, id:%s, time:%s\n", rowno, rowid, (ctype?ctype:snull),(id?id:snull), (colltime?colltime:snull));
      }
      break;
    case SQLITE_DONE:	rowno = -100;	break;
    default:			fprintf(stderr, "rowno = %d, sqlite3 error %d\n", rowno, rc);	break;
    }
    if (rowno >= 0) {
        
         rowno++; }
  }
  while (rowno >= 0);
  sqlite3_finalize(stmt);
  

  Output:

  rowno: 950129, rowid: 4478623, ctype: 5002, id:001548532328, time:20191220105300
  rowno: 950130, rowid: 4478624, ctype: 5002, id:001548532328, time:20191220105400
  ...
  rowno: 950297, rowid: 4478791, ctype: 5002, id:001548532342, time:20191220105800
  rowno: 950298, rowid: 4478792, ctype: 5002, id:001548532342, time:20191220105900
  rowno: 950299, rowid: 4478793, ctype: 5002, id:001548532342, time:20191220110100
  rowno = 950300, sqlite3 error 11
  rowno: 950301, rowid: 3526517, ctype: 5002, id:005410401490, time:20191214202200
  

Log

• 20191220.log

  #12-20 11:21:30.098: 写入[38] 20191220-111700  ...
  #12-20 11:21:30.433: 写入[39] 20191220-111800  ...
  #12-20 11:21:30.434: 写入[40] 20191220-111900  ...
  

Check the database for the next day

• 2019/12/21 database
• Database file size: 210 MB
  • Same database size
• Check menu
  • OK

analysis

• The 2019/12/20 database has a malformed problem, but the 2019/12/21 database does not have this problem. It may be:
  • Possible reason 1: The database has been automatically restored
    • Data cycle has not arrived: automatic recovery without cleaning
    • No repair function in the device
  • Possible cause two: database copy problem
    • The final log time stamp: 12-20 11:24:21.057# ...
    • If you pull the database with ftp first, and then save the shell log, when you pull the database, it is possible that the database is executing a transaction, which causes the problem

in conclusion

• When ftp pulls the database, the database is executing a data write transaction, which causes the malformed problem