A certain project needs to query clickhousethe database. This article is some records and notes of the usage process and is not authoritative.
Overview
The author has never been familiar with databases and only knows a few selectstatements. Just like not being familiar with Androiddevelopment, Javadevelopment. But the recent project has actually used it, and the project has been promoted to a very high level. It is said that the director-level leader personally intervened. This project needs to query the database and do some necessary statistics and analysis, but oraclethere are too many databases and the query is very time-consuming. After inquiry, I was able to deploy a set clickhouseof databases, which may be clustered or distributed. But to me it is just a database. The test found that the speed is much, much faster. Except for the last 1 or 2 hours, the historical data is basically the same. So, I decided to use this library in the project. There are many related clients, gosome Didn't work, but eventually found a suitable client.
- Some clients support
clickhousea newer version, but the production database version is older, so it cannot be used. - Production databases only support
httpconnection methods. - To facilitate testing, the database was redeployed in a container.
Use github.com/ClickHouse/clickhouse-go/v2
The github repository is here , download:
go get github.com/ClickHouse/clickhouse-go/v2
Test code:
import (
"database/sql"
_ "github.com/ClickHouse/clickhouse-go/v2"
)
func CreateClickHouse_bad(dbstr string) (sqldb *sql.DB, err error) {
rdbstr = "http://latelee.cn:8082/default?username=default&password=123456"
klog.Println("connn ", rdbstr)
sqldb, err = sql.Open("clickhouse", rdbstr) // 去掉前缀
if err != nil {
return nil, errors.New("open database failed: " + err.Error())
}
err = sqldb.Ping()
if err != nil {
return nil, errors.New("connect database failed: " + err.Error())
}
klog.Println("connect to clickhouse ok")
//log.Println("connect to ", dbParam.server, dbParam.database, "ok")
// test...
results, err := sqldb.Query(`
select id, age, name from userinfo u order by u.id
`)
if err != nil {
klog.Println("Query error: ", err)
return
}
for results.Next() {
var item1, item3 sql.NullString
var item2 sql.NullInt64
err := results.Scan(&item1, &item2, &item3)
if err != nil {
klog.Println("scan error: ", err)
break
}
if !item1.Valid {
continue
}
klog.Println("dddddddddd ", item1.String, item2.Int64, item3.String)
// break
}
os.Exit(0)
return
}
result:
2023-03-22 00:19:05.031 INFO - connecting db...
2023-03-22 00:19:05.031 INFO - connn http://latelee.cn:8082/default?username=default&password=123456
WARNING: version 21.9.6 of ClickHouse is not supported by this client
2023-03-22 00:19:05.131 INFO - connect to clickhouse ok
2023-03-22 00:19:05.154 INFO - scan error: sql: Scan error on column index 0, name "id": unsupported Scan, storing driver.Value type *string into type *string
try
According to
of ClickHouse is not supported by this clientthe source code search, it was found that it supports a specific version of the database.Find
github.com\ClickHouse\clickhouse-go\v2\resources\meta.ymlthe file with the following content:clickhouse_versions: - 22.3 - 22.8 - 22.9 - 22.10 - 22.11 go_versions: - 1.18 - 1.19Manually added the version number to the file, the error remains.
This version is commonly used on the Internet, but it cannot be used in projects and is discarded.
Use github.com/uptrace/go-clickhouse/ch
The github repository is here , download:
go get github.com/uptrace/go-clickhouse/ch
go get github.com/uptrace/go-clickhouse/chdebug
Test code:
import (
"database/sql"
"github.com/uptrace/go-clickhouse/ch"
"github.com/uptrace/go-clickhouse/chdebug"
)
func CreateClickHouse_bad1(dbstr string) (sqldb *sql.DB, err error) {
ctx := context.Background()
constr := "clickhouse://default:[email protected]:9000/default?sslmode=disable"
aaa := fmt.Sprintf(
"clickhouse://%s:%s@%s:%d/%s?sslmode=disable",
"default",
"",
"localhost",
9000,
"testUSER",
)
klog.Println("connn ", constr)
klog.Println("aaa ", aaa)
db := ch.Connect(
ch.WithDSN(constr),
ch.WithTimeout(5*time.Second),
ch.WithDialTimeout(5*time.Second),
ch.WithReadTimeout(5*time.Second),
ch.WithWriteTimeout(5*time.Second),
ch.WithPoolSize(100),
)
//打印Query Error
db.AddQueryHook(chdebug.NewQueryHook(chdebug.WithVerbose(true)))
//联通测试
if err = db.Ping(ctx); err != nil {
return
}
// 测试打印Query
klog.Println("db ", db)
// results, err := sqldb.QueryContext(ctx, `
// select id, age, name from userinfo u order by u.id
// `)
results, err := db.Query(`
select id, age, name from userinfo u order by u.id
`)
if err != nil {
klog.Println("Query error: ", err)
return
}
for results.Next() {
var item1, item3 sql.NullString
var item2 sql.NullInt64
err := results.Scan(&item1, &item2, &item3)
if err != nil {
klog.Println("scan error: ", err)
break
}
if !item1.Valid {
continue
}
klog.Println("dddddddddd ", item1.String, item2.Int64, item3.String)
}
os.Exit(0)
return
}
result:
2023-03-22 00:28:51.456 INFO - db DB<addr: latelee.cn:9000>
[ch] 00:28:51.464
select 7.588ms
select id, age, name from userinfo u order by u.id
panic: reflect: call of reflect.Value.IsNil on struct Value
goroutine 1 [running]:
reflect.Value.IsNil(...)
D:/go/src/reflect/value.go:1506
github.com/uptrace/go-clickhouse/ch/chschema.(*NullableColumn).ConvertAssign(0xc0004a4080, 0x0, {0x15d9600?, 0xc0000965b8?, 0x23195500598?})
E:/project/golang_test/dbtool/vendor/github.com/uptrace/go-clickhouse/ch/chschema/column_nullable.go:85 +0x19d
github.com/uptrace/go-clickhouse/ch.(*Rows).Scan(0xc00052e210, {0xc0007a3b50, 0x3, 0x1692676?})
Reviews:
According to the project home page, its performance ratio ClickHouse/clickhouse-gois better.
According to the official manual , it is supported model, but the library fields involved by the author exceed 100, and the access to this mode has not been studied and discarded.
May not be supported http. The sample code all uses 9000ports. Not found in sample code.
Use github.com/mailru/go-clickhouse/v2
The github repository is here , download:
go get github.com/mailru/go-clickhouse/v2
Test code:
import (
"database/sql"
_ "github.com/mailru/go-clickhouse/v2"
)
func CreateClickHouse(dbstr string) (sqldb *sql.DB, err error) {
rdbstr := dbstr[len("clickhouse:"):]
rdbstr = "http://default:[email protected]:8082/default"
klog.Println("connn ", rdbstr)
sqldb, err = sql.Open("chhttp", rdbstr) // 去掉前缀
if err != nil {
return nil, errors.New("open database failed: " + err.Error())
}
err = sqldb.Ping()
if err != nil {
return nil, errors.New("connect database failed: " + err.Error())
}
klog.Println("connect to clickhouse ok")
//log.Println("connect to ", dbParam.server, dbParam.database, "ok")
results, err := sqldb.Query(`
select id, age, name from userinfo u order by u.id
`)
if err != nil {
klog.Println("Query error: ", err)
return
}
for results.Next() {
var item1, item3 sql.NullString
var item2 sql.NullInt64
err := results.Scan(&item1, &item2, &item3)
if err != nil {
klog.Println("scan error: ", err)
break
}
if !item1.Valid {
continue
}
klog.Println("dddddddddd ", item1.String, item2.Int64, item3.String)
// break
}
os.Exit(0)
return
}
result:
2023-03-22 00:34:35.435 INFO - connecting db...
2023-03-22 00:34:35.435 INFO - connn http://default:[email protected]:8082/default
2023-03-22 00:34:35.495 INFO - connect to clickhouse ok
2023-03-22 00:34:35.520 INFO - dddddddddd 001 250 latelee
2023-03-22 00:34:35.520 INFO - dddddddddd 002 252 latelee2
2023-03-22 00:34:35.520 INFO - dddddddddd 003 253 latelee3
2023-03-22 00:34:35.520 INFO - dddddddddd 004 254 latelee4
2023-03-22 00:34:35.520 INFO - dddddddddd 005 255 latelee5
Finally use this client
summary
At present, there is no clickhousetoo in-depth research on gothe query, and it remains oracleconsistent. In the future, we may study data warehouse and data analysis, and we will see again then.