背景

PostgreSQL是一个历史悠久的数据库,历史可以追溯到1973年,最早由2014计算机图灵奖得主,关系数据库的鼻祖Michael_Stonebraker 操刀设计,PostgreSQL具备与Oracle类似的功能、性能、架构以及稳定性。

pic

PostgreSQL社区的贡献者众多,来自全球各个行业,历经数年,PostgreSQL 每年发布一个大版本,以持久的生命力和稳定性著称。

2017年10月,PostgreSQL 推出10 版本,携带诸多惊天特性,目标是胜任OLAP和OLTP的HTAP混合场景的需求:

《最受开发者欢迎的HTAP数据库PostgreSQL 10特性》

1、多核并行增强

2、fdw 聚合下推

3、逻辑订阅

4、分区

5、金融级多副本

6、json、jsonb全文检索

7、还有插件化形式存在的特性,如 向量计算、JIT、SQL图计算、SQL流计算、分布式并行计算、时序处理、基因测序、化学分析、图像分析 等。

pic

在各种应用场景中都可以看到PostgreSQL的应用:

pic

PostgreSQL近年来的发展非常迅猛,从知名数据库评测网站dbranking的数据库评分趋势,可以看到PostgreSQL向上发展的趋势:

pic

从每年PostgreSQL中国召开的社区会议,也能看到同样的趋势,参与的公司越来越多,分享的公司越来越多,分享的主题越来越丰富,横跨了 传统企业、互联网、医疗、金融、国企、物流、电商、社交、车联网、共享XX、云、游戏、公共交通、航空、铁路、军工、培训、咨询服务等 行业。

接下来的一系列文章,将给大家介绍PostgreSQL的各种应用场景以及对应的性能指标。

环境

环境部署方法参考:

《PostgreSQL 10 + PostGIS + Sharding(pg_pathman) + MySQL(fdw外部表) on ECS 部署指南(适合新用户)》

阿里云 ECS:56核,224G,1.5TB*2 SSD云盘

操作系统:CentOS 7.4 x64

数据库版本:PostgreSQL 10

PS:ECS的CPU和IO性能相比物理机会打一定的折扣,可以按下降1倍性能来估算。跑物理主机可以按这里测试的性能乘以2来估算。

场景 - 高并发空间位置更新、多属性KNN搜索并测(含空间索引)

1、背景

越来越多的应用正在接入空间数据属性,例如 物联网、车辆轨迹管理、公安系统的稽侦系统、O2O应用、LBS交友应用、打车应用等等。

被管理的对象携带空间属性,对象的运动形成了轨迹,最后一个位置点表示对象的最终位置。

PostgreSQL在空间数据库管理领域有这几十年的发展历程,例如PostGIS空间数据库,pgrouting路由插件等,GiST空间索引,SP-GiST空间分区索引等。

本文要测试的是空间数据的合并更新性能(携带空间索引),例如,更新对象的最终空间位置,同时根据用户输入,搜索附近N米内满足条件的对象(用到了btree_gist插件以及GiST索引)。

2、设计

2000万个被跟踪对象,2000万个点,含空间索引。

1、实时合并更新被跟踪对象的位置。

2、同时根据用户输入,搜索附近N米内满足条件的对象。

3、准备测试表

创建测试表、索引。

  1. create table tbl_pos(id int primary key, att1 int, att2 int, att3 int, mod_time timestamp, pos geometry);
  2. create extension btree_gist;
  3. create index idx_tbl_pos_1 on tbl_pos using gist(att1, att2, pos);

查询为多维度搜索,除了空间相近,还需要满足某些查询条件,例如:

注意,order by的两侧需要对齐类型,例如geometry <-> geometry,这样才会走索引哦,否则效率差的很。

  1. explain select *, ST_DistanceSpheroid(pos , st_setsrid(st_makepoint(120,60), 4326), 'SPHEROID["WGS84",6378137,298.257223563]') from tbl_pos -- 120, 60表示经纬度
  2. where att1=3 and att2<=3 and
  3. st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(120, 60), 4326)),5000)), pos) -- 5000 表示5000
  4. order by pos <-> st_setsrid(st_makepoint(120, 60), 4326) limit 100; -- 这里不要使用pos <-> geography(st_setsrid(st_makepoint(120, 60), 4326))
  5. QUERY PLAN
  6. -----------------------------------------------------------------------------------------------------------------------
  7. Limit (cost=0.42..469.18 rows=100 width=72)
  8. -> Index Scan using idx_tbl_pos_1 on tbl_pos (cost=0.42..7125.52 rows=1520 width=72)
  9. Index Cond: ((att1 = 3) AND (att2 <= 3) AND ('0103000020E61000000100000021000000267F6991BA055E40F420D353BCFF4D40AE7F7C0791055E4004D7AB0D9FFE4D405B8C22C830055E409DBC6E5F8FFD4D40D24FE28E9D045E40FB4032B697FC4D4043F65109DD035E40B3AD 6691C1FB4D407907729EF6025E40CD022A2615FB4D40FCD5B525F3015E401498630F99FA4D40FD598C90DC005E400571980D52FA4D40C3B0A789BDFF5D40F14DD0D842FA4D40F12 00DA1FE5D408B 38066CFA4D4059C7410092FD5D40C53C7702CCFA4D40AFC2A9C AFC5D40A28FEA205FFB4D40D F6AD2D2C4FB5D408 A3DBF1FFC4D40B73C6B6518FB5D400FF2147C06FD4D40112D1B1F9CFA5D40D0CACF7E0AFE4D404E26B0CD54FA5D40CCE2BECD21FF4D4087E3C53845FA5D403B51A0AF41004E408079E9046EFA5D40CDB8BA145F014E40CF0290ABCDFA5D406B5C9D036F024E409222728860FB5D 40D84C5A0567034E40DB9B09FC20FC5D409F06088D3D044E40B06412A207FD5D40FBB07A56EA044E40B78A179A0BFE5D40963273B866054E40A25449DF22FF5D40D0DCF9E6AD054E4024FC28AC42005E402D883723BD054E4025240AE65F015E40A173DCD693054E4033F60B8C6F025E4068D3F799330 54E4023F6012567035E4013FB0323A0044E4069F3D1273D045E40E1C5CE21DF034E40D515105AE9045E402E7AC106F8024E407F25172165055E40579ED4B8F3014E40D5FF79C2AB055E40E0132A3DDC004E40267F6991BA055E40F420D353BCFF4D40'::geometry ~ pos))
  10. Order By: (pos <-> '0101000020E61000000000000000005E400000000000004E40'::geometry)
  11. Filter: _st_contains('0103000020E61000000100000021000000267F6991BA055E40F420D353BCFF4D40AE7F7C0791055E4004D7AB0D9FFE4D405B8C22C830055E409DBC6E5F8FFD4D40D24FE28E9D045E40FB4032B697FC4D4043F65109DD035E40B3AD6691C1FB4D407907729EF602 5E40CD022A2615FB4D40FCD5B525F3015E401498630F99FA4D40FD598C90DC005E400571980D52FA4D40C3B0A789BDFF5D40F14DD0D842FA4D40F12 00DA1FE5D408B 38066CFA4D4059C7410092FD5D40C53C7702CCFA4D40AFC2A9C AFC5D40A28FEA205FFB4D40DF6AD2D2C4FB5D408 A3DBF1 FFC4D40B73C6B6518FB5D400FF2147C06FD4D40112D1B1F9CFA5D40D0CACF7E0AFE4D404E26B0CD54FA5D40CCE2BECD21FF4D4087E3C53845FA5D403B51A0AF41004E408079E9046EFA5D40CDB8BA145F014E40CF0290ABCDFA5D406B5C9D036F024E409222728860FB5D40D84C5A0567034E40DB9B09 FC20FC5D409F06088D3D044E40B06412A207FD5D40FBB07A56EA044E40B78A179A0BFE5D40963273B866054E40A25449DF22FF5D40D0DCF9E6AD054E4024FC28AC42005E402D883723BD054E4025240AE65F015E40A173DCD693054E4033F60B8C6F025E4068D3F79933054E4023F6012567035E4013F B0323A0044E4069F3D1273D045E40E1C5CE21DF034E40D515105AE9045E402E7AC106F8024E407F25172165055E40579ED4B8F3014E40D5FF79C2AB055E40E0132A3DDC004E40267F6991BA055E40F420D353BCFF4D40'::geometry, pos)
  12. (5 rows)
  1. id | att1 | att2 | att3 | mod_time | pos | st_distancespheroid
  2. ----------+------+------+------+----------------------------+----------------------------------------------------+---------------------
  3. 8892973 | 3 | 3 | 1 | 2018-07-10 17:44:08.386618 | 0101000020E61000000000031067FF5D4000001A383C014E40 | 1194.64901625583
  4. 2083046 | 3 | 1 | 2 | 2018-07-17 15:42:55.031903 | 0101000020E610000000002B99AE005E400000E007D5014E40 | 1701.93484541633
  5. 13441821 | 3 | 2 | 3 | 2018-07-10 17:41:49.504894 | 0101000020E610000000006675EF005E400000E04E74FB4D40 | 40 .84460729299
  6. 8662140 | 3 | 1 | 4 | 2018-07-17 15:41:42.761599 | 0101000020E61000000000D0F49AFF5D400000DC3C0BFB4D40 | 4327.54163693541
  7. 78 452 | 3 | 3 | 1 | 2018-07-17 15:42:15.954411 | 0101000020E61000000000CA94E7FF5D400000F27727054E40 | 4487.02042256402
  8. 16796301 | 3 | 2 | 4 | 2018-07-10 17:15:10.231126 | 0101000020E6100000000008F571025E400000D2A562024E40 | 2975. 699500948
  9. 1587379 | 3 | 1 | 1 | 2018-07-10 17:53:31.308692 | 0101000020E61000000000ABBFBBFC5D400000EC0B93FB4D40 | 4791.49425173447
  10. 8560096 | 3 | 3 | 2 | 2018-07-17 15:41:46.907464 | 0101000020E610000000001B707F035E4000002A5B09FC4D40 | 4605.4604334459
  11. 5540068 | 3 | 1 | 2 | 2018-07-10 17:42:29.689334 | 0101000020E610000000004C330C055E400000F02624FE4D40 | 4689.80080183583
  12. 17813180 | 3 | 1 | 5 | 2018-07-10 17:50:18.297117 | 0101000020E61000000000B88C95FA5D400000A6C915004E40 | 4722.45290664137
  13. 6424827 | 3 | 3 | 4 | 2018-07-10 17:50:54.958542 | 0101000020E61000000000564E8EFA5D4000002C28BA004E40 | 4788.20027459238
  14. (11 rows)

4、准备测试函数(可选)

5、准备测试数据

6、准备测试脚本

测试仅使用一般的CPU资源(28核)。

1、更新用户的最终位置,由于空间移动有一定的速比,所以更新后是基于原有位置的一个相对位移位置。

  1. vi test.sql
  2. \set att1 random(1,5)
  3. \set att2 random(1,5)
  4. \set att3 random(1,5)
  5. \set id random(1,20000000)
  6. \set x random(120,130)
  7. \set y random(70,80)
  8. insert into tbl_pos (id, att1, att2, att3, mod_time, pos) values (:id, :att1, :att2, :att3, now(), st_setsrid(st_makepoint(:x,:y), 4326)) on conflict (id) do update set pos=st_setsrid(st_makepoint( st_x(tbl_pos.pos)+5-random()*10, st_y(tbl_pos.pos)+5-random()*10), 4326), mod_time=excluded.mod_time;

压测

  1. CONNECTS=20
  2. TIMES=120
  3. export PGHOST=$PGDATA
  4. export PGPORT=1999
  5. export PGUSER=postgres
  6. export PGPASSWORD=postgres
  7. export PGDATABASE=postgres
  8. pgbench -M prepared -n -r -f ./test.sql -P 5 -c $CONNECTS -j $CONNECTS -T $TIMES

2、根据用户输入的att1, att2条件,搜索满足条件的附近5公里内的对象,根据距离顺序返回100条。

  1. vi test1.sql
  2. \set att1 random(1,5)
  3. \set att2 random(1,5)
  4. \set x random(120,130)
  5. \set y random(70,80)
  6. select * from tbl_pos where att1=:att1 and att2=:att2 and st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(:x,:y), 4326)),5000)), pos) order by pos <-> st_setsrid(st_makepoint(:x,:y), 4326) limit 100;

压测

  1. CONNECTS=8
  2. TIMES=120
  3. export PGHOST=$PGDATA
  4. export PGPORT=1999
  5. export PGUSER=postgres
  6. export PGPASSWORD=postgres
  7. export PGDATABASE=postgres
  8. pgbench -M prepared -n -r -f ./test1.sql -P 5 -c $CONNECTS -j $CONNECTS -T $TIMES

7、测试

1、更新对象位置

  1. transaction type: ./test.sql
  2. scaling factor: 1
  3. query mode: prepared
  4. number of clients: 20
  5. number of threads: 20
  6. duration: 120 s
  7. number of transactions actually processed: 13271261
  8. latency average = 0.181 ms
  9. latency stddev = 0.196 ms
  10. tps = 110592.138000 (including connections establishing)
  11. tps = 110597.618184 (excluding connections establishing)
  12. script statistics:
  13. - statement latencies in milliseconds:
  14. 0.001 \set att1 random(1,5)
  15. 0.000 \set att2 random(1,5)
  16. 0.000 \set att3 random(1,5)
  17. 0.000 \set id random(1,20000000)
  18. 0.000 \set x random(120,130)
  19. 0.000 \set y random(70,80)
  20. 0.178 insert into tbl_pos (id, att1, att2, att3, mod_time, pos) values (:id, :att1, :att2, :att3, now(), st_setsrid(st_makepoint(:x,:y), 4326)) on conflict (id) do update set pos=st_setsrid(st_makepoint( st_x(tbl_pos.pos)+5-random()*10, st_y(tbl_pos.pos)+5-random()*10), 4326), mod_time=excluded.mod_time;

TPS: 110592

平均响应时间: 0.178 毫秒

2、根据用户输入的att1, att2条件,搜索满足条件的附近5公里内的对象,根据距离顺序返回100条。

  1. transaction type: ./test1.sql
  2. scaling factor: 1
  3. query mode: prepared
  4. number of clients: 8
  5. number of threads: 8
  6. duration: 120 s
  7. number of transactions actually processed: 1136703
  8. latency average = 0.845 ms
  9. latency stddev = 0.3 ms
  10. tps = 9472.446079 (including connections establishing)
  11. tps = 9472.793841 (excluding connections establishing)
  12. script statistics:
  13. - statement latencies in milliseconds:
  14. 0.002 \set att1 random(1,5)
  15. 0.000 \set att2 random(1,5)
  16. 0.000 \set x random(120,130)
  17. 0.000 \set y random(70,80)
  18. 0.842 select * from tbl_pos where att1=:att1 and att2=:att2 and st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(:x,:y), 4326)),5000)), pos) order by pos <-> st_setsrid(st_makepoint(:x,:y), 4326) limit 100;

TPS: 9472

平均响应时间: 0.842 毫秒

小结1

1、注意,为了提高过滤性能,同时为了支持米为单位的距离,我们存储时使用4326 srid, 同时距离过滤时使用以下表达式

  1. st_contains(
  2. geometry(
  3. ST_Buffer( -- 圆形转换为对称多边形,所以边缘部分会有一定的缺失,不过大部分场景不是非得要一个圆形覆盖,调整半径即可放大多边形覆盖的区域。
  4. geography(
  5. st_setsrid(st_makepoint(:x,:y), 4326) -- 中心点
  6. ),
  7. 5000 -- st_buffer生成以某个POINT为中心的,半径距离为5000米的polygon空间对对象
  8. )
  9. ), -- geography转换为geometry
  10. pos -- st_contains, polygon包含point
  11. )

st_buffer输出的多边形精度(边的锯齿),可以通过第三个参数指定

http://postgis.net/docs/manual-2.4/ST\_Buffer.html

  1. geometry ST_Buffer(geometry g1, float radius_of_buffer);
  2. geometry ST_Buffer(geometry g1, float radius_of_buffer, integer num_seg_quarter_circle);
  3. geometry ST_Buffer(geometry g1, float radius_of_buffer, text buffer_style_parameters);
  4. geography ST_Buffer(geography g1, float radius_of_buffer_in_meters);
  5. geography ST_Buffer(geography g1, float radius_of_buffer, integer num_seg_quarter_circle);
  6. geography ST_Buffer(geography g1, float radius_of_buffer, text buffer_style_parameters);
  1. postgres=# select ST_Buffer( -- 圆形转换为对称多边形,所以边缘部分会有一定的缺失,不过大部分场景不是非得要一个圆形覆盖,调整半径即可放大多边形覆盖的区域。
  2. postgres(# geography(
  3. postgres(# st_setsrid(st_makepoint(:x,:y), 4326) -- 中心点
  4. postgres(# ),
  5. postgres(# 5000 -- st_buffer生成以某个POINT为中心的,半径距离为5000米的polygon空间对对象
  6. postgres(# );
  7. ERROR: syntax error at or near ":"
  8. LINE 3: st_setsrid(st_makepoint(:x,:y), 4326)
  9. ^
  10. postgres=# select st_astext(ST_Buffer( -- 圆形转换为对称多边形,所以边缘部分会有一定的缺失,不过大部分场景不是非得要一个圆形覆盖,调整半径即可放大多边形覆盖的区域。
  11. geography(
  12. st_setsrid(st_makepoint(120,60), 4326) -- 中心点
  13. ),
  14. 5000 -- st_buffer生成以某个POINT为中心的,半径距离为5000米的polygon空间对对象
  15. ));
  16. st_astext
  17. -----------------------------------------------------------------------------------------------------------------
  18. POLYGON((120.089512208008 59.997934797373,120.086976882535 59.9892289246618,120.081102403391 59.9809378901202,120.072116585739 59.9733798737888,120.060365991631 59.9668447257385,120.046302424771 59.9615829186565,120.030465533804 59.9577
  19. 960239696,120.013462197363 59.9556290621131,119.995943464051 59.9551650063621,119.978579893839 59.9564216398749,119.962036194074 59.9593508798803,119.946946063787 59.9638405938451,119.933888154493 59.9697188425826,119.923364023899 59.976
  20. 7603971742,119.915778900661 59.9846952929125,119.911425992986 59.9932191068707,119.91047496152 60.0020045788574,119.912965038302 60.0107141410568,119.918803110729 60.0190128820767,119.927766906223 60.0265814486426,119.939513215431 60.033
  21. 1283845128,119.953590887034 60.0384014224128,119.969458125105 60.0421972811211,119.98650343095 60.0443695755035,120.004069366478 60.0448345204498,120.021478185562 60.0435741974281,120.038058292085 60.0406372509086,120.053170444408 60.036
  22. 1369866788,120.066232638388 60.0302469501146,120.076742664059 60.0231941647256,120.084297440151 60.0152503050019,120.088608378544 60.0067211585481,120.089512208008 59.997934797373))
  23. (1 row)
  24. postgres=# \df st_buffer
  25. List of functions
  26. Schema | Name | Result data type | Argument data types | Type
  27. --------+-----------+------------------+--------------------------------------+--------
  28. public | st_buffer | geography | geography, double precision | normal
  29. public | st_buffer | geography | geography, double precision, integer | normal
  30. public | st_buffer | geography | geography, double precision, text | normal
  31. public | st_buffer | geometry | geometry, double precision | normal
  32. public | st_buffer | geometry | geometry, double precision, integer | normal
  33. public | st_buffer | geometry | geometry, double precision, text | normal
  34. public | st_buffer | geometry | text, double precision | normal
  35. public | st_buffer | geometry | text, double precision, integer | normal
  36. public | st_buffer | geometry | text, double precision, text | normal
  37. (9 rows)
  38. postgres=# select st_astext(ST_Buffer( -- 圆形转换为对称多边形,所以边缘部分会有一定的缺失,不过大部分场景不是非得要一个圆形覆盖,调整半径即可放大多边形覆盖的区域。
  39. geography(
  40. st_setsrid(st_makepoint(120,60), 4326) -- 中心点
  41. ),
  42. 5000 -- st_buffer生成以某个POINT为中心的,半径距离为5000米的polygon空间对对象
  43. , 2 ));
  44. st_astext
  45. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  46. ------------------------------------------------------------------------------
  47. POLYGON((120.089512208008 59.997934797373,120.060365991631 59.9668447257385,119.995943464051 59.9551650063621,119.933888154493 59.9697188425826,119.91047496152 60.0020045788574,119.939513215431 60.0331283845128,120.004069366478 60.04483
  48. 45204498,120.066232638388 60.0302469501146,120.089512208008 59.997934797373))
  49. (1 row)
  50. postgres=# select st_astext(ST_Buffer( -- 圆形转换为对称多边形,所以边缘部分会有一定的缺失,不过大部分场景不是非得要一个圆形覆盖,调整半径即可放大多边形覆盖的区域。
  51. geography(
  52. st_setsrid(st_makepoint(120,60), 4326) -- 中心点
  53. ),
  54. 5000 -- st_buffer生成以某个POINT为中心的,半径距离为5000米的polygon空间对对象
  55. , 20 ));
  56. st_astext
  57. ---------------------------------------------------------------------------------------------------------------------
  58. POLYGON((120.089512208008 59.997934797373,120.088908111921 59.9944238512618,120.087756162016 59.9909474584064,120.086063625849 59.9875270364661,120.083841092477 59.9841836517995,120.081102403391 59.9809378901202,120.077864563782 59.9778
  59. 097302563,120.074147634755 59.9748184217725,120.069974607239 59.9719823671917,120.065371258421 59.9693190095193,120.060365991631 59.9668447257385,120.054989660701 59.9645747269098,120.049275379886 59.9625229654624,120.043258320518 59.960
  60. 7020502247,120.036975495624 59.9591231696903,120.030465533804 59.9577960239696,120.023768443702 59.956728765823,120.016925370471 59.9559279511195,120.009978345645 59.9553984990076,120.002970031881 59.9551436620311,119.995943464051 59.955
  61. 1650063621,119.988941788165 59.9554624022667,119.982007999661 59.9560340248591,119.975184682535 59.9568763651408,119.968513750845 59.9579842512615,119.962036194074 59.9593508798803,119.955791827821 59.9609678574459,119.949819051296 59.96
  62. 28252511582,119.944154613012 59.9649116493158,119.938833386091 59.9672142306985,119.933888154493 59.9697188425826,119.929349411479 59.9724100869304,119.925245171517 59.9752714142513,119.921600796809 59.9782852245804,119.918438839511 59.9
  63. 814329749779,119.915778900661 59.9846952929125,119.913637506731 59.9880520948514,119.912028004629 59.9914827093477,119.910960475872 59.9949660038842,119.910441670538 59.9984805147068,119.91047496152 60.0020045788574,119.911060319439 60.0
  64. 055164676015,119.912194308496 60.0089945204305,119.913870103388 60.0124172788112,119.916077527281 60.0157636188547,119.918803110729 60.0190128820767,119.922030171247 60.02214500343,119.925738913146 60.0251406358055,119.929906547094 60.02
  65. 7981270212,119.934507428714 60.030649350874,119.939513215431 60.0331283845128,119.944893040613 60.0354030431106,119.950613703968 60.0374592594989,119.956639877008 60.0392843151536,119.962934322297 60.0408669196299,119.969458125105 60.042
  66. 1972811211,119.976170935976 60.0432671676831,119.983031222659 60.0440699587263,119.989996529766 60.0446006864375,119.997023744464 60.0448560668633,120.004069366478 60.0448345204498,120.011089780615 60.044536181907,120.018041530037 60.043
  67. 9628993317,120.024881588488 60.043118222597,120.03156762968 60.042007381085,120.038058292085 60.0406372509086,120.044313437398 60.0390163118395,120.050294400996 60.0371545942238,120.055964232773 60.0350636162323,120.061287926806 60.03275
  68. 63118559,120.066232638388 60.0302469501146,120.070767887067 60.027551046003,120.074865744419 60.0246852637505,120.078501005406 60.0216673130169,120.081651342285 60.0185158386919,120.084297440151 60.0152503050019,120.086423113343 60.01189
  69. 08746616,120.088015402048 60.0084582838374,120.0890646486 60.0049737137097,120.089564553084 60.0014586594418,120.089512208008 59.997934797373))
  70. (1 row)

2、本文使用的插件btree_gin, btree_gist用于加速数组搜索,空间数据与其他普通字段的搜索。

3、使用的索引接口gist用于KNN搜索,距离排序。

4、unionall用于普通字段(可枚举)+gis字段的复合排序输出。

需求升华1

除了空间搜索,还引入普通字段筛选条件

1、如果业务方要求按普通字段(当可以枚举时) + 空间字段排序,可以这样来操作,以达到最好的性能。

例如先返回空闲状态的骑手,其次返回最近7天活跃的骑手,其次。。。。

  1. with
  2. a as (
  3. select * from tbl_pos where att1=1 and -- 普通条件(假设att1可枚举)(首先输出att1=1,然后att1=2,然后att1=3
  4. att2<=3 and st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(120,60), 4326)),5000)), pos)
  5. order by pos <-> st_setsrid(st_makepoint(120,60), 4326) limit 100 -- 只按空间排
  6. ),
  7. b as (
  8. select * from tbl_pos where att1=2 and
  9. att2<=3 and st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(120,60), 4326)),5000)), pos)
  10. order by pos <-> st_setsrid(st_makepoint(120,60), 4326) limit 100
  11. ),
  12. c as (
  13. select * from tbl_pos where att1=3 and
  14. att2<=3 and st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(120,60), 4326)),5000)), pos)
  15. order by pos <-> st_setsrid(st_makepoint(120,60), 4326) limit 100
  16. )
  17. select * from a union all select * from b union all select * from c limit 100; -- 按指定顺序写ALIAS union all,执行计划会从先到后对query进行append
  18. 以上效果等效如下SQL,但是以上SQL性能比下面这条高很多很多。
  19. select * from tbl_pos where att2<=3 and st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(120,60), 4326)),5000)), pos)
  20. order by att1, pos <-> st_setsrid(st_makepoint(120,60), 4326) -- att1, knn 复合排序。
  21. limit 100;

效果:

  1. explain (analyze,verbose,timing,costs,buffers) with
  2. a as (
  3. select * from tbl_pos where att1=1 and -- 普通条件(首先输出1,然后2,然后3
  4. att2<=3 and st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(120,60), 4326)),5000)), pos) order by pos <-> st_setsrid(st_makepoint(120,60), 4326) limit 100
  5. ),
  6. b as (
  7. select * from tbl_pos where att1=2 and
  8. att2<=3 and st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(120,60), 4326)),5000)), pos) order by pos <-> st_setsrid(st_makepoint(120,60), 4326) limit 100
  9. ),
  10. c as (
  11. select * from tbl_pos where att1=3 and
  12. att2<=3 and st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(120,60), 4326)),5000)), pos) order by pos <-> st_setsrid(st_makepoint(120,60), 4326) limit 100
  13. )
  14. select * from a union all select * from b union all select * from c limit 100;

自动跳过不需要执行的SQL,类似如下

  1. Limit (cost=1282.26..1284.26 rows=100 width=56) (actual time=0.663..2.295 rows=32 loops=1)
  2. Output: a.id, a.att1, a.att2, a.att3, a.mod_time, a.pos
  3. Buffers: shared hit=324
  4. CTE a
  5. -> Limit (cost=0.42..427.51 rows=100 width=64) (actual time=0.651..0.768 rows=10 loops=1)
  6. Output: tbl_pos.id, tbl_pos.att1, tbl_pos.att2, tbl_pos.att3, tbl_pos.mod_time, tbl_pos.pos, ((tbl_pos.pos <-> '0101000020E61000000000000000005E400000000000004E40'::geometry))
  7. Buffers: shared hit=97
  8. -> Index Scan using idx_tbl_pos_1 on public.tbl_pos (cost=0.42..3481.20 rows=815 width=64) (actual time=0.650..0.766 rows=10 loops=1)
  9. Output: tbl_pos.id, tbl_pos.att1, tbl_pos.att2, tbl_pos.att3, tbl_pos.mod_time, tbl_pos.pos, (tbl_pos.pos <-> '0101000020E61000000000000000005E400000000000004E40'::geometry)
  10. Index Cond: ((tbl_pos.att1 = 1) AND (tbl_pos.att2 <= 3) AND ('0103000020E61000000100000021000000267F6991BA055E40F420D353BCFF4D40AE7F7C0791055E4004D7AB0D9FFE4D405B8C22C830055E409DBC6E5F8FFD4D40D24FE28E9D045E40FB4032B697FC 4D4043F65109DD035E40B3AD6691C1FB4D407907729EF6025E40CD022A2615FB4D40FCD5B525F3015E401498630F99FA4D40FD598C90DC005E400571980D52FA4D40C3B0A789BDFF5D40F14DD0D842FA4D40F12 00DA1FE5D408B 38066CFA4D4059C7410092FD5D40C53C7702CCFA4D40AFC2A9C AFC5D40A28FEA205FFB4D40DF6AD2D2C4FB5D408 A3DBF1FFC4D40B73C6B6518FB5D400FF2147C06FD4D40112D1B1F9CFA5D40D0CACF7E0AFE4D404E26B0CD54FA5D40CCE2BECD21FF4D4087E3C53845FA5D403B51A0AF41004E408079E9046EFA5D40CDB8BA145F014E40CF0290ABCDFA5D406B5C9D 036F024E409222728860FB5D40D84C5A0567034E40DB9B09FC20FC5D409F06088D3D044E40B06412A207FD5D40FBB07A56EA044E40B78A179A0BFE5D40963273B866054E40A25449DF22FF5D40D0DCF9E6AD054E4024FC28AC42005E402D883723BD054E4025240AE65F015E40A173DCD693054E4033F 60B8C6F025E4068D3F79933054E4023F6012567035E4013FB0323A0044E4069F3D1273D045E40E1C5CE21DF034E40D515105AE9045E402E7AC106F8024E407F25172165055E40579ED4B8F3014E40D5FF79C2AB055E40E0132A3DDC004E40267F6991BA055E40F420D353BCFF4D40'::geometry ~ tb l_pos.pos))
  11. Order By: (tbl_pos.pos <-> '0101000020E61000000000000000005E400000000000004E40'::geometry)
  12. Filter: _st_contains('0103000020E61000000100000021000000267F6991BA055E40F420D353BCFF4D40AE7F7C0791055E4004D7AB0D9FFE4D405B8C22C830055E409DBC6E5F8FFD4D40D24FE28E9D045E40FB4032B697FC4D4043F65109DD035E40B3AD6691C1FB4D407907 729EF6025E40CD022A2615FB4D40FCD5B525F3015E401498630F99FA4D40FD598C90DC005E400571980D52FA4D40C3B0A789BDFF5D40F14DD0D842FA4D40F12 00DA1FE5D408B 38066CFA4D4059C7410092FD5D40C53C7702CCFA4D40AFC2A9C AFC5D40A28FEA205FFB4D40DF6AD2D2C4FB5D408 A3DBF1FFC4D40B73C6B6518FB5D400FF2147C06FD4D40112D1B1F9CFA5D40D0CACF7E0AFE4D404E26B0CD54FA5D40CCE2BECD21FF4D4087E3C53845FA5D403B51A0AF41004E408079E9046EFA5D40CDB8BA145F014E40CF0290ABCDFA5D406B5C9D036F024E409222728860FB5D40D84C5A0567034E 40DB9B09FC20FC5D409F06088D3D044E40B06412A207FD5D40FBB07A56EA044E40B78A179A0BFE5D40963273B866054E40A25449DF22FF5D40D0DCF9E6AD054E4024FC28AC42005E402D883723BD054E4025240AE65F015E40A173DCD693054E4033F60B8C6F025E4068D3F79933054E4023F60125670 35E4013FB0323A0044E4069F3D1273D045E40E1C5CE21DF034E40D515105AE9045E402E7AC106F8024E407F25172165055E40579ED4B8F3014E40D5FF79C2AB055E40E0132A3DDC004E40267F6991BA055E40F420D353BCFF4D40'::geometry, tbl_pos.pos)
  13. Rows Removed by Filter: 3
  14. Buffers: shared hit=97
  15. CTE b
  16. -> Limit (cost=0.42..427.17 rows=100 width=64) (actual time=0.624..0.758 rows=11 loops=1)
  17. Output: tbl_pos_1.id, tbl_pos_1.att1, tbl_pos_1.att2, tbl_pos_1.att3, tbl_pos_1.mod_time, tbl_pos_1.pos, ((tbl_pos_1.pos <-> '0101000020E61000000000000000005E400000000000004E40'::geometry))
  18. Buffers: shared hit=114
  19. -> Index Scan using idx_tbl_pos_1 on public.tbl_pos tbl_pos_1 (cost=0.42..3478.46 rows=815 width=64) (actual time=0.623..0.756 rows=11 loops=1)
  20. Output: tbl_pos_1.id, tbl_pos_1.att1, tbl_pos_1.att2, tbl_pos_1.att3, tbl_pos_1.mod_time, tbl_pos_1.pos, (tbl_pos_1.pos <-> '0101000020E61000000000000000005E400000000000004E40'::geometry)
  21. Index Cond: ((tbl_pos_1.att1 = 2) AND (tbl_pos_1.att2 <= 3) AND ('0103000020E61000000100000021000000267F6991BA055E40F420D353BCFF4D40AE7F7C0791055E4004D7AB0D9FFE4D405B8C22C830055E409DBC6E5F8FFD4D40D24FE28E9D045E40FB4032B6 97FC4D4043F65109DD035E40B3AD6691C1FB4D407907729EF6025E40CD022A2615FB4D40FCD5B525F3015E401498630F99FA4D40FD598C90DC005E400571980D52FA4D40C3B0A789BDFF5D40F14DD0D842FA4D40F12 00DA1FE5D408B 38066CFA4D4059C7410092FD5D40C53C7702CCFA4D40AFC2A 9C AFC5D40A28FEA205FFB4D40DF6AD2D2C4FB5D408 A3DBF1FFC4D40B73C6B6518FB5D400FF2147C06FD4D40112D1B1F9CFA5D40D0CACF7E0AFE4D404E26B0CD54FA5D40CCE2BECD21FF4D4087E3C53845FA5D403B51A0AF41004E408079E9046EFA5D40CDB8BA145F014E40CF0290ABCDFA5D406B 5C9D036F024E409222728860FB5D40D84C5A0567034E40DB9B09FC20FC5D409F06088D3D044E40B06412A207FD5D40FBB07A56EA044E40B78A179A0BFE5D40963273B866054E40A25449DF22FF5D40D0DCF9E6AD054E4024FC28AC42005E402D883723BD054E4025240AE65F015E40A173DCD693054E4 033F60B8C6F025E4068D3F79933054E4023F6012567035E4013FB0323A0044E4069F3D1273D045E40E1C5CE21DF034E40D515105AE9045E402E7AC106F8024E407F25172165055E40579ED4B8F3014E40D5FF79C2AB055E40E0132A3DDC004E40267F6991BA055E40F420D353BCFF4D40'::geometry ~ tbl_pos_1.pos))
  22. Order By: (tbl_pos_1.pos <-> '0101000020E61000000000000000005E400000000000004E40'::geometry)
  23. Filter: _st_contains('0103000020E61000000100000021000000267F6991BA055E40F420D353BCFF4D40AE7F7C0791055E4004D7AB0D9FFE4D405B8C22C830055E409DBC6E5F8FFD4D40D24FE28E9D045E40FB4032B697FC4D4043F65109DD035E40B3AD6691C1FB4D407907 729EF6025E40CD022A2615FB4D40FCD5B525F3015E401498630F99FA4D40FD598C90DC005E400571980D52FA4D40C3B0A789BDFF5D40F14DD0D842FA4D40F12 00DA1FE5D408B 38066CFA4D4059C7410092FD5D40C53C7702CCFA4D40AFC2A9C AFC5D40A28FEA205FFB4D40DF6AD2D2C4FB5D408 A3DBF1FFC4D40B73C6B6518FB5D400FF2147C06FD4D40112D1B1F9CFA5D40D0CACF7E0AFE4D404E26B0CD54FA5D40CCE2BECD21FF4D4087E3C53845FA5D403B51A0AF41004E408079E9046EFA5D40CDB8BA145F014E40CF0290ABCDFA5D406B5C9D036F024E409222728860FB5D40D84C5A0567034E 40DB9B09FC20FC5D409F06088D3D044E40B06412A207FD5D40FBB07A56EA044E40B78A179A0BFE5D40963273B866054E40A25449DF22FF5D40D0DCF9E6AD054E4024FC28AC42005E402D883723BD054E4025240AE65F015E40A173DCD693054E4033F60B8C6F025E4068D3F79933054E4023F60125670 35E4013FB0323A0044E4069F3D1273D045E40E1C5CE21DF034E40D515105AE9045E402E7AC106F8024E407F25172165055E40579ED4B8F3014E40D5FF79C2AB055E40E0132A3DDC004E40267F6991BA055E40F420D353BCFF4D40'::geometry, tbl_pos_1.pos)
  24. Rows Removed by Filter: 5
  25. Buffers: shared hit=114
  26. CTE c
  27. -> Limit (cost=0.42..427.58 rows=100 width=64) (actual time=0.624..0.720 rows=11 loops=1)
  28. Output: tbl_pos_2.id, tbl_pos_2.att1, tbl_pos_2.att2, tbl_pos_2.att3, tbl_pos_2.mod_time, tbl_pos_2.pos, ((tbl_pos_2.pos <-> '0101000020E61000000000000000005E400000000000004E40'::geometry))
  29. Buffers: shared hit=113
  30. -> Index Scan using idx_tbl_pos_1 on public.tbl_pos tbl_pos_2 (cost=0.42..3464.70 rows=811 width=64) (actual time=0.623..0.717 rows=11 loops=1)
  31. Output: tbl_pos_2.id, tbl_pos_2.att1, tbl_pos_2.att2, tbl_pos_2.att3, tbl_pos_2.mod_time, tbl_pos_2.pos, (tbl_pos_2.pos <-> '0101000020E61000000000000000005E400000000000004E40'::geometry)
  32. Index Cond: ((tbl_pos_2.att1 = 3) AND (tbl_pos_2.att2 <= 3) AND ('0103000020E61000000100000021000000267F6991BA055E40F420D353BCFF4D40AE7F7C0791055E4004D7AB0D9FFE4D405B8C22C830055E409DBC6E5F8FFD4D40D24FE28E9D045E40FB4032B6 97FC4D4043F65109DD035E40B3AD6691C1FB4D407907729EF6025E40CD022A2615FB4D40FCD5B525F3015E401498630F99FA4D40FD598C90DC005E400571980D52FA4D40C3B0A789BDFF5D40F14DD0D842FA4D40F12 00DA1FE5D408B 38066CFA4D4059C7410092FD5D40C53C7702CCFA4D40AFC2A 9C AFC5D40A28FEA205FFB4D40DF6AD2D2C4FB5D408 A3DBF1FFC4D40B73C6B6518FB5D400FF2147C06FD4D40112D1B1F9CFA5D40D0CACF7E0AFE4D404E26B0CD54FA5D40CCE2BECD21FF4D4087E3C53845FA5D403B51A0AF41004E408079E9046EFA5D40CDB8BA145F014E40CF0290ABCDFA5D406B 5C9D036F024E409222728860FB5D40D84C5A0567034E40DB9B09FC20FC5D409F06088D3D044E40B06412A207FD5D40FBB07A56EA044E40B78A179A0BFE5D40963273B866054E40A25449DF22FF5D40D0DCF9E6AD054E4024FC28AC42005E402D883723BD054E4025240AE65F015E40A173DCD693054E4 033F60B8C6F025E4068D3F79933054E4023F6012567035E4013FB0323A0044E4069F3D1273D045E40E1C5CE21DF034E40D515105AE9045E402E7AC106F8024E407F25172165055E40579ED4B8F3014E40D5FF79C2AB055E40E0132A3DDC004E40267F6991BA055E40F420D353BCFF4D40'::geometry ~ tbl_pos_2.pos))
  33. Order By: (tbl_pos_2.pos <-> '0101000020E61000000000000000005E400000000000004E40'::geometry)
  34. Filter: _st_contains('0103000020E61000000100000021000000267F6991BA055E40F420D353BCFF4D40AE7F7C0791055E4004D7AB0D9FFE4D405B8C22C830055E409DBC6E5F8FFD4D40D24FE28E9D045E40FB4032B697FC4D4043F65109DD035E40B3AD6691C1FB4D407907 729EF6025E40CD022A2615FB4D40FCD5B525F3015E401498630F99FA4D40FD598C90DC005E400571980D52FA4D40C3B0A789BDFF5D40F14DD0D842FA4D40F12 00DA1FE5D408B 38066CFA4D4059C7410092FD5D40C53C7702CCFA4D40AFC2A9C AFC5D40A28FEA205FFB4D40DF6AD2D2C4FB5D408 A3DBF1FFC4D40B73C6B6518FB5D400FF2147C06FD4D40112D1B1F9CFA5D40D0CACF7E0AFE4D404E26B0CD54FA5D40CCE2BECD21FF4D4087E3C53845FA5D403B51A0AF41004E408079E9046EFA5D40CDB8BA145F014E40CF0290ABCDFA5D406B5C9D036F024E409222728860FB5D40D84C5A0567034E 40DB9B09FC20FC5D409F06088D3D044E40B06412A207FD5D40FBB07A56EA044E40B78A179A0BFE5D40963273B866054E40A25449DF22FF5D40D0DCF9E6AD054E4024FC28AC42005E402D883723BD054E4025240AE65F015E40A173DCD693054E4033F60B8C6F025E4068D3F79933054E4023F60125670 35E4013FB0323A0044E4069F3D1273D045E40E1C5CE21DF034E40D515105AE9045E402E7AC106F8024E407F25172165055E40579ED4B8F3014E40D5FF79C2AB055E40E0132A3DDC004E40267F6991BA055E40F420D353BCFF4D40'::geometry, tbl_pos_2.pos)
  35. Rows Removed by Filter: 3
  36. Buffers: shared hit=113
  37. -> Append (cost=0.00..6.00 rows=300 width=56) (actual time=0.663..2.288 rows=32 loops=1)
  38. Buffers: shared hit=324
  39. -> CTE Scan on a (cost=0.00..2.00 rows=100 width=56) (actual time=0.663..0.786 rows=10 loops=1)
  40. Output: a.id, a.att1, a.att2, a.att3, a.mod_time, a.pos
  41. Buffers: shared hit=97
  42. -> CTE Scan on b (cost=0.00..2.00 rows=100 width=56) (actual time=0.625..0.766 rows=11 loops=1)
  43. Output: b.id, b.att1, b.att2, b.att3, b.mod_time, b.pos
  44. Buffers: shared hit=114
  45. -> CTE Scan on c (cost=0.00..2.00 rows=100 width=56) (actual time=0.625..0.728 rows=11 loops=1)
  46. Output: c.id, c.att1, c.att2, c.att3, c.mod_time, c.pos
  47. Buffers: shared hit=113
  48. Planning time: 1.959 ms
  49. Execution time: 2.362 ms
  50. (49 rows)
  1. id | att1 | att2 | att3 | mod_time | pos
  2. ----------+------+------+------+----------------------------+----------------------------------------------------
  3. 5097942 | 1 | 3 | 4 | 2018-07-10 17:51:32.653585 | 0101000020E610000000007B4FFB005E4000006056A2004E40
  4. 16158515 | 1 | 1 | 5 | 2018-07-17 15:43:00.621385 | 0101000020E61000000000C32AFE005E40000068FD69034E40
  5. 11518286 | 1 | 2 | 4 | 2018-07-17 15:42:34.189407 | 0101000020E61000000000A89FF6005E400000104E7EFC4D40
  6. 13313866 | 1 | 2 | 4 | 2018-07-10 17:40: .385905 | 0101000020E610000000001F3097005E4000008C9E2C044E40
  7. 7959337 | 1 | 2 | 1 | 2018-07-10 17:53:14.8 877 | 0101000020E610000000002D5A60FF5D400000AC3B8AFB4D40
  8. 12076193 | 1 | 2 | 2 | 2018-07-17 15:37:19.79298 | 0101000020E61000000000A0F570025E4000009658EEFF4D40
  9. 3666469 | 1 | 2 | 3 | 2018-07-17 15:41:21.49508 | 0101000020E6100000000075875DFD5D4000003CC529024E40
  10. 11836353 | 1 | 2 | 4 | 2018-07-17 15:41:49.73175 | 0101000020E610000000005A636A025E400000420506FC4D40
  11. 2562725 | 1 | 3 | 5 | 2018-07-17 15:42:43.744631 | 0101000020E6100000000022EEBF025E40000088 65044E40
  12. 2433530 | 1 | 1 | 5 | 2018-07-10 17:49:04.626915 | 0101000020E610000000004F0226FC5D400000BE99C7FE4D40
  13. 5129924 | 2 | 2 | 1 | 2018-07-17 15:42:13.010257 | 0101000020E610000000000468CD005E400000D4ACB9FC4D40
  14. 5759027 | 2 | 3 | 5 | 2018-07-17 15:42:37.054746 | 0101000020E61000000000C1ADEF005E4000002A5751FC4D40
  15. 7844609 | 2 | 2 | 5 | 2018-07-10 17:42:32.851153 | 0101000020E61000000000E9F593025E4000005864A0FE4D40
  16. 12243642 | 2 | 3 | 5 | 2018-07-17 15:41:33.378954 | 0101000020E61000000000D2AAF9005E4000009E7352054E40
  17. 347785 | 2 | 3 | 5 | 2018-07-17 15:42:28.101822 | 0101000020E610000000003EFC6BFE5D4000001801F2044E40
  18. 16587252 | 2 | 1 | 3 | 2018-07-17 15:43:00.253373 | 0101000020E610000000006EF1ADFD5D40000012A1D0034E40
  19. 15918799 | 2 | 2 | 2 | 2018-07-10 17:43:54.153194 | 0101000020E610000000009B1005FE5D400000944F79FB4D40
  20. 8183081 | 2 | 2 | 4 | 2018-07-17 15:41:29.903525 | 0101000020E61000000000608A70FC5D400000D0E3F8FB4D40
  21. 5698100 | 2 | 1 | 2 | 2018-07-17 15:42:56.053288 | 0101000020E610000000002FAEB6FB5D400000B21B47004E40
  22. 1806142 | 2 | 2 | 4 | 2018-07-10 17:53:06.627044 | 0101000020E61000000000D8479A045E400000BC7331FE4D40
  23. 386427 | 2 | 2 | 5 | 2018-07-10 17:52:26.568013 | 0101000020E610000000001E428BFA5D400000087438014E40
  24. 8892973 | 3 | 3 | 1 | 2018-07-10 17:44:08.386618 | 0101000020E61000000000031067FF5D4000001A383C014E40
  25. 2083046 | 3 | 1 | 2 | 2018-07-17 15:42:55.031903 | 0101000020E610000000002B99AE005E400000E007D5014E40
  26. 13441821 | 3 | 2 | 3 | 2018-07-10 17:41:49.504894 | 0101000020E610000000006675EF005E400000E04E74FB4D40
  27. 8662140 | 3 | 1 | 4 | 2018-07-17 15:41:42.761599 | 0101000020E61000000000D0F49AFF5D400000DC3C0BFB4D40
  28. 78 452 | 3 | 3 | 1 | 2018-07-17 15:42:15.954411 | 0101000020E61000000000CA94E7FF5D400000F27727054E40
  29. 16796301 | 3 | 2 | 4 | 2018-07-10 17:15:10.231126 | 0101000020E6100000000008F571025E400000D2A562024E40
  30. 1587379 | 3 | 1 | 1 | 2018-07-10 17:53:31.308692 | 0101000020E61000000000ABBFBBFC5D400000EC0B93FB4D40
  31. 8560096 | 3 | 3 | 2 | 2018-07-17 15:41:46.907464 | 0101000020E610000000001B707F035E4000002A5B09FC4D40
  32. 5540068 | 3 | 1 | 2 | 2018-07-10 17:42:29.689334 | 0101000020E610000000004C330C055E400000F02624FE4D40
  33. 17813180 | 3 | 1 | 5 | 2018-07-10 17:50:18.297117 | 0101000020E61000000000B88C95FA5D400000A6C915004E40
  34. 6424827 | 3 | 3 | 4 | 2018-07-10 17:50:54.958542 | 0101000020E61000000000564E8EFA5D4000002C28BA004E40
  35. (32 rows)

需求升华2

除了空间搜索,普通字段筛选条件,还引入数组字段过滤条件

如果业务上还有数组条件的包含查询过滤,可以创建intarray插件,把数组、普通字段、空间字段放到一个GIST索引里面

异或使用单独的gin+gist索引(看哪种效率高)

《PostgreSQL bitmapAnd, bitmapOr, bitmap index scan, bitmap heap scan》

例子(使用数组、普通字段、空间字段放到一个GIST索引里面):

1、建表,索引

  1. create table tbl_pos(id int primary key, att1 int, att2 int, att3 int, att4 int[], mod_time timestamp, pos geometry);
  2. create extension btree_gist;
  3. create extension intarray;
  4. create index idx_tbl_pos_1 on tbl_pos using gist(att1, att2, att4, pos);
  5. 假设ATT4的取值空间在029, 随机选取6个元素

2、更新用户的最终位置(同样2000万骑手),由于空间移动有一定的速比,所以更新后是基于原有位置的一个相对位移位置。

  1. vi test.sql
  2. \set att1 random(1,5)
  3. \set att2 random(1,5)
  4. \set att3 random(1,5)
  5. \set id random(1,20000000)
  6. \set x random(120,130)
  7. \set y random(70,80)
  8. insert into tbl_pos (id, att1, att2, att3, att4, mod_time, pos) values (:id, :att1, :att2, :att3, array(select (random()*29)::int from generate_series(1,6)), now(), st_setsrid(st_makepoint(:x,:y), 4326)) on conflict (id) do update set pos=st_setsrid(st_makepoint( st_x(tbl_pos.pos)+5-random()*10, st_y(tbl_pos.pos)+5-random()*10), 4326), mod_time=excluded.mod_time;

压测

  1. pgbench -M prepared -n -r -f ./test.sql -P 5 -c 28 -j 28 -T 120

数据样本

  1. postgres=# select * from tbl_pos limit 10;
  2. id | att1 | att2 | att3 | att4 | mod_time | pos
  3. ----------+------+------+------+---------------------+----------------------------+----------------------------------------------------
  4. 10688124 | 1 | 5 | 4 | {20,5,24,29,4,13} | 2018-07-25 17:57:37.846193 | 0101000020E61000000000000000C05E400000000000C05140
  5. 11168933 | 3 | 5 | 3 | {14,26,3,25,3,14} | 2018-07-25 17:57:37.920361 | 0101000020E61000000000000000C05F400000000000805240
  6. 18166259 | 3 | 2 | 3 | {16,28,3,22,1,0} | 2018-07-25 17:57:37.920514 | 0101000020E61000000000000000C05E400000000000405340
  7. 285635 | 5 | 4 | 4 | {20,16,1,20,8,27} | 2018-07-25 17:57:37.920668 | 0101000020E61000000000000000005E400000000000405240
  8. 16686877 | 3 | 4 | 1 | {25,15,7,8,20,2} | 2018-07-25 17:57:37.920848 | 0101000020E61000000000000000805F400000000000005240
  9. 13619811 | 5 | 5 | 5 | {12,5,4,28,3,6} | 2018-07-25 17:57:37.921826 | 0101000020E610000000000000000060400000000000C05140
  10. 19075025 | 1 | 2 | 1 | {22,4,7,27,24,23} | 2018-07-25 17:57:37.921984 | 0101000020E61000000000000000805F400000000000405340
  11. 14410958 | 2 | 1 | 4 | {15,2,25,14,27,11} | 2018-07-25 17:57:37.922141 | 0101000020E61000000000000000C05F400000000000405340
  12. 280895 | 3 | 4 | 5 | {27,16,20,12,28,24} | 2018-07-25 17:57:37.922475 | 0101000020E610000000000000000060400000000000C05240
  13. 2697548 | 3 | 5 | 5 | {28,27,22,20,3,29} | 2018-07-25 17:57:37.922745 | 0101000020E61000000000000000005E400000000000C05340
  14. (10 rows)

3、根据用户输入的att2, att4, 空间 条件,搜索满足条件的附近5公里内的对象,根据距离顺序返回100条。

  1. create or replace function get_res(i_att2 int, i_att4_1 int, i_att4_2 int, dis int)
  2. returns setof tbl_pos as $$
  3. declare
  4. x float8 := 120+random()*1;
  5. y float8 := 59+random()*1;
  6. begin
  7. return query
  8. with
  9. a as (
  10. select * from tbl_pos where att1=1 and
  11. att2<=i_att2 and
  12. st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(x,y), 4326)),dis)), pos) and
  13. att4 @> array[i_att4_1, i_att4_2]
  14. order by pos <-> st_setsrid(st_makepoint(x,y), 4326) limit 100
  15. ),
  16. b as (
  17. select * from tbl_pos where att1=2 and
  18. att2<=i_att2 and
  19. st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(x,y), 4326)),dis)), pos) and
  20. att4 @> array[i_att4_1, i_att4_2]
  21. order by pos <-> st_setsrid(st_makepoint(x,y), 4326) limit 100
  22. ),
  23. c as (
  24. select * from tbl_pos where att1=3 and
  25. att2<=i_att2 and
  26. st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(x,y), 4326)),dis)), pos) and
  27. att4 @> array[i_att4_1, i_att4_2]
  28. order by pos <-> st_setsrid(st_makepoint(x,y), 4326) limit 100
  29. )
  30. select * from a union all select * from b union all select * from c limit 100;
  31. end;
  32. $$ language plpgsql strict;

执行计划

  1. load 'auto_explain';
  2. set auto_explain.log_buffer =on;
  3. set auto_explain.log_min_duration =0;
  4. set auto_explain.log_nested_statements =on;
  5. set auto_explain.log_timing =on;
  6. set auto_explain.log_verbose =on;
  7. set client_min_messages =log;
  8. postgres=# select * from get_res(1,2,9,500000);
  9. LOG: duration: 0.020 ms plan:
  10. Query Text: SELECT proj4text FROM public.spatial_ref_sys WHERE srid = 4326 LIMIT 1
  11. Limit (cost=0.28..2.50 rows=1 width=116) (actual time=0.017..0.017 rows=1 loops=1)
  12. Output: proj4text
  13. -> Index Scan using spatial_ref_sys_pkey on public.spatial_ref_sys (cost=0.28..2.50 rows=1 width=116) (actual time=0.016..0.016 rows=1 loops=1)
  14. Output: proj4text
  15. Index Cond: (spatial_ref_sys.srid = 4326)
  16. LOG: duration: 0.010 ms plan:
  17. Query Text: SELECT proj4text FROM public.spatial_ref_sys WHERE srid = 4326 LIMIT 1
  18. Limit (cost=0.28..2.50 rows=1 width=116) (actual time=0.007..0.007 rows=1 loops=1)
  19. Output: proj4text
  20. -> Index Scan using spatial_ref_sys_pkey on public.spatial_ref_sys (cost=0.28..2.50 rows=1 width=116) (actual time=0.007..0.007 rows=1 loops=1)
  21. Output: proj4text
  22. Index Cond: (spatial_ref_sys.srid = 4326)
  23. LOG: duration: 0.985 ms plan:
  24. Query Text: SELECT public.geography(public.ST_Transform(public.ST_Buffer(public.ST_Transform(public.geometry($1), public._ST_BestSRID($1)), $2), 4326))
  25. Result (cost=0.00..0.03 rows=1 width=32) (actual time=0.982..0.982 rows=1 loops=1)
  26. Output: geography(st_transform(st_buffer(st_transform(geometry($1), _st_bestsrid($1, $1)), $2), 4326))
  27. LOG: duration: 0.009 ms plan:
  28. Query Text: SELECT proj4text FROM public.spatial_ref_sys WHERE srid = 4326 LIMIT 1
  29. Limit (cost=0.28..2.50 rows=1 width=116) (actual time=0.007..0.007 rows=1 loops=1)
  30. Output: proj4text
  31. -> Index Scan using spatial_ref_sys_pkey on public.spatial_ref_sys (cost=0.28..2.50 rows=1 width=116) (actual time=0.007..0.007 rows=1 loops=1)
  32. Output: proj4text
  33. Index Cond: (spatial_ref_sys.srid = 4326)
  34. LOG: duration: 0.008 ms plan:
  35. Query Text: SELECT proj4text FROM public.spatial_ref_sys WHERE srid = 4326 LIMIT 1
  36. Limit (cost=0.28..2.50 rows=1 width=116) (actual time=0.007..0.007 rows=1 loops=1)
  37. Output: proj4text
  38. -> Index Scan using spatial_ref_sys_pkey on public.spatial_ref_sys (cost=0.28..2.50 rows=1 width=116) (actual time=0.006..0.006 rows=1 loops=1)
  39. Output: proj4text
  40. Index Cond: (spatial_ref_sys.srid = 4326)
  41. LOG: duration: 0.524 ms plan:
  42. Query Text: SELECT public.geography(public.ST_Transform(public.ST_Buffer(public.ST_Transform(public.geometry($1), public._ST_BestSRID($1)), $2), 4326))
  43. Result (cost=0.00..0.03 rows=1 width=32) (actual time=0.522..0.522 rows=1 loops=1)
  44. Output: geography(st_transform(st_buffer(st_transform(geometry($1), _st_bestsrid($1, $1)), $2), 4326))
  45. LOG: duration: 0.010 ms plan:
  46. Query Text: SELECT proj4text FROM public.spatial_ref_sys WHERE srid = 4326 LIMIT 1
  47. Limit (cost=0.28..2.50 rows=1 width=116) (actual time=0.007..0.007 rows=1 loops=1)
  48. Output: proj4text
  49. -> Index Scan using spatial_ref_sys_pkey on public.spatial_ref_sys (cost=0.28..2.50 rows=1 width=116) (actual time=0.007..0.007 rows=1 loops=1)
  50. Output: proj4text
  51. Index Cond: (spatial_ref_sys.srid = 4326)
  52. LOG: duration: 0.009 ms plan:
  53. Query Text: SELECT proj4text FROM public.spatial_ref_sys WHERE srid = 4326 LIMIT 1
  54. Limit (cost=0.28..2.50 rows=1 width=116) (actual time=0.007..0.007 rows=1 loops=1)
  55. Output: proj4text
  56. -> Index Scan using spatial_ref_sys_pkey on public.spatial_ref_sys (cost=0.28..2.50 rows=1 width=116) (actual time=0.007..0.007 rows=1 loops=1)
  57. Output: proj4text
  58. Index Cond: (spatial_ref_sys.srid = 4326)
  59. LOG: duration: 0.550 ms plan:
  60. Query Text: SELECT public.geography(public.ST_Transform(public.ST_Buffer(public.ST_Transform(public.geometry($1), public._ST_BestSRID($1)), $2), 4326))
  61. Result (cost=0.00..0.03 rows=1 width=32) (actual time=0.548..0.548 rows=1 loops=1)
  62. Output: geography(st_transform(st_buffer(st_transform(geometry($1), _st_bestsrid($1, $1)), $2), 4326))
  63. LOG: duration: 8.519 ms plan:
  64. Query Text: with
  65. a as (
  66. select * from tbl_pos where att1=1 and
  67. att2<=i_att2 and
  68. st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(x,y), 4326)),dis)), pos) and
  69. att4 @> array[i_att4_1, i_att4_2]
  70. order by pos <-> st_setsrid(st_makepoint(x,y), 4326) limit 100
  71. ),
  72. b as (
  73. select * from tbl_pos where att1=2 and
  74. att2<=i_att2 and
  75. st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(x,y), 4326)),dis)), pos) and
  76. att4 @> array[i_att4_1, i_att4_2]
  77. order by pos <-> st_setsrid(st_makepoint(x,y), 4326) limit 100
  78. ),
  79. c as (
  80. select * from tbl_pos where att1=3 and
  81. att2<=i_att2 and
  82. st_contains(geometry(ST_Buffer(geography(st_setsrid(st_makepoint(x,y), 4326)),dis)), pos) and
  83. att4 @> array[i_att4_1, i_att4_2]
  84. order by pos <-> st_setsrid(st_makepoint(x,y), 4326) limit 100
  85. )
  86. select * from a union all select * from b union all select * from c limit 100
  87. Limit (cost=103.79..104.27 rows=24 width=88) (actual time=0.579..8.479 rows=100 loops=1)
  88. Output: a.id, a.att1, a.att2, a.att3, a.att4, a.mod_time, a.pos
  89. CTE a
  90. -> Limit (cost=0.42..34.60 rows=8 width=109) (actual time=0.575..7.345 rows=85 loops=1)
  91. Output: tbl_pos.id, tbl_pos.att1, tbl_pos.att2, tbl_pos.att3, tbl_pos.att4, tbl_pos.mod_time, tbl_pos.pos, ((tbl_pos.pos <-> '0101000020E61000000080A216312D5E400000B15EF8FC4D40'::geometry))
  92. -> Index Scan using idx_tbl_pos_1 on public.tbl_pos (cost=0.42..34.60 rows=8 width=109) (actual time=0.575..7.329 rows=85 loops=1)
  93. Output: tbl_pos.id, tbl_pos.att1, tbl_pos.att2, tbl_pos.att3, tbl_pos.att4, tbl_pos.mod_time, tbl_pos.pos, (tbl_pos.pos <-> '0101000020E61000000080A216312D5E400000B15EF8FC4D40'::geometry)
  94. Index Cond: ((tbl_pos.att1 = 1) AND (tbl_pos.att2 <= 1) AND (tbl_pos.att4 @> '{2,9}'::integer[]) AND ('0103000020E6100000010000002100000008956E948B34604005B6F7880FEA4D406AD812B7D429604033471D5EB97B4D4088C40953AC156040B4751DAF65134D40775090B7A5F25F40A738CEDB85B44C401511453031AC5F40B867373A13624C40688DF186A75A5F40C706A1C6891E4C408B8A2558B7005F402385A953E3EB4B40553DF2680DA15E40260EB9C892CB4B4087862191533E5E40C342B6ED7FBE4B40EADA06E730DB5D40ED5B41A204C54B4005DF006E4A7A5D40BF19753CECDE4B403B6DA206441E5D40FB6AB34E750B4C4024BB556BC1C95C402B536B7655494C40693A2B9F667F5C40515B3A56BE964C40B976E367D6415C4015E8CAB362F14C40155E7D4BAC135C40100997027B564D4054B2BA3A6EF75B409B3A047FC9C24D4098594DE570EF5B404ED001599F324E40F1925A4DA9FD5B40DC4ACF26E6A14E4075FF27B267235C409837C69D320C4F401283CEECFC605C40BFF7D3BBE66C4F40D7CD8F2C55B55C408949A20C69BF4F40BAC94AC0A01D5D40ACCED43073FF4F40C0B7C4A62F955D408416E63EB9145040EB780B55A4155E403901B1E0751D50400592A9238C975E40DD1387B065195040F7229DCF43135F40DB128194C50850400C5C1A29EA815F40CA3AD2F121D94F401BA10DD11BDE5F405474586DC28C4F4094A4631E2712604052096DA47C304F401625C60A652960406E5A3804DAC84E40FB9EF3C3B13460409548397D865A4E4008956E948B34604005B6F7880FEA4D40'::geometry ~ tbl_pos.pos))
  95. Order By: (tbl_pos.pos <-> '0101000020E61000000080A216312D5E400000B15EF8FC4D40'::geometry)
  96. Filter: _st_contains('0103000020E6100000010000002100000008956E948B34604005B6F7880FEA4D406AD812B7D429604033471D5EB97B4D4088C40953AC156040B4751DAF65134D40775090B7A5F25F40A738CEDB85B44C401511453031AC5F40B867373A13624C40688DF186A75A5F40C706A1C6891E4C408B8A2558B7005F402385A953E3EB4B40553DF2680DA15E40260EB9C892CB4B4087862191533E5E40C342B6ED7FBE4B40EADA06E730DB5D40ED5B41A204C54B4005DF006E4A7A5D40BF19753CECDE4B403B6DA206441E5D40FB6AB34E750B4C4024BB556BC1C95C402B536B7655494C40693A2B9F667F5C40515B3A56BE964C40B976E367D6415C4015E8CAB362F14C40155E7D4BAC135C40100997027B564D4054B2BA3A6EF75B409B3A047FC9C24D4098594DE570EF5B404ED001599F324E40F1925A4DA9FD5B40DC4ACF26E6A14E4075FF27B267235C409837C69D320C4F401283CEECFC605C40BFF7D3BBE66C4F40D7CD8F2C55B55C408949A20C69BF4F40BAC94AC0A01D5D40ACCED43073FF4F40C0B7C4A62F955D408416E63EB9145040EB780B55A4155E403901B1E0751D50400592A9238C975E40DD1387B065195040F7229DCF43135F40DB128194C50850400C5C1A29EA815F40CA3AD2F121D94F401BA10DD11BDE5F405474586DC28C4F4094A4631E2712604052096DA47C304F401625C60A652960406E5A3804DAC84E40FB9EF3C3B13460409548397D865A4E4008956E948B34604005B6F7880FEA4D40'::geometry, tbl_pos.pos)
  97. Rows Removed by Filter: 37
  98. CTE b
  99. -> Limit (cost=0.42..34.60 rows=8 width=109) (actual time=0.469..1.018 rows=15 loops=1)
  100. Output: tbl_pos_1.id, tbl_pos_1.att1, tbl_pos_1.att2, tbl_pos_1.att3, tbl_pos_1.att4, tbl_pos_1.mod_time, tbl_pos_1.pos, ((tbl_pos_1.pos <-> '0101000020E61000000080A216312D5E400000B15EF8FC4D40'::geometry))
  101. -> Index Scan using idx_tbl_pos_1 on public.tbl_pos tbl_pos_1 (cost=0.42..34.60 rows=8 width=109) (actual time=0.468..1.015 rows=15 loops=1)
  102. Output: tbl_pos_1.id, tbl_pos_1.att1, tbl_pos_1.att2, tbl_pos_1.att3, tbl_pos_1.att4, tbl_pos_1.mod_time, tbl_pos_1.pos, (tbl_pos_1.pos <-> '0101000020E61000000080A216312D5E400000B15EF8FC4D40'::geometry)
  103. Index Cond: ((tbl_pos_1.att1 = 2) AND (tbl_pos_1.att2 <= 1) AND (tbl_pos_1.att4 @> '{2,9}'::integer[]) AND ('0103000020E6100000010000002100000008956E948B34604005B6F7880FEA4D406AD812B7D429604033471D5EB97B4D4088C40953AC156040B4751DAF65134D40775090B7A5F25F40A738CEDB85B44C401511453031AC5F40B867373A13624C40688DF186A75A5F40C706A1C6891E4C408B8A2558B7005F402385A953E3EB4B40553DF2680DA15E40260EB9C892CB4B4087862191533E5E40C342B6ED7FBE4B40EADA06E730DB5D40ED5B41A204C54B4005DF006E4A7A5D40BF19753CECDE4B403B6DA206441E5D40FB6AB34E750B4C4024BB556BC1C95C402B536B7655494C40693A2B9F667F5C40515B3A56BE964C40B976E367D6415C4015E8CAB362F14C40155E7D4BAC135C40100997027B564D4054B2BA3A6EF75B409B3A047FC9C24D4098594DE570EF5B404ED001599F324E40F1925A4DA9FD5B40DC4ACF26E6A14E4075FF27B267235C409837C69D320C4F401283CEECFC605C40BFF7D3BBE66C4F40D7CD8F2C55B55C408949A20C69BF4F40BAC94AC0A01D5D40ACCED43073FF4F40C0B7C4A62F955D408416E63EB9145040EB780B55A4155E403901B1E0751D50400592A9238C975E40DD1387B065195040F7229DCF43135F40DB128194C50850400C5C1A29EA815F40CA3AD2F121D94F401BA10DD11BDE5F405474586DC28C4F4094A4631E2712604052096DA47C304F401625C60A652960406E5A3804DAC84E40FB9EF3C3B13460409548397D865A4E4008956E948B34604005B6F7880FEA4D40'::geometry ~ tbl_pos_1.pos))
  104. Order By: (tbl_pos_1.pos <-> '0101000020E61000000080A216312D5E400000B15EF8FC4D40'::geometry)
  105. Filter: _st_contains('0103000020E6100000010000002100000008956E948B34604005B6F7880FEA4D406AD812B7D429604033471D5EB97B4D4088C40953AC156040B4751DAF65134D40775090B7A5F25F40A738CEDB85B44C401511453031AC5F40B867373A13624C40688DF186A75A5F40C706A1C6891E4C408B8A2558B7005F402385A953E3EB4B40553DF2680DA15E40260EB9C892CB4B4087862191533E5E40C342B6ED7FBE4B40EADA06E730DB5D40ED5B41A204C54B4005DF006E4A7A5D40BF19753CECDE4B403B6DA206441E5D40FB6AB34E750B4C4024BB556BC1C95C402B536B7655494C40693A2B9F667F5C40515B3A56BE964C40B976E367D6415C4015E8CAB362F14C40155E7D4BAC135C40100997027B564D4054B2BA3A6EF75B409B3A047FC9C24D4098594DE570EF5B404ED001599F324E40F1925A4DA9FD5B40DC4ACF26E6A14E4075FF27B267235C409837C69D320C4F401283CEECFC605C40BFF7D3BBE66C4F40D7CD8F2C55B55C408949A20C69BF4F40BAC94AC0A01D5D40ACCED43073FF4F40C0B7C4A62F955D408416E63EB9145040EB780B55A4155E403901B1E0751D50400592A9238C975E40DD1387B065195040F7229DCF43135F40DB128194C50850400C5C1A29EA815F40CA3AD2F121D94F401BA10DD11BDE5F405474586DC28C4F4094A4631E2712604052096DA47C304F401625C60A652960406E5A3804DAC84E40FB9EF3C3B13460409548397D865A4E4008956E948B34604005B6F7880FEA4D40'::geometry, tbl_pos_1.pos)
  106. CTE c
  107. -> Limit (cost=0.42..34.60 rows=8 width=109) (never executed)
  108. Output: tbl_pos_2.id, tbl_pos_2.att1, tbl_pos_2.att2, tbl_pos_2.att3, tbl_pos_2.att4, tbl_pos_2.mod_time, tbl_pos_2.pos, ((tbl_pos_2.pos <-> '0101000020E61000000080A216312D5E400000B15EF8FC4D40'::geometry))
  109. -> Index Scan using idx_tbl_pos_1 on public.tbl_pos tbl_pos_2 (cost=0.42..34.60 rows=8 width=109) (never executed)
  110. Output: tbl_pos_2.id, tbl_pos_2.att1, tbl_pos_2.att2, tbl_pos_2.att3, tbl_pos_2.att4, tbl_pos_2.mod_time, tbl_pos_2.pos, (tbl_pos_2.pos <-> '0101000020E61000000080A216312D5E400000B15EF8FC4D40'::geometry)
  111. Index Cond: ((tbl_pos_2.att1 = 3) AND (tbl_pos_2.att2 <= 1) AND (tbl_pos_2.att4 @> '{2,9}'::integer[]) AND ('0103000020E6100000010000002100000008956E948B34604005B6F7880FEA4D406AD812B7D429604033471D5EB97B4D4088C40953AC156040B4751DAF65134D40775090B7A5F25F40A738CEDB85B44C401511453031AC5F40B867373A13624C40688DF186A75A5F40C706A1C6891E4C408B8A2558B7005F402385A953E3EB4B40553DF2680DA15E40260EB9C892CB4B4087862191533E5E40C342B6ED7FBE4B40EADA06E730DB5D40ED5B41A204C54B4005DF006E4A7A5D40BF19753CECDE4B403B6DA206441E5D40FB6AB34E750B4C4024BB556BC1C95C402B536B7655494C40693A2B9F667F5C40515B3A56BE964C40B976E367D6415C4015E8CAB362F14C40155E7D4BAC135C40100997027B564D4054B2BA3A6EF75B409B3A047FC9C24D4098594DE570EF5B404ED001599F324E40F1925A4DA9FD5B40DC4ACF26E6A14E4075FF27B267235C409837C69D320C4F401283CEECFC605C40BFF7D3BBE66C4F40D7CD8F2C55B55C408949A20C69BF4F40BAC94AC0A01D5D40ACCED43073FF4F40C0B7C4A62F955D408416E63EB9145040EB780B55A4155E403901B1E0751D50400592A9238C975E40DD1387B065195040F7229DCF43135F40DB128194C50850400C5C1A29EA815F40CA3AD2F121D94F401BA10DD11BDE5F405474586DC28C4F4094A4631E2712604052096DA47C304F401625C60A652960406E5A3804DAC84E40FB9EF3C3B13460409548397D865A4E4008956E948B34604005B6F7880FEA4D40'::geometry ~ tbl_pos_2.pos))
  112. Order By: (tbl_pos_2.pos <-> '0101000020E61000000080A216312D5E400000B15EF8FC4D40'::geometry)
  113. Filter: _st_contains('0103000020E6100000010000002100000008956E948B34604005B6F7880FEA4D406AD812B7D429604033471D5EB97B4D4088C40953AC156040B4751DAF65134D40775090B7A5F25F40A738CEDB85B44C401511453031AC5F40B867373A13624C40688DF186A75A5F40C706A1C6891E4C408B8A2558B7005F402385A953E3EB4B40553DF2680DA15E40260EB9C892CB4B4087862191533E5E40C342B6ED7FBE4B40EADA06E730DB5D40ED5B41A204C54B4005DF006E4A7A5D40BF19753CECDE4B403B6DA206441E5D40FB6AB34E750B4C4024BB556BC1C95C402B536B7655494C40693A2B9F667F5C40515B3A56BE964C40B976E367D6415C4015E8CAB362F14C40155E7D4BAC135C40100997027B564D4054B2BA3A6EF75B409B3A047FC9C24D4098594DE570EF5B404ED001599F324E40F1925A4DA9FD5B40DC4ACF26E6A14E4075FF27B267235C409837C69D320C4F401283CEECFC605C40BFF7D3BBE66C4F40D7CD8F2C55B55C408949A20C69BF4F40BAC94AC0A01D5D40ACCED43073FF4F40C0B7C4A62F955D408416E63EB9145040EB780B55A4155E403901B1E0751D50400592A9238C975E40DD1387B065195040F7229DCF43135F40DB128194C50850400C5C1A29EA815F40CA3AD2F121D94F401BA10DD11BDE5F405474586DC28C4F4094A4631E2712604052096DA47C304F401625C60A652960406E5A3804DAC84E40FB9EF3C3B13460409548397D865A4E4008956E948B34604005B6F7880FEA4D40'::geometry, tbl_pos_2.pos)
  114. -> Append (cost=0.00..0.48 rows=24 width=88) (actual time=0.579..8.461 rows=100 loops=1)
  115. -> CTE Scan on a (cost=0.00..0.16 rows=8 width=88) (actual time=0.578..7.413 rows=85 loops=1)
  116. Output: a.id, a.att1, a.att2, a.att3, a.att4, a.mod_time, a.pos
  117. -> CTE Scan on b (cost=0.00..0.16 rows=8 width=88) (actual time=0.471..1.028 rows=15 loops=1)
  118. Output: b.id, b.att1, b.att2, b.att3, b.att4, b.mod_time, b.pos
  119. -> CTE Scan on c (cost=0.00..0.16 rows=8 width=88) (never executed)
  120. Output: c.id, c.att1, c.att2, c.att3, c.att4, c.mod_time, c.pos
  121. LOG: duration: 12.365 ms plan:
  122. Query Text: select * from get_res(1,2,9,500000);
  123. Function Scan on public.get_res (cost=0.25..10.25 rows=1000 width=88) (actual time=12.175..12.185 rows=100 loops=1)
  124. Output: id, att1, att2, att3, att4, mod_time, pos
  125. Function Call: get_res(1, 2, 9, 500000)

压测脚本

  1. vi test1.sql
  2. \set att2 random(1,5)
  3. \set att4_1 random(0,29)
  4. \set att4_2 random(0,29)
  5. select * from get_res(:att2, :att4_1, :att4_2, 500000);

压测

  1. pgbench -M prepared -n -r -f ./test1.sql -P 5 -c 56 -j 56 -T 300

压测结果

1、更新对象位置

  1. transaction type: ./test.sql
  2. scaling factor: 1
  3. query mode: prepared
  4. number of clients: 28
  5. number of threads: 28
  6. duration: 120 s
  7. number of transactions actually processed: 11605219
  8. latency average = 0.290 ms
  9. latency stddev = 0.957 ms
  10. tps = 96708.804105 (including connections establishing)
  11. tps = 96714.596970 (excluding connections establishing)
  12. script statistics:
  13. - statement latencies in milliseconds:
  14. 0.002 \set att1 random(1,5)
  15. 0.000 \set att2 random(1,5)
  16. 0.000 \set att3 random(1,5)
  17. 0.000 \set id random(1,20000000)
  18. 0.000 \set x random(120,130)
  19. 0.000 \set y random(70,80)
  20. 0.286 insert into tbl_pos (id, att1, att2, att3, att4, mod_time, pos) values (:id, :att1, :att2, :att3, array(select (random()*29)::int from generate_series(1,6)), now(), st_setsrid(st_makepoint(:x,:y), 4326)) on conflict (id) do update set pos=st_setsrid(st_makepoint( st_x(tbl_pos.pos)+5-random()*10, st_y(tbl_pos.pos)+5-random()*10), 4326), mod_time=excluded.mod_time;

TPS: 96708

平均响应时间: 0.286 毫秒

2、根据用户输入的att2, att4,空间过滤组合条件,搜索满足条件的附近500000米内的对象,根据距离顺序返回100条。

  1. transaction type: ./test1.sql
  2. scaling factor: 1
  3. query mode: prepared
  4. number of clients: 56
  5. number of threads: 56
  6. duration: 120 s
  7. number of transactions actually processed: 600815
  8. latency average = 11.184 ms
  9. latency stddev = 2.410 ms
  10. tps = 5005.566075 (including connections establishing)
  11. tps = 5006.063092 (excluding connections establishing)
  12. script statistics:
  13. - statement latencies in milliseconds:
  14. 0.003 \set att2 random(1,5)
  15. 0.001 \set att4_1 random(0,29)
  16. 0.000 \set att4_2 random(0,29)
  17. 11.182 select * from get_res(:att2, :att4_1, :att4_2, 500000);

TPS: 5005

平均响应时间: 11 毫秒

小结2

通过intarray , btree_gist插件,使得gist索引接口同时支持了普通类型、数组类型、空间类型的复合索引。查询效率上大幅提升。

性能对比

经过用户实测,相比ES集群,PG的“普通类型、数组类型、空间类型”的复合搜索性能提升了50倍。

参考

《PostgreSQL、Greenplum 应用案例宝典《如来神掌》 - 目录》

《数据库选型之 - 大象十八摸 - 致 架构师、开发者》

《PostgreSQL 使用 pgbench 测试 sysbench 相关case》

《数据库界的华山论剑 tpc.org》

https://www.postgresql.org/docs/10/static/pgbench.html

原文:http://mysql.taobao.org/monthly/2018/08/10/