words-match

words-match组件是基于字典树(DFA)并利用UnixSock通讯和自定义进程实现，开发本组件的目的是帮小伙伴们快速部署内容检测服务。

使用场景

  • 跟文字内容相关的产品都有应用场景。

  • 博客类的文章，评论的检测

  • 聊天内容的检测

  • 对垃圾内容的屏蔽

组件要求

None

安装方法

composer require easyswoole/words-match

仓库地址

easyswoole/words-match

基本使用

准备词库

服务启动的时候会一行一行将数据读出来，每一行的第一列为敏感词，其它列为附属信息

  1. php,是世界上,最好的语言
  2. java
  3. golang
  4. 程序员
  5. 代码
  6. 逻辑

服务注册

  1. <?php
  3. namespace EasySwoole\EasySwoole;
  5. use EasySwoole\Component\Di;
  6. use EasySwoole\EasySwoole\AbstractInterface\Event;
  7. use EasySwoole\EasySwoole\Swoole\EventRegister;
  8. use EasySwoole\Http\Request;
  9. use EasySwoole\Http\Response;
  10. use EasySwoole\WordsMatch\WMServer;
  12. class EasySwooleEvent implements Event
  13. {
  14.     public static function initialize()
  15.     {
  16.         date_default_timezone_set('Asia/Shanghai');
  18.         Di::getInstance()->set(SysConst::HTTP_GLOBAL_ON_REQUEST, function (Request $request, Response $response): bool {
  19.             // TODO: Implement onRequest() method.
  20.             return true;
  21.         });
  23.         Di::getInstance()->set(SysConst::HTTP_GLOBAL_AFTER_REQUEST, function (Request $request, Response $response): void {
  24.             // TODO: Implement onRequest() method.
  25.         });
  26.     }
  28.     public static function mainServerCreate(EventRegister $register)
  29.     {
  30.         // 配置 words-match
  31.         $wdConfig = new \EasySwoole\WordsMatch\Config();
  32.         $wdConfig->setDict(__DIR__ . '/dictionary.txt'); // 配置 词库地址
  33.         $wdConfig->setMaxMEM(1024); // 配置 每个进程最大占用内存(M)，默认为 512 M
  34.         $wdConfig->setTimeout(3.0); // 配置 内容检测超时时间。默认为 3.0 s
  35.         $wdConfig->setWorkerNum(3); // 配置 进程数
  36.         // $wdConfig->setSockDIR(sys_get_temp_dir()); // (不建议修改)配置 socket 存放地址，默认为 sys_get_temp_dir()，即 '/tmp'
  38.         // 注册服务
  39.         WMServer::getInstance($wdConfig)->attachServer(ServerManager::getInstance()->getSwooleServer());
  40.     }
  41. }

客户端使用

  1. <?php
  3. namespace App\HttpController;
  5. use EasySwoole\Http\AbstractInterface\Controller;
  6. use EasySwoole\WordsMatch\WMServer;
  8. class Index extends Controller
  9. {
  10.     function detect()
  11.     {
  12.         // 需要检测的内容敏感词
  13.         $content = 'php是世界上最好的语言';
  14.         // 检测结果（返回 -1 表示检测超时，匹配检测到时返回检测到的敏感词内容）
  15.         $result = WMServer::getInstance()->detect($content, 3);
  16.         var_dump($result);
  17.         /**
  18.          * 输出结果：
  19.          * array(1) {
  20.             [0]=>
  21.             object(EasySwoole\WordsMatch\Dictionary\DetectResult)#96 (5) {
  22.             ["word"]=>
  23.             string(30) "php是世界上最好的语言"
  24.             ["location"]=>
  25.             array(1) {
  26.               [0]=>
  27.               array(3) {
  28.                 ["word"]=>
  29.                 string(30) "php是世界上最好的语言"
  30.                 ["length"]=>
  31.                 int(12)
  32.                 ["location"]=>
  33.                 array(1) {
  34.                   [0]=>
  35.                   int(0)
  36.                 }
  37.               }
  38.             }
  39.             ["count"]=>
  40.             int(1)
  41.             ["remark"]=>
  42.             string(0) ""
  43.             ["type"]=>
  44.             int(1)
  45.             }
  46.          * }
  47.          */
  48.     }
  49. }

压测结果

对此组件分别进行1.5万、13万等级的词库测试，服务默认开启3个进程。

仅做参考，具体还以线上验证

电脑配置

  1. MacBook Air (13-inch, 2017)
  2. 处理器 1.8 GHz Intel Core i5
  3. 内存 8 GB 1600 MHz DDR3

1.5万词

并发10总请求数100
  1. 10 100
  2. Concurrency Level:      10
  3. Time taken for tests:   0.067 seconds
  4. Complete requests:      100
  5. Failed requests:        0
  6. Non-2xx responses:      100
  7. Total transferred:      17300 bytes
  8. HTML transferred:       2600 bytes
  9. Requests per second:    1492.49 [#/sec] (mean)
  10. Time per request:       6.700 [ms] (mean)
  11. Time per request:       0.670 [ms] (mean, across all concurrent requests)
  12. Transfer rate:          252.15 [Kbytes/sec] received
并发100总请求数1000
  1. Concurrency Level:      100
  2. Time taken for tests:   0.239 seconds
  3. Complete requests:      1000
  4. Failed requests:        0
  5. Non-2xx responses:      1000
  6. Total transferred:      173000 bytes
  7. HTML transferred:       26000 bytes
  8. Requests per second:    4189.17 [#/sec] (mean)
  9. Time per request:       23.871 [ms] (mean)
  10. Time per request:       0.239 [ms] (mean, across all concurrent requests)
  11. Transfer rate:          707.74 [Kbytes/sec] received

13万词

并发10总请求数100
  1. Concurrency Level:      10
  2. Time taken for tests:   0.057 seconds
  3. Complete requests:      100
  4. Failed requests:        0
  5. Non-2xx responses:      100
  6. Total transferred:      17300 bytes
  7. HTML transferred:       2600 bytes
  8. Requests per second:    1751.71 [#/sec] (mean)
  9. Time per request:       5.709 [ms] (mean)
  10. Time per request:       0.571 [ms] (mean, across all concurrent requests)
  11. Transfer rate:          295.94 [Kbytes/sec] received
并发100总请求数1000
  1. Concurrency Level:      100
  2. Time taken for tests:   0.225 seconds
  3. Complete requests:      1000
  4. Failed requests:        0
  5. Non-2xx responses:      1000
  6. Total transferred:      173000 bytes
  7. HTML transferred:       26000 bytes
  8. Requests per second:    4444.84 [#/sec] (mean)
  9. Time per request:       22.498 [ms] (mean)
  10. Time per request:       0.225 [ms] (mean, across all concurrent requests)
  11. Transfer rate:          750.93 [Kbytes/sec] received