elasticsearch学习

1.介绍

Elasticsearch是一个基于Lucene的搜索服务器，提供了一个分布式多用户能力的全文搜索引擎，基于Restful web接口。仅支持json格式，自带有分布式协调管理功能。

2.lucene与elasticsearch的关系

lucene只是一个提供全文搜索功能类库的核心工具包，而真正使用它还需要一个完善的服务框架搭建起来的应用。

3.与其他数据存储进行比较

	redis	mysql	es	hbase	hadoop/hive
容量	低	中	较大	海量	海量
查询时效性	极高	中等	较高	中等	低
查询灵活性	较差k-v模式	非常好，支持sql	较好，关联查询弱，但是可以全文检索，DSL语言可以处理过滤，匹配，排序，聚合等各种操作	较差，主要靠rowkey,scan的话性能不行，或者建立二级索引	非常好，支持sql
写入速度	较快	中等	较快	较快	慢
一致性/事务	弱	强	弱	弱	弱

4.特点

• 天然集群，天然分片
  es把数据分成多个shard,多个shard可以组成一份完整的数据，这些shard可以分布在集群的各个节点上，随着数据的不断增加，集群可以增加多个分片，把多个分片放到多个机子上，已达到负载均衡，横向扩展。
  在实际运算过程中，每个查询任务提交到某一个节点，该节点必须负责将数据进行整理汇聚，在返回给客户端，也就是一个简单的节点上进行map计算，在一个固定的节点上进行reduces得到最终结果向客户端返回。

• 天然索引
  所有数据默认进行倒排索引（全文搜索引擎主流方式）
  每个记录保存数据时，都不会直接存入数据库，系统先 会对数据分词，然后按照倒排索引结构保存
  搜索时同样会先把搜索关键词进行分词，还会根据匹配程度进行打分，次数越多位置越靠前
  lucenu为倒排索引(Term Dictionary)部分又增加了一层Term Index 结构，用于快速定位，而Term Index是缓存在内存中的，所以速度快但同时更消耗资源（内存，磁盘）。

5.修改es配置文件

• /config/elasticsearch.yml
  ①集群名称必须相同
  
  ②同一集群，单个节点名称必须不同；是否有资格选举为master;是否存储数据
  
  ③网络访问(bind_host内网)和内部通信(publish_host外网)接口
  
  ④自发现配置：新节点向集群报道的主机名（集群通信端口9300，外部访问端口9200）
  
  ⑤把bootstrap自检程序关掉
  
  ⑥集群中最少的master数，如果集群中最少master数少于指定的数，将无法启动，官方推荐master数设置为:集群数/2+1
  

6.修改linux系统配置（改完后重启）

问题1：max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536] elasticsearch
原因：系统允许 Elasticsearch 打开的最大文件数需要修改成65536
解决：vi /etc/security/limits.conf
添加内容：
*soft nofile 65536
*hard nofile 131072
*soft nproc 2048
*hard nproc 65536

问题2：max number of threads [1024] for user [judy2] likely too low, increase to at least [2048] （CentOS7.x 不用改）
原因：允许最大进程数修该成4096
解决：vi /etc/security/limits.d/90-nproc.conf
修改如下内容：
*soft nproc 4096

问题3：max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144] （CentOS7.x 不用改）
原因：一个进程可以拥有的虚拟内存区域的数量。
解决：可零时提高vm.max_map_count的大小
命令：sysctl -w vm.max_map_count=262144
上述方法修改之后，如果重启虚拟机将失效，所以：
在 vim /etc/sysctl.conf文件最后添加一行
vm.max_map_count=262144
即可永久修改

7.集群启动脚本

8.集群验证

curl http://kqkj101:9200/_cat/nodes?v(查询各个节点状态)

9.head插件

• 下载head插件
  Get https://github.com/mobz/elasticsearch-head/archive/elasticsearch-head-master.zip
  也可以用git下载，前提yum install git
  unzip elasticsearch-head-master.zip
• 安装node.js
  wget https://npm.taobao.org/mirrors/node/latest-v4.x/node-v4.4.7-linux-x64.tar.gz
  tar -zxvf node-v9.9.0-linux-x64.tar.gz
• 添加node.js到环境变量
  
  source /etc/profile
• 测试 node -v | npm -v
• 安装grunt（grunt是一个很方便的构建工具，可以进行打包压缩、测试、执行等等的工作）
  进入到elasticsearch-head-master
  npm install -g grunt-cli
  npm install
  (npm install -g cnpm --registry=https://registry.npm.taobao.org)
• 修改elasticearch.yml（head插件才能访问es）
  添加：http.cors.enabled: true
  http.cors.allow-origin: “*”
• 修改Gruntfile.js（注意’，’）
  打开elasticsearch-head-master/Gruntfile.js，找到下面connect属性，新增hostname:’’：
  connect: {
  server: {
  options: {
  hostname: '’,
  port: 9100,
  base: ‘.’,
  keepalive: true
  }
  }
  }
• 启动elasticsearch-head
  进入elasticsearch-head目录，执行命令：grunt server
  后台启动 nohup grunt server &exit
• 关闭head插件
  ps -aux|grep head
  kill 进程号

10.logstash

• vim ./config/log4j_to_es.conf

//创建nginx_logstash.conf文件
input {//输入/*file {//从文件中读取数据path=>[""]//文件位置exclue=>"*.gz"//不读取哪些文件sincedb_path=>""//记录sincedb文件路径type=>""start_position=>"beginning"//或end,是否重头读取文件}*/stdin{//输入插件，可以管道插入，也可以终端交互插入codec=>"plain"tags=>["test"]//类型为array,自定义事件的tagtype=>"std"//类型为string,自定义该事件类型，用于后续判断add_field=>{"key":"value"}//类型为hash,为该事件添加字段}
}
filter {grok {//①基于正则表达式提供丰富可重用模式pattern;②基于此可以将非结构化数据做结构化处理match => {"message" => '%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] "%{WORD:request_action} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"'}}date {//将字符串类型的时间字段转换为时间戳类型，方便后续处理match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ]locale => en}geoip {//添加地理位置信息source => "remote_ip"target => "geoip"}mutate{}进行增加修改删除替换等字段相关处理
output {//输出stdout {codec=>rubydebug}
}
logstash启动解析nginx文件
head -n 2 /home/elk1/nginx_logs|./logstash -f ../config/nginx_logstash.conf

• ./bin/logstash -f config/log4j_to_es.conf

10.kibana(端口：5601)

• /config/kibana.yml
  

11.IK分词

• ①在/plugins/目录下建立ik文件夹
• ②把elasticsearch-analysis-ik-6.6.0.zip拷贝到该目录下，执行unzip进行解压缩
• ③修改分词词库/ik/config/IKAnalyzer.cfg.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties><comment>IK Analyzer 扩展配置</comment><!--用户可以在这里配置自己的扩展字典 --><entry key="ext_dict"></entry><!--用户可以在这里配置自己的扩展停止词字典--><entry key="ext_stopwords"></entry><!--用户可以在这里配置远程扩展字典 --><entry key="remote_ext_dict">http://192.168.67.163/fenci/myword.txt </entry><!--用户可以在这里配置远程扩展停止词字典--><!-- <entry key="remote_ext_stopwords">words_location</entry> -->
</properties>
远程扩展词典可以利用nginx发布静态资源nginx.conf
server{listen 80;server_name 127.0.0.1;location /fenci/{root es;}并在/usr/local/nginx/下建/es/fenci目录下加myword.txt,每一行代表一个词（重启es,nginx）,历史数据不会进行分词，如想对历史数据重新分词，执行POST /dul/_doc/_update_by_query?conflicts=proceed当遇到冲突而导致整个更新过程失败时，更新过程是不会回滚的。如果不想因为冲突导致整个更新过程终止，可以在url中添加参数conflicts=proceed

• ④重启es

12.ES增删改查

• text支持分词，keyword只能全部内容匹配。
• ik_smart粗粒度分词器（不重复），ik_max_word细粒度分词
• es客户端选择
  ①TransportClient为代表的ES的原生客户端，不能执行原生dsl语句必须使用它的java api方法
  ②以Rest Api为主的missing client,最典型的jest。这种客户端可以直接使用dsl语句拼接成字符串，直接传给服务端，然后返回json字符串在解析
  

1）创建索引语句

PUT /dul/
{"mappings": {"_doc":{"properties":{"user_id":{"type":"keyword"},"sku_name":{"type":"text","analyzer": "ik_max_word"},"sku_category1_name":{"type":"text","analyzer": "ik_max_word"},"dt":{"type":"keyword","index":"false"} }}}
}

2）查看mapping

GET  /dul/_mapping/
结果：
{"dul" : {"mappings" : {"_doc" : {"properties" : {"dt" : {"type" : "keyword","index" : false},"sku_category1_name" : {"type" : "text","analyzer" : "ik_max_word"},"sku_name" : {"type" : "text","analyzer" : "ik_max_word"},"user_id" : {"type" : "keyword"}}}}}
}

3）新增文档（和修改-整体替换无区别）

PUT /dul/_doc/3  //POST自动生成22位uuid
{"user_id":"2","sku_name":"我是美国人","sku_category1_name":"红米 平板电脑","dt":"K"
}

4）修改某一个字段

POST /dul/_doc/3/_update
{"doc":{"dt":"D"}
}

5）用id查找

GET /dul/_doc/3

6）删除一个文档

DELETE /dul/_doc/3

7）搜索全部数据

GET /dul/_doc/_search
结果：
{"took" : 11, //耗费时间（毫秒）"timed_out" : false,//是否超时"_shards" : {"total" : 5,//发送给全部5个分片"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 3,//命中3条数据"max_score" : 1.0,//最大评分"hits" : [//结果{"_index" : "dul","_type" : "_doc","_id" : "2","_score" : 1.0,"_source" : {"user_id" : "2","sku_name" : "我是美国人","sku_category1_name" : "红米 平板电脑","dt" : "K"}},{"_index" : "dul","_type" : "_doc","_id" : "1","_score" : 1.0,"_source" : {"user_id" : "1","sku_name" : "我是中国人","sku_category1_name" : "小米手机 平板手机","dt" : "K"}}]}
}

8）分词查询

GET /dul/_doc/_search
{"query":{"match":{"sku_category1_name":"平板手机"}}
}
结果：
{"took" : 19,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 2,"max_score" : 0.970927,"hits" : [{"_index" : "dul","_type" : "_doc","_id" : "1","_score" : 0.970927,"_source" : {"user_id" : "1","sku_name" : "我是中国人","sku_category1_name" : "小米手机 平板手机","dt" : "K"}},{"_index" : "dul","_type" : "_doc","_id" : "2","_score" : 0.2876821,"_source" : {"user_id" : "2","sku_name" : "我是美国人","sku_category1_name" : "红米 平板电脑","dt" : "K"}}]}
}

9）短语查询：不使用分词，直接原始数据中匹配（指定顺序）

GET /dul/_doc/_search
{"query":{"match_phrase":{"sku_category1_name":"平板电脑"}}
}
结果：
{"took" : 6,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 1,"max_score" : 0.5753642,"hits" : [{"_index" : "dul","_type" : "_doc","_id" : "2","_score" : 0.5753642,"_source" : {"user_id" : "2","sku_name" : "我是美国人","sku_category1_name" : "红米 平板电脑","dt" : "K"}}]}
}

10）fuzzy查询（对非常接近的词进行查询匹配，消耗性能）

GET /dul/_doc/_search
{"query":{"fuzzy":{"sku_category1_name":"rad"}}
}

11）查询前过滤

Bool Query

filter	只过滤符合条件的文档，不计算相关性得分，es有缓存功能，提升效率
must	文档必须符合must中的所有条件，会影响相关性得分
must_not	文档中必须不符合must_not中的所有条件
should	文档可以符合should中的条件(一个或多个)，会影响相关性得分

GET /dul/_doc/_search
{"query":{"bool":{"filter":[{"term":{"user_id":2}}],"must":{"match":{"sku_category1_name":"平板"}}}}
}
结果：
{"took" : 24,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 1,"max_score" : 0.2876821,"hits" : [{"_index" : "dul","_type" : "_doc","_id" : "2","_score" : 0.2876821,"_source" : {"user_id" : "2","sku_name" : "我是美国人","sku_category1_name" : "红米 平板电脑","dt" : "K"}}]}
}

12）按范围过滤

操作符介绍：
gt--------------------------------------大于
lt---------------------------------------小于
gte------------------------------------大于等于
lte-------------------------------------小于等于

GET /dul/_doc/_search
{"query":{"bool":{"filter":{"range":{"user_id":{"gte":3}}}}}
}
结果：
{"took" : 17,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 1,"max_score" : 0.0,"hits" : [{"_index" : "dul","_type" : "_doc","_id" : "3","_score" : 0.0,"_source" : {"user_id" : "3","sku_name" : "我是法国人","sku_category1_name" : "红米 电脑","dt" : "K"}}]}
}

13）排序

GET /dul/_doc/_search
{"query":{"match":{"sku_category1_name":"平板"}},/** "query":{"match":{"sku_category1_name":{"query":"平板电脑","operator": "and"//合并(or或者)控制单词间匹配关系（顺序不指定）}}}, **/"sort":{"user_id":{"order":"asc"}}
}
结果：
{"took" : 16,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 2,"max_score" : null,"hits" : [{"_index" : "dul","_type" : "_doc","_id" : "1","_score" : null,"_source" : {"user_id" : "1","sku_name" : "我是中国人","sku_category1_name" : "小米手机 平板手机","dt" : "K"},"sort" : ["1"]},{"_index" : "dul","_type" : "_doc","_id" : "2","_score" : null,"_source" : {"user_id" : "2","sku_name" : "我是美国人","sku_category1_name" : "红米 平板电脑","dt" : "K"},"sort" : ["2"]}]}
}

14）分页查询

GET /dul/_doc/_search
{"query":{"match_all":{}},"from":0,"size":2
}
结果：
{"took" : 5,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 3,"max_score" : 1.0,"hits" : [{"_index" : "dul","_type" : "_doc","_id" : "2","_score" : 1.0,"_source" : {"user_id" : "2","sku_name" : "我是美国人","sku_category1_name" : "红米 平板电脑","dt" : "K"}},{"_index" : "dul","_type" : "_doc","_id" : "1","_score" : 1.0,"_source" : {"user_id" : "1","sku_name" : "我是中国人","sku_category1_name" : "小米手机 平板手机","dt" : "K"}}]}
}

15）查询指定字段

GET /dul/_doc/_search
{"query":{"match_all":{}},"from":0,"size":3,"sort":{"user_id":{"order":"desc"}},"_source":["sku_name","dt"]
}
结果：
{"took" : 16,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 2,"max_score" : null,"hits" : [{"_index" : "dul","_type" : "_doc","_id" : "1","_score" : null,"_source" : {"user_id" : "1","sku_name" : "我是中国人","sku_category1_name" : "小米手机 平板手机","dt" : "K"},"sort" : ["1"]},{"_index" : "dul","_type" : "_doc","_id" : "2","_score" : null,"_source" : {"user_id" : "2","sku_name" : "我是美国人","sku_category1_name" : "红米 平板电脑","dt" : "K"},"sort" : ["2"]}]}
}

16）聚合

GET /dul/_doc/_search
{"aggs":{"groupby_name":{"terms":{"field":"user_id"}}}
}
结果：
{"took" : 13,"timed_out" : false,"_shards" : {"total" : 5,"successful" : 5,"skipped" : 0,"failed" : 0},"hits" : {"total" : 3,"max_score" : 1.0,"hits" : [{"_index" : "dul","_type" : "_doc","_id" : "2","_score" : 1.0,"_source" : {"user_id" : "2","sku_name" : "我是美国人","sku_category1_name" : "红米 平板电脑","dt" : "K"}},{"_index" : "dul","_type" : "_doc","_id" : "1","_score" : 1.0,"_source" : {"user_id" : "1","sku_name" : "我是中国人","sku_category1_name" : "小米手机 平板手机","dt" : "K"}},{"_index" : "dul","_type" : "_doc","_id" : "3","_score" : 1.0,"_source" : {"user_id" : "3","sku_name" : "我是法国人","sku_category1_name" : "红米 电脑","dt" : "K"}}]},"aggregations" : {"groupby_name" : {"doc_count_error_upper_bound" : 0,"sum_other_doc_count" : 0,"buckets" : [{"key" : "1","doc_count" : 1},{"key" : "2","doc_count" : 1},{"key" : "3","doc_count" : 1}]}}
}

17）检索多个文档(检索确定的字段，也可以定义一个_source参数)

POST /_mget
{"docs" : [{"_index" : "website","_type" :  "blog","_id" :    2},{"_index" : "website","_type" :  "pageviews","_id" :    1,"_source": "views"}]
}
结果：
{"docs" : [{"_index" :   "website","_id" :      "2","_type" :    "blog","found" :    true,"_source" : {"text" :  "This is a piece of cake...","title" : "My first external blog entry"},"_version" : 10},{"_index" :   "website","_id" :      "1","_type" :    "pageviews","found" :    true,"_version" : 2,"_source" : {"views" : 2}}]
}