ELK下载

https://www.elastic.co/downloads

 

因为我用的是windows操作系统，所以下面的介绍都基于windows环境。功能方面应该和Linux是一样的。

=======================================================================================

Getting Started with ELK for Apache Logs (这个例子展示了怎样用logstash把log发送到Elastic Search, 然后用Kibana做图形化展示）

https://github.com/aagea/elk-example

 

我修改了上面的例子，用stdin作为logstash的输入（输入了hello world)，用Elastic Search作为输出。

那么为了验证，可以用下面这个命令搜索Elastic Search的内容

http://localhost:9200/applog/_search?q=hello

http://127.0.0.1:9200/applog/doc/_search

 

当然也可以直接在Kibana的Discovery界面搜索。具体方法如下。这篇文章写的很好，尤其是里面的图画得很清晰，缺点是软件版本有点老了。

https://my.oschina.net/itblog/blog/547250/

 

下面介绍一下ELK各个组件安装后的目录结构和help命令返回的结果。这个有助于了解组件的功能覆盖和基本使用方法。

Elastic Search 6.6

目录

bin -- 这里存放Elastic Search的执行程序.  这里我用elasticsearch -help /all > rex_output.txt命令得到所有的可选项

config -- 这里存放配置文件，主要是一些yml文件。 比如elasticsearch.yml (内容如下)

cluster.name=my-application
node.name=node-1
path.data=/path/to/data
path.logs=/path/to/logs
network.host=127.0.0.1
http.port=9200

其余配置文件如下

    - log4j2.properties

    - role_mapping.yml

    - roles.yml

    - users

    - users_roles

data -- 这里面好像是跟cluster有关的文件，比如默认的是cluster node-0, 所以这里面就有一个名字是“0”的文件夹

lib -- 就是elasticsearch依赖的一些JAR包

logs -- 日志文件

modules  -- 这里面是ElasticSearch用到的主要模块，包括X-Pack (lets you set up watches (or rules) to detect and alert on changes in your ElasticSearch data) ，也是一些jar包。

plugins --  插件 sudo bin/elasticsearch-plugin install [plugin_name]

 

Help命令返回的内容

Option                Description                                               
------                -----------                                               
-E <KeyValuePair>     Configure a setting                                       
-V, --version         Prints elasticsearch version information and exits        
-d, --daemonize       Starts Elasticsearch in the background                    
-h, --help            show help                                                 
-p, --pidfile <Path>  Creates a pid file in the specified path on start         
-q, --quiet           Turns off standard output/error streams logging in console
-s, --silent          show minimal output                                       
-v, --verbose         show verbose output        

 

ElasticSearch的官网上有一些例子（https://github.com/elastic/examples），还没来得及细看，下一篇文章再细说吧。

 

Logstash 6.6

目录 （这里有详细介绍 https://www.elastic.co/guide/en/logstash/current/dir-layout.html）

bin -- 可执行程序

config -- 配置文件。可以参考logstash-sample.conf创建你自己的conf文件

data -- 这里面的文件有些看不懂。比如一个uuid文件，一个.lock文件，几个queue文件夹。好像是跟持久化有关的内容。

lib -- 这里面有bootstrap, pluginmaster, secretstore, systeminstall几个文件夹。也都是ruby程序。

logstash-core -- 这里面有一些ruby程序和JAR包，应该是logstash的核心代码。

logstash-core-plugin-api -- 插件。也是ruby程序。

modules -- 好像是用来存放打包后的ruby module的。ps:  Ruby通常用rubygem来打包

tools -- 这里有一个ingest-converter文件夹，里面是一个JAR文件。

vendor -- 这里有bundle和jruby文件夹

x-pack -- 这里面有很多文件夹。可能和ElasticSearch一样，作用是lets you set up watches (or rules) to detect and alert on changes in your Logstash data

 

Help命令返回的内容 (初学者常用的命令用粗体表示)

Usage:
    bin/logstash [OPTIONS]

Options:
    -n, --node.name NAME          Specify the name of this logstash instance, if no value is given
                                  it will default to the current hostname.
                                   (default: 这里指的是主机名)
    -f, --path.config 指定配置文件 比如这样：config/log4j_to_es.conf

   CONFIG_PATH Load the logstash config from a specific file
                                  or directory.  If a directory is given, all
                                  files in that directory will be concatenated
                                  in lexicographical order and then parsed as a
                                  single config file. You can also specify
                                  wildcards (globs) and any matched files will
                                  be loaded in the order described above.
    -e, --config.string 从命令行指定配置 比如：logstash -e “input { stdin { } } output { stdout {} }

”CONFIG_STRING Use the given string as the configuration
                                  data. Same syntax as the config file. If no
                                  input is specified, then the following is
                                  used as the default input:
                                  "input { stdin { type => stdin } }"
                                  and if no output is specified, then the
                                  following is used as the default output:
                                  "output { stdout { codec => rubydebug } }"
                                  If you wish to use both defaults, please use
                                  the empty string for the '-e' flag.
                                   (default: nil)
    --field-reference-parser MODE Use the given MODE when parsing field
                                  references.
                                  The field reference parser is used to expand
                                  field references in your pipeline configs,
                                  and will be becoming more strict to better
                                  handle illegal and ambbiguous inputs in a
                                  future release of Logstash.
                                  
                                  Available MODEs are:
                                   - `LEGACY`: parse with the legacy parser,
                                     which is known to handle ambiguous- and
                                     illegal-syntax in surprising ways;
                                     warnings will not be emitted.
                                   - `COMPAT`: warn once for each distinct
                                     ambiguous- or illegal-syntax input, but
                                     continue to expand field references with
                                     the legacy parser.
                                   - `STRICT`: parse in a strict manner; when
                                     given ambiguous- or illegal-syntax input,
                                     raises a runtime exception that should
                                     be handled by the calling plugin.
                   
                                   The MODE can also be set with
                                   `config.field_reference.parser`
                                  
                                   (default: "COMPAT")
    --modules MODULES             Load Logstash modules.
                                  Modules can be defined using multiple instances
                                  '--modules module1 --modules module2',
                                     or comma-separated syntax
                                  '--modules=module1,module2'
                                  Cannot be used in conjunction with '-e' or '-f'
                                  Use of '--modules' will override modules declared
                                  in the 'logstash.yml' file.
    -M, --modules.variable MODULES_VARIABLE Load variables for module template.
                                  Multiple instances of '-M' or
                                  '--modules.variable' are supported.
                                  Ignored if '--modules' flag is not used.
                                  Should be in the format of
                                  '-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"'
                                  as in
                                  '-M "example.var.filter.mutate.fieldname=fieldvalue"'
    --setup                       Load index template into Elasticsearch, and saved searches,
                                  index-pattern, visualizations, and dashboards into Kibana when
                                  running modules.
                                   (default: false)
    --cloud.id CLOUD_ID           Sets the elasticsearch and kibana host settings for
                                  module connections in Elastic Cloud.
                                  Your Elastic Cloud User interface or the Cloud support
                                  team should provide this.
                                  Add an optional label prefix '<label>:' to help you
                                  identify multiple cloud.ids.
                                  e.g. 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy'
    --cloud.auth CLOUD_AUTH       Sets the elasticsearch and kibana username and password
                                  for module connections in Elastic Cloud
                                  e.g. 'username:<password>'
    --pipeline.id ID              Sets the ID of the pipeline.
                                   (default: "main")
    -w, --pipeline.workers COUNT  Sets the number of pipeline workers to run.
                                   (default: 4)
    --java-execution              Use Java execution engine.
                                   (default: false)
    -b, --pipeline.batch.size SIZE Size of batches the pipeline is to work in.
                                   (default: 125)
    -u, --pipeline.batch.delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling
                                  for the next event.
                                   (default: 50)
    --pipeline.unsafe_shutdown    Force logstash to exit during shutdown even
                                  if there are still inflight events in memory.
                                  By default, logstash will refuse to quit until all
                                  received events have been pushed to the outputs.
                                   (default: false)
    --path.data PATH              This should point to a writable directory. Logstash
                                  will use this directory whenever it needs to store
                                  data. Plugins will also have access to this path.
                                   (default: "C:/Users/IBM_ADMIN/Downloads/logstash-6.6.0/data")
    -p, --path.plugins PATH       A path of where to find plugins. This flag
                                  can be given multiple times to include
                                  multiple paths. Plugins are expected to be
                                  in a specific directory hierarchy:
                                  'PATH/logstash/TYPE/NAME.rb' where TYPE is
                                  'inputs' 'filters', 'outputs' or 'codecs'
                                  and NAME is the name of the plugin.
                                   (default: [])
    -l, --path.logs PATH          Write logstash internal logs to the given
                                  file. Without this flag, logstash will emit
                                  logs to standard output.
                                   (default: "C:/Users/IBM_ADMIN/Downloads/logstash-6.6.0/logs")
    --log.level LEVEL             Set the log level for logstash. Possible values are:
                                    - fatal
                                    - error
                                    - warn
                                    - info
                                    - debug
                                    - trace
                                   (default: "info")
    --config.debug     Print the compiled config ruby code out as a debug log (you must also have --log.level=debug enabled).
                                  WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result
                                  in plaintext passwords appearing in your logs!
                                   (default: false)
    -i, --interactive SHELL       Drop to shell instead of running as normal.
                                  Valid shells are "irb" and "pry"
    -V, --version                 Emit the version of logstash and its friends,
                                  then exit.
    -t, --config.test_and_exit    Check configuration for valid syntax and then exit.
                                   (default: false)
    -r, --config.reload.automatic Monitor configuration changes and reload
                                  whenever it is changed.
                                  NOTE: use SIGHUP to manually reload the config
                                   (default: false)
    --config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location
                                  for changes, in seconds.
                                   (default: 3000000000)
    --http.host HTTP_HOST         Web API binding host (default: "127.0.0.1")
    --http.port HTTP_PORT         Web API http port (default: 9600..9700)
    --log.format FORMAT           Specify if Logstash should write its own logs in JSON form (one
                                  event per line) or in plain text (using Ruby's Object#inspect)
                                   (default: "plain")
    --path.settings SETTINGS_DIR  Directory containing logstash.yml file. This can also be
                                  set through the LS_SETTINGS_DIR environment variable.
                                   (default: config目录的地址)
    --verbose                     Set the log level to info.
                                  DEPRECATED: use --log.level=info instead.
    --debug                       Set the log level to debug.
                                  DEPRECATED: use --log.level=debug instead.
    --quiet                       Set the log level to info.
                                  DEPRECATED: use --log.level=info instead.
    -h, --help                    print help

 

以我初学者的眼光看，Logstash是ELK里面需要配置最多的组件。通过配置,  Logstash可以搜集不同类型的日志供我们分析。下面是网上的一些例子：

Logstash收集RESTful API日志

https://howtodoinjava.com/microservices/elk-stack-tutorial-example/

 

Logstash收集log4j日志

https://my.oschina.net/itblog/blog/547250/

 

 

Kibana 6.6

目录

bin  -- 这个目录存放Kibana主程序

config -- 这个目录存放kibana.yml配置文件

data -- 好像是存放主程序生成的数据？

dlls (这应该只有windows才有) --  windows需要的动态链接库文件

node -- 这里面存放一个node.exe程序

node_modules -- 这里存放node插件

optimize -- 这里存放了一些JS和CSS文件，可能是Kibana界面所使用的

plugins -- 插件

src -- 这里存放的是Kibana的源程序，比如core文件夹，里面的README描述是这样的Core is a set of systems (frontend, backend etc.) that Kibana and its plugins are built on top of.  通过修改源程序可以对Kibana进行定制。

target -- 这里面好像是一些.ts.map文件，应该是src编译之后产生的

webpackShims -- 这里面是一些JS文件，好像是一些common module, 比如ui-bootstrap.js,  angular.js,  elasticsearch.js

 

Help命令返回的内容

Usage: bin/kibana [command=serve] [options]
 
  Kibana is an open source (Apache Licensed), browser based analytics and search dashboard for Elasticsearch.
 
  Commands:
    serve  [options]  Run the kibana server
    help  <command>   Get the help for a specific command
 
  "serve" Options:
 
    -h, --help                       output usage information
    -e, --elasticsearch <uri1,uri2>  Elasticsearch instances
    -c, --config <path>              Path to the config file, can be changed with the CONFIG_PATH environment variable as well. Use multiple --config args to include multiple config files.
    -p, --port <port>                The port to bind to
    -q, --quiet                      Prevent all logging except errors
    -Q, --silent                     Prevent all logging
    --verbose                        Turns on verbose logging
    -H, --host <host>                The host to bind to
    -l, --log-file <path>            The file to log to
    --plugin-dir <path>              A path to scan for plugins, this can be specified multiple times to specify multiple directories
    --plugin-path <path>             A path to a plugin which should be included by the server, this can be specified multiple times to specify multiple paths
    --plugins <path>                 an alias for --plugin-dir
    --optimize                       Optimize and then stop the server

 

在实际使用中，我们可能需要自定义Kibana界面。 我在网上找到两篇文章，先收藏着吧。

Kibana界面风格重制

https://www.jianshu.com/p/1edbc952a913

 

Kibana国际化

https://www.jianshu.com/p/57e0475fda90