By
Securitynik on 2020-10-05 12:57:35
In the first post, we installed Elasticsearch. In the second post, we installed Kibana. In this post, we now provide some basic security to the communication between the Elasticsearch and Kibana. Note, there is a lot more you can do to secure this environment, taking advantage of keystores, etc.
In generating the Certification Authority (CA) certificate, I choose to use PKCS#12 format. In this format, this file contains both the CA certificate and its private key. This may be a cause for concern in some environments.
Additionally, I created one certificate file to be used by each component. This means the various components need to be able to read the file. This obviously makes the data in the file accessible by those users. In this post, I am keeping it simple. If you are looking at thoroughly securing your environment, this post is something you can essentially build on if you wish.
If you are wondering why we need to secure the communication between Elastic and Kibana, here is a simple reason why.
root@securitynik-monitoring:~# tcpdump -nnti any host 10.0.0.1 and port 9200 -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
IP 10.0.0.1.60100 > 10.0.0.1.9200: Flags [P.], seq 3727928959:3727929473, ack 22660352, win 9203, options [nop,nop,TS val 4105791837 ecr 4105786838], length 514
E..6.G@.@.!"..........#..3...Y....#........
..i]..U.POST /.reporting-*/_search HTTP/1.1
content-type: application/json
Host: 10.0.0.1:9200
Content-Length: 374
Connection: keep-alive
{"seq_no_primary_term":true,"_source":{"excludes":["output.content"]},"query":{"bool":{"filter":{"bool":{"minimum_should_match":1,"should":[{"term":{"status":"pending"}},{"bool":{"must":[{"term":{"status":"processing"}},{"range":{"process_expiration":{"lte":"2020-08-14T01:02:35.764Z"}}}]}}]}}}},"sort":[{"priority":{"order":"asc"}},{"created_at":{"order":"asc"}}],"size":1}
IP 10.0.0.1.9200 > 10.0.0.1.60100: Flags [P.], seq 1:247, ack 514, win 512, options [nop,nop,TS val 4105791839 ecr 4105791837], length 246
E..*..@.@...........#....Y...3.......u.....
..i_..i]HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 159
....
root@securitynik-monitoring:~# cat /etc/elasticsearch/elasticsearch.yml | grep "xpack.security.enabled: true" xpack.security.enabled: true
root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-certutil ca ca-dn securitynik.local This tool assists you in the generation of X.509 certificates and certificate signing requests for use with SSL/TLS in the Elastic stack. The 'ca' mode generates a new 'certificate authority'. This will create a new X.509 certificate and private key that can be used to sign certificate when running in 'cert' mode. Use the 'ca-dn' option if you wish to configure the 'distinguished name' of the certificate authority By default the 'ca' mode produces a single PKCS#12 output file which holds: * The CA certificate * The CA's private key If you elect to generate PEM format certificates (the -pem option), then the output will be a zip file containing individual files for the CA certificate and private key Please enter the desired output file [elastic-stack-ca.p12]: SecurityNik-CA.p12 Enter password for SecurityNik-CA.p12 :
root@securitynik-monitoring:~# ls /usr/share/elasticsearch/SecurityNik-CA.p12 -al -rw------- 1 root root 2527 Aug 13 21:49 /usr/share/elasticsearch/SecurityNik-CA.p12
root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /usr/share/elasticsearch/SecurityNik-CA.p12 --days 1825 --dns monitoring,monitoring.securitynik.local --ip 10.0.0.1 --keysize 2048 --name 10.0.0.1 --out /usr/share/elasticsearch/10.0.0.1.p12 --pass "" --silent Enter password for CA (/usr/share/elasticsearch/SecurityNik-CA.p12) :
root@securitynik-monitoring:~# ls -al /usr/share/elasticsearch/10.0.0.1.p12 -al -rw------- 1 root root 3529 Aug 13 22:23 /usr/share/elasticsearch/10.0.0.1.p12
root@securitynik-monitoring:~# cp /usr/share/elasticsearch/10.0.0.1.p12 /etc/elasticsearch/ -v
'/usr/share/elasticsearch/10.0.0.1.p12' -> '/etc/elasticsearch/10.0.0.1.p12'
root@securitynik-monitoring:~# chmod 644 /etc/elasticsearch/10.0.0.1.p12
root@securitynik-monitoring:~# ls -al /etc/elasticsearch/10.0.0.1.p12 -rw-r--r-- 1 root elasticsearch 3529 Aug 13 22:32 /etc/elasticsearch/10.0.0.1.p12
xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: 10.0.0.1.p12 xpack.security.transport.ssl.truststore.path: 10.0.0.1.p12
root@securitynik-monitoring:~# tail --lines 6 /etc/elasticsearch/elasticsearch.yml # Configuration added by Nik for security xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: "10.0.0.1.p12" xpack.security.transport.ssl.truststore.path: "10.0.0.1.p12"
root@securitynik-monitoring:~# systemctl status elasticsearch.service ● elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-08-13 22:40:44 EDT; 3min 50s ago Docs: https://www.elastic.co Main PID: 24533 (java) Tasks: 61 (limit: 4563) Memory: 1.2G CGroup: /system.slice/elasticsearch.service ├─24533 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddres> └─24725 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller Aug 13 22:40:17 securitynik-monitoring systemd[1]: Starting Elasticsearch... Aug 13 22:40:44 securitynik-monitoring systemd[1]: Started Elasticsearch.
root@securitynik-monitoring:~# ss --numeric --listen --tcp | grep 9200 LISTEN 0 4096 [::ffff:10.0.0.1]:9200 *:*
root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-certutil http --silent ## Elasticsearch HTTP Certificate Utility ## Do you wish to generate a Certificate Signing Request (CSR)? Generate a CSR? [y/N]N ## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate? Use an existing CA? [y/N]Y ## What is the path to your CA? CA Path: /usr/share/elasticsearch/SecurityNik-CA.p12 Password for SecurityNik-CA.p12: ## How long should your certificates be valid? For how long should your certificate be valid? [5y] ## Do you wish to generate one certificate per node? Generate a certificate per node? [y/N]N ## Which hostnames will be used to connect to your nodes? securitynik-monitoring securitynik-monitoring.securitynik.local You entered the following hostnames. - securitynik-monitoring - securitynik-monitoring.securitynik.local Is this correct [Y/n]y ## Which IP addresses will be used to connect to your nodes? 10.0.0.1 You entered the following IP addresses. - 10.0.0.1 Is this correct [Y/n]y ## Other certificate options Key Name: securitynik-monitoring Subject DN: CN=securitynik-monitoring Key Size: 2048 Do you wish to change any of these options? [y/N]n ## What password do you want for your private key(s)? Provide a password for the "http.p12" file: [<ENTER> for none] ## Where should we save the generated files? What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]
root@securitynik-monitoring:~# ls /usr/share/elasticsearch/elasticsearch-ssl-http.zip -l -rw------- 1 root root 7334 Aug 13 23:22 /usr/share/elasticsearch/elasticsearch-ssl-http.zip
root@securitynik-monitoring:/usr/share/elasticsearch# apt install unzip
root@securitynik-monitoring:/usr/share/elasticsearch# unzip -l elasticsearch-ssl-http.zip Archive: elasticsearch-ssl-http.zip Length Date Time Name --------- ---------- ----- ---- 0 2020-08-13 23:22 elasticsearch/ 1091 2020-08-13 23:22 elasticsearch/README.txt 3499 2020-08-13 23:22 elasticsearch/http.p12 657 2020-08-13 23:22 elasticsearch/sample-elasticsearch.yml 0 2020-08-13 23:22 kibana/ 1306 2020-08-13 23:22 kibana/README.txt 1200 2020-08-13 23:22 kibana/elasticsearch-ca.pem 1056 2020-08-13 23:22 kibana/sample-kibana.yml --------- ------- 8809 8 files
root@securitynik-monitoring:/usr/share/elasticsearch# unzip -d certs/ elasticsearch-ssl-http.zip Archive: elasticsearch-ssl-http.zip creating: certs/elasticsearch/ inflating: certs/elasticsearch/README.txt inflating: certs/elasticsearch/http.p12 inflating: certs/elasticsearch/sample-elasticsearch.yml creating: certs/kibana/ inflating: certs/kibana/README.txt inflating: certs/kibana/elasticsearch-ca.pem inflating: certs/kibana/sample-kibana.yml root@securitynik-monitoring:/usr/share/elasticsearch# ls certs/ elasticsearch kibana
root@securitynik-monitoring:~# cp /usr/share/elasticsearch/certs/elasticsearch/http.p12 /etc/elasticsearch/ -v
'/usr/share/elasticsearch/certs/elasticsearch/http.p12' -> '/etc/elasticsearch/http.p12'
# This turns on SSL for the HTTP (Rest) interface xpack.security.http.ssl.enabled: true # This configures the keystore to use for SSL on HTTP xpack.security.http.ssl.keystore.path: "http.p12"
root@securitynik-monitoring:/usr/share/elasticsearch/certs# cd /etc/kibana/ root@securitynik-monitoring:/etc/kibana# mv elasticsearch-ca.pem SecurityNik-CA.pem root@securitynik-monitoring:/etc/kibana#
root@securitynik-monitoring:/usr/share/elasticsearch/certs# cp /usr/share/elasticsearch/certs/kibana/SecurityNik-ca.pem /etc/kibana/ -v '/usr/share/elasticsearch/certs/kibana/SecurityNik-ca.pem' -> '/etc/kibana/SecurityNik-ca.pem'
root@securitynik-monitoring:/etc/kibana# systemctl stop elasticsearch.service root@securitynik-monitoring:/etc/kibana# systemctl start elasticsearch.service root@securitynik-monitoring:/etc/kibana# systemctl stop kibana.service root@securitynik-monitoring:/etc/kibana# systemctl start kibana.service
root@securitynik-monitoring:~# cp /etc/elasticsearch/10.0.0.1.p12 /etc/kibana/ -v
'/etc/elasticsearch/10.0.0.1.p12' -> '/etc/kibana/10.0.0.1.p12'
server.ssl.keystore.path: "/etc/kibana/10.0.0.1.p12" server.ssl.enabled: true server.ssl.keystore.password: ""
root@securitynik-monitoring:/etc/kibana# systemctl stop kibana.service root@securitynik-monitoring:/etc/kibana# systemctl start kibana.service
root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive --silent Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [apm_system]: Reenter password for [apm_system]: Enter password for [kibana_system]: Reenter password for [kibana_system]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Enter password for [beats_system]: Reenter password for [beats_system]: Enter password for [remote_monitoring_user]: Reenter password for [remote_monitoring_user]:
root@securitynik-monitoring:~# cat /etc/kibana/kibana.yml | grep "kibana_system" --after-context 1 elasticsearch.username: "kibana_system" elasticsearch.password: "WelcomeToSecurityNikElastic"
root@securitynik-monitoring:~# tail --lines 8 /etc/kibana/kibana.yml # Below added by Nik server.ssl.enabled: true elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/SecurityNik-CA.pem"] server.ssl.keystore.path: "/etc/kibana/10.0.0.1.p12" server.ssl.keystore.password: "" xpack.encryptedSavedObjects.encryptionKey: 'fhjskloppd678ehkdfdlliverpoolfcr' xpack.security.session.idleTimeout: "30m" xpack.security.session.lifespan: "8h"
root@n3-monitoring:/etc/elasticsearch# tcpdump -nnti any host 10.0.0.1 and port 9200 -A -c 5 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes IP 10.0.0.1.36344 > 10.0.0.1.9200: Flags [P.], seq 2879060748:2879061003, ack 1452891267, win 512, options [nop,nop,TS val 2724200057 ecr 2724168569], length 255 E..3y.@.@.>...........#.....V.\......~..... ._.y._.y.........@.../....Ls}..E... .l.;........x.^..u..3S...A.,.O...2e.h...../.).*@.a15ou.u.^D...~.3..Hy\W'.../....D.ahL..H...q;...j....$..L.0<%J..._..k]..TMQj.B m....n.id.O....5.....S...=1.Iq.|Ox...}.mOP......z@..L.&...&X.>....R.u'I.-U*!|.Z..._.Z.[%7=...K.6m... IP 10.0.0.1.9200 > 10.0.0.1.36344: Flags [P.], seq 1:238, ack 255, win 512, options [nop,nop,TS val 2724200061 ecr 2724200057], length 237 E..!..@.@...........#...V.\..........l..... ._.}._.y..............{.m6!....ZK^... .j..$-.......4(..\Y6...y=j.....T0v..J.yG..8.I.....'.+%w.6...y..z...r...*K..L......t..cc.Pk.\..RZ.%.N.....l...Zq......qw.W..!05."...{...I..g..)\(.H..........Q...H..<X...1...X4.W..). .eoB...w..i.....[......+.W IP 10.0.0.1.36344 > 10.0.0.1.9200: Flags [.], ack 238, win 511, options [nop,nop,TS val 2724200061 ecr 2724200061], length 0 E..4y.@.@.?...........#.....V.]p........... ._.}._.} IP 10.0.0.1.36294 > 10.0.0.1.9200: Flags [.], ack 2500636561, win 512, options [nop,nop,TS val 2724200281 ecr 2724199268], length 0 E..43.@.@.............#.3.1................ ._.Y._.d IP 10.0.0.1.9200 > 10.0.0.1.36294: Flags [.], ack 1, win 512, options [nop,nop,TS val 2724200281 ecr 2724199268], length 0 E..4..@.@...........#.......3.1............
Posts in this series: