ranger审计日志对接CDH solr

作者：耀灵

一、准备条件

1、已安装完毕ranger-admin
2、已在CDH上部署solr（注意在安装solr时更改下solr在zk上的节点信息）

请添加图片描述

二、更改相关配置

1、修改ranger-2.1.0-admin/contrib/solr_for_audit_setup/install.properties

SOLR_USER=solr
SOLR_GROUP=solr
MAX_AUDIT_RETENTION_DAYS=90

SOLR_INSTALL=false



SOLR_DOWNLOAD_URL=


SOLR_INSTALL_FOLDER=/opt/cloudera/parcels/CDH/lib/solr

SOLR_RANGER_HOME=/opt/cloudera/parcels/CDH/lib/solr/ranger_audit_server

SOLR_RANGER_PORT=8983

SOLR_DEPLOYMENT=solrcloud

SOLR_RANGER_DATA_FOLDER=/opt/cloudera/parcels/CDH/lib/solr/ranger_audit_server/data


SOLR_ZK=100.116.3.226:2181,100.116.3.227,100.116.3.228/solr/configs/ranger_audits
SOLR_HOST_URL=http://`hostname -f`:8983
SOLR_SHARDS=3
SOLR_REPLICATION=2

SOLR_LOG_FOLDER=/opt/logs/solr/ranger_audits

SOLR_RANGER_COLLECTION=ranger_audits

SOLR_MAX_MEM=2g

2、修改ranger-2.1.0-admin/contrib/solr_for_audit_setup/conf/solrconfig.xml

请添加图片描述

此处修改为

EEE MMM dd HH:mm:ss [z ]yyyy

请添加图片描述

此处直接改为cdh上solr jar对应的目录，这个配置文件之前是基于非集成的solr配置的，不做修改的话会报配置文件找不到的错误。

3、修改ranger-admin的install.propertities

PYTHON_COMMAND_INVOKER=python

DB_FLAVOR=MYSQL

SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar


db_root_user=root
db_root_password=VYi2OfrsO0Mz
db_host=100.116.3.249
db_ssl_enabled=false
db_ssl_required=false
db_ssl_verifyServerCertificate=false
db_ssl_auth_type=2-way
javax_net_ssl_keyStore=
javax_net_ssl_keyStorePassword=
javax_net_ssl_trustStore=
javax_net_ssl_trustStorePassword=
db_name=ranger
db_user=ranger
db_password=VYi2OfrsO0Mz

rangerAdmin_password=
rangerTagsync_password=
rangerUsersync_password=
keyadmin_password=


audit_store=solr

audit_elasticsearch_urls=
audit_elasticsearch_port=
audit_elasticsearch_protocol=
audit_elasticsearch_user=
audit_elasticsearch_password=
audit_elasticsearch_index=
audit_elasticsearch_bootstrap_enabled=false


audit_solr_urls=http://100.116.3.227:8983/solr/ranger_audits
audit_solr_user=
audit_solr_password=
audit_solr_zookeepers=100.116.3.226:2181,100.116.3.227:2181,100.116.3.228:2181/ranger_audits

audit_solr_collection_name=ranger_audits
audit_solr_config_name=ranger_audits
audit_solr_no_shards=
audit_solr_no_replica=
audit_solr_max_shards_per_node=
audit_solr_acl_user_list_sasl=solr,infra-solr
audit_solr_bootstrap_enabled=true



policymgr_external_url=http://localhost:6080
policymgr_http_enabled=true
policymgr_https_keystore_file=
policymgr_https_keystore_keyalias=rangeradmin
policymgr_https_keystore_password=

policymgr_supportedcomponents=



unix_user=ranger
unix_user_pwd=ranger
unix_group=ranger


authentication_method=NONE
remoteLoginEnabled=true
authServiceHostName=localhost
authServicePort=5151
ranger_unixauth_keystore=keystore.jks
ranger_unixauth_keystore_password=password
ranger_unixauth_truststore=cacerts
ranger_unixauth_truststore_password=changeit


xa_ldap_url=
xa_ldap_userDNpattern=
xa_ldap_groupSearchBase=
xa_ldap_groupSearchFilter=
xa_ldap_groupRoleAttribute=
xa_ldap_base_dn=
xa_ldap_bind_dn=
xa_ldap_bind_password=
xa_ldap_referral=
xa_ldap_userSearchFilter=

xa_ldap_ad_domain=
xa_ldap_ad_url=
xa_ldap_ad_base_dn=
xa_ldap_ad_bind_dn=
xa_ldap_ad_bind_password=
xa_ldap_ad_referral=
xa_ldap_ad_userSearchFilter=

spnego_principal=
spnego_keytab=
token_valid=30
cookie_domain=
cookie_path=/
admin_principal=
admin_keytab=
lookup_principal=
lookup_keytab=
hadoop_conf=/etc/hadoop/conf
sso_enabled=false
sso_providerurl=https://127.0.0.1:8443/gateway/knoxsso/api/v1/websso
sso_publickey=


RANGER_ADMIN_LOG_DIR=$PWD

RANGER_PID_DIR_PATH=/var/run/ranger

XAPOLICYMGR_DIR=$PWD
app_home=$PWD/ews/webapp
TMPFILE=$PWD/.fi_tmp
LOGFILE=$PWD/logfile
LOGFILES="$LOGFILE"

JAVA_BIN='java'
JAVA_VERSION_REQUIRED='1.8'
JAVA_ORACLE='Java(TM) SE Runtime Environment'

ranger_admin_max_heap_size=1g
PATCH_RETRY_INTERVAL=120
STALE_PATCH_ENTRY_HOLD_TIME=10

mysql_core_file=db/mysql/optimized/current/ranger_core_db_mysql.sql
mysql_audit_file=db/mysql/xa_audit_db.sql

oracle_core_file=db/oracle/optimized/current/ranger_core_db_oracle.sql
oracle_audit_file=db/oracle/xa_audit_db_oracle.sql
postgres_core_file=db/postgres/optimized/current/ranger_core_db_postgres.sql
postgres_audit_file=db/postgres/xa_audit_db_postgres.sql
sqlserver_core_file=db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql
sqlserver_audit_file=db/sqlserver/xa_audit_db_sqlserver.sql
sqlanywhere_core_file=db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql
sqlanywhere_audit_file=db/sqlanywhere/xa_audit_db_sqlanywhere.sql
cred_keystore_filename=$app_home/WEB-INF/classes/conf/.jceks/rangeradmin.jceks

三、初始化

1、修改完成后，初始化solr、ranger-admin
ranger-2.1.0-admin/contrib/solr_for_audit_setup

./setup.sh

请添加图片描述

./setup.sh

请添加图片描述

四、上传相关配置到zookeeper并创建对应collection

1、相关指令如下

查看创建的solrhome的实例 
solrctl instancedir --list 


创建 collection1 实例并将配置文件上传到 zookeeper: 
#conf必须是一个目录，这里的目录是 ranger-2.1.0-admin/contrib/solr_for_audit_setup/conf
solrctl instancedir --create ranger_audits conf 


可以通过下面命令查看上传的实体： 
solrctl instancedir --list 
创建 collection 
1）创建ranger_audits
solrctl collection --create ranger_audits -s 3 -c ranger_audits -r 2 -m 3