在CentOS6.9中 搭建 Flume

前端之家收集整理的这篇文章主要介绍了在CentOS6.9中 搭建 Flume前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。

之前的基本环境准备我就不再说了,请参照我之前的搭建hadoop的文章

在CentOS 6.9中搭建Hadoop

配置 flume 环境变量

  1. export FLUME_HOME=/opt/apache-flume-1.7.0-bin
  2. export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$FLUME_HOME/bin:$HOME/bin

然后 记得 source ~/.bash_profile

根据需求,配置不同的 source/channel/sink,添加配置文件到 conf/中

  • flume_exec_hdfs.conf
  1. logAgent.sources = logSource
  2. logAgent.channels = fileChannel
  3. logAgent.sinks = hdfsSink
  4.  
  5. logAgent.sources.logSource.type = exec
  6. logAgent.sources.logSource.command = tail -F /aura/data/flume-search/logs
  7. logAgent.sources.logSource.channels = fileChannel
  8.  
  9. logAgent.sinks.hdfsSink.type = hdfs
  10. logAgent.sinks.hdfsSink.hdfs.path = hdfs://bigdata:9000/flume/record/%Y-%m-%d/%H%M
  11. logAgent.sinks.hdfsSink.hdfs.rollCount= 10000
  12. logAgent.sinks.hdfsSink.hdfs.rollSize= 0
  13. logAgent.sinks.hdfsSink.hdfs.batchSize= 1000
  14. logAgent.sinks.hdfsSink.hdfs.filePrefix= transaction_log
  15. logAgent.sinks.hdfsSink.hdfs.rollInterval= 600
  16. logAgent.sinks.hdfsSink.hdfs.roundUnit = minute
  17. logAgent.sinks.hdfsSink.hdfs.fileType = DataStream
  18. logAgent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
  19. logAgent.sinks.hdfsSink.channel = fileChannel
  20.  
  21. logAgent.channels.fileChannel.type = memory
  22. logAgent.channels.logSource.capacity=1000
  23. logAgent.channels.logSource.transactionCapacity=100
  • flume_avro_hdfs.conf
  1. logAgent.sources = logSource
  2. logAgent.channels = fileChannel
  3. logAgent.sinks = hdfsSink
  4.  
  5. logAgent.sources.logSource.type = avro
  6. logAgent.sources.logSource.bind = 127.0.0.1
  7. logAgent.sources.logSource.port = 44444
  8. logAgent.sources.logSource.channels = fileChannel
  9.  
  10. logAgent.sinks.hdfsSink.type = hdfs
  11. logAgent.sinks.hdfsSink.hdfs.path = hdfs://bigdata:9000/flume/record/%Y-%m-%d/%H%M
  12. logAgent.sinks.hdfsSink.hdfs.rollCount= 10000
  13. logAgent.sinks.hdfsSink.hdfs.rollSize= 0
  14. logAgent.sinks.hdfsSink.hdfs.batchSize= 1000
  15. logAgent.sinks.hdfsSink.hdfs.filePrefix= transaction_log
  16. logAgent.sinks.hdfsSink.hdfs.rollInterval= 600
  17. logAgent.sinks.hdfsSink.hdfs.roundUnit = minute
  18. logAgent.sinks.hdfsSink.hdfs.fileType = DataStream
  19. logAgent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
  20. logAgent.sinks.hdfsSink.channel = fileChannel
  21.  
  22. logAgent.channels.fileChannel.type = memory
  23. logAgent.channels.logSource.capacity=1000
  24. logAgent.channels.logSource.transactionCapacity=100
  • flume_dir_hdfs.conf
  1. logAgent.sources = logSource
  2. logAgent.channels = fileChannel
  3. logAgent.sinks = hdfsSink
  4.  
  5. logAgent.sources.logSource.type = spooldir
  6. logAgent.sources.logSource.spoolDir =/aura/data/flume-search
  7. logAgent.sources.logSource.channels = fileChannel
  8.  
  9. logAgent.sinks.hdfsSink.type = hdfs
  10. logAgent.sinks.hdfsSink.hdfs.path = hdfs://bigdata:9000/flume/record/%Y-%m-%d/%H%M
  11. logAgent.sinks.hdfsSink.hdfs.rollCount= 10000
  12. logAgent.sinks.hdfsSink.hdfs.rollSize= 0
  13. logAgent.sinks.hdfsSink.hdfs.batchSize= 1000
  14. logAgent.sinks.hdfsSink.hdfs.filePrefix= transaction_log
  15. logAgent.sinks.hdfsSink.hdfs.rollInterval= 600
  16. logAgent.sinks.hdfsSink.hdfs.roundUnit = minute
  17. logAgent.sinks.hdfsSink.hdfs.fileType = DataStream
  18. logAgent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
  19. logAgent.sinks.hdfsSink.channel = fileChannel
  20.  
  21. logAgent.channels.fileChannel.type = memory
  22. logAgent.channels.logSource.capacity=1000
  23. logAgent.channels.logSource.transactionCapacity=100
  1. bin/flume-ng agent -n logAgent -c conf -f conf/flume_exec_hdfs.conf -Dflume.root.logger=INFO,console

猜你在找的CentOS相关文章