公司应用es有一段时间,今天接触了一个相对复杂的业务,针对每隔几分钟,几小时,几天进行统计折线图,具体逻辑如下图:
如图,系统必须要支持查询,每小时(每隔10分钟),每日(每隔4小时统计),每周(每隔1日),每月(每隔5日)进行统计,找到最大值显示到折线图上。
首先4张图像使用term聚合,每张图像上有两条线,表示cpu和内存,也属于term聚合方式,整个折线图采用dateHistogram聚合方式。
使用语句如下:
GET /system-audit1/auditEvent/_search { "aggs": { "sales": { "terms": { "field": "psName.keyword" },"aggs": { "type": { "terms": { "field": "type.keyword" },"aggs": { "staticTime": { "date_histogram": { "field": "statisticTime","interval": "4h" },"aggs": { "maxValue": { "max": { "field": "value" } } } } } } } } } }执行结果:
"aggregations": { "sales": { "doc_count_error_upper_bound": 0,"sum_other_doc_count": 0,"buckets": [ { "key": "192.168.1.241:es","doc_count": 7516,"type": { "doc_count_error_upper_bound": 0,"buckets": [ { "key": "cpu","doc_count": 3763,"staticTime": { "buckets": [ { "key_as_string": "2018-01-05T16:00:00.000Z","key": 1515168000000,"doc_count": 2067,"maxValue": { "value": 23.100000381469727 } },{ "key_as_string": "2018-01-05T20:00:00.000Z","key": 1515182400000,"doc_count": 132,"maxValue": { "value": 22.799999237060547 } },{ "key_as_string": "2018-01-06T00:00:00.000Z","key": 1515196800000,"doc_count": 0,"maxValue": { "value": null } }...
java代码实现:
List<SystemDistribution> list = new ArrayList<>(); //统计最终的数据 BoolQueryBuilder boolQueryBuilder=QueryBuilders.boolQuery(); boolQueryBuilder.must(QueryBuilders.rangeQuery("createTime").lte(endTime).gt(startTime)); //createTime是YYYYMMDDHHMMSSSSS格式字符串 DateHistogramInterval dateHistogramInterval=getDateHistogramInterval(timeType); //聚合时间类型 TermsAggregationBuilder termAggregation=AggregationBuilders.terms("psName").field("psName.keyword"); //服务器名称聚合 TermsAggregationBuilder typeAggregation=AggregationBuilders.terms("type").field("type.keyword"); AggregationBuilder timeAggregation = AggregationBuilders .dateHistogram("agg") .field("statisticTime")//统计时间聚合 .dateHistogramInterval(dateHistogramInterval); MaxAggregationBuilder maxAggregation = AggregationBuilders.max("maxValue").field("value");//最大值聚合 timeAggregation.subAggregation(maxAggregation); typeAggregation.subAggregation(timeAggregation); termAggregation.subAggregation(typeAggregation); SearchResponse response = client.prepareSearch(INDEX_NAME).setTypes(TYPE) .setQuery(boolQueryBuilder).addAggregation(termAggregation).execute().actionGet(); Terms genders = response.getAggregations().get("psName"); for (Terms.Bucket entry : genders.getBuckets()) { SystemDistribution systemDistribution=new SystemDistribution(); String psName=entry.getKey().toString(); systemDistribution.setHostName(psName); Terms typeTerm = entry.getAggregations().get("type"); List<RiskStatisticsVo> memRiskStatistics=new ArrayList<>(); List<RiskStatisticsVo> cpuRiskStatisTics=new ArrayList<>(); for (Terms.Bucket entry1 : typeTerm.getBuckets()) { String type = entry1.getKeyAsString(); // Key as String 2017-12-27T00:00:00.000Z Histogram histogram=entry1.getAggregations().get("agg"); for(Histogram.Bucket entry2 : histogram.getBuckets()){ RiskStatisticsVo riskStatisticsVo=new RiskStatisticsVo(); riskStatisticsVo.setRiskType(type); String statisTime=entry2.getKeyAsString(); Max max=entry2.getAggregations().get("maxValue"); Double maxValue=max.getValue(); if(maxValue.equals(Double.NEGATIVE_INFINITY)){ //如果为无穷大,赋值为0 maxValue=0.0; }//-Infinity riskStatisticsVo.setStatisticTime(formatReturnTime(statisTime,timeType));//2018-01-08T11:00:00.000Z riskStatisticsVo.setCount(maxValue.toString()); if("mem".equals(type)){ memRiskStatistics.add(riskStatisticsVo); }else{ cpuRiskStatisTics.add(riskStatisticsVo); } } } systemDistribution.setcpuStatisticList(cpuRiskStatisTics); systemDistribution.setEmeStatisticList(memRiskStatistics); list.add(systemDistribution); } return list;获得时间类型:
private DateHistogramInterval getDateHistogramInterval(String dateType) { if(StatisticTimeTypeEnum.HOUR.getName().equals(dateType)){ return DateHistogramInterval.minutes(10);//统计一个小时内数据,每隔10分钟一个显示 }else if(StatisticTimeTypeEnum.Day.getName().equals(dateType)){ return DateHistogramInterval.hours(4); //统计每日,每隔4小时统计 }else if(StatisticTimeTypeEnum.WEEK.getName().equals(dateType)){ return DateHistogramInterval.days(1); //每周,统计每天的数据统计 }else{ return DateHistogramInterval.days(5); //每月,每隔5天一个统计数据 } }格式化返回时间:
private String formatReturnTime(String time,String dateType){ if(StatisticTimeTypeEnum.HOUR.getName().equals(dateType)){ return time.substring(11,16); }else if(StatisticTimeTypeEnum.Day.getName().equals(dateType)){ return time.substring(8,10)+"日"+time.substring(11,13)+"时"; }else{ return time.substring(8,10)+"日"; } }