我有一个文本文件与100000对:字和频率.
> 1行 – 所有字频对的总计数
> 2行〜〜100 001 – 字频对
> 100 002行 – 用户输入字的总数
>从100 003到最终用户输入字
我解析这个文件并把这些单词放进去
Dictionary<string,double> dictionary;
for(int i=0;i<15000;i++) { tempInputWord = //take data from file(or other sources) var adviceWords = dictionary .Where(p => p.Key.StartsWith(searchWord,StringComparison.Ordinal)) .OrderByDescending(ks => ks.Value) .ThenBy(ks => ks.Key,StringComparer.Ordinal) .Take(10) .ToList(); //some output }
问题:该代码必须在不到10秒的时间内运行.
在我的电脑(核心i5 2400,8gb RAM)与Parallel.For() – 约91秒.
你能给我一些如何提高性能的建议吗?
更新:
万岁!我们做到了!
谢谢@CodesInChaos,@usr,@T_D和所有参与解决问题的人.
最后的代码:
var kvList = dictionary.OrderBy(ks => ks.Key,StringComparer.Ordinal).ToList(); var strComparer = new MyStringComparer(); var intComparer = new MyIntComparer(); var kvListSize = kvList.Count; var allUserWords = new List<string>(); for (int i = 0; i < userWordQuantity; i++) { var searchWord = Console.ReadLine(); allUserWords.Add(searchWord); } var result = allUserWords .AsParallel() .AsOrdered() .Select(searchWord => { int startIndex = kvList.BinarySearch(new KeyValuePair<string,int>(searchWord,0),strComparer); if (startIndex < 0) startIndex = ~startIndex; var matches = new List<KeyValuePair<string,int>>(); bool isNotEnd = true; for (int j = startIndex; j < kvListSize ; j++) { isNotEnd = kvList[j].Key.StartsWith(searchWord,StringComparison.Ordinal); if (isNotEnd) matches.Add(kvList[j]); else break; } matches.Sort(intComparer); var res = matches.Select(s => s.Key).Take(10).ToList(); return res; }); foreach (var adviceWords in result) { foreach (var adviceWord in adviceWords) { Console.WriteLine(adviceWord); } Console.WriteLine(); }
6 sec (9 sec without manual loop (with linq)))
解决方法
>按键排序列表< KeyValuePair< string,decimal>>替换字典.
对于搜索,我使用一个子字符串直接在其前缀之前进行序数比较.所以我可以用二进制搜索找到第一个候选人.由于候选人是连续的,我可以用TakeWhile代替Where.
int startIndex = dictionary.BinarySearch(searchWord,comparer); if(startIndex < 0) startIndex = ~startIndex; var adviceWords = dictionary .Skip(startIndex) .TakeWhile(p => p.Key.StartsWith(searchWord,StringComparison.Ordinal)) .OrderByDescending(ks => ks.Value) .ThenBy(ks => ks.Key) .Select(s => s.Key) .Take(10).ToList();
>确保对所有操作使用顺序比较,包括初始排序,二进制搜索和StartsWith检查.>我会在并行循环之外调用Console.ReadLine.可能使用AsParallel().在搜索单词的集合上选择(…)而不是Parallel.For.