博客园_TODAY ^_^

Lucene Boost 精度表

2012-05-22T07:31:00Z

Lucene中索引阶段Boost的值会被存放在.nrm文件中，用16进制打开就能看的到。

每个Boost值最终会被保存于一个Byte中，如果细心的同学就会发现在SetBoost的时候参数类型是float，float是四个字节的，也就是说最后会有精度丢失。
废了方便查询给出代码和0-255对应的float

下面是换算的代码：

public static sbyte FloatToByte315(float f)
{
int num = BitConverter.ToInt32(BitConverter.GetBytes(f), 0);
int num2 = num >> 0x15;
if (num2 < 0x180)
{
if (num > 0) return 1;
return 0;
}
if (num2 >= 640) return -1;
return (sbyte) (num2 - 0x180);
}

public static float Byte315ToFloat(byte b)
{
if (b == 0) return 0f;
int num = (b & 0xff) << 0x15;
num += 0x30000000;
return BitConverter.ToSingle(BitConverter.GetBytes(num), 0);
}

0
5.820766E-10
6.984919E-10
8.149073E-10
9.313226E-10
1.164153E-09
1.396984E-09
1.629815E-09
1.862645E-09
2.328306E-09
2.793968E-09
3.259629E-09
3.72529E-09
4.656613E-09
5.587935E-09
6.519258E-09
7.450581E-09
9.313226E-09
1.117587E-08
1.303852E-08
1.490116E-08
1.862645E-08
2.235174E-08
2.607703E-08
2.980232E-08
3.72529E-08
4.470348E-08
5.215406E-08
5.960464E-08
7.450581E-08
8.940697E-08
1.043081E-07
1.192093E-07
1.490116E-07
1.788139E-07
2.086163E-07
2.384186E-07
2.980232E-07
3.576279E-07
4.172325E-07
4.768372E-07
5.960464E-07
7.152557E-07
8.34465E-07
9.536743E-07
1.192093E-06
1.430511E-06
1.66893E-06
1.907349E-06
2.384186E-06
2.861023E-06
3.33786E-06
3.814697E-06
4.768372E-06
5.722046E-06
6.67572E-06
7.629395E-06
9.536743E-06
1.144409E-05
1.335144E-05
1.525879E-05
1.907349E-05
2.288818E-05
2.670288E-05
3.051758E-05
3.814697E-05
4.577637E-05
5.340576E-05
6.103516E-05
7.629395E-05
9.155273E-05
0.0001068115
0.0001220703
0.0001525879
0.0001831055
0.000213623
0.0002441406
0.0003051758
0.0003662109
0.0004272461
0.0004882813
0.0006103516
0.0007324219
0.0008544922
0.0009765625
0.001220703
0.001464844
0.001708984
0.001953125
0.002441406
0.002929688
0.003417969
0.00390625
0.004882813
0.005859375
0.006835938
0.0078125
0.009765625
0.01171875
0.01367188
0.015625
0.01953125
0.0234375
0.02734375
0.03125
0.0390625
0.046875
0.0546875
0.0625
0.078125
0.09375
0.109375
0.125
0.15625
0.1875
0.21875
0.25
0.3125
0.375
0.4375
0.5
0.625
0.75
0.875
1
1.25
1.5
1.75
2
2.5
3
3.5
4
5
6
7
8
10
12
14
16
20
24
28
32
40
48
56
64
80
96
112
128
160
192
224
256
320
384
448
512
640
768
896
1024
1280
1536
1792
2048
2560
3072
3584
4096
5120
6144
7168
8192
10240
12288
14336
16384
20480
24576
28672
32768
40960
49152
57344
65536
81920
98304
114688
131072
163840
196608
229376
262144
327680
393216
458752
524288
655360
786432
917504
1048576
1310720
1572864
1835008
2097152
2621440
3145728
3670016
4194304
5242880
6291456
7340032
8388608
1.048576E+07
1.258291E+07
1.468006E+07
1.677722E+07
2.097152E+07
2.516582E+07
2.936013E+07
3.355443E+07
4.194304E+07
5.033165E+07
5.872026E+07
6.710886E+07
8.388608E+07
1.006633E+08
1.174405E+08
1.342177E+08
1.677722E+08
2.013266E+08
2.34881E+08
2.684355E+08
3.355443E+08
4.026532E+08
4.69762E+08
5.368709E+08
6.710886E+08
8.053064E+08
9.395241E+08
1.073742E+09
1.342177E+09
1.610613E+09
1.879048E+09
2.147484E+09
2.684355E+09
3.221225E+09
3.758096E+09
4.294967E+09
5.368709E+09
6.442451E+09
7.516193E+09

本文链接

Lucene Document getBoost(float) 和 setBoost(float)

2012-05-15T09:56:00Z

今天写了个单元测试，看看权重的变化，结果发现索引的时候啥都好好的，索引完拿出来看都是1，困惑了好一会，还好lucene的文档给出了解释：

getBoost
public float getBoost()
Returns, at indexing time, the boost factor as set by setBoost(float).
Note that once a document is indexed this value is no longer available from the index. At search time, for retrieved documents, this method always returns 1. This however does not mean that the boost value set at indexing time was ignored - it was just combined with other indexing time factors and stored elsewhere, for better indexing and search performance. (For more information see the "norm(t,d)" part of the scoring formula in Similarity.)

翻译：在索引的时候，激励因子由函数SetBoost（float）设置。注意一旦文档被写入索引，这个值将不再有效（或者说不可逆），在搜索时取出的文档中这个方法总是返回1。然而这并不意味着索引时设置的激励因子被无视-它只是与各种因子合并并存储在了某处，这是为了更好的索引和高效的搜索。

顺便贴上合并的文档：

public abstract class Similarityextends Objectimplements Serializable
Expert: Scoring API.
Similarity defines the components of Lucene scoring. Overriding computation of these components is a convenient way to alter Lucene scoring.
Suggested reading: Introduction To Information Retrieval, Chapter 6.
The following describes how Lucene scoring evolves from underlying information retrieval models to (efficient) implementation. We first brief on VSM Score, then derive from it Lucene's Conceptual Scoring Formula, from which, finally, evolves Lucene's Practical Scoring Function (the latter is connected directly with Lucene classes and methods).
Lucene combines Boolean model (BM) of Information Retrieval with Vector Space Model (VSM) of Information Retrieval - documents "approved" by BM are scored by VSM.
In VSM, documents and queries are represented as weighted vectors in a multi-dimensional space, where each distinct index term is a dimension, and weights are Tf-idf values.
VSM does not require weights to be Tf-idf values, but Tf-idf values are believed to produce search results of high quality, and so Lucene is using Tf-idf. Tf and Idf are described in more detail below, but for now, for completion, let's just say that for given term t and document (or query) x, Tf(t,x) varies with the number of occurrences of term t in x (when one increases so does the other) and idf(t) similarly varies with the inverse of the number of index documents containing term t.
VSM score of document d for query q is the Cosine Similarity of the weighted query vectors V(q) and V(d):

cosine-similarity(q,d) =
V(q) · V(d)
–––––––––
|V(q)| |V(d)|

VSM Score

Where V(q) · V(d) is the dot product of the weighted vectors, and |V(q)| and |V(d)| are their Euclidean norms.
Note: the above equation can be viewed as the dot product of the normalized weighted vectors, in the sense that dividing V(q) by its euclidean norm is normalizing it to a unit vector.
Lucene refines VSM score for both search quality and usability:
Normalizing V(d) to the unit vector is known to be problematic in that it removes all document length information. For some documents removing this info is probably ok, e.g. a document made by duplicating a certain paragraph 10 times, especially if that paragraph is made of distinct terms. But for a document which contains no duplicated paragraphs, this might be wrong. To avoid this problem, a different document length normalization factor is used, which normalizes to a vector equal to or larger than the unit vector: doc-len-norm(d).
At indexing, users can specify that certain documents are more important than others, by assigning a document boost. For this, the score of each document is also multiplied by its boost value doc-boost(d).
Lucene is field based, hence each query term applies to a single field, document length normalization is by the length of the certain field, and in addition to document boost there are also document fields boosts.
The same field can be added to a document during indexing several times, and so the boost of that field is the multiplication of the boosts of the separate additions (or parts) of that field within the document.
At search time users can specify boosts to each query, sub-query, and each query term, hence the contribution of a query term to the score of a document is multiplied by the boost of that query term query-boost(q).
A document may match a multi term query without containing all the terms of that query (this is correct for some of the queries), and users can further reward documents matching more query terms through a coordination factor, which is usually larger when more terms are matched: coord-factor(q,d).
Under the simplifying assumption of a single field in the index, we get Lucene's Conceptual scoring formula:

score(q,d) = coord-factor(q,d) · query-boost(q) ·
V(q) · V(d)
–––––––––
|V(q)|

· doc-len-norm(d) · doc-boost(d)

Lucene Conceptual Scoring Formula

The conceptual formula is a simplification in the sense that (1) terms and documents are fielded and (2) boosts are usually per query term rather than per query.
We now describe how Lucene implements this conceptual scoring formula, and derive from it Lucene's Practical Scoring Function.
For efficient score computation some scoring components are computed and aggregated in advance:
Query-boost for the query (actually for each query term) is known when search starts.
Query Euclidean norm |V(q)| can be computed when search starts, as it is independent of the document being scored. From search optimization perspective, it is a valid question why bother to normalize the query at all, because all scored documents will be multiplied by the same |V(q)|, and hence documents ranks (their order by score) will not be affected by this normalization. There are two good reasons to keep this normalization:
Recall that Cosine Similarity can be used find how similar two documents are. One can use Lucene for e.g. clustering, and use a document as a query to compute its similarity to other documents. In this use case it is important that the score of document d3 for query d1 is comparable to the score of document d3 for query d2. In other words, scores of a document for two distinct queries should be comparable. There are other applications that may require this. And this is exactly what normalizing the query vector V(q) provides: comparability (to a certain extent) of two or more queries.
Applying query normalization on the scores helps to keep the scores around the unit vector, hence preventing loss of score data because of floating point precision limitations.

Document length norm doc-len-norm(d) and document boost doc-boost(d) are known at indexing time. They are computed in advance and their multiplication is saved as a single value in the index: norm(d). (In the equations below, norm(t in d) means norm(field(t) in doc d) where field(t) is the field associated with term t.)
Lucene's Practical Scoring Function is derived from the above. The color codes demonstrate how it relates to those of the conceptual formula:

score(q,d) = coord(q,d) · queryNorm(q) · ∑ ( tf(t in d) · idf(t)² · t.getBoost() · norm(t,d) )
t in q

Lucene Practical Scoring Function

where
tf(t in d) correlates to the term's frequency, defined as the number of times term t appears in the currently scored document d. Documents that have more occurrences of a given term receive a higher score. Note that tf(t in q) is assumed to be 1 and therefore it does not appear in this equation, However if a query contains twice the same term, there will be two term-queries with that same term and hence the computation would still be correct (although not very efficient). The default computation for tf(t in d) in DefaultSimilarity is:

tf(t in d) = frequency^½

idf(t) stands for Inverse Document Frequency. This value correlates to the inverse of docFreq (the number of documents in which the term t appears). This means rarer terms give higher contribution to the total score. idf(t) appears for t in both the query and the document, hence it is squared in the equation. The default computation for idf(t) in DefaultSimilarity is:

idf(t) = 1 + log (
numDocs
–––––––––
docFreq+1

)

coord(q,d) is a score factor based on how many of the query terms are found in the specified document. Typically, a document that contains more of the query's terms will receive a higher score than another document with fewer query terms. This is a search time factor computed in coord(q,d) by the Similarity in effect at search time.

queryNorm(q) is a normalizing factor used to make scores between queries comparable. This factor does not affect document ranking (since all ranked documents are multiplied by the same factor), but rather just attempts to make scores from different queries (or even different indexes) comparable. This is a search time factor computed by the Similarity in effect at search time. The default computation in DefaultSimilarity produces a Euclidean norm:

queryNorm(q) = queryNorm(sumOfSquaredWeights) =
1
––––––––––––––
sumOfSquaredWeights^½

The sum of squared weights (of the query terms) is computed by the query Weight object. For example, a boolean query computes this value as:

sumOfSquaredWeights = q.getBoost() ² · ∑ ( idf(t) · t.getBoost() ) ²
t in q

t.getBoost() is a search time boost of term t in the query q as specified in the query text (see query syntax), or as set by application calls to setBoost(). Notice that there is really no direct API for accessing a boost of one term in a multi term query, but rather multi terms are represented in a query as multi TermQuery objects, and so the boost of a term in the query is accessible by calling the sub-query getBoost().

norm(t,d)encapsulates a few (indexing time) boost and length factors:
Document boost - set by calling doc.setBoost() before adding the document to the index.
Field boost - set by calling field.setBoost() before adding the field to a document.
lengthNorm(field) - computed when the document is added to the index in accordance with the number of tokens of this field in the document, so that shorter fields contribute more to the score. LengthNorm is computed by the Similarity class in effect at indexing.
When a document is added to the index, all the above factors are multiplied. If the document has multiple fields with the same name, all their boosts are multiplied together:

norm(t,d) = doc.getBoost() · lengthNorm(field) · ∏ f.getBoost()
field f in d named as t

However the resulted norm value is encoded as a single byte before being stored. At search time, the norm byte value is read from the index directory and decoded back to a float norm value. This encoding/decoding, while reducing index size, comes with the price of precision loss - it is not guaranteed that decode(encode(x)) = x. For instance, decode(encode(0.89)) = 0.75.

Compression of norm values to a single byte saves memory at search time, because once a field is referenced at search time, its norms - for all documents - are maintained in memory.

The rationale supporting such lossy compression of norm values is that given the difficulty (and inaccuracy) of users to express their true information need by a query, only big differences matter.

Last, note that search time is too late to modify this norm part of scoring, e.g. by using a different Similarity for search.

本文链接

对《LINQ能不能用系列（一）数组筛选效率对比》中测试的几个问题

2012-05-08T10:28:00Z

Stone_W 同学写了一篇《LINQ能不能用系列（一）数组筛选效率对比》

错误一：作为对比测试，测试数组应该为同一个，否则测试数据没有可比性

错误二：对比组中对List的使用不对，List默认创建的数组大小为4，每次增长为4，那么这里就会反复重新创建新的数组次数为log10000000次左右当然会比Linq慢很多

错误三：面对Linq接近0毫秒的处理能力，稍微有点经验的同学就知道这是不可能的，除非是很强很强的计算机，至于为什么后面给出答案，总之linq查询里肯定有猫腻，直接调用tolist()强制返回结果再说;//这里Stone W在评论中对ToList有质疑，我之所以ToList是为了和第二组进行对比，因为第二组得到的结果是一个List，我很奇怪，这次的对比测试到底是为了测试得到两个结果集的算法对比呢还是测试Count算法的对比呢？如果是前者，一个拿到的是IEnumerable的对象一个是List对象，牛跟闹钟怎么对比哪个跑的快呢？也只有在调用ToList的时候才会真正执行Linq的算法也就是下面的嵌套类WhereListIterator；当然如果是为了进行Count对比的话那么对比组二中的算法真的有点拙劣，我想不会有谁会用方法二来统计。

下面是修改了如上三个明显错误后的代码，如果哪位同学有补充欢迎留言：

[Fact]
public void LinqTest()
{
TestLinq(1);
TestLinq(2);
TestLinq(3);
}

public void TestLinq(int time)
{
const int listCount = 10000000; // 数组长度
Random random = new Random(); // 数据随机构建值

// 数组构建
List<int> list1 = new List<int>();
for (int i = 0; i < listCount; i++)
{
list1.Add(random.Next(10000));
}

// 效率测试内容：提取数组中数值大于的100的数组

// LINQ 测试
Stopwatch linq_Stopwatch = new Stopwatch();
linq_Stopwatch.Start();

var linqList = (from num in list1
where num > 100
select num).ToList();
linq_Stopwatch.Stop();
// 普通方式测试
Stopwatch before_Stopwatch = new Stopwatch();
before_Stopwatch.Start();

List<int> beforeList = new List<int>(10000000);
for (int i = 0; i < list1.Count(); i++)
{
if (list1[i] > 100)
beforeList.Add(list1[i]);
}
before_Stopwatch.Stop();

Console.WriteLine(
String.Format("第{0}次测试,测试：{5}条数据。\n\r \t LINQ用时：{1}毫秒,筛选了{2}条数据。\n\r\t 普通用时：{3}毫秒,筛选了{4}条数据。\r\n",
time, linq_Stopwatch.ElapsedMilliseconds, linqList.Count(),
before_Stopwatch.ElapsedMilliseconds, beforeList.Count(), listCount));

}

测试结果：

第1次测试,测试：10000000条数据。
LINQ用时：448毫秒,筛选了9898832条数据。
普通用时：437毫秒,筛选了9898832条数据。

第2次测试,测试：10000000条数据。
LINQ用时：516毫秒,筛选了9899569条数据。
普通用时：460毫秒,筛选了9899569条数据。

第3次测试,测试：10000000条数据。
LINQ用时：608毫秒,筛选了9899231条数据。
普通用时：470毫秒,筛选了9899231条数据。

结论：Linq在实现灵活性提高编写效率的时候牺牲了一定的性能，当然这个是必须的，有的必有失嘛。

我的选择：绝大部分时候使用Linq，在对性能要求高的时候使用普通的迭代；

0毫秒的秘密：

var linqList = (from num in list1 where num > 100 select num)

先看看这个LinqList的类型（Console.WriteLine(linqList.GetType().FullName);）：System.Linq.Enumerable+WhereListIterator`1[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]

可以看到这是一个嵌套类，作用是对Where 条件进行迭代操作，贴上它的源代码：

class WhereListIterator<TSource> : Iterator<TSource>
{
List<TSource> source;
Func<TSource, bool> predicate;
List<TSource>.Enumerator enumerator;

public WhereListIterator(List<TSource> source, Func<TSource, bool> predicate) {
this.source = source;
this.predicate = predicate;
}

public override Iterator<TSource> Clone() {
return new WhereListIterator<TSource>(source, predicate);
}

public override bool MoveNext() {
switch (state) {
case 1:
enumerator = source.GetEnumerator();
state = 2;
goto case 2;
case 2:
while (enumerator.MoveNext()) {
TSource item = enumerator.Current;
if (predicate(item)) {
current = item;
return true;
}
}
Dispose();
break;
}
return false;
}

public override IEnumerable<TResult> Select<TResult>(Func<TSource, TResult> selector) {
return new WhereSelectListIterator<TSource, TResult>(source, predicate, selector);
}

public override IEnumerable<TSource> Where(Func<TSource, bool> predicate) {
return new WhereListIterator<TSource>(source, CombinePredicates(this.predicate, predicate));
}
}

真相大白于天下。

ps：下面是原文代码的截图

本文链接

几种Lucene.Net打开IndexReader的方式

2012-05-03T10:27:00Z

入门第一式：

IndexReader.Open(Dir,readOnly);

当readOnly为true的时候效率最高，最好不要用Reader删除修改记录；

入门第二式：

reader.Reopen(readOnly);

优点：比第一式效率高，推荐使用；

缺点：前提是需要有Reader存在；

入门第三式：

writer.GetReader();

优点：NRL的实现，对实时性要求高的不二选择；

缺点：writer说它很痛苦，因为每次调用内部都会flush一次以保证所有index会被启用；

追求效率（中阶）：

public class IndexReaderProxy
{
private IndexReader _indexReader;
private readonly object _indexReaderLock = new object();

public IndexReaderProxy(Directory directory, bool readOnly)
{
_indexReader = IndexReader.Open(directory, readOnly);
}

public IndexReader GetCurrentIndexReader()
{
ReopenIndexReaderIfNotCurrent();
return _indexReader;
}

private void ReopenIndexReaderIfNotCurrent()
{
if (_indexReader.IsCurrent()) return;
lock (_indexReaderLock)
{
if (_indexReader.IsCurrent()) return;
var newIndexReader = _indexReader.Reopen();
_indexReader.Close();
_indexReader = newIndexReader;
}
}
}

优点：效率和实时的一个平衡，在对实时要求不高的地方使用；

缺点：不属于NRL范畴，对实时性要求较高的地方就无法使用了；

本文链接

缓存管理的帮助类（更新clear）

2012-04-20T14:46:00Z

public class DictionaryCacheManager<TK,TV>
{
private Dictionary<TK, TV> cacheDic = new Dictionary<TK,TV>();
private Dictionary<TK, object> lockDic = new Dictionary<TK, object>();

public TV Get(TK key,Func<TV> getValue)
{
if(!CheckKey(key))
{
CheckLock(key);
}

if(!CheckKey(key))
{
lock(lockDic[key])
{
if(!CheckKey(key)||cacheDic[key]==null)
{
cacheDic[key] = getValue();
}
}
}
return cacheDic[key];
}

private object _lock = new object();
private void CheckLock(TK key)
{
lock(_lock)
{
if(!lockDic.ContainsKey(key))
{
lockDic.Add(key, new object());
}
}
}

private bool CheckKey(TK key)
{
return cacheDic.ContainsKey(key);
}

public void Clear()
{
lock(_lock)
{
lockDic.Clear();
cacheDic.Clear();
}
}
public void Clear(TK key)
{
lock(lockDic[key])
{
cacheDic.Remove(key);
}
}
}

很简单

本文链接

未处理的异常导致基于 ASP.NET 的应用程序意外退出

2012-04-19T05:14:00Z

问题具体见：http://q.cnblogs.com/q/34874/

相应站点的w3wp.exe 会意外终止

EventLog中的信息：

An unhandled exception occurred and the process was terminated.
Application ID: DefaultDomain
Process ID: 43644
Exception: System.Runtime.Serialization.SerializationException
Message: Unable to find assembly 'Lucene.Net, Version=2.9.4.1, Culture=neutral, PublicKeyToken=85089178b9ac3181'.
StackTrace: at System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryAssemblyInfo assemblyInfo, String name)
at System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA, Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo, SizedArray assemIdToAssemblyTable)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.AppDomain.Deserialize(Byte[] blob) at System.AppDomain.UnmarshalObject(Byte[] blob)

具体原因和解决方案如下：

http://support.microsoft.com/kb/911816

出现这个问题的原因在于未处理异常的默认策略在 .NET Framework 2.0 中已发生更改。默认情况下，未处理异常的策略是结束工作进程。
在 Microsoft .NET Framework 1.1 和 Microsoft .NET Framework 1.0 中，会忽略托管线程上的未处理异常。除非附加调试程序以捕获异常，否则您可能意识不到出错。
ASP.NET 在 .NET Framework 2.0 中使用未处理异常的默认策略。引发未处理的异常时，基于 ASP.NET 的应用程序将会意外退出。
如果在请求上下文中出现异常，则不会发生上述行为。这类异常仍由 HttpException 对象进行处理和包装。在请求上下文中出现的异常不会导致工作进程结束。但是，请求上下文之外的未处理异常（如计时器线程上或回调函数中的异常）会导致工作进程结束。

如果嫌麻烦用已经做好的msi程序安装httpmodule：http://unhandledexception.codeplex.com/

本文链接

[转]Lucene 性能优化带数据

2012-04-12T10:35:00Z

虽然是很久了的数据，还是有很好的参考价值的：

lucene.commit.batch.size=0
lucene.commit.time.interval=0

These properties allow commits in batch, you can either set how many document changes a batch will contain (commit will happen after X docs are modified) or set a time interval in milliseconds (commit will happen every X milliseconds).

lucene.buffer.size=16

This will call IndexWriter's setRAMBufferSizeMB method, this the max memory in megabytes used by Lucene before flushing documents, a higher number means less disk writes.

These 3 new properties will make reindexing faster, I've made stress tests and I could perceive a 30% improvement in certain configurations, I also learned the following while tweaking Lucene properties:

A higher lucene.buffer.size helps a lot during reindex, but there is a up bound limit, over that limit, reindex won't get faster, for example, setting to 32MB or 48MB gave the same results, but 32MB was much better than the default, 16MB.

It's a bad idea to set lucene.merge.factor to a very high number, there will be much more disk write accesses and this will degrade performance, keep it at 10.

Set lucene.autocommit.documents.interval to the number of documents you have in the index, this means only 1 commit will happen. I thought this property would bring better performance results, but it made it faster around 15% only.

Setting a higher lucene.optimize.interval can make some improvement, but since the reindex process also make searches, it's important that you also optimize the index often during reindex, you need to find a balance.

My stress tests consisted in reindexing 30.000 blog entries, I tested on a Intel Quadcore 2.66GHz, 4MB of RAM, Ubuntu 32 bits. The best result was around 4 minutes with this configuration:

lucene.commit.time.interval=30000
lucene.merge.factor=10
lucene.optimize.interval=10000
lucene.buffer.size=48

The worst result was 7:35 minutes (not considering the one I set lucene.merge.factor to 1000):

lucene.commit.time.interval=0
lucene.merge.factor=50
lucene.optimize.interval=1000
lucene.buffer.size=16

Here are the other results:

lucene.commit.time.interval=0
lucene.merge.factor=15
lucene.optimize.interval=30000
lucene.buffer.size=16
7:30 minutes

lucene.commit.time.interval=0
lucene.merge.factor=10
lucene.optimize.interval=100
lucene.buffer.size=16
7:18 minutes

lucene.commit.time.interval=10000
lucene.merge.factor=10
lucene.optimize.interval=100
lucene.buffer.size=16
06:23 minutes

lucene.commit.time.interval=1000
lucene.merge.factor=10
lucene.optimize.interval=100
lucene.buffer.size=16
6:00 minutes

lucene.commit.time.interval=30000
lucene.merge.factor=10
lucene.optimize.interval=100
lucene.buffer.size=32
5:00 minutes

lucene.commit.time.interval=30000
lucene.merge.factor=10
lucene.optimize.interval=100
lucene.buffer.size=48
5:00 minutes

lucene.commit.time.interval=30000
lucene.merge.factor=10
lucene.optimize.interval=50000
lucene.buffer.size=48
5:00 minutes

lucene.commit.time.interval=15000
lucene.merge.factor=10
lucene.optimize.interval=10000
lucene.buffer.size=48
5:00 minutes

lucene.commit.time.interval=30000
lucene.merge.factor=10
lucene.optimize.interval=1000
lucene.buffer.size=48
4:30 minutes

本文链接

【转】URL短地址压缩算法微博短地址原理解析

2012-04-07T16:20:00Z

引自 http://www.nowamagic.net/webdesign/webdesign_ShortUrlInTwitter.php

短网址应用已经在全国各大微博上开始流行了起来。例如QQ微博的url.cn，新郎的sinaurl.cn等。

我们在QQ微博上发布网址的时候，微博会自动判别网址，并将其转换，例如：http://url.cn/2hytQx

为什么要这样做的，原因我想有这样几点：

微博限制字数为140字一条，那么如果我们需要发一些连接上去，但是这个连接非常的长，以至于将近要占用我们内容的一半篇幅，这肯定是不能被允许的，所以短网址应运而生了。
短网址可以在我们项目里可以很好的对开放级URL进行管理。有一部分网址可以会涵盖色情，暴力，广告等信息，这样我们可以通过用户的举报，完全管理这个连接将不出现在我们的应用中，应为同样的URL通过加密算法之后，得到的地址是一样的。
我们可以对一系列的网址进行流量，点击等统计，挖掘出大多数用户的关注点，这样有利于我们对项目的后续工作更好的作出决策。

其实以上三点纯属个人观点，因为在我接下来的部分项目中会应用到，所以就了解了一下，下面先来看看短网址映射算法的理论（网上找到的资料）：

将长网址md5生成32位签名串，分为4段，每段8个字节；
对这四段循环处理，取8个字节，将他看成16进制串与0x3fffffff(30位1)与操作，即超过30位的忽略处理；
这30位分成6段，每5位的数字作为字母表的索引取得特定字符，依次进行获得6位字符串；
总的md5串可以获得4个6位串；取里面的任意一个就可作为这个长url的短url地址；

很简单的理论，我们并不一定说得到的URL是唯一的，但是我们能够取出4组URL，这样几乎不会出现太大的重复。

下面来看看程序部分：

[c-sharp:nogutter] view plain copy

public static string[] ShortUrl(string url)
{
//可以自定义生成MD5加密字符传前的混合KEY
string key = "Leejor";
//要使用生成URL的字符
string[] chars = new string[]{
"a","b","c","d","e","f","g","h",
"i","j","k","l","m","n","o","p",
"q","r","s","t","u","v","w","x",
"y","z","0","1","2","3","4","5",
"6","7","8","9","A","B","C","D",
"E","F","G","H","I","J","K","L",
"M","N","O","P","Q","R","S","T",
"U","V","W","X","Y","Z"
};
//对传入网址进行MD5加密
string hex = System.Web.Security.FormsAuthentication.HashPasswordForStoringInConfigFile(key + url, "md5");
string[] resUrl = new string[4];
for (int i = 0; i < 4; i++)
{
//把加密字符按照8位一组16进制与0x3FFFFFFF进行位与运算
int hexint = 0x3FFFFFFF & Convert.ToInt32("0x" + hex.Substring(i * 8, 8), 16);
string outChars = string.Empty;
for (int j = 0; j < 6; j++)
{
//把得到的值与0x0000003D进行位与运算，取得字符数组chars索引
int index = 0x0000003D & hexint;
//把取得的字符相加
outChars += chars[index];
//每次循环按位右移5位
hexint = hexint >> 5;
}
//把字符串存入对应索引的输出数组
resUrl[i] = outChars;
}
return resUrl;
}

现在可以直接使用该方法，可以等到下面四组值：

[java:nogutter] view plain copy

ShortUrl(http://www.me3.cn)[0]; //得到值fAVfui
ShortUrl(http://www.me3.cn)[1]; //得到值3ayQry
ShortUrl(http://www.me3.cn)[2]; //得到值UZzyUr
ShortUrl(http://www.me3.cn)[3]; //得到值36rQZn

在存放这个URL的数据方面，我个人推荐TTServer，有的朋友可以没有听说过，下面是这个数据库的介绍：

Tokyo Cabinet 是日本人 Mikio Hirabayashi（平林幹雄）のページ开发的一款DBM数据库(注：大名鼎鼎的DBM数据库qdbm就是他开发的)，该数据库读写非常快。insert:0.4sec/1000000 recordes(2500000qps)，写入100万数据只需要0.4秒。search:0.33sec/1000000 recordes (3000000 qps)，读取100万数据只需要0.33秒。

可以看到对于字典类型的数据Key/Value的查询，这个数据库可以说是我目前见过效率非常高的，况且他如此的小巧，用来对short url/long url的配对再好不过。

该系统使用6个短码字符来表示任何长度的网址。有效的字符代码是ASCII 'A'到'Z'和'0'的'5'，其中每个字符包含2 ^ 5（32）状态。 6短码字符可用于绘制32 ^ 6（1073741824）的网址

首先，你需要一个数据库表来存储和检索你映射的网址。

[c-sharp:nogutter] view plain copy

CREATE TABLE mappedURL (的CREATE TABLE mappedURL（
shortCode char(6) not null,
lognURL text not null,
PRIMARY KEY shortCodeInd (shortCode),
);

其次，你需要定义一个算法将长的URL映射到短的URL。算法上面已经介绍过了。

第三，你需要创建一个网页，从数据库的短网址的映射找到原始的URL,并重定向之。

本文链接

QueryParse后的查询：“中华共和国”查不到，但是“中华人民共和国”查的到

2012-04-05T06:44:00Z

我想被这个东西拌到的人估计不少，问题就出在QueryParse会将分词结果用语法查询PhraseQuery，而语法查询默认slop是0（关于slop请看：http://www.blogjava.net/freeman1984/archive/2011/02/25/345116.html）。

泪奔，一直以为索引的问题，感谢luke：http://luke.codeplex.com/

下面是一个分解查询设置slop为99的帮助方法 :

        private void SetSlop(Query q)
        {
            var bQuery = q as BooleanQuery;
            if (bQuery != null)
            {
                foreach (var bq in bQuery.Clauses())
                {
                    var query = bq as BooleanClause;
                    if(query!=null)
                    {SetSlop(query.GetQuery());}
                }
                return;
            }
            var pQuery = q as PhraseQuery;
            if (pQuery != null)
            {
                pQuery.SetSlop(99);
            }
        }

本文链接

Dot NET 内存泄漏

2012-04-01T05:33:00Z

因为正好解决了手头一个泄漏的问题，正好做个总结。一开始我认为托管代码是不存在泄漏问题的（不使用指针，不操作unsafe的代码），我想跟我这样想的人应该不少，呵呵。

传统的内存泄漏主要是因为使用后没有释放造成的（维基解释http://en.wikipedia.org/wiki/Memory_leak），而在托管平台下内存的回收是由gc完成，所以托管中出现内存泄漏一般都是因为gc无法分辨对象是否可以被回收造成的，而最常见的就是循环引用，另外委托也是重灾区，4.0中引入的task也成为新的增长点（插播新闻http://news.cnblogs.com/n/124603/）。
（插播新闻：

在我的代码中大量使用了task，造成泄漏是因为我使用了TaskCreationOptions.AttachedToParent，它会保持引用关系而不被回收。

本文链接

`sumOfSquaredWeights` = `q.getBoost()` ² ·	∑	( idf(t) · t.getBoost() ) ²
	t in q

norm(t,d) = `doc.getBoost()` · `lengthNorm(field)` ·	∏	`f.getBoost`()
	field f in d named as t

博客园_TODAY ^_^

Lucene Boost 精度表

Lucene Document getBoost(float) 和 setBoost(float)

对《LINQ能不能用系列（一）数组筛选效率对比》中测试的几个问题

几种Lucene.Net打开IndexReader的方式

缓存管理的帮助类（更新clear）

未处理的异常导致基于 ASP.NET 的应用程序意外退出

[转]Lucene 性能优化带数据

【转】URL短地址压缩算法 微博短地址原理解析

QueryParse后的查询：“中华 共和国”查不到，但是“中华人民共和国”查的到

Dot NET 内存泄漏

【转】URL短地址压缩算法微博短地址原理解析

QueryParse后的查询：“中华共和国”查不到，但是“中华人民共和国”查的到