Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 947

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/samsongli/meirendaddy.com/blog/wp-includes/kses.php on line 948
December 9, 2010

//更新:彼岸翻译了第5章的后面部分

6.优化
前面一章描述了BT的实现,我们还需要很多优化工作来获得用户需要的高性能,高可用性和高可靠性.本章描述实现的一些部分,以强调这些优化.

局部性群组

客户可以将多个列族组合成局部性群族.对每个子表中的每个局部性群组都会生成一个单独的SSTable.将通常不会一起访问的列族分割成不同的局部 性群组,将会提高读取效率.例如,Webtable中的网页元数据(语言和校验和之类的)可以在一个局部性群组,网页内容可以在另外一个群组:如果一个应 用要读取元数据,它就没有必要去访问页面内容.

此外,每个群组可以设定不同的调试参数.例如,一个局部性群组可以被设定在内存中.内存中的局部性群组的SSTable在被子表服务器装载进内存的时候,使用的装载策略是懒惰型的.一旦属于该局部性群组的列族被装载进内存,再访问它们就不必通过硬盘了{读不懂?知道机器翻译有多难了吧?人翻译都不行}.这个特性对于需要频繁访问的小块数据特别有用:在BT内部,我们用这个特性来访问元数据表中的地址列族.

压缩

客户可以控制是否压缩一个局部性群组的SSTable.每个SSTable块(块的大小由局部性群组的调试参数确定),都会使用用户指定的压缩格式.尽管这样分块压缩{比全表压缩}浪费了少量空间,但是在读取SSTable的一小部分数据的时候,就不必解压整个文件了{那是,你们文件巨大,硬盘又便宜}. 很多客户使用两遍的订制压缩方式.第一遍是Bentley and McIlroy’s方式[6],该方式在一个大的扫描窗口中将常见的长串进行压缩.第二遍是一种快速压缩算法,在一个16KB的小扫描窗口中寻找重复数 据.两个算法都很快,现有机器上压缩速率为100到200MB/s,解压速率是400到1000MB/s.

尽管我们选择压缩算法的重点是速率,而非空间效率,这种两遍的压缩方式空间效率也令人惊叹{老大,别吹了,您老是在压字符串哪!你去压压运行代码看看?}. 例如,在Webtable,我们用这种压缩方式来存储网页内容.针对实验的目的,我们对每个文档只存储一个版本,而非存储所有能访问的版本.该模式获得了 10比1的压缩率.这比一般的Gzip的3比1或者4比1的HTML页面压缩率好很多,因为Webtable的行是这样安排的:从一个主机获取的页面都存 在临近的地方,这种特性让Bentley-McIlroy算法可以从一个主机那里来的页面里找到大量的重复内容.不仅是Webtable,其他很多应用也 通过选择行名来将类似内容聚集在一起,因此压缩效果非常的好{针对数据和程序特性选择的压缩算法}.当在BT中存储同一数据的多个版本的时候,压缩效率更高.

使用缓存来提高读取性能

为了提高读操作性能,子表服务机构使用两层缓存.扫描缓存是高层,它缓存子表服务器代码从SSTable获取的关键字-值对.块缓存是底层,缓存的 是从GFS读取的SSTable块.对于经常要重复读取同一部分数据的应用程序来说,扫描缓存很有用.对于经常要读取前面刚读过的数据附近的其他数据(例 如,顺序读{性能提升老花招:预取},或者在一个热门的行中的同一局部性群组中,随机读取不同的列)的应用程序来说,块缓存很有用{后面这句比较拗口,是说一个局部性群组里的列都在缓存里,可以随机读取}.

Bloom过滤器{需要读参考文献才知道是什么意思.从标题看,bloom是一种杂凑函数,有相对低的不命中率,所以可以用它来猜测一个关键字对应的存储数据在哪里,从而减少访问硬盘的次数,以提高性能}

如5.3节所述,一个读操作要知道一个子表的所有状态,必须从所有SSTable中读取数据.如果这些SSTable不在内存,那么就需要多次访问 硬盘.我们通过允许客户对特定局部性群组的SSTable指定bloom过滤器[7],来降低访问硬盘的次数.使用bloom过滤器,我们就可以猜测一个 SSTable是否可能包含特定行和列的数据.对于某些特定应用程序,使用少量内存来存储bloom过滤器换来的,是显著减少的访问磁盘次数.我们使用 bloom过滤器也使当应用程序要求访问不存在的行或列时,我们不会访问硬盘.

修改日志{commit-log}的实现

如果我们把对每个子表的修改日志都另存一个文件的话,就会产生非常多的文件,这些文件会并行的写入GFS.根据每个GFS底层实现的细节,这些写操 作在最终写入实际日志文件的时候,可能造成大量的硬盘寻道动作.此外,由于群组不大,为每个子表建立一个新的日志文件也降低了群组修改的优化程度.为了避 免这些问题,我们将对每个子表服务器的修改动作添加到一个修改日志中,将多个子表的修改混合到一个实际日志文件中[18,20].

使用单个日志显著提高了正常使用中的性能,但是将恢复的工作复杂化了.当一个子表服务器死掉时,它以前服务的子表们将会被移动到很多其他的子表服务 器上:它们每个都装载很少的几个子表.要恢复子表的状态,新的子表服务器要按照原来的服务器写的修改日志来重新进行修改.但是,这些修改记录是混合在一个 实际日志文件中的.一种方法是把日志文件都读出来,然后只重复需要进行的修改项.但是,用这种方法,假如有100台机器都装载了要恢复的子表,那么这个日 志文件要读取100次(每个服务器一次).

避免这个问题的方法是先把日志按照关键字排序.在排序以后,所有的修改项都是连续的了,只要一次寻道操作,然后顺序读取.为了并行的排序,我们将日 志分割成64MB的段,并在不同的子表服务器上并行的排序.这个排序工作是由主服务器来协同的,当一个子表服务器表示需要从某些日志文件中开始恢复修改, 这个过程就开始了.

有时,向GFS中写修改日志文件会导致性能不稳定,原因很多(例如,正在写的时候,一个GFS服务器不行了,或者访问某三个GFS服务器的网络路由 断了或者拥塞).为了在GFS延迟的高峰时还能保证修改顺利进行,每个子表服务器实际上有两个线程:各自写不同的日志文件;两个线程里只有一个活跃.如果 一个线程的写操作性能不好,就切换到另外一个线程,修改的记录就写入新的活跃线程的日志文件.每个日志项都有序列号,在恢复的时候,由于线程切换而导致的 重复的序列号将被忽略.

加速子表的恢复

如果主服务器将一个子表从一个子表服务器移动到另外一个服务器,第一个子表服务器对子表进行轻度压缩.该压缩减少了子表服务器的日志文件中没有被紧 缩的状态,从而减少了恢复时间.压缩完成以后,该服务器就停止服务该子表.然后,在卸载该子表前,该服务器再次进行一次(通常很快)轻度压缩,以消除在前 面一次压缩时遗留下来的未紧缩的状态.第二次压缩做完以后,子表就可以被装载到另外一个服务器上,而不必请求从日志中恢复了.

利用不变性{immutability,不可写,可以并行读取}

除了SSTable缓存以外,由于所有生成的SSTable都是不变的,所以BT的很多其他部分都变的简单了.例如,当从SSTable读的时候, 就不必进行同步.这样一来,对行的并行操作就可以非常有效的实现了.内存表是唯一一个被读和写操作同时访问的可变数据结构.为了减少在读操作中对内存表的 竞争,内存表是写复制的,这样一来就可以并行进行读写操作.

因为SSTable是不变的,因此永久消除被删除的数据的问题,就转换成对过时的SSTable进行垃圾收集的问题了.每个子表的SSTable们 都在元数据表进行注册.主服务器对SSTable集合进行标记-扫除的垃圾收集工作[25],元数据表保存了根SSTable集合。

最后,SSTable的不变性使分裂子表的操作更加快速。我们不必为每个分裂出来的子表建立新的SSTable集合,而是让分裂的子表集合共享原来子表的SSTable集合。

3 Comments »

  1. “我々は、商品の品質を確保し、速度を送信するために、主に中国での販売代理店と大手ブランドです。

    Comment by ナイキシューズ — September 19, 2012 @ 11:58 pm

  2. It will look very much creative. Really good blog.

    Comment by Ugg Boots Sale — December 10, 2012 @ 8:12 pm

  3. iceable, gor0uei innovative accompanied by delicate types. Like a an attractive additional Mulberry components, Pandora UK an infant bookbag variety of are just regardin

    Comment by Pandora Jewellery — January 22, 2013 @ 12:04 am

RSS feed for comments on this post. TrackBack URI

Leave a comment 说两句吧