博客园_feisky

解决linux下/etc/rc.local开机器不执行的原因

2012-04-27T04:39:00Z

前不久因项目需要写了开机启动其他程序的shell脚本,因工作忙，调试完给了技术支持人员，也没去注意过。

到后来，有几台服务器突然被重启了，这时候领导问，怎么开机启动的脚本没起作用，还被批了一顿，哎，做事虎头蛇尾,活该！

然后去查，一直没查到原因，

手动执行都是正确的，在排除了权限原因和脚本问题后，很郁闷，怎么放到开机启动项目中就无法执行了呢。

于是去查linux开机启动过程信息。google下，看了下关于一些开机启动启的资料，发现了开机启动程序的级别为3.

果断顺藤莫瓜，找到了这个目录 /etc/rc.d/rc3.d，进去一看恍然大悟。

我的rc.local文件是在 /etc/rc.local中（网上都是说写在这里，我是按照网上来弄的），而我查看了系统指向rc.local文件的路径是 /etc/rc.d/rc.local

后来把开机启动的脚本配置到/etc/rc.d/rc.local中（或者把S99local指向 /etc/rc.local也行），重启检查，成功！

总结:主要还是对linux系统不熟悉，这是个漫长的过程，有时间得多看看这方面的书籍。

原文：http://www.cnblogs.com/montya/archive/2011/09/09/2172001.html

本文链接

关于dd命令输出结果无法重定向到文件的问题

2012-04-27T03:20:00Z

先以time为例：

解决time命令输出信息的重定向问题

time命令的输出信息是打印在标准错误输出上的，我们通过一个简单的尝试来验证一下。

[root@web186 root]# time find . -name "mysql.sh" >1.txt

real    0m0.081s
user    0m0.060s
sys     0m0.020s
[root@web186 root]# time find . -name "mysql.sh" 2>2.txt
./work186/sms/bin/mysql.sh
./work186/sms/src/scripts/mysql.sh
./work186/sms/src/scripts1/mysql.sh

real    0m0.068s
user    0m0.040s
sys     0m0.030s

通过上面的尝试，发现无法将time的输出信息重定向到文件里面，为什么？因为time是shell的关键字，shell做了特殊处理，它会把time命令后面的命令行作为一个整体来进行处理，在重定向时，实际上是针对后面的命令来的，time命令本身的输出并不会被重定向的。那现在怎么办呢？网上提供了两种解决方法【2，3】，我们一一尝试一下。

第一种解决方法，就是将time命令和将要执行的命令行放到一个shell代码块中，也就是一对大括号中，要注意空格和分号的使用。
[root@web186 root]# {time find . -name "mysql.sh"} 2>2.txt

好像成功了。慢，看一下对不对。
[root@web186 root]# cat 2.txt
-bash: {time: command not found

原来bash把 {time 作为一个整体来处理了，前后都加上空格试试。
[root@web186 root]# { time find . -name "mysql.sh" } 2>2.txt
> Ctrl+C

这次Bash认为命令都没有输入完成，少了分号。因为Bash认为后面的 } 是find命令的参数。
[root@web186 root]# { time find . -name "mysql.sh"; } 2>2.txt
./work186/sms/bin/mysql.sh
./work186/sms/src/scripts/mysql.sh
./work186/sms/src/scripts1/mysql.sh
./work186/sms1/bin/mysql.sh
./work186/sms1/src/scripts/mysql.sh
./temp/sms/bin/mysql.sh
./temp/sms/src/scripts/mysql.sh
[root@web186 root]# cat 2.txt

real    0m0.068s
user    0m0.030s
sys     0m0.040s

第一种方式的尝试成功了，总结起来就是 { time command-line; } 2>file 注意分隔符的使用。

另外一种方式就是使用子Shell的方式，如下所示：

[root@web186 root]# (time find . -name "mysql.sh") 2>2.txt
./work186/sms/bin/mysql.sh
./work186/sms/src/scripts/mysql.sh
./work186/sms/src/scripts1/mysql.sh
./work186/sms1/bin/mysql.sh
./work186/sms1/src/scripts/mysql.sh
./temp/sms/bin/mysql.sh
./temp/sms/src/scripts/mysql.sh
[root@web186 root]# cat 2.txt

real    0m0.083s
user    0m0.040s
sys     0m0.020s
[root@web186 root]#

第二种方式的尝试也成功了，总结起来就是 (time command-line) 2>file 这里time紧贴着小括号(也可以的，命令行结束也不必带分号。当然最好还是用第一种方式，毕竟启动一个子shell是要多占些资源的。

针对dd命令。同time命令，dd命令包括了time命令。

故 dd if=/dev/zero of=/mnt/test bs=1k count=100 >>output.txt

此时并不能把dd命令显示的时间和速度保存到output.txt里。因为dd命令的输出是到标准错误输出上的。

必须采用dd if=/dev/zero of=/mnt/test bs=1k count=100 2>>output.txt 才可以。

转自：http://dylan602.blog.163.com/blog/static/167974276201172121823753/

本文链接

一个诡异的bash问题

2012-04-24T06:44:00Z

一个诡异的bash问题

今天写的一个bash脚本运行发现：
test.sh: 1: #!/bin/bash: not found

本以为是由于在windows下编辑导致的问题，执行dos2unix之后问题依旧。

后来想到用file命令查看，文件居然是UTF-8 Unicode (with BOM) text格式

# file test.sh
test.sh: UTF-8 Unicode (with BOM) text

而正常的bash脚本格式应该为 Bourne-Again shell script text executable

用vim -b test.sh删除开头的<feff>即可。

本文链接

用 grep 恢复误删的文本文件

2012-04-16T02:54:00Z

作为长期的电脑使用者，肯定会有误删文件的经历，在 Mac OS X 和 Windows 上删除的文件都会默认进 “回收站”。在 Linux 上如果事先没有用别名（alias）修改默认的 rm 功能，rm 后文件就会丢失，幸运的是，在一般的删除文件操作中，Linux 并不会立即清空存储该文件的 block 内容，而只会释放该文件所占用的 inode 和 data block，Linux 上 rm 的过程其实就是将 inode bitmap 和 data block bitmap 中的相应标识设为空闲状态的过程，并不涉及到真正的数据，这也是为什么在 Linux 下删除大文件这么快速的原因，因为大文件所占的磁盘根本就没有清空。所以，如果我们能找到文件对应的 inode，由此查到相应的 data block，就可能从磁盘上把已删除的文件恢复出来，很多文件/磁盘恢复工具都是这么做的。

一般来说二进制文件、库文件等删除后都可以从其他 Linux 拷贝恢复，不是很要紧，如果自己辛苦写的脚本或者配置文件误删了就杯具了。误删这种蠢事经常在 VPSee 身上发生，最近一次是在今年年初的时候，开了 screen 多次 ssh 到不同服务器上，在前后切换的过程中删除了一个很肯定的配置文件，后来发现删的文件是对的，可惜在一台错误的服务器上，本来应该在 A 服务器上删除 file.txt 结果在 B 服务器上删除了 file.txt，多个 screen 和 ssh 把自己搞晕了，而且主机名没有安排好，多台机器都是用的 localhost，不利于识别当时的环境。如果使用 Linux 的时候不幸误删了一个文本文件怎么办呢？

先临时建一个文本文件 vpsee.log 做测试，然后删除这个文件：

$ echo "important log file for vpsee.com" > vpsee.log

$ cat vpsee.log
important log file for vpsee.com

$ rm vpsee.log

如果能记住刚才删除文件中的某个关键字的话可以用 grep 搜索整个 /dev/sda1，-a 标志位的意思是把 /dev/sda1 这个分区看成是文本形式的（分区本身是二进制形式的），-B 10 -A 100 的意思是如果找到关键字就打印出其前10行和后100行的内容：

# grep -a -B 10 -A 100 'vpsee.com' /dev/sda1 > tmp.txt

在一堆 @ 之间可以找到我们刚才删除的内容：

$ vi tmp.txt
...
@$^@^@^@^@^@^@^@^@^@^@^@^A^@^@^@^@^@^@^@Ç^@^@^@^A^@^@^@^@^@^@^@^@^@^@^@È^K^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^A^@^@^@^@^@^@^@^Q^@^@^@^C^@^@^@^@^@^@^@^@^@^@^@È^K^@^@×^@
^@^@important log file for vpsee.com
@
@
...

当然，如果误删了二进制、doc、png/jpg/gif 之类的文件，可以用一些第三方 ext2/ext3 文件恢复工具帮助恢复文件，比如 TestDisk, PhotoRec 等。

原文：http://www.vpsee.com/2010/08/using-grep-to-recover-text-files/

本文链接

转一篇Linux可用内存的统计方法

2012-04-14T12:11:00Z

背景
由于Linux缓存机制的设计，系统对缓存的使用是非常狠的，所以经常会看到某些环境内存只剩几十兆了，而应用只用了不到一半。所以在计算可用内存的时候，一定要算上缓存的部分。
通常方法，就是通过free命令首行free+cached+buffers计算，或者直接使用第二行的free字段。但这个方法有时仍然会造成比较大的误差，导致性能监控等方面的问题。
比如系统中使用了大量的共享内存会造成多计算可用内存；再比如对大量的文件做了查询（find / ?!!!），会导致少计算可用内存。对于这点我在《说说free命令》中有详细的说明。这里就不再赘述了。

SUSE11 SP1基于2.6.32内核，内核暴露了更多的统计接口给用户空间，把slab分为可回收和不可回收两类指标来统计。free命令也对应做了修改。解决了free命令少计算可用内存的问题。但多计算的问题还是存在。
因此，在这里对统计可用内存的方法做了个总结。供需要的同学参考。（后面有空可能会开发一个统计工具）
其中SUSE10由于内核版本过低（2.6.16）暴露信息不足，下面的方法仍然不能很精确，但相比通过free命令简单统计而言，一般不会造成比较大的误差。

可用内存定义
包括未被使用的空闲内存，以及已经被使用但用作缓存可以自动回收的部分。

SUSE 10可用内存统计方法
总内存：free命令首行total字段。
空闲内存：free命令首行free字段。
缓存：free命令首行buffers字段+cached字段。
修正值1：cached字段包含了共享内存和tmpfs内存文件系统占用的内存。需要减去这两部分。这两部分内存可通过ipcs -m -u和df 命令获取。
修正值2：cached字段漏掉了内核slab中可以自动回收的内存，比如xxx_inode_cache和dentry_cache。这两部分的内存的计算方法是解析/proc/slabinfo。
最终的可用内存计算方法：
空闲内存+缓存-修正值1+修正值2

SUSE 11可用内存统计方法
总内存：free命令首行total字段。
空闲内存：free命令首行free字段。
缓存：free命令首行buffers字段+cached字段。
修正值1：cached字段包含了共享内存和tmpfs内存文件系统占用的内存。需要减去这两部分。这两部分内存之和可通过/proc/meminfo的Shmem字段直接获取。
最终的可用内存计算方法：
空闲内存+缓存-修正值1

附1：ipcs获取共享内存占用物理内存大小
# ipcs -m -u
------ Shared Memory Status --------
segments allocated 4
pages allocated 786433
pages resident 2 #使用这个字段，单位是页，X86下一个页是4kB
pages swapped 0
Swap performance: 0 attempts 0 successes

附2：df获取tmpfs占用物理内存大小
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 20972152 4427900 16544252 22% /
devtmpfs 24711780 160 24711620 1% /dev #Used字段表示实际占用物理内存
tmpfs 24711780 0 24711780 0% /dev/shm #Used字段表示实际占用物理内存
/dev/sda5 1052184 59188 992996 6% /boot
/dev/sda9 83888824 16852500 67036324 21% /iso
/dev/sda7 10490040 1142876 9347164 11% /opt
/dev/sda10 83888824 9421200 74467624 12% /src
/dev/sda8 20972152 3475104 17497048 17% /usr
/dev/sda6 5245016 328392 4916624 7% /var

///////////////////增加统计工具////////2011.6.29///////////////////////////////////////////////////
统计工具(在SUSE10、SUSE11验证通过)
1. 修正可用内存多计算的情况（SUSE10 SUSE11）
如下环境，在进行了大量共享内存创建使用后，free命令统计可用内存，出现了很大的误差。
#echo 3 >/proc/sys/vm/drop_caches #清理可回收的内存
# free
total used free shared buffers cached
Mem: 49423560 13609680 35813880 0 26764 11686608 #仍有11G不可以回收，因为是共享内存，不具备回收属性。
-/+ buffers/cache: 1896308 47527252 # free计算可用内存，算入了共享内存，得到47G
Swap: 2104472 0 2104472
# ipcs -m -u
------ Shared Memory Status --------
segments allocated 20
pages allocated 4980737
pages resident 2883591 #共享内存有2883591 * 4 = 11534364 kB ~= 11G
pages swapped 0
Swap performance: 0 attempts 0 successes
# afree
Total: 49423560 kB #物理内存总计
Free: 35816908 kB #未被使用的内存
Reclaimable: 179348 kB #被使用了但是可以自动回收的内存
Available: 35996256 kB #afree统计出来的可用内存去除了11G的共享内存。

2. 修正可用内存少计算的情况（SUSE10）
如下环境，在进行了大量文件访问操作后，系统中缓存了大概600M的inode和dentry。
# free
total used free shared buffers cached
Mem: 3987316 1080908 2906408 0 163056 163848
-/+ buffers/cache: 754004 3233312 #没有统计slab中的可以回收的inode和dentry。
Swap: 2104472 0 2104472
# afree
Total: 3987316 kB
Free: 2906384 kB
Reclaimable: 958888 kB
Available: 3865272 kB #afree统计了slab中的可以回收的inode和dentry。
# echo 3 >/proc/sys/vm/drop_caches #释放600M的inode和dentry缓存。
# free
total used free shared buffers cached
Mem: 3987316 93628 3893688 0 17780 8948
-/+ buffers/cache: 66900 3920416 #执行drop_caches后，增加了600M，说明cached少统计了可回收的slab缓存。这600M即使不执行drop_caches也都是可用的，系统在需要的时候会自动回收。

注：afree通过解析slabinfo文件统计可回收的slab，其实suse10的内核有一个全局变量slab_reclaim_pages维护了准确的可回收数量，但没有暴露给给用户空间，因此更准确的方法其实就是写一个模块，把这个变量导出到来。

afree代码如下：

#!/bin/sh
#
#Get accurate available memory.
#

function get_meminfo()
{
   grep -w $1 /proc/meminfo | awk -F' ' '{print $2}'
}
function show_meminfo()
{
   printf "%s\t%10d kB\n" "$1" "$2"
}

PAGE_SIZE=4 #kB, for x86
function get_shmem_from_ipcs()
{
   local _shm=0
   _shm=$(ipcs -m -u | grep 'pages resident' | awk -F' ' '{print $3}')
   echo $((_shm * PAGE_SIZE))
}
function get_tmpfs_from_df()
{
   local _size=""
   _size=$(df -k | awk -F' ' 'BEGIN{total=0} {if ($1 == "tmpfs" || $1 == "devtmpfs" || $1 == "shm")total+=$3}END{print total}')
   echo $_size
}

#inode, dentry and buffer_head is reclaimable
function get_slab_reclaimable_from_slabinfo()
{
   local _size=""
   _size=$(awk -F' ' 'BEGIN{total=0} {if ($1~/inode/ || $1~"dentry" || $1 == "buffer_head")total+=($3 * $4)}END{printf "%d\n", total / 1024}' /proc/slabinfo)
   [ -z "$_size" ] && _size=0
   echo $_size
}

free=$(get_meminfo MemFree)
total=$(get_meminfo MemTotal)
cached=$(get_meminfo Cached)
buffer=$(get_meminfo Buffers)
swapcached=$(get_meminfo SwapCached)
shmem=$(get_meminfo Shmem)
slab_reclaimable=$(get_meminfo SReclaimable)
nfs_unstable=$(get_meminfo NFS_Unstable)

#the kernel does not support, no 'Shmem' field in /proc/meminfo, we use ipc and df.
if [ -z "$shmem" ] ; then
   shmem=$(( $(get_shmem_from_ipcs) + $(get_tmpfs_from_df) ))
fi

#the kernel does not support, no 'SReclaimable' field in /proc/meminfo, we use /proc/slabinfo.
if [ -z "$slab_reclaimable" ]; then
   slab_reclaimable=$(get_slab_reclaimable_from_slabinfo)
fi

#the kernel does not support, no 'NFS_Unstable' field in /proc/meminfo, we use null. :)
if [ -z "$nfs_unstable" ]; then
   nfs_unstable=0
fi

reclaimable=$((cached + buffer + slab_reclaimable + swapcached + nfs_unstable - shmem))
available=$((free + reclaimable))
show_meminfo "Total: "

说说free命令

free是个常用命令，几乎每个接触、使用linux的用户都会用到它。但往往对它的统计输出会有一些困惑，这一方面和Linux内存管理机制有关，另一方面Linux在内存统计上也确实有些不足和问题。

关键在于两个字段，buffers和cached。

你经常会发现Linux系统用了一段时间后，内存所剩无几，free命令，一看，内存全跑到 buffers和cached里面了；这个现象是正常的。访问过的磁盘文件的元数据及内容，内核都会缓存起来。这些缓存就是磁盘缓存。

Linux磁盘缓存设计特点（设计理念）：
除了系统运行必须的一小部分保留外，只要有剩余内存，只要需要，就会用给磁盘缓存。（没有一个参数可以让你限定缓存的上限。2.6内核之前有一个限定参数，后来给取消了）

所以会经常看到内存所剩无几的现象，这是缓存机制导致的，对应用是透明，在有内存需要时，这些内存会释放。这个过程对应用是透明的，应用可以认为系统的可用内存包括buffers和cached。

这种设计，在大多数服务器应用场景下都有比较好的性能表现。可以说是比较可取的。

设计本身没有问题，但free命令显示的buffers和cached并不能和磁盘缓存完全对应，这是实现细节上的不足和问题。

1. buffers和cached包含了不属于磁盘缓存的内容。
由于buffers和cached实际上就是内核为所有文件映射分配的物理页的总和（page cache）。
但内核中的“文件概念”是广泛的，不仅包含了真正位于磁盘上的文件，还包含了为特殊需要创建的虚拟文件，比如：

进程间的共享内存（通过shmget API创建的内存），内核建立一个虚拟的文件和共享内存关联起来。（通过pmap命令你可以看到进程拥有的共享内存地址空间的映射字段是/SYSVXXXX字样，不是匿名的）。

非常的不幸，这些虚拟文件映射关联的page，也被算入了free命令显示的cached字段。但这部分内存没有缓存属性，在内存不足时不能按缓存的方式来回收。(使用echo 3 >/proc/sys/vm/drop_caches，无法释放掉这部分内存）

这个问题，会带来一些麻烦。比如，按照常规理解，某产品设计内存占用过高告警的条件是，空闲内存+buffers+cached小于内存总和的20% 。一般情况下没有问题，但如果产品使用了大量的共享内存，告警将失去作用。

2. buffers和cached遗漏了部分属于磁盘缓存的内容。
还是由于buffers和cached只是内核为所有文件映射分配的物理页的总和。在文件系统方面，还有一部分缓存是不和文件映射相关联的，比如内核分配的 inode对象，在文件关闭时，并不会立即释放，具备缓存的属性。这部分不在基于文件映射的页面里，而是通过slab（内核内存池）分配的。

原文：http://yalung929.blog.163.com/blog/static/203898225201223102617451/

本文链接

转一篇Instagram的架构分享

2012-04-14T11:46:00Z

One of the questions we always get asked at meet-ups and conversations with other engineers is, “what’s your stack?” We thought it would be fun to give a sense of all the systems that power Instagram, at a high-level; you can look forward to more in-depth descriptions of some of these systems in the future. This is how our system has evolved in the just-over-1-year that we’ve been live, and while there are parts we’re always re-working, this is a glimpse of how a startup with a small engineering team can scale to our 14 million+ users in a little over a year. Our core principles when choosing a system are:

Keep it very simple
Don’t re-invent the wheel
Go with proven and solid technologies when you can

We’ll go from top to bottom:

OS / Hosting

We run Ubuntu Linux 11.04 (“Natty Narwhal”) on Amazon EC2. We’ve found previous versions of Ubuntu had all sorts of unpredictable freezing episodes on EC2 under high traffic, but Natty has been solid. We’ve only got 3 engineers, and our needs are still evolving, so self-hosting isn’t an option we’ve explored too deeply yet, though is something we may revisit in the future given the unparalleled growth in usage.

Load Balancing

Every request to Instagram servers goes through load balancing machines; we used to run 2 nginx machines and DNS Round-Robin between them. The downside of this approach is the time it takes for DNS to update in case one of the machines needs to get decomissioned. Recently, we moved to using Amazon’s Elastic Load Balancer, with 3 NGINX instances behind it that can be swapped in and out (and are automatically taken out of rotation if they fail a health check). We also terminate our SSL at the ELB level, which lessens the CPU load on nginx. We use Amazon’s Route53 for DNS, which they’ve recently added a pretty good GUI tool for in the AWS console.

Application Servers

Next up comes the application servers that handle our requests. We run Django on Amazon High-CPU Extra-Large machines, and as our usage grows we’ve gone from just a few of these machines to over 25 of them (luckily, this is one area that’s easy to horizontally scale as they are stateless). We’ve found that our particular work-load is very CPU-bound rather than memory-bound, so the High-CPU Extra-Large instance type provides the right balance of memory and CPU.

We use http://gunicorn.org/ as our WSGI server; we used to use mod_wsgi and Apache, but found Gunicorn was much easier to configure, and less CPU-intensive. To run commands on many instances at once (like deploying code), we use Fabric, which recently added a useful parallel mode so that deploys take a matter of seconds.

Data storage

Most of our data (users, photo metadata, tags, etc) lives in PostgreSQL; we’ve previously written about how we shard across our different Postgres instances. Our main shard cluster involves 12 Quadruple Extra-Large memory instances (and twelve replicas in a different zone.)

We’ve found that Amazon’s network disk system (EBS) doesn’t support enough disk seeks per second, so having all of our working set in memory is extremely important. To get reasonable IO performance, we set up our EBS drives in a software RAID using mdadm.

As a quick tip, we’ve found that vmtouch is a fantastic tool for managing what data is in memory, especially when failing over from one machine to another where there is no active memory profile already. Here is the script we use to parse the output of a vmtouch run on one machine and print out the corresponding vmtouch command to run on another system to match its current memory status.

All of our PostgreSQL instances run in a master-replica setup using Streaming Replication, and we use EBS snapshotting to take frequent backups of our systems. We use XFS as our file system, which lets us freeze & unfreeze the RAID arrays when snapshotting, in order to guarantee a consistent snapshot (our original inspiration came from ec2-consistent-snapshot. To get streaming replication started, our favorite tool is repmgr by the folks at 2ndQuadrant.

To connect to our databases from our app servers, we made early on that had a huge impact on performance was using Pgbouncer to pool our connections to PostgreSQL. We found Christophe Pettus’s blog to be a great resource for Django, PostgreSQL and Pgbouncer tips.

The photos themselves go straight to Amazon S3, which currently stores several terabytes of photo data for us. We use Amazon CloudFront as our CDN, which helps with image load times from users around the world (like in Japan, our second most-popular country).

We also use Redis extensively; it powers our main feed, our activity feed, our sessions system (here’s our Django session backend), and other related systems. All of Redis’ data needs to fit in memory, so we end up running several Quadruple Extra-Large Memory instances for Redis, too, and occasionally shard across a few Redis instances for any given subsystem. We run Redis in a master-replica setup, and have the replicas constantly saving the DB out to disk, and finally use EBS snapshots to backup those DB dumps (we found that dumping the DB on the master was too taxing). Since Redis allows writes to its replicas, it makes for very easy online failover to a new Redis machine, without requiring any downtime.

For our geo-search API, we used PostgreSQL for many months, but once our Media entries were sharded, moved over to using Apache Solr. It has a simple JSON interface, so as far as our application is concerned, it’s just another API to consume.

Finally, like any modern Web service, we use Memcached for caching, and currently have 6 Memcached instances, which we connect to using pylibmc & libmemcached. Amazon has an Elastic Cache service they’ve recently launched, but it’s not any cheaper than running our instances, so we haven’t pushed ourselves to switch quite yet.

Task Queue & Push Notifications

When a user decides to share out an Instagram photo to Twitter or Facebook, or when we need to notify one of our Real-time subscribers of a new photo posted, we push that task into Gearman, a task queue system originally written at Danga. Doing it asynchronously through the task queue means that media uploads can finish quickly, while the ‘heavy lifting’ can run in the background. We have about 200 workers (all written in Python) consuming the task queue at any given time, split between the services we share to. We also do our feed fan-out in Gearman, so posting is as responsive for a new user as it is for a user with many followers.

For doing push notifications, the most cost-effective solution we found was https://github.com/samuraisam/pyapns, an open-source Twisted service that has handled over a billion push notifications for us, and has been rock-solid.

Monitoring

With 100+ instances, it’s important to keep on top of what’s going on across the board. We use Munin to graph metrics across all of our system, and also alert us if anything is outside of its normal range. We write a lot of custom Munin plugins, building on top of Python-Munin, to graph metrics that aren’t system-level (for example, signups per minute, photos posted per second, etc). We use Pingdom for external monitoring of the service, and PagerDuty for handling notifications and incidents.

For Python error reporting, we use Sentry, an awesome open-source Django app written by the folks at Disqus. At any given time, we can sign-on and see what errors are happening across our system, in real time.

You?

If this description of our systems interests you, or if you’re hopping up and down ready to tell us all the things you’d change in the system, we’d love to hear from you. We’re looking for a DevOps person to join us and help us tame our EC2 instance herd.

原文：http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances-dozens-of

本文链接

xm list和xl list显示内存不一致的问题

2012-04-13T09:16:00Z

xm list和xl list显示内存不一致的问题

在xen4.1.2上一台windows虚机碰到这个问题：
# xm list test6
Name                                        ID   Mem VCPUs      State   Time(s)
test6                                       16 512     2     -b----    158.0
# xl list test6
Name                                        ID   Mem VCPUs      State   Time(s)
test6                                       16   2048     2     -b----     158.6

同一台虚机，用两个不同的工具查看，内存居然差了将近2GB。

由于xm和xl的实现机制不同，所以从源码分析开始，分别查看xm list和xl list的源码

xm list:

    前文（http://www.cnblogs.com/feisky/archive/2011/11/27/2265053.html）已经分析过xm list的源码，它得到的内存是xend和xenstore中的内存（两者总是保持一致）

    xenstore中的内存可以通过下面的命令读出：

    xenstore-read /vm/[uuid]/memory

xl list:
    ==> xl.c::main() --> cmdtable_lookup() --> main_list()
    ==> xl_cmdimpl.c::main_list()
    ==> libxl_domain_info(&ctx, &info_buf, domid);
          ==> xc_domain_getinfolist(ctx->xch, domid, 1, &xcinfo)
          ==> xcinfo2xlinfo(&xcinfo, info_r)
          ==> xlinfo->current_memkb = PAGE_TO_MEMKB(xcinfo->tot_pages);
    ==> xc_domain_getinfolist(ctx->xch, domid, 1, &xcinfo);
         可以看到xl list最终是通过XEN_SYSCTL_getdomaininfolist超级调用实现的
    ==> xen/common/sysctl.c::do_sysctl()
    ==> xen/common/domctl.c::getdomaininfo(d, &info)
         info->tot_pages         = d->tot_pages;
         info->max_pages         = d->max_pages;
         info->shr_pages         = atomic_read(&d->shr_pages);
         info->shared_info_frame = mfn_to_gmfn(d, __pa(d->shared_info)>>PAGE_SHIFT);

可以看出，xl list得到的是hypervisor为该domain分配的内存。

通常情况下hypervisor为该domain分配的内存比xend传进去的内存多2MB（在xen3.4.3上也有同样的问题，多余的2MB应该是hypervisor占用）
[root@localhost ~]# xm li test6
Name                                        ID   Mem VCPUs      State   Time(s)
test6                                       21   512     2     -b----     79.4
[root@localhost ~]# xl list test6
Name                                        ID   Mem VCPUs      State   Time(s)
test6                                       21   514     2     -b----      79.4

上面碰到的问题是由于虽然通过xm mem-set为虚机重新调整了内存，新的内存值写入到了xend和xenstore中，但hypervisor并没有立即为该虚机ballon内存。此时，如果多次执行xl list可以看到test6虚机的内存一点点减少，直至514MB。

本文链接

Xen4.2增加新的CPU调度参数

2012-04-11T05:01:00Z

Xen 4.2 will contain two new scheduling parameters for the credit1 scheduler which significantly increase its confurability and performance for cloud-based workloads:timeslice_ms and ratelimit_us. This blog post describes what they do, and how to configure them for best performance.

Timeslice

The timeslice for the credit1 has historically been fixed at 30ms. This is actually a fairly long time — it’s great for computationally-intensive workloads, but not so good for latency-sensitive workloads, particularly ones involving network traffic or audio.

Xen 4.2 introduces the tslice_ms parameter, which sets the timeslice of the scheduler in milliseconds. This can be set either using the Xen command-line option,sched_credit_tslice_ms, or by using the new scheduling parameter interface to xl sched-credit:
# xl sched-credit -t [n]

Interesting values you might give try are 10ms, 5ms, and 1ms. One millisecond might be a good choice for particularly latency-sensitive workloads; but beware that reducing the timeslice also increases the overhead from context switching and reduces the effectiveness of the CPU cache. Values of 5ms or 10ms give a good balance. The default, 30ms, is probably too long; but we’re going to do some more experimentation and probably switch the default in 4.3. If you try any values that turn out to be particularly good or bad, let us know.

Schedule rate limiting

Hui Lv over at Intel did some fascinating work analyzing the performance overhead of running SpecVirt inside of Xen. What he discovered was that under certain circumstances, some VMs were waking up, doing a few microseconds worth of work, and going back to sleep, only to be woken up microseconds later. The credit1 scheduler correctly identified that these were probably latency-sensitive applications and gave them priority to run whenever they needed to. The problem was that they were causing thousands of schedules per second — in some cases up to 15,000 schedules per second. This meant that there was a very significant amount of time actually spent in the scheduler switching back and forth between the two tasks, rather than doing the actual work.

The credit1 scheduler was giving these processes microsecond latency; but there are very few network-based workloads that require sub-millisecond latency. Hui worked with the Xen development community to introduce a simple mechanism that would be predictable and robust, and improve the performance for this workload without degrading the performance of other workloads. The result was ratelimit_us. The ratelimit is a value in microseconds. It is a minimum amount of time which a VM is allowed to run without being preempted. The default value is 1000 (that is, 1ms). So if a VM starts running, and another VM with higher priority wakes up, if the first VM has run for less than 1ms, it is allowed to continue to run until its 1ms is up; only after that will the higher-priority VM be allowed to run.

One millisecond is not generally too long for network-based workloads to wait; and the effect is to have more “batching”, so the whole system is used more effectively. This caused significant increase in SpecVirt performance in Hui’s tests.

This feature can be disabled by setting the ratelimit to 0. One could imagine, on a computation-heavy workload, setting this to something higher, like 5ms or 10ms; or if you have a particularly latency-sensitive workload, bringing it down to 500us or even 100us.

This value can be set either on the Xen command-line using sched_ratelimit_us (Note no “credit” in this one — it’s meant to be consumed by other schedulers as well) or the xl command-line:
# xl sched-credit -r [n]

Curent values of both can also be viewed using xl:

# xl sched-credit
Cpupool Pool-0: tslice=30ms ratelimit=1000us
Name ID Weight Cap
Domain-0 0 256 0

Wait, what’s this “cpupool” thing, you say? That’s another new feature in Xen 4.2 — but the topic of another blog post. Watch this space!

来自：http://blog.xen.org/index.php/2012/04/10/xen-4-2-new-scheduler-parameters-2/

本文链接

python epoll

2012-04-10T13:28:00Z

python epoll用法

import socket, select

EOL1 = b'\n\n'
EOL2 = b'\n\r\n'
response = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
response += b'Hello, world!'

serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serversocket.bind(('0.0.0.0', 8080))
serversocket.listen(1)
serversocket.setblocking(0)

epoll = select.epoll()
epoll.register(serversocket.fileno(), select.EPOLLIN | select.EPOLLET)

try:
connections = {}; requests = {}; responses = {}
while True:
events = epoll.poll(1)
for fileno, event in events:
if fileno == serversocket.fileno():
try:
while True:
connection, address = serversocket.accept()
connection.setblocking(0)
epoll.register(connection.fileno(), select.EPOLLIN | select.EPOLLET)
connections[connection.fileno()] = connection
requests[connection.fileno()] = b''
responses[connection.fileno()] = response
except socket.error:
pass
elif event & select.EPOLLIN:
try:
while True:
requests[fileno] += connections[fileno].recv(1024)
except socket.error:
pass
if EOL1 in requests[fileno] or EOL2 in requests[fileno]:
epoll.modify(fileno, select.EPOLLOUT | select.EPOLLET)
print('-'*40 + '\n' + requests[fileno].decode()[:-2])
elif event & select.EPOLLOUT:
try:
while len(responses[fileno]) > 0:
byteswritten = connections[fileno].send(responses[fileno])
responses[fileno] = responses[fileno][byteswritten:]
except socket.error:
pass
if len(responses[fileno]) == 0:
epoll.modify(fileno, select.EPOLLET)
connections[fileno].shutdown(socket.SHUT_RDWR)
elif event & select.EPOLLHUP:
epoll.unregister(fileno)
connections[fileno].close()
del connections[fileno]
finally:
epoll.unregister(serversocket.fileno())
epoll.close()
serversocket.close()

参考：http://scotdoyle.com/python-epoll-howto.html

本文链接

xen-4.1.2+Linux-3.1.2源码编译安装

2012-04-10T13:10:00Z

xen-4.1.2+Linux-3.1.2源码编译安装

以下所有命令均在CentOS6.2系统上操作。

下载xen源码

wget http://bits.xensource.com/oss-xen/release/4.1.2/xen-4.1.2.tar.gz

安装必备软件包

yum groupinstall "Development Libraries"yum groupinstall "Development Tools"yum install transfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial libidn-develyum -y install glibc-devel.i686texinfo libuuid-devel iasl python-lxmlyum -y install openssl openssl-develyum -y install ncurses ncurses-*yum install python-devel -y 编译安装xen hypervisor

tar zxvf xen-4.1.2.tar.gz make worldmake install /boot下多了xen-4.1.2.gz和xen-syms-4.1.2文件将Xen加入到启动脚本：

/sbin/chkconfig --add xend
/sbin/chkconfig --add xencommons
/sbin/chkconfig --add xendomains
/sbin/chkconfig xend on
/sbin/chkconfig xendomains on
/sbin/chkconfig xencommons on

编译安装Linux3.1.2内核

wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.1.2.tar.bz2

#make oldconfig # 全部默认

#make menuconfig

Processor type and features --- >

选中Paravirtualized Guest Support

Device Drivers --->

Xen driver support --->

全部选*

修改：CONFIG_XEN_DEV_EVTCHN=y（如果是m，开机时无法启动xencommons）

make
make modules
make modules_install
make install （会报找不到xen-evtchn模块，忽略）

depmod 3.1.2

mkinitrd -v -f --with=aacraid --with=sd_mod --with=scsi_mod initramfs-3.1.2.img 3.1.2

配置grub：

title CentOS (3.1.2-xen)
        root (hd0,0)
        kernel /boot/xen-4.1.2.gz dom0_mem=1024M
        module /boot/vmlinuz-3.1.2 ro root=/dev/sda1
        module /boot/initramfs-3.1.2.img

参考：
http://linuxme.blog.51cto.com/1850814/390778
http://share.blog.51cto.com/278008/723828
http://blog.chinaunix.net/space.php?uid=147142&do=blog&id=2788048
http://wiki.xen.org/xenwiki/2.6.18-to-2.6.31-and-higher

http://wiki.xen.org/wiki/Migrate_from_Linux_2.6.18_to_2.6.31_and_higher

http://wiki.xen.org/wiki/Mainline_Linux_Kernel_Configs

http://wiki.xen.org/wiki/HostConfiguration/Networking

本文链接