Tag Archives: apache

Mailcatcher to test your email subsystem

You have probably heard of Mailcatcher already. It’s a great piece of software that implements neat idea.
I am using Vagrant and Apache on Debian, so my development box is build using puphpet.com. It allows mailcatcher installation btw, but we’ll do it manually and with a step forward:

  1. Get sqlite3 development libs: sudo apt-get install libsqlite3-dev;
  2. Got ruby? Install mailcatcher: sudo gem install mailcatcher;
  3. Now, let’s make it running after box restart:
    • Create /etc/init.d/mailcatcher script with this code:
      #! /bin/sh
      
      ### BEGIN INIT INFO
      # Provides:          mailcatcher
      # Required-Start:    $all
      # Required-Stop:     $network
      # Default-Start:     2 3 4 5
      # Default-Stop:      0 1 6
      # Short-Description: Start daemon at boot time
      # Description:       Enable service provided by daemon.
      ### END INIT INFO
      
      PID_FILE=/var/run/mailcatcher.pid
      NAME=mailcatcher
      PROG=/usr/local/bin/mailcatcher
      USER=mailcatcher
      GROUP=mailcatcher
       
      start() {
      	echo -n "Starting MailCatcher"
      	if start-stop-daemon --stop --quiet --pidfile $PID_FILE --signal 0
      	then
      		echo " already running."
      		exit
      	fi
      	start-stop-daemon \
      		--start \
      		--pidfile $PID_FILE \
      		--make-pidfile \
      		--background \
      		--exec $PROG \
      		--user $USER \
      		--group $GROUP \
      		--chuid $USER \
      		-- \
      		--foreground
      	echo "."
      	return $?
      }
       
      stop() {
      	echo -n "Stopping MailCatcher"
      	start-stop-daemon \
      		--stop \
      		--oknodo \
      		--pidfile $PID_FILE
      	echo "."
      	return $?
      }
       
      restart() {
      	stop
      	start
      }
       
      case "$1" in
      	start)
      		start
      		;;
      	stop)
      		stop
      		;;
      	restart)
      		restart
      		;;
      	*)
      		echo "Usage: $0 {start|stop|restart}"
      		exit 1
      		;;
      esac
      
    • Make the script executable: sudo chmod +x /etc/init.d/mailcatcher;
    • Create the user: sudo adduser mailcatecher;
    • Add the script for upstart: update-rc.d mailcatcher enable;

Ok, so mailcatcher will run at startup, but will not be visible at host machine. Now, you can do it just by forwarding guest’s 1080 port to host’s 1080 port using Vagrant config. But it didn’t work for me for no apparent reason. And i am not a big fan of typing in port numbers into browser address bar. So next steps will solve that:

  1. Create a new Apache “site”, let’s say “/etc/apache2/sites-available/35-mailcatcher.conf“;
  2. Let’s use Apache proxy module to let us access to mailcatcher (on port 80). Add this to the newly created “site” file:
    <VirtualHost *:80>
         ServerName mailcatcher.local
         ProxyPass / http://127.0.0.1:1080/
         ProxyPassReverse / http://127.0.0.1:1080/
    </VirtualHost>
    
  3. On your host machine update /etc/hosts file with this line (replace %GUEST_IP% with your Vagrant box IP address):
    %GUEST_IP% mailcatcher.local
    
  4. Enable mailcatcher site with a2ensite command, enable proxy module with a2enmod proxy_http command and restart apache;
  5. Point your browser to http://mailcatcher.local and your should see mailcatcher interface.
  6. If you see 503 error, it’s probably because mailcatcher did not start.

    Oh, yes, refer to the manual on how to add support to your language of choice. For PHP it’s simple:

    ; php.ini
    sendmail_path = /usr/bin/env catchmail -f [email protected]
    

Hunting memory leaks in Apache (and finding them elsewhere)

Testing your stuff is very important. That’s why we do performance testing at work, and usually everything turns out to be fine. But one day an unpleasant news have come: server under load have used all available RAM and went to swap. That is not good. Not at all.

Ganglia graph
That huge spike is how swapping server looks like

We installed xhprof on the server in question and began profiling.
BTW i have used this nice manual to get it working.

Day one. Profiling.
Profiler is a great tool, but it did not give us answer why the memory usage grows like crazy. PHP interpreter was using ~18M or memory on each request in question (we are using Symfony 2 with some bundles, that’s why). I was looking at numbers and scratching my head. The day have ended there.

Days two and three. More profiling.
I knew that some extensions could leak. Like cURL for example. Here is a bug report #65458. It’s for 5.5.2, but there are report for 5.4 somewhere, and that’s exactly what we are using. So i had put down simple test – command line script – trying to catch cURL with a hand in cookie jar. Well, you guessed it, right? xhprof reported same memory consumption, no matter how long the script was working. Back to square one.

Day four. Time for low level sorcery.
So PHP works in interesting and safe way: at the end of each request/script run it will drop all memory that have been allocated by your code and call it a day. Or at least that’s what i read several times in manuals. Knowing how things are working (or not working) it was time to get to real tools. Digging through the internet i have found mentions of Valgrind, a powerful set of tools for monitoring applications. Just what i need!

Apparently Valgrind runs any command you want to check for memory leaks and tracks how much memory allocated/freed. This has some (significant) overhead, but it totally worth it.

Now, there is this page in PHP manual: https://bugs.php.net/bugs-getting-valgrind-log.php. According to it, you cannot just use Valgrind with PHP because Zend uses it’s own memory management, but that feature can be disables (oh, blessing) during runtime. Let’s do it!

$ export ZEND_DONT_UNLOAD_MODULES=1; export USE_ZEND_ALLOC=0; valgrind --tool=memcheck --num-callers=30 --track-origins=yes --leak-check=full --log-file=php.log php ~/1.php

Valgrind will generate huge log file, filled with 1,2, 100 bytes lost messages, we have to look for those, marked ‘definitely lost’. And that’s what I have found:

==24504== 16 bytes in 1 blocks are definitely lost in loss record 24 of 383
==24504== at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==24504== by 0x126A6708: recode_malloc (in /usr/lib64/librecode.so.0.0.0)
==24504== by 0x126A42E5: declare_implied_surface (in /usr/lib64/librecode.so.0.0.0)
==24504== by 0x126A55FA: recode_new_outer (in /usr/lib64/librecode.so.0.0.0)
==24504== by 0x123FD11A: ??? (in /opt/rh/php54/root/usr/lib64/php/modules/recode.so)
==24504== by 0x5E48E2: zend_startup_module_ex (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x5ED9B4: zend_hash_apply (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x5E8039: zend_startup_modules (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x584949: php_module_startup (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x689DBC: ??? (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x68B728: ??? (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x3828A1ED1C: (below main) (in /lib64/libc-2.12.so)

And the summary wasn’t pleasant at all:

==24504== LEAK SUMMARY:
==24504== definitely lost: 10,794 bytes in 1,033 blocks
==24504== indirectly lost: 720 bytes in 7 blocks
==24504== possibly lost: 0 bytes in 0 blocks
==24504== still reachable: 240,825 bytes in 3,254 blocks
==24504== suppressed: 0 bytes in 0 blocks
==24504== Reachable blocks (those to which a pointer was found) are not shown.

What is ‘recode’ extension? Never heard of it. Let’s disable it and see what happens!
$ export ZEND_DONT_UNLOAD_MODULES=1; export USE_ZEND_ALLOC=0; valgrind --tool=memcheck --num-callers=30 --track-origins=yes --leak-check=full --log-file=php-no-recode.log php ~/1.php

Bingo!

==32272== LEAK SUMMARY:
==32272== definitely lost: 0 bytes in 0 blocks
==32272== indirectly lost: 0 bytes in 0 blocks
==32272== possibly lost: 0 bytes in 0 blocks
==32272== still reachable: 238,194 bytes in 3,251 blocks
==32272== suppressed: 0 bytes in 0 blocks
==32272== Reachable blocks (those to which a pointer was found) are not shown.

So it turns out not PHP code or Apache was leaking, but librecode itself. Fun stuff.

Yeah, those were four intense days. But some lessons learned:

  • use xhprof for a long running php scripts, daemons etc. ;
  • valgrind is your friend;
  • disable unused PHP extensions;
  • test. Test. Test.