Tag Archives: php

Mailcatcher to test your email subsystem

You have probably heard of Mailcatcher already. It’s a great piece of software that implements neat idea.
I am using Vagrant and Apache on Debian, so my development box is build using puphpet.com. It allows mailcatcher installation btw, but we’ll do it manually and with a step forward:

  1. Get sqlite3 development libs: sudo apt-get install libsqlite3-dev;
  2. Got ruby? Install mailcatcher: sudo gem install mailcatcher;
  3. Now, let’s make it running after box restart:
    • Create /etc/init.d/mailcatcher script with this code:
      #! /bin/sh
      
      ### BEGIN INIT INFO
      # Provides:          mailcatcher
      # Required-Start:    $all
      # Required-Stop:     $network
      # Default-Start:     2 3 4 5
      # Default-Stop:      0 1 6
      # Short-Description: Start daemon at boot time
      # Description:       Enable service provided by daemon.
      ### END INIT INFO
      
      PID_FILE=/var/run/mailcatcher.pid
      NAME=mailcatcher
      PROG=/usr/local/bin/mailcatcher
      USER=mailcatcher
      GROUP=mailcatcher
       
      start() {
      	echo -n "Starting MailCatcher"
      	if start-stop-daemon --stop --quiet --pidfile $PID_FILE --signal 0
      	then
      		echo " already running."
      		exit
      	fi
      	start-stop-daemon \
      		--start \
      		--pidfile $PID_FILE \
      		--make-pidfile \
      		--background \
      		--exec $PROG \
      		--user $USER \
      		--group $GROUP \
      		--chuid $USER \
      		-- \
      		--foreground
      	echo "."
      	return $?
      }
       
      stop() {
      	echo -n "Stopping MailCatcher"
      	start-stop-daemon \
      		--stop \
      		--oknodo \
      		--pidfile $PID_FILE
      	echo "."
      	return $?
      }
       
      restart() {
      	stop
      	start
      }
       
      case "$1" in
      	start)
      		start
      		;;
      	stop)
      		stop
      		;;
      	restart)
      		restart
      		;;
      	*)
      		echo "Usage: $0 {start|stop|restart}"
      		exit 1
      		;;
      esac
      
    • Make the script executable: sudo chmod +x /etc/init.d/mailcatcher;
    • Create the user: sudo adduser mailcatecher;
    • Add the script for upstart: update-rc.d mailcatcher enable;

Ok, so mailcatcher will run at startup, but will not be visible at host machine. Now, you can do it just by forwarding guest’s 1080 port to host’s 1080 port using Vagrant config. But it didn’t work for me for no apparent reason. And i am not a big fan of typing in port numbers into browser address bar. So next steps will solve that:

  1. Create a new Apache “site”, let’s say “/etc/apache2/sites-available/35-mailcatcher.conf“;
  2. Let’s use Apache proxy module to let us access to mailcatcher (on port 80). Add this to the newly created “site” file:
    <VirtualHost *:80>
         ServerName mailcatcher.local
         ProxyPass / http://127.0.0.1:1080/
         ProxyPassReverse / http://127.0.0.1:1080/
    </VirtualHost>
    
  3. On your host machine update /etc/hosts file with this line (replace %GUEST_IP% with your Vagrant box IP address):
    %GUEST_IP% mailcatcher.local
    
  4. Enable mailcatcher site with a2ensite command, enable proxy module with a2enmod proxy_http command and restart apache;
  5. Point your browser to http://mailcatcher.local and your should see mailcatcher interface.
  6. If you see 503 error, it’s probably because mailcatcher did not start.

    Oh, yes, refer to the manual on how to add support to your language of choice. For PHP it’s simple:

    ; php.ini
    sendmail_path = /usr/bin/env catchmail -f [email protected]
    

Insert title here

You know how you supposed to make backups to keep all your data safe and warm? If you don’t – trust me, you have to do them. But here is a catch (how come that there is always a catch in this world?) – even if you do make reserve copy (or even several of them) – does not necessarily mean you are on the safe side. How is that? Very simple: software is broken. Which brings me to this comprehensive post from Scott Hanselman. Go on, read it and don’t tell me he is not right.

I’ve been using Macs for at least five years now. And rarely had any problems. Until now. Bought a new Mac (retina if you ask me) and certainly wanted to get all my stuff quickly from Time Machine. Except that Migration Assistant did not find any valid data on the storage. Weird. Tried several things here and there which did not lead to anything… I don’t say there is some critical stuff, but photos and documents are nice to have. Also rsa keys. And Kerbal Space Program saves.

After some digging found James Pond site, stuffed with Time Machine troubleshooting information. According to him, what happened is backup process failed to finish for any reason (although i do remember waiting for it to stop) and creating invalid records. Every other (finished) copy of data should be fine, just the last one is corrupted. So here i am, copying my home folder from network storage (which is my TM disk) manually. It will take just three hours or nine. And it better work.

Will see tomorrow.

On the code side: been playing with PHP stuff all day at work. Got a good piece done, but it doesn’t work because of some crazy issue with Document mapping in Symfony CMF. That stuff is seriously decoupled. Have to spend lifetime to get it to behave. I will write a separate post about it someday maybe to help guys like myself (or future me) understand it better. But like i said yesterday – best docs for open source is the code itself.

That’s it for today.

One day one code #13-14

Yep, i skipped a day. But because I was writing code, a lot. But mostly PHP. A lot of PHP and YAML. Not much time to learn C though.

So, the summary:
– DI is a bitch – sometimes it’s really hard to get it right;
– open source documentation is a bitch – have to read code, that’s the best documentation;
– i do complain too much.

Next step – get that calculator done. At least the is no DI containers.

That’s it for today.

Photolib 2

Today I was doing household stuff for Photolib. Well, it still needs some facelifting, but the good stuff was happening on the background.

I am using PHP built-in web server (available since 5.4) to test photolib. But since ‘photos’ is a folder and I run server like this “php -S 127.0.0.1:8080 index.php”, to ensure all request captured by index.php, thus my development environment lacks base URL (/photos/) and everything breaks. The solution would be to use separate configuration for development and for production. That’s kind of usual stuff, I just never actually did it with Silex before. Turns out it’s pretty easy.

First, have to get configuration service provider, like ConfigServiceProvider.
Second, create two configuration files: prod.yml and dev.yml.
Third, add configuration to the application:

// index.php
$app = new Silex\Application();
$env = getenv('APP_ENV') ?: 'prod';
$app->register(new Igorw\Silex\ConfigServiceProvider(__DIR__."/app/conf/{$env}.yml"));

Now, we can enable debug for development, by simply putting the line

# dev.yml
debug: true

into dev.yml. Alright, moving on to Twig.

Which is a little bit trickier. TwigServiceProvider in Silex knows nothing about configuration provider, so need to tell it that we have some settings for it:

// index.php

// Save globals for future use
$twigGlobals = $app['twig']['globals'];

// Register twig as a service
$app->register(new Silex\Provider\TwigServiceProvider(), array(
    'twig.path' => __DIR__."/app/twig",
));
// Extend twig service by adding global variables
$app['twig'] = $app->share($app->extend('twig', function($twig, $app) use ($twigGlobals) {
    foreach ($twigGlobals as $k => $g) {
        $twig->addGlobal($k, $g);
    }

    return $twig;
}));

And in config file:

# prod.yml
twig:
  globals:
    base_url: /photos/

Now just use {{ base_url }} in templates. Best way to do it would be to allow Twig service provider to accept ‘twig.globals’ as an option (like it does with ‘twig.path’) but that will have to wait until next time.

That’s it for today.

As a bonus, here is an excellent example of banner double hit in iAlien application:


I want adblock for iOS too.

imagine reading that on iPhone 4 screen

Fold. Not the end.

Really lazy weekend. Barely touched a keyboard, maybe just to chat for some time.

Still. I have rewritten `fold` in PHP, which is complete cheating, but at least i know how it should work now. Next step – implement it in C. And this is not because I am learning C, but because of this:

$ time php fold.php 10 syntax.text
real 0m0.179s
user 0m0.151s
sys 0m0.017s

$ time fold -w 10 syntax.text
real 0m0.041s
user 0m0.010s
sys 0m0.010s

Right…

PHP is not fast at all. But i have spent 20 minutes hacking that piece of code together and it totally works. That’s why dynamic languages rule. I still want to learn C, so have to rewrite that again.

Also, here is the picture of a guy stealing electricity on 42nd-Bryant Park:
He is just charging his iPhone.

Hunting memory leaks in Apache (and finding them elsewhere)

Testing your stuff is very important. That’s why we do performance testing at work, and usually everything turns out to be fine. But one day an unpleasant news have come: server under load have used all available RAM and went to swap. That is not good. Not at all.

Ganglia graph
That huge spike is how swapping server looks like

We installed xhprof on the server in question and began profiling.
BTW i have used this nice manual to get it working.

Day one. Profiling.
Profiler is a great tool, but it did not give us answer why the memory usage grows like crazy. PHP interpreter was using ~18M or memory on each request in question (we are using Symfony 2 with some bundles, that’s why). I was looking at numbers and scratching my head. The day have ended there.

Days two and three. More profiling.
I knew that some extensions could leak. Like cURL for example. Here is a bug report #65458. It’s for 5.5.2, but there are report for 5.4 somewhere, and that’s exactly what we are using. So i had put down simple test – command line script – trying to catch cURL with a hand in cookie jar. Well, you guessed it, right? xhprof reported same memory consumption, no matter how long the script was working. Back to square one.

Day four. Time for low level sorcery.
So PHP works in interesting and safe way: at the end of each request/script run it will drop all memory that have been allocated by your code and call it a day. Or at least that’s what i read several times in manuals. Knowing how things are working (or not working) it was time to get to real tools. Digging through the internet i have found mentions of Valgrind, a powerful set of tools for monitoring applications. Just what i need!

Apparently Valgrind runs any command you want to check for memory leaks and tracks how much memory allocated/freed. This has some (significant) overhead, but it totally worth it.

Now, there is this page in PHP manual: https://bugs.php.net/bugs-getting-valgrind-log.php. According to it, you cannot just use Valgrind with PHP because Zend uses it’s own memory management, but that feature can be disables (oh, blessing) during runtime. Let’s do it!

$ export ZEND_DONT_UNLOAD_MODULES=1; export USE_ZEND_ALLOC=0; valgrind --tool=memcheck --num-callers=30 --track-origins=yes --leak-check=full --log-file=php.log php ~/1.php

Valgrind will generate huge log file, filled with 1,2, 100 bytes lost messages, we have to look for those, marked ‘definitely lost’. And that’s what I have found:

==24504== 16 bytes in 1 blocks are definitely lost in loss record 24 of 383
==24504== at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==24504== by 0x126A6708: recode_malloc (in /usr/lib64/librecode.so.0.0.0)
==24504== by 0x126A42E5: declare_implied_surface (in /usr/lib64/librecode.so.0.0.0)
==24504== by 0x126A55FA: recode_new_outer (in /usr/lib64/librecode.so.0.0.0)
==24504== by 0x123FD11A: ??? (in /opt/rh/php54/root/usr/lib64/php/modules/recode.so)
==24504== by 0x5E48E2: zend_startup_module_ex (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x5ED9B4: zend_hash_apply (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x5E8039: zend_startup_modules (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x584949: php_module_startup (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x689DBC: ??? (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x68B728: ??? (in /opt/rh/php54/root/usr/bin/php)
==24504== by 0x3828A1ED1C: (below main) (in /lib64/libc-2.12.so)

And the summary wasn’t pleasant at all:

==24504== LEAK SUMMARY:
==24504== definitely lost: 10,794 bytes in 1,033 blocks
==24504== indirectly lost: 720 bytes in 7 blocks
==24504== possibly lost: 0 bytes in 0 blocks
==24504== still reachable: 240,825 bytes in 3,254 blocks
==24504== suppressed: 0 bytes in 0 blocks
==24504== Reachable blocks (those to which a pointer was found) are not shown.

What is ‘recode’ extension? Never heard of it. Let’s disable it and see what happens!
$ export ZEND_DONT_UNLOAD_MODULES=1; export USE_ZEND_ALLOC=0; valgrind --tool=memcheck --num-callers=30 --track-origins=yes --leak-check=full --log-file=php-no-recode.log php ~/1.php

Bingo!

==32272== LEAK SUMMARY:
==32272== definitely lost: 0 bytes in 0 blocks
==32272== indirectly lost: 0 bytes in 0 blocks
==32272== possibly lost: 0 bytes in 0 blocks
==32272== still reachable: 238,194 bytes in 3,251 blocks
==32272== suppressed: 0 bytes in 0 blocks
==32272== Reachable blocks (those to which a pointer was found) are not shown.

So it turns out not PHP code or Apache was leaking, but librecode itself. Fun stuff.

Yeah, those were four intense days. But some lessons learned:

  • use xhprof for a long running php scripts, daemons etc. ;
  • valgrind is your friend;
  • disable unused PHP extensions;
  • test. Test. Test.