Troubleshooting mail queue on Postfix, Zimbra, Exim
Recently our Zimbra opensource mail server ground to a halt. As it did, the Godaddy smarthost reported we'd exceeded our 1000 daily smtp relay limit. This was an important clue: unfortunately that email was lost among hundreds of bounce messages, or perhaps stuck on the queue.
The reason was straightforward and common enough. We had been targeted for delivering thousands of spam messages.
The Zimbra admin web ui became rather useless. First, it is slow at the best of times, and with the server struggling for life, the web ui just didn't respond. Partially fixing things on the server using good old command line tools got zimbra responding. However, with 80000 messages on hold, the Zimbra admin web ui again could do nothing useful with them. Back to the command line.
Zimbra would be better off just having a fancy ajax shell console, perhaps with some nicely integrated instructions for the already existing command line tools, rather than their complex and ineffective ui, imho.
First thing that needed to be done was to stop Zimbra.
With zimbra stopped, the server became nicely responsive and I proceeded (as root) to manage the postfix queued messages with the postfix tools.
On our web server, which runs exim, from bradthemad.org/tech/notes/exim_cheatsheet.php, I did the following (as root).
Now I repeated using postsuper on the zimbra server (with zimbra stopped) to clear the queues. Restarted zimbra, and all was well. Except for the thousands of emails now held. These I could not so easily delete, as some legitimate emails were in there.
The zimbra admin web ui was ineffective for dealing with the thousands of held mails, so it was back to the postfix command line tools.
Ultimately I used the following.
The reason was straightforward and common enough. We had been targeted for delivering thousands of spam messages.
The Zimbra admin web ui became rather useless. First, it is slow at the best of times, and with the server struggling for life, the web ui just didn't respond. Partially fixing things on the server using good old command line tools got zimbra responding. However, with 80000 messages on hold, the Zimbra admin web ui again could do nothing useful with them. Back to the command line.
Zimbra would be better off just having a fancy ajax shell console, perhaps with some nicely integrated instructions for the already existing command line tools, rather than their complex and ineffective ui, imho.
First thing that needed to be done was to stop Zimbra.
su - zimbraThe zimbra documentation hinted that "zmcontrol stop mta" might just stop a particular service, but I found that not to be the case. Everything starts or stops.
zmcontrol stop
With zimbra stopped, the server became nicely responsive and I proceeded (as root) to manage the postfix queued messages with the postfix tools.
cd /opt/zimbra/postfix/sbinThis transferred all the messages out of "deferred", "incoming" and "active" into "hold" queue. I was then able to start zimbra and it worked for a bit. Unbeknown to me, we had a big queue of messages on our web server, and after zimbra started, thousands more messages arrived and choked zimbra, so it had to be stopped again. I had wanted to stop just the zimbra mta and keep the imap daemon running, so I could study the nature of these mails easily, but zimbra doesn't support that.
./postsuper -h ALL
On our web server, which runs exim, from bradthemad.org/tech/notes/exim_cheatsheet.php, I did the following (as root).
exiqgrep -i > /tmp/xThis deleted everything that was queued. I updated apache, and exim, and php and disabled our forums - as it was not immediately obvious how the spam was being generated, only that it was starting out as root from the local machine, which was distressing. And, this server is actually a guest on a Xen host. It's a tangled web we weave.
for f in $(cat /tmp/x); do exim -Mrm $f; done;
Now I repeated using postsuper on the zimbra server (with zimbra stopped) to clear the queues. Restarted zimbra, and all was well. Except for the thousands of emails now held. These I could not so easily delete, as some legitimate emails were in there.
The zimbra admin web ui was ineffective for dealing with the thousands of held mails, so it was back to the postfix command line tools.
postqueue -p prints message ids (all of them) with a little bit of envelope info.
postcat -q prints the actual message
Ultimately I used the following.
./postqueue -p | awk '/root@server1.homerent.ca/ {print $1}' > /tmp/x
./postsuper -d - < /tmp/x