PHP at Scale #13
Welcome to the thirteenth edition of PHP at Scale. I am diving deep into the principles, best practices and practical lessons learned from scaling PHP projects — not only performance-wise but also code quality and maintainability.
This edition focuses on essential async PHP techniques and basic queueing usage. Changes, you can often implement within minutes. I will touch on a bit more complicated topics, but I have intentionally left out more sophisticated things.
Looking ahead, I plan to delve deeper into real-time PHP capabilities, including WebSockets, server-sent events, and other async patterns that are transforming how we build modern PHP applications. In this edition, I will focus mostly on queues.
—
Let me start with some basics. I know a lot of you might feel annoyed by that, as you might already work with some kind of queues, but I feel it’s good to understand the basics. Queues are very easily packed into something that has an easy-to-use interface, which you can implement within minutes (I will link them later).
Do we need queues?
Imagine a user uploading a 50MB PDF that needs OCR processing, thumbnail generation, and archiving. Keeping them waiting 90 seconds for this to complete synchronously is a poor experience. Even worse, if 10 users trigger this simultaneously, you might crash your server.
Queues are often implemented to handle longer-running or resource-heavy flows. Instead of keeping a user waiting or hoping not too many users will trigger a heavy job at once, we can just create a queue. A task/job is created and sent to a queue. Next, a worker is picking jobs from the queue, processing them as fast as possible, but within the capacity it has. Having one worker means one job at a time, etc. So, queues are a way to implement asynchronous processing in our systems.
But there is another option, just adding a job to a database table, right? Yes, it is, and it is especially used in older systems.
So instead of processing the request synchronously, you just write it to a database table, and then you have a cronjob that picks it up to process. I think many modern developers might think this approach is a joke and should never be considered.
Yes, queues are better than the DB+cronjob approach, because:
A cronjob is fired every N minutes, which means there is a problem of finding the right number of jobs that fit within that period. You will always end up with not enough or too many, as a job processing time can vary.
Queuing systems usually come with some additional features like re-delivery, dead letter queue, etc. You might need to reimplement some of them.
It will deliver a message only once, even if you have many servers, workers, etc. With a DB job approach, that needs to be added manually.
Consider using the DB approach when you need:
Audit trails with complex queries (e.g., ‘show me all failed invoice jobs for customer X in the last month’)
A simple system with <100 jobs/day, where adding infrastructure isn’t justified
Strong transactional guarantees with your business logic
Basic queue implementation
A couple of years ago, we had to implement the queues “manually”, by using libraries like predis or php-amqplib. Nowadays, we have many options and ready-to-use libraries like Symfony Messenger or Laravel Queues.
How to set up Symfony Messenger in 5 minutes
The article nicely shows what I like in Messenger - being able to connect a lot of different backends for queues. RabbitMQ, SQS, DB, Redis, and the number of drivers should allow for easy integration with your stack.
If you have a look at the Doctrine driver, it is using LISTEN/NOTIFY by default. That means it is not pulling changes every N seconds; it actively listens for new jobs, similar to how a dedicated queueing backend works (like RabbitMQ). Thanks to that, you can actually consider Doctrine as a useful driver, especially for lower-traffic systems.
What I dislike in this article is the example flow. Although it brings the benefit of moving the “long” running code to an async flow, it does not follow my preferred architecture choices when it comes to splitting code. But this is something for another discussion.
Mastering Message Brokering in Symfony: A Practical Guide to Three Essential Patterns
This article goes a bit deeper into what Symfony Messenger offers. It is fully configurable. You have multiple different transports, eg. one for commands (exactly one handler), one for events (0-N handlers). Also, you can customize the delivery per message. The article shows it in the Publisher-Subscriber pattern part.
There is still a lot more to uncover in queueing and messaging, but this release is focused on simple use-cases.
Laravel’s approach is different, and the philosophy behind it is also different. Symfony Messenger is a robust, highly configurable tool for messaging that also happens to act as a job queue. Laravel Queues, on the other hand, are a very simple, easy-to-use tool to create background Jobs. It comes with some nice features like rate limiting, preventing job overlap, and ensuring job uniqueness.
Depending on your needs, one or the other might work better for you.
Basic monitoring
I often see teams just “adding” queueing, and not considering the problems it might bring. What if the queue gets stuck, as there are not enough workers? Will you get an alert, or will you learn about it when customers start to complain?
As with adding any other solution to your system, it is good to consider gathering some metrics, and it is especially important with queues.
There are lots of ways to achieve that, and it will depend a lot on your existing observability/metrics stack. The most basic thing to monitor is your queue depth - how many messages are waiting in the queue. It is also good to monitor your failed messages.
If you don’t have a system in place, there are tools that integrate with a messenger, like zenstruck/messenger-monitor-bundle.
For Laravel, there is Horizon, but for some reason, it works only with Redis.
AWS SQS comes with its own monitoring that you can use.
Consider introducing such alerts:
if queue depth > 1000 messages for 5+ minutes
if failed message rate > 5% over 15 minutes
if average processing time increases by 50%
Tracking flows using correlation ID
Not required for the simpler flows, but sometimes, when the system grows, you might find yourself in a situation where it would be good to see why a job/message was triggered. This article shows an approach to implementing a request ID that allows you to combine log lines that were used in one “flow”, no matter if it was handled in an async message handler or not. This can be easily extended to your needs. In the system I work on, we have a correlation and causation ID. This way, we can both check all things that were run within a single flow, and also monitor the exact flow.
This is also helpful when introducing separate services that communicate with each other. Just pass the request ID to the service being called.
In-depth performance monitoring
In some cases, you might find that some additional monitoring would be nice. How long does a PDF export take? Are there any outliers recently? What is the mean time to process an XML import? Is there a performance degradation in any of the backend processes recently?
In such cases, the metrics that RabbitMQ (or any other) offers might not be enough. You might need to implement something on your side.
There are ways to hook into the process, like using worker events, but this might not be enough in some cases. You can also make use of the middleware pattern in Symfony Messenger
Queues are the backbone of scalable PHP applications, but they’re only as good as your implementation and monitoring. Start simple — even a DB queue with proper monitoring beats a misconfigured RabbitMQ setup. As your needs grow, you’ll know exactly when and how to evolve your approach. Stuff that works for Netflix is not necessarily the best option for a 20-user SaaS MVP ;)
Why is this newsletter for me?
If you are passionate about well-crafted software products and despise poor software design, this newsletter is for you! With a focus on mature PHP usage, best practices, and effective tools, you'll gain valuable insights and techniques to enhance your PHP projects and keep your skills up to date.
I hope this edition of PHP at Scale is informative and inspiring. I aim to provide the tools and knowledge you need to excel in your PHP development journey. As always, I welcome your feedback and suggestions for future topics. Stay tuned for more insights, tips, and best practices in our upcoming issues.
May thy software be mature!