Easily Deduplicate Jobs Using Laravel Queues

TL:DR; use a delayed dispatch plus the ShouldBeUnique contract on your job, and let the framework deduplicate it for you!

Picture this scenario: you receive webhooks from a third-party service that you need to handle somehow in your application. Due to multiple changes at the third party, you may receive multiple webhooks about the same resource within a short timeframe (2 minutes, for example), but you only want to process it a single time.

You could add a processed_at timestamp and track when you last processed the resource, and bypass additional processing using that.

Or you could combine several tools the Laravel framework already provides:

  1. Determine the expected timeframe for multiple updates
    • e.g., if somebody is manually updating multiple fields and you get a webhook for each change, estimate how long a user might be working before you want to process the changes
  2. Add a delay when you dispatch the job to cover that expected timeframe, plus a little bit extra: SomeJob::dispatch($data)->delay(now()->addMinutes(3))
  3. Add the ShouldBeUnique interface to your job
  4. Add the uniqueId() method to your job with a unique ID or some other key that will be used to find a match
  5. Voila!

Now when you receive an incoming webhook, your app will delay the processing for 3 minutes, and if there is already a job on the queue for that unique ID, the framework will not dispatch a second job.

One additional thing to consider: within that job, you may wish to perform an API call to retrieve the current state of the resource, since the dispatched job will contain the state as of the first webhook, not the most recent.

<?php

// somewhere in your app

SomeJob::dispatch($data)->delay(
    now()->addMinutes(3)
);
<?php
 
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Contracts\Queue\ShouldBeUnique;
 
class SomeJob implements ShouldQueue, ShouldBeUnique
{
    /**
     * Get the unique ID for the job.
     */
    public function uniqueId(): string
    {
        return $this->data->id;
    }

    ...
}

Fatal Errors with PHPUnit Test Suite: Laravel 10, PHPUnit 10, ParaTest 7

TL;DR: PHPUnit 10.5.32 is broken, 10.5.31 works.

After a composer update one one of my projects today, I started getting this error when running my test suite: Fatal error: Uncaught Illuminate\Contracts\Container\BindingResolutionException: Target [Illuminate\Contracts\Debug\ExceptionHandler] is not instantiable.

All the tests passed just fine…no errors or warnings. At the end of the test execution output, it displayed this stacktrace:

> php artisan test --parallel --stop-on-failure --stop-on-error
ParaTest v7.4.5 upon PHPUnit 10.5.32 by Sebastian Bergmann and contributors.

Processes:     10
Runtime:       PHP 8.3.10
Configuration: /Users/andrewminion/Sites/site-domain/phpunit.xml

.......................................................SS....   61 / 1014 (  6%)
.............................................................  122 / 1014 ( 12%)
.............................................................  183 / 1014 ( 18%)
.............................................................  244 / 1014 ( 24%)
.............................................................  305 / 1014 ( 30%)
.............................................................  366 / 1014 ( 36%)
..................................S...I......................  427 / 1014 ( 42%)
.............................................................  488 / 1014 ( 48%)
.............................................................  549 / 1014 ( 54%)
.............................................................  610 / 1014 ( 60%)
.............................................................  671 / 1014 ( 66%)
.............................................................  732 / 1014 ( 72%)
.............................................................  793 / 1014 ( 78%)
.............................................................  854 / 1014 ( 84%)
.............................................................  915 / 1014 ( 90%)
.........................I...................................  976 / 1014 ( 96%)
......................................                        1014 / 1014 (100%)

Time: 02:04.815, Memory: 70.50 MB

OK, but there were issues!
Tests: 1014, Assertions: 6462, Skipped: 3, Incomplete: 2.

Fatal error: Uncaught Illuminate\Contracts\Container\BindingResolutionException: Target [Illuminate\Contracts\Debug\ExceptionHandler] is not instantiable. in /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Container/Container.php on line 1126

Illuminate\Contracts\Container\BindingResolutionException: Target [Illuminate\Contracts\Debug\ExceptionHandler] is not instantiable. in /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Container/Container.php on line 1126

Call Stack:
  125.2423   77506896   1. Illuminate\Foundation\Bootstrap\HandleExceptions->Illuminate\Foundation\Bootstrap\{closure:/Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:254-256}() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:0
  125.2423   77507336   2. Illuminate\Foundation\Bootstrap\HandleExceptions->handleException() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:255
  125.2423   77470472   3. Illuminate\Foundation\Bootstrap\HandleExceptions->getExceptionHandler() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:183
  125.2423   77470472   4. Illuminate\Foundation\Application->make() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:288
  125.2423   77470472   5. Illuminate\Foundation\Application->make() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:946
  125.2423   77470472   6. Illuminate\Foundation\Application->resolve() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Container/Container.php:731
  125.2423   77470472   7. Illuminate\Foundation\Application->resolve() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:961
  125.2423   77470528   8. Illuminate\Foundation\Application->build() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Container/Container.php:795
  125.2423   77470624   9. Illuminate\Foundation\Application->notInstantiable() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Container/Container.php:921


Fatal error: Uncaught Illuminate\Contracts\Container\BindingResolutionException: Target [Illuminate\Contracts\Debug\ExceptionHandler] is not instantiable. in /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Container/Container.php on line 1126

Illuminate\Contracts\Container\BindingResolutionException: Target [Illuminate\Contracts\Debug\ExceptionHandler] is not instantiable. in /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Container/Container.php on line 1126

Call Stack:
  125.2457   77507224   1. Illuminate\Foundation\Bootstrap\HandleExceptions->Illuminate\Foundation\Bootstrap\{closure:/Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:254-256}() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:0
  125.2457   77507448   2. Illuminate\Foundation\Bootstrap\HandleExceptions->handleShutdown() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:255
  125.2460   77519232   3. Illuminate\Foundation\Bootstrap\HandleExceptions->handleException() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:231
  125.2460   77519232   4. Illuminate\Foundation\Bootstrap\HandleExceptions->getExceptionHandler() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:183
  125.2460   77519232   5. Illuminate\Foundation\Application->make() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:288
  125.2460   77519232   6. Illuminate\Foundation\Application->make() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:946
  125.2460   77519232   7. Illuminate\Foundation\Application->resolve() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Container/Container.php:731
  125.2460   77519232   8. Illuminate\Foundation\Application->resolve() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:961
  125.2460   77519232   9. Illuminate\Foundation\Application->build() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Container/Container.php:795
  125.2460   77519328  10. Illuminate\Foundation\Application->notInstantiable() /Users/andrewminion/Sites/site-domain/vendor/laravel/framework/src/Illuminate/Container/Container.php:921

After some trial-and-error, I narrowed it down to PHPUnit 10.5.32. Something in the changeset from 10.5.31 to 10.5.32 caused this error.

This also happens only when running tests in parallel (php artisan test --parallel), suggesting to me that perhaps the PHPUnit output changed in some way that ParaTest was not expecting.

My fix, at least for now, is to stick with PHPUnit 10.5.31, and hopefully an imminent upgrade to Laravel 11 and PHPUnit 11 will resolve it.

Laravel Testing Tip: Reset UUID Creation

TL;DR: if you call Str::createUuidsUsing(…) in a test method, don’t forget to call Str::createUuidsNormally() later in that same method (or the tearDown method), or the rest of your test suite will continue to use that same UUID.

Laravel’s string helper provides a nice interface for generating uuids, as well as a nice way to fake the UUIDs during a test:

While this is very useful, I expected that it would reset between tests, similar to Queue::fake(), Http::fake(), etc.

However, because of how the Str helper generates UUIDs, whatever you provide will be used for the rest of the test run. If you have other tests or app code that expects a unique UUID each time Str::uuid() is used, you may get unexpected results.

There are a couple of options to work around this:

  1. After running the code that needs a UUID, call Str::createUuidsNormally() to reset the Str helper.
  2. If you don’t actually need the value of the UUID for testing, you can wrap your code in the freezeUuids() method instead. Once your code in the callback finishes running, the framework will call createUuidsNormally() to reset everything for you:

Laravel Tip: Generating Signed URLs with Ignored Parameters

TL;DR: don’t use ignored URL parameters when building signed URLs or the resulting signed URL will be invalid. Instead, manually append them to the resulting URL.

Laravel includes some really nice helpers for building signed URLs: https://laravel.com/docs/master/urls#signed-urls

They allow you to generate a URL containing a signature that prevents anybody from modifying the URL to access something you didn’t intend (e.g., you could provide a signed URL for a specific post with ID 123; if somebody changed that ID to 124, then Laravel will display a 403 Signature Invalid error rather than happily displaying post 124).

Occasionally you may wish to ignore certain URL parameters when validating the signature (e.g., a pagination or print parameter).

In this case, you cannot include the ignored parameter when generating the signed URL, or the URL will be invalid.

Here’s an example. This route ignores the print parameter when verifying the signature:

If you generate a signed URL without the print parameter, it will be valid. But if you include print in the URL parameters for the helper method, the resulting signature will be invalid, because Laravel uses all of those parameters to generate the signature. Instead, just add the new parameter to the end of the resulting URL:

Note how examples 1 and 3 have the same signature; that is the signature that Laravel calculates when determining what the correct signature should be to verify that the URL has not been modified. The example 2 use print=true when generating the signature, but will remove that parameter when verifying the signature, so they don’t match.

Update: I submitted a PR to the framework to pass ignored parameters to the signed route methods to make this easier.

Debugging HTTP Client Request Assertions in Laravel Test Suites

The Http::assertSent(), Event::assertDispatched(), and Queue::assertPushed()) test methods are perhaps a bit unintuitive.

They seem like they would run just once and check the assertions provided in your callback.

In fact, that callback runs once for each HTTP request (or dispatched event or queued job) and it evaluates the logic inside the callback for each. So if you have 10 requests (or events or jobs) and 3 of them return true it passes. If you have 10 requests and 9 of them return true it passes. It only fails if none of the callbacks return true.

So using dd($request) in one of those assertions is only dumping out the first one and then killing the test. You might not see the one you actually need.

If you’re trying to see what data is in the request, you’re better off either doing something like Log::debug($request->body()) or ray($request->url())1 or putting an xdebug breakpoint on the first line of the callback, running the test, and pressing the “continue” button until you get to the request you’re trying to inspect (possibly the second or fourth or tenth, depending on what happens before the one you want to inspect).

Here is a few code samples that may clarify this a bit more:

Adding Sentry to a Laravel/Inertia/Vue 3 app

I’m in the process of adding Sentry to a Laravel app that uses Laravel Jetstream with Inertia.js and Vue 3, and the Sentry Vue 3 documentation wasn’t working for me because the app setup was wrapped inside a createInertiaApp function.

The key is to add Sentry in the setup method of that function:

Shimming MySQL Functions into SQLite for Laravel CI/CD Testing

Colin DeCarlo presented a talk at Laracon Online where among other useful tips, he demonstrated how to shim MySQL functions in an SQLite database (e.g., add functions that MySQL has but SQLite does not).

Here are two examples that I just needed in a project (FLOOR and DATEDIFF):

use Illuminate\Support\Facades\DB;

DB::getPdo()->sqliteCreateFunction('floor', fn ($value) => floor($value));
DB::getPdo()->sqliteCreateFunction('datediff', fn ($date1, $date2) => Carbon::parse($date1)->diff(Carbon::parse($date2))->days);

Redirect to Original URL with Laravel Socialite

We’re using Laravel Socialite with a self-hosted GitLab instance to authenticate users for an internal tool.

Every time the session times out, the OAuth flow redirects the user back to the default dashboard, regardless of what URL the user originally requested.

However, Laravel provides a url.intended session key with the original URL; here’s how I used it, with a fallback to the dashboard URL:

return redirect()->to(
    session()->get('url.intended', route('dashboard')
);

Database Platform Comparisons for Laravel Feature Tests

TL;DR: MySQL significantly outperforms MariaDB in my automated test suite.

The Problem

This Twitter thread prompted me to do a bit of research on database platforms for Laravel automated tests.

I’ve recently been building an ecommerce app based on Laravel. Partway through development, we added geometry fields to a couple of tables in order to determine distances. I’ve been using this spatial package, so SQLite was not an option for my test suite.

As soon as I switched the testing database driver from SQLite to MariaDB, my tests immediately took an extra 12–13 seconds to run, regardless of whether I ran the entire test suite, a single file, or just one test.

This significantly lengthened the feedback loop when making changes to code and re-running tests.

So when I saw Jack Ellis mention that he uses MySQL for his test suite, it made me curious if he had the same issue.

He said that one of his test files runs 39 tests in < 2 seconds, so apparently it’s not been a problem for him.

Context

  • I’m using the LazilyRefreshDatabase trait added in Laravel 8.62.0 on my entire test suite
  • I’m using squashed migrations
  • Many of my tables have constrained foreign keys referencing other tables

Comparisons

I decided to do some digging; here are comparisons using four different platforms for the same test in my application.

MariaDB

I’ve been using MariaDB as the main database platform on my development machine for years. Currently I’m on version 10.6.4.

In-Memory SQLite Database

I temporarily disabled the geometry features and tried the in-memory SQLite database (DB_CONNECTION=:memory:); it performed much better for the same tests:

SQLite File Database

I then tried with an SQLite file (DB_CONNECTION=sqlite), and it performed about the same:

MySQL 8

I have an installation of MySQL 8 set up for one app that uses some specific MySQL 8 and I figured why not give that a try too.

Here are the results:

Summary

For some reason, MariaDB takes approximately 12–13 seconds to tear down and recreate the database before starting to run tests, but MySQL is much faster.

While testing MariaDB, I opened the raw data directory for the database, and noticed chunks of files being removed and recreated at a time, so perhaps the foreign key constraints are (part of) the culprit here.

I do have 77 databases with ~3800 tables in my MariaDB installation built up from various projects over the years. It seems unlikely, but theoretically possible, that the server size could be part of the problem too.

I think I’ll experiment with switching back to MySQL as my development platform of choice.

Have you run into this same issue? Have any tips or tricks? Let me know in the comments.

Retrieving Route and Parameters from an Arbitrary URL in Laravel

I build an oEmbed provider in a Laravel application the other day and needed to parse an arbitrary URL to determine the route and parameters passed in order to determine the response.

Since I already had the routes built for the possible URLs, I didn’t want to duplicate code and re-parse them.

Here’s how I ended up retrieving the route and parameters: