Queues

Introduction

With TRAX LRS, you can use queues to optimize the writing flow on the standard Statement API.

When queues are enabled, each web request on the API performs the minimum tasks: it validates the request, records the statements, and return a response immediatly.

At this time, the recorded statements are pending statements. They are not available for retrieval because some additional tasks still have to be done.

That's where queues come into play. The statements processing queue takes a batch of statements, and then performs some tasks depending of the LRS configuration:

  • It may update agents, verbs and activity definitions.
  • It may record their relations with statements.
  • It may pseudonymize the statements.
  • It may void some statements.

Finally, the recorded statements are no longer "pending" and become available for retrieval.

Benefits

There are a number of benefits using queues.

  • Web requests are processed faster by the API, so it means a better reactivity for your xAPI clients.
  • The statements processing is optimized with batches, so it means less pressure on your database.
  • A single centralized process is responsible for updating activity definitions, so it solves concurrency issues that may arise with simultaneous requests.

Pitfalls

When an xAPI client records statements thru the Statement API and receives a success response, it does not mean the recorded statements are immediatly available for retrieval because the LRS needs a delay to process the data asynchronously.

This should not be an issue for most of the xAPI clients because they rarely need to get back the statements they just pushed to the LRS. However, this may be an issue when performing tests, because we may want to immediatly check that pushed statements have been recorded by the LRS.

This is something the xAPI specification has anticipated and that's why the LRS returns a X-Experience-API-Consistent-Through header in some situations.

Unfortunatly, our unit tests ignore this behavior, so you should disable queues to run the unit tests.

The ADL test suite takes into account a delay between recording and getting statements, but this delay is short, so the LRS must be reactive to pass the test. Check the running the queue worker options to solve this issue. Furthermore, the ADL test suite fails to test the Activity API when queues are enables. This is a testing issue which has been reported and which is not solved today.

Despite these issues, using queues brings great benefits in most of the situations.

Configuration

First of all, you need to enable the Statement API queueing in your .env file with:

DEFER_STATEMENTS_PROCESSING=true

Then you need to choose a queue driver which may be database or redis.

Database driver

The database driver uses your primary database to record queues information, which is fine for developement, testing, or if you don't want to use Redis at all.

QUEUE_CONNECTION=database

Redis driver

If you have the opportunity to use Redis, that may be the best choice for production.

  1. Install and start the Redis server.
  2. Check that Redis PHP extension is installed and enabled in your php.ini.
  3. Change the Redis settings in you .env file if needed. The following example shows the default value for each setting.
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0

Finally, enable the Redis queue in your .env file.

QUEUE_CONNECTION=redis

You can open http://your-traxlrs-root-url/test/queue to check your queueing configuration.

Running the queue worker

To start the queue worker, you can use the following command.

php artisan queue:work

This commands takes a --sleep option which determines how many seconds the worker will "sleep" if there are no new jobs available. The default value is 3, which means that you accept a delay of 3 seconds before recorded statements may become available for retrieval. This is usually fine, but you may define a shorter delay as illustrated in the following example.

php artisan queue:work --sleep=0.1

As recommended in the Lavarel documentation, you should use a process monitor such as Supervisor to ensure that the queue worker does not stop running.

Be aware the queue uses the application cache to store restart signals, so you should verify that a cache driver is properly configured for your application before using this feature. If the queue stops working and you can't restart it, try to clear the cache with: 

php artisan cache:clear

Refer to the Laravel documentation to go further with queues.