Queuing Tasks with Redis

Introduction and background

Redis is an open-source data structure that is used for in-memory storage and helps developers across the globe with the quick and efficient organization and utilization of data. Even though many developers worldwide are still struggling to decide which open-source software application to use, Redis is quickly growing to be a widely popular choice. Currently, more than 3000 tech joints, including our team, are using Redis.

Redis supports several data structures, including lists, sets, sorted sets, hashes, binary-safe strings, and HyperLogLogs. Our team uses Redis to support queuing in this project.

Queuing is the storing or deferring of tasks of operation inside a queue so that they can be used later. It comes into use for operations which are large in number and/or takes up a lot of time. Tasks can be executed in two different methods –

Challenge

For this project, we needed to download large files, which is extremely time-consuming. To make the process more time-efficient, we decided to use queuing to effectively manage the download request. These download requests were added and served in the FIFO order.

Moreover, we wanted to retry the request in the time interval of one hour if it fails, until it fails three times. After this, the request is marked as failed and then removed from the queue. Our team soon found that manually creating and managing separate queues was rather inefficient, time-consuming, and troublesome, which hinted that we needed a new solution. This is where Redis comes in.

Solution

To create and manage separate queues more effectively, we put Kue npm package to the test. We hoped that it would make our task less time-consuming and more efficient.

And what exactly is Kue? Kue is a priority job queue package that is built for node.js and backed by Redis. What makes Kue so appealing for developers is that it provides us with a UI where the status of queues is displayed. This means that we can see the current status of the queues in real-time, thus helping us work better and smarter.

To use Kue, you have to first install it, then create a job Queue with kue.createQueue(). The next step is to create a job of type email with arbitrary job data using create() method. This enables the return of a job, which will be saved in Redis using save() method.

Then, after the jobs are created, the next step is to process them using process() method, after which failed jobs should be removed. You can then add Kue UI if you choose and install kue-ui package.

With this, you will be able to store your request in the Redis queue and then process them in FIFO order.

Related posts:

How do you figure out what career you should pursue, especially when you’re not even sure what exactly you’re passionate about? Many people, even after years of working, are not sure what they’re…

Viene o non viene? Intanto fuori piove e tira vento e a me - se viene- tocca pure uscire. Non venire!! Prego mentre tengo d’occhio i suoi movimenti dalla pagina social. Perchè se non viene lui…

The race to build autonomous vehicles is underway and the actual fleet planning process often gets left behind. This is unfortunate because planning the fleets presents a really cool and complex…