Aggregation

A common need for applications, in addition to retrieving information as a document list, like you have done with the “FT.SEARCH“ command, is to do some “aggregation”.

For example if we look at the movie documents, you may want to retrieve the number of movies grouped by release year starting with the most recent ones.

For this, RediSearch provides the FT.AGGREGATE command, with aggregations described as a data processing pipeline.

Let’s check out some examples.

Group By & Sort By

Number of movies by year
  1. > FT.AGGREGATE "idx:movie" "*" GROUPBY 1 @release_year REDUCE COUNT 0 AS nb_of_movies
  2. 1) (integer) 60
  3. 2) 1) "release_year"
  4. 2) "1964"
  5. 3) "nb_of_movies"
  6. 4) "9"
  7. ...
  8. 61) 1) "release_year"
  9. 2) "2010"
  10. 3) "nb_of_movies"
  11. 4) "15"

Number of movies by year from the most recent to the oldest
  1. > FT.AGGREGATE "idx:movie" "*" GROUPBY 1 @release_year REDUCE COUNT 0 AS nb_of_movies SORTBY 2 @release_year DESC
  2. 1) (integer) 60
  3. 2) 1) "release_year"
  4. 2) "2019"
  5. 3) "nb_of_movies"
  6. 4) "14"
  7. ...
  8. 11) 1) "release_year"
  9. 2) "2010"
  10. 3) "nb_of_movies"
  11. 4) "15"

Number of movies by genre, with the total number of votes, and average rating
  1. > FT.AGGREGATE idx:movie "*" GROUPBY 1 @genre REDUCE COUNT 0 AS nb_of_movies REDUCE SUM 1 votes AS nb_of_votes REDUCE AVG 1 rating AS avg_rating SORTBY 4 @avg_rating DESC @nb_of_votes DESC
  2. 1) (integer) 26
  3. 2) 1) "genre"
  4. 2) "fantasy"
  5. 3) "nb_of_movies"
  6. 4) "1"
  7. 5) "nb_of_votes"
  8. 6) "1500090"
  9. 7) "avg_rating"
  10. 8) "8.8"
  11. ...
  12. 11) 1) "genre"
  13. 2) "romance"
  14. 3) "nb_of_movies"
  15. 4) "2"
  16. 5) "nb_of_votes"
  17. 6) "746"
  18. 7) "avg_rating"
  19. 8) "6.65"

Count the number of females by country sorted from the biggest to smallest number.
  1. > FT.AGGREGATE idx:user "@gender:{female}" GROUPBY 1 @country REDUCE COUNT 0 AS nb_of_users SORTBY 2 @nb_of_users DESC
  2. 1) (integer) 193
  3. 2) 1) "country"
  4. 2) "china"
  5. 3) "nb_of_users"
  6. 4) "537"
  7. ...
  8. 11) 1) "country"
  9. 2) "ukraine"
  10. 3) "nb_of_users"
  11. 4) "72"

Apply Functions

Number of logins per year and month

The idx:user index contains the last_login field. This field stores the last login time as an EPOC timestamp.

RediSearch aggregation allows you to apply transformations to each record. This is done using the APPLY parameter.

For this example you have to use a date/time function to extract the month and year from the timestamp.

  1. > FT.AGGREGATE idx:user * APPLY year(@last_login) AS year APPLY "monthofyear(@last_login) + 1" AS month GROUPBY 2 @year @month REDUCE count 0 AS num_login SORTBY 4 @year ASC @month ASC
  2. 1) (integer) 13
  3. 2) 1) "year"
  4. 2) "2019"
  5. 3) "month"
  6. 4) "9"
  7. 5) "num_login"
  8. 6) "230"
  9. ...
  10. 14) 1) "year"
  11. 2) "2020"
  12. 3) "month"
  13. 4) "9"
  14. 5) "num_login"
  15. 6) "271"

Number of logins per weekday

Using the date/time Apply functions it is possible to extract the day of the week from the timestamp, so let’s see how the logins are distributed over the week.

  1. > FT.AGGREGATE idx:user * APPLY "dayofweek(@last_login) +1" AS dayofweek GROUPBY 1 @dayofweek REDUCE count 0 AS num_login SORTBY 2 @dayofweek ASC
  2. 1) (integer) 7
  3. 2) 1) "dayofweek"
  4. 2) "1"
  5. 3) "num_login"
  6. 4) "815"
  7. ...
  8. 8) 1) "dayofweek"
  9. 2) "7"
  10. 3) "num_login"
  11. 4) "906"

Filter

In the previous example you used the query string parameter to select all documents ("*") or a subset of the documents ("@gender:{female}")

It is also possible to filter the results using a predicate expression relating to values in each result. This is applied post-query and relates to the current state of the pipeline. This is done using the FILTER parameter.

Count the number of females by country, except China, with more than 100 users, and sorted from the biggest to lowest number
  1. > FT.AGGREGATE idx:user "@gender:{female}" GROUPBY 1 @country REDUCE COUNT 0 AS nb_of_users FILTER "@country!='china' && @nb_of_users > 100" SORTBY 2 @nb_of_users DESC
  2. 1) (integer) 163
  3. 2) 1) "country"
  4. 2) "indonesia"
  5. 3) "nb_of_users"
  6. 4) "309"
  7. ...
  8. 6) 1) "country"
  9. 2) "brazil"
  10. 3) "nb_of_users"
  11. 4) "108"

Number of login per month, for year 2020

This is similar to the previous query with the addition of a filter on the year.

  1. > FT.AGGREGATE idx:user * APPLY year(@last_login) AS year APPLY "monthofyear(@last_login) + 1" AS month GROUPBY 2 @year @month REDUCE count 0 AS num_login FILTER "@year==2020" SORTBY 2 @month ASC
  2. 1) (integer) 13
  3. 2) 1) "year"
  4. 2) "2020"
  5. 3) "month"
  6. 4) "1"
  7. 5) "num_login"
  8. 6) "520"
  9. ...
  10. 10) 1) "year"
  11. 2) "2020"
  12. 3) "month"
  13. 4) "9"
  14. 5) "num_login"
  15. 6) "271"


Next: Advanced Options