MySQL 8 Calculating Average by Partitioning By Date

Question

I've setup a fiddle here: https://www.db-fiddle.com/f/snDGExYZgoYASvWkDGHKDC/2

But also:

Schema:

CREATE TABLE `scores` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `shift_id` int unsigned NOT NULL,
  `employee_name` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
  `score` double(8,2) unsigned NOT NULL,
  `created_at` timestamp NOT NULL,
  PRIMARY KEY (`id`)
);

INSERT INTO scores(shift_id, employee_name, score, created_at) VALUES
(1, "John",   6.72, "2020-04-01 00:00:00"),
(1, "Bob",   15.71, "2020-04-01 00:00:00"),
(1, "Bob",   54.02, "2020-04-01 08:00:00"),
(1, "John",  23.55, "2020-04-01 13:00:00"),
(2, "John",   9.13, "2020-04-02 00:00:00"),
(2, "Bob",   44.76, "2020-04-02 00:00:00"),
(2, "Bob",   33.40, "2020-04-02 08:00:00"),
(2, "James", 20,    "2020-04-02 00:00:00"),
(3, "John",  20,    "2020-04-02 00:00:00"),
(3, "Bob",   20,    "2020-04-02 00:00:00"),
(3, "Bob",   30,    "2020-04-02 08:00:00"),
(3, "James", 10,    "2020-04-02 00:00:00")

Query 1:

-- This doesn't work

SELECT
    employee_name,
    DATE_FORMAT(created_at, '%Y-%m-%d') AS `date`,
    ANY_VALUE(AVG(score) OVER(PARTITION BY(ANY_VALUE(created_at)))) AS `average_score`
FROM
  scores
GROUP BY
    employee_name, date;

Query 2:

SELECT
    employee_name,
    DATE_FORMAT(created_at, '%Y-%m-%d') AS `date`,
    ANY_VALUE(AVG(score)) AS `average_score`
FROM
  scores
GROUP BY
    employee_name, date;

Query 3:

-- This works but scales very poorly with millions of rows

SELECT
    t1.employee_name,
    ANY_VALUE(DATE_FORMAT(t1.created_at, '%Y-%m-%d')) AS `date`,
    ANY_VALUE(SUM(t1.score) / (
      SELECT SUM(t2.score)
      FROM scores t2
      WHERE date(t2.created_at) = date(t1.created_at)
    ) * 100) AS `average_score`
FROM
  scores t1
GROUP BY
    t1.employee_name, date;

The third query executes correctly but in my testing has been very slow when scaling to millions of rows. I think this is because it is a correlated subquery and runs millions of times.

The first two attempts are me trying to created to use MySQL 8 Window Functions to partition the average calculation. However, these are giving unexpected results. The total average_scores for a given day should add up to 100, like it does in the 3rd query.

Does anyone know of a more efficient way to calculate this?

It's also worth noting that in reality, there will also be a WHERE IN on the queries to filter by specific shift_ids. The number of shift_ids given could be in the hundreds of thousands, up to a million.

One other thing being considered is ElasticSearch. Would it help with calculating these in a quicker way?

@GMB query three gives the correct result and is shown in the fiddle. But it doesn’t scale well at all. — Shiv
– Shiv, Commented Apr 12, 2020 at 9:34

GMB · Accepted Answer · 2020-04-12 09:33:34Z

1

You can use window functions. The trick is to take a window sum of the total score per employee for each day, like so:

select
    employee_name,
    date(created_at) created_date,
    100 * sum(score) / sum(sum(score)) over(partition by date(created_at)) monthly_score
from scores
group by employee_name, date(created_at)

In your DB Fiddle, this yields:

| employee_name | created_date | monthly_score |
| ------------- | ------------ | ------------- |
| John          | 2020-04-01   | 30.27         |
| Bob           | 2020-04-01   | 69.73         |
| John          | 2020-04-02   | 15.55342      |
| Bob           | 2020-04-02   | 68.42864      |
| James         | 2020-04-02   | 16.01794      |

answered Apr 12, 2020 at 9:33

GMB

224k25 gold badges103 silver badges151 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

MySQL 8 Calculating Average by Partitioning By Date

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related