AWS Glue Script Scanning Entire Table Despite Date Filter

Question

I have written a small Glue script that fetches some data between two dates, but I found that it scans the entire table instead of just the data within the specified time range. I also tried creating an index on my SQL database, but it didn’t make any difference.

SELECT * 
FROM employee_management.employees 
WHERE created_at >= '2025-11-11 23:00:00'
  AND created_at < '2025-11-12 23:00:00';

Are a large portion of the rows in your table are in the date range you show in this example? If the MySQL optimizer sees that your query condition would fetch, say, at least 20% of the rows, it calculates that using an index to get those rows indirectly would probably be more costly than the table-scan, so it skips using the index. The 20% figure isn't an official threshold, it's just my observation. — Bill Karwin
– Bill Karwin, Commented Nov 13 at 18:20
Although the table has little data, MySQL performs a full table scan according to the execution plan. — new coderrrr
– new coderrrr, Commented Nov 13 at 18:32
Can you be specific about what you mean by "little data?" For example: SELECT COUNT(*) FROM employee_management.employees; — Bill Karwin
– Bill Karwin, Commented Nov 13 at 18:52
You said you tried creating an index, but you also said you don't know how to create an index. Can you confirm whether your table has an index? Here's a query that can do that: select index_name from information_schema.statistics where table_schema='employee_management' and table_name='employees'; — Bill Karwin
– Bill Karwin, Commented Nov 13 at 19:08

O. Jones · Accepted Answer · 2025-11-20 23:22:58Z

0

Try this index creation data definition language:

CREATE INDEX created_at_dex ON  employee_management.employees(created_at);

If you still get a full table scan, you have some choices.

trust the query planner when it decides scanning the table is going to generate your result set faster than using the index.
Try ANALYZE TABLE employee_management.employees; to update the index statistics, in case the query planner is using stale stats.
(Best) Get rid of SELECT * and name the columns you actually need.

answered yesterday

O. Jones

110k17 gold badges134 silver badges187 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

AWS Glue Script Scanning Entire Table Despite Date Filter

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related