0

We have a delta table in databricks and de-duplicate the rows with dropDuplicates

We merge data into this table in batches and use

.whenMatchedUpdateAll()
                    .whenNotMatchedInsertAll()
                    .execute()

to prevent further duplicates in the table. Sometimes the table schema needs to change so I need to enable column mapping in mode name to be able to add a new column of the correct data type, drop the old column and change the name of the newly added column so the column names remain intact for the table

when I do this, the size of the table doubles... I think the other column mapping mode is id and I guess when we switch to name it messes with the de-duplication logic?

Improve this question
Related questions
21
Databricks - How to change a partition of an existing Delta table?
1
Access the existing Delta lake table data in another Azure Databricks
2
Can a Delta table be created using a merge where the source table is empty?
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.