1

What could be a cause of the following error of my code in a Databricks notebook, and how can we fix the error?

ImportError: cannot import name 'pipelines' from 'pyspark' (/databricks/python/lib/python3.12/site-packages/pyspark/__init__.py)

This is the top line of the Databricks notebook that throws the error:

from pyspark import pipelines as dp

According to the following quote from Basics of Python for pipeline development from Databricks' team, we need to import the above module for creating Lakeflow Declarative pipelines using Python:

All Lakeflow Declarative Pipelines Python APIs are implemented in the pyspark.pipelines module.

Also, as we know PySpark is an integral and primary programming interface used within the Databricks platform. So, what I may be missing here that causes the error?

Improve this question
3
  • What version of pyspark do you have installed? Commented Oct 17 at 20:10
  • @JohnGordon I'm using Latest Free edition that has psyspark version 4.0.0 and Databricks runtime version 17.2 Commented Oct 17 at 23:04
  • when I used AI on Databrick to fix problem then it suggested "module is only available in Databricks environments with Lakeflow Declarative Pipelines enabled`, so it seems it not problem with code but with environment/workspace (hardware and probably account which you use). So it needs to chagen some settings before you can use "Lakeflow Declarative". Commented Oct 19 at 12:33

1 Answer 1

Reset to default
2

You are trying to run the command on a generic notebook as a generic pyspark import.

The pipeline module can be accessed only within a context of a pipeline.

Please refer this documentation for clarity:

https://docs.databricks.com/aws/en/ldp/developer/python-ref/#gsc.tab=0

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Your referenced link helped. Quote from this section of your link: The pipelines module is only available in the context of a pipeline. It is not available in Python running outside of pipelines.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.