dask apply: AttributeError: 'DataFrame' object has no attribute 'name'

Question

I have a dataframe of params and apply a function to each row. this function is essentially a couple of sql_queries and simple calculations on the result.

I am trying to leverage Dask's multiprocessing while keeping structure and ~ interface. The example below works and indeed has a significant boost:

def get_metrics(row):

    record = {'areaName': row['name'],
              'areaType': row.area_type,
              'borough': row.Borough,
              'fullDate': row['start'],
              'yearMonth': row['start'],
              }


    Q = Qsi.format(unittypes=At,
                   start_date=row['start'],
                   end_date=row['end'],
                   freq='Q',
                   area_ids=row['descendent_ids'])

    sales = _get_DF(Q)
    record['salesInventory'] = len(sales)
    record['medianAskingPrice'] = sales.price.median()
    R.append(record)

R = []
x = ddf.map_partition(lambda x: x.apply(_metric, axis=1), meta={'result': None})
    x.compute()

result2 = pd.DataFrame(R)

However, when I try to use .apply method instead (see below), it throws me 'DataFrame' object has no attribute 'name'...

R = list()
y = ddf.apply(_metrics, axis=1, meta={'result': None})

Yet, ddf.head() shows that there is a name column in the dataframe

You write dask_DF.apply() but say that ddf has a name column. Try ddf.apply(). — Mike Müller
– Mike Müller, Commented Oct 13, 2017 at 20:04
thanks, but that is just (resolved) misspelling, as I try to simplify the code here. It has nothing to do with the issue — Philipp_Kats
– Philipp_Kats, Commented Oct 13, 2017 at 20:06
The accepted answer also works for me. But the code sample in the question is too complex, and most of the code is not related to the problem. — Gary Wang
– Gary Wang, Commented Jun 22, 2021 at 15:34

Jean-François Corbett · Accepted Answer · 2018-06-27 10:11:22Z

10

If the output of your _metric function is a Series, maybe you should use meta=('your series's columns name','output's dtype')

This worked for me.

edited Jun 27, 2018 at 10:11

Jean-François Corbett

38.7k30 gold badges144 silver badges192 bronze badges

answered Jun 27, 2018 at 9:45

Cherrymelon

4532 gold badges8 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ta8 Over a year ago

Could you please explain why using a tuple makes a difference here? This is not apparent from the documentation.

Cherrymelon Over a year ago

I'm sorry,I hadn't use dask for nearly 2 years,I guess meta parameter tell dask the part you want use and output type,maybe because if u don't set dtype,dask maybe infer an error dtpye

Collectives™ on Stack Overflow

dask apply: AttributeError: 'DataFrame' object has no attribute 'name'

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related