How to create Pandas Dataframe from Google API Response Object

Question

I'm trying to get a Google Analytics Admin API response into a Pandas dataframe, and I'm having trouble parsing the results. I'm working with the list customDimensions method within the admin_v1beta library. I can see the results, but I haven't yet figured out how to get them into a format that's usable within Pandas.

My function:

def get_custom_dimensions(property_filter):
    client = admin_v1beta.AnalyticsAdminServiceClient()

    request = admin_v1beta.ListCustomDimensionsRequest(
        parent=property_filter
    )

    return client.list_custom_dimensions(request=request)

And the output:

    ga4_custom_dimensions = get_custom_dimensions("properties/xxx")

    print(type(ga4_custom_dimensions))

<class 'google.analytics.admin_v1beta.services.analytics_admin_service.pagers.ListCustomDimensionsPager'>

    print(ga4_custom_dimensions)

 
ListCustomDimensionsPager<custom_dimensions {
  name: "properties/xxx/customDimensions/yy1"
  parameter_name: "custom_dimension_zz1"
  display_name: "Custom Dimension ZZ1"
  description: "The dimension ZZ1 useful for filtering."
   scope: EVENT
}
custom_dimensions {
  name: "properties/xxx/customDimensions/yy2"
  parameter_name: "custom_dim...

The response has its own class, so before I can do anything with it I need to convert it and I haven't yet figured out how. My first thought was to convert to JSON:

custom_dimensions_json_1 = json.dumps(ga4_custom_dimensions.__dict__)
custom_dimensions_json_2 = json.dumps(vars(ga4_custom_dimensions))

These produce the same error,

TypeError: Object of type _GapicCallable is not JSON serializable.

My next attempt was via the Pandas json_normalize method:

custom_dimension_df = pd.json_normalize(ga4_custom_dimensions)

The result is a dataframe with only an index.

Lastly, I tried parsing the results directly:

temp_df = pd.DataFrame(ga4_custom_dimensions['custom_dimensions'])[['name',
    'parameter_name',
    'display_name',
    'description',
    'scope']]

This produces "TypeError:

ListCustomDimensionsPager' object is not subscriptable.

Can anyone guide me on how to parse the API response within Pandas?

Lenwood · Accepted Answer · 2023-12-14 20:36:24Z

I was able to solve this and move on. I updated my function to iterate through a series of str.replace() steps to make the response valid json then create the pandas dataframe from there.

def get_custom_dimensions(property_filter):
    client = admin_v1beta.AnalyticsAdminServiceClient()

    request = admin_v1beta.ListCustomDimensionsRequest(
        parent=property_filter
    )

    full_response = client.list_custom_dimensions(request=request)

    df_list = []

    for response in full_response:
        step1 = response.__dict__
        step2 = str(step1)
        step3 = step2.replace(': name:', ': "name" :')
        step4 = step3.replace('parameter_name:', ', "parameter_name" :')
        step5 = step4.replace('display_name:', ', "display_name" :')
        step6 = step5.replace('description:', ', "description" :')
        step7 = step6.replace('scope:', ', "scope" :')
        step8 = step7.replace('disallow_ads_personalization: true', ', "disallow_ads_personalization" : "true"')
        step9 = step8.replace("'_pb': ", "")
        step10 = step9.replace(' : EVENT', ' : "EVENT"')
        step11 = step10.replace(' : USER', ' : "USER"')
        step12 = step11.encode('utf-8').decode('unicode_escape')
        step13 = json.loads(step12)
        df_list.append(step13)

    return pd.DataFrame(df_list)

Calling this produces a Pandas dataframe with the response that can then be saved, manipulated, etc.

custom_dimension_df = get_custom_dimensions("xxx")

I don't have much experience working with class objects in python, my solution isn't very elegant but it works.

Collectives™ on Stack Overflow

How to create Pandas Dataframe from Google API Response Object

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related