FastAPI issues with MongoDB - TypeError: 'ObjectId' object is not iterable

Question

I am having some issues inserting into MongoDB via FastAPI.

The below code works as expected. Notice how the response variable has not been used in response_to_mongo().

The model is an sklearn ElasticNet model.

app = FastAPI()


def response_to_mongo(r: dict):
    client = pymongo.MongoClient("mongodb://mongo:27017")
    db = client["models"]
    model_collection = db["example-model"]
    model_collection.insert_one(r)


@app.post("/predict")
async def predict_model(features: List[float]):

    prediction = model.predict(
        pd.DataFrame(
            [features],
            columns=model.feature_names_in_,
        )
    )

    response = {"predictions": prediction.tolist()}
    response_to_mongo(
        {"predictions": prediction.tolist()},
    )
    return response

However when I write predict_model() like this and pass the response variable to response_to_mongo():

@app.post("/predict")
async def predict_model(features: List[float]):

    prediction = model.predict(
        pd.DataFrame(
            [features],
            columns=model.feature_names_in_,
        )
    )

    response = {"predictions": prediction.tolist()}
    response_to_mongo(
        response,
    )
    return response

I get an error stating that:

TypeError: 'ObjectId' object is not iterable

From my reading, it seems that this is due to BSON/JSON issues between FastAPI and Mongo. However, why does it work in the first case when I do not use a variable? Is this due to the asynchronous nature of FastAPI?

While it seems like a stretch, does ObjectId gets populated inside the response object when sent to insert_one? If that is the case, your first example ends up with it being inserted in a throw away dict, while in the second example it gets inserted into a dict you're still referencing. — MatsLindh
– MatsLindh, Commented Mar 14, 2022 at 12:24
@MatsLindh I wouldn't think thats the case because the response object is not being changed in-place — SoftwareThings
– SoftwareThings, Commented Mar 14, 2022 at 12:50
Sounds like that's exactly what's happening based on the answer below :-) — MatsLindh
– MatsLindh, Commented Mar 14, 2022 at 14:01

Chris · Accepted Answer · 2024-12-05 18:07:21Z

As per the documentation:

When a document is inserted a special key, "_id", is automatically added if the document doesn’t already contain an "_id" key. The value of "_id" must be unique across the collection. insert_one() returns an instance of InsertOneResult. For more information on "_id", see the documentation on _id.

Thus, in the second case of the example you provided, when you pass the dictionary to the insert_one() function, Pymongo will add to your dictionary the unique identifier (i.e., ObjectId) necessary to retrieve the data from the database; and hence, when returning the response from the endpoint, the ObjectId fails getting serialized—since, as described in this answer in detail, FastAPI, by default, will automatically convert that return value into JSON-compatible data using the jsonable_encoder (to ensure that objects that are not serializable are converted to a str), and then return a JSONResponse, which uses the standard json library to serialize the data.

Solution 1

Use the approach demonstrated here, by having the ObjectId converted to str by default, and hence, you can return the response as usual inside your endpoint.

# place these at the top of your .py file
import pydantic
from bson import ObjectId
pydantic.json.ENCODERS_BY_TYPE[ObjectId]=str

return response # as usual

Solution 2

Dump the loaded BSON to valid JSON string and then reload it as dict, as described here and here.

from bson import json_util
import json

response = json.loads(json_util.dumps(response))
return response

Solution 3

Define a custom JSONEncoder, as described here, to convert the ObjectId into str:

import json
from bson import ObjectId

class JSONEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, ObjectId):
            return str(o)
        return json.JSONEncoder.default(self, o)


response = JSONEncoder().encode(response)
return response

Solution 4

You can have a separate output model without the 'ObjectId' (_id) field, as described in the documentation. You can declare the model used for the response with the parameter response_model in the decorator of your endpoint. Example:

from pydantic import BaseModel

class ResponseBody(BaseModel):
    name: str
    age: int


@app.get('/', response_model=ResponseBody)
def main():
    # response sample
    response = {'_id': ObjectId('53ad61aa06998f07cee687c3'), 'name': 'John', 'age': '25'}
    return response

Solution 5

Remove the "_id" entry from the response dictionary before returning it (see here on how to remove a key from a dict):

response.pop('_id', None)
return response

juanmirocks · Accepted Answer · 2023-10-16 08:47:31Z

1

Solution 4, from Chris's excellent answer, can also be accomplished with function output type hints. Thus:

from pydantic import BaseModel


class ResponseBody(BaseModel):
    name: str
    age: int


@app.get('/')
def example() -> ResponseBody:
    # you'd need to await this if you were using Motor (the Async MongoDB Driver)
    return db.my_collection.find_one(...)

answered Oct 16, 2023 at 8:47

juanmirocks

6,6197 gold badges56 silver badges53 bronze badges

Comments

zex_rectooor · Accepted Answer · 2025-04-13 07:23:44Z

I have use custom class to conver ObjectId used in mondodb in pydantic models
If you encounter this error :

ValueError: [TypeError("'ObjectId' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]

```

from typing import Any, Callable
from typing_extensions import Annotated
from bson import ObjectId as _ObjectId
from pydantic_core import core_schema
from pydantic.functional_serializers import PlainSerializer

class _ObjectIdPydanticAnnotation:
    # Based on https://docs.pydantic.dev/latest/usage/types/custom/#handling-third-party-types.

    @classmethod
    def __get_pydantic_core_schema__(
        # Generate a Pydantic core schema for the custom `_ObjectId` type.
        # This method is used to define how the `_ObjectId` type should be validated
        # and serialized when used with Pydantic models.
        # Args:
        #     cls: The class on which this method is defined.
        #     _source_type (Any): The source type for schema generation.
        #     _handler (Callable[[Any], core_schema.CoreSchema]): A handler function
        #         for generating core schemas.
        # Returns:
        #     core_schema.CoreSchema: A Pydantic core schema that validates and
        #     serializes the `_ObjectId` type.
        # The schema performs the following:
        #     1. Checks if the input is an instance of `_ObjectId`.
        #     2. If not, validates the input by attempting to create an `_ObjectId`
        #        instance from a string.
        #     3. Serializes the `_ObjectId` instance to a string representation.
        
        cls,
        _source_type: Any,
        _handler: Callable[[Any], core_schema.CoreSchema],
    ) -> core_schema.CoreSchema:
        def validate_from_str(input_value: str) -> _ObjectId:
            return _ObjectId(input_value)

        return core_schema.union_schema(
            [
                # check if it's an instance first before doing any further work
                core_schema.is_instance_schema(_ObjectId),
                core_schema.no_info_plain_validator_function(validate_from_str),
            ],
            serialization=core_schema.to_string_ser_schema(),
        )

def decode_object_id(id: _ObjectId):
    """
    Converts an ObjectId instance to its string representation.

    Args:
        id (_ObjectId): The ObjectId instance to be converted.

    Returns:
        str: The string representation of the given ObjectId.
    """
    return str(id)

ObjectId = Annotated[
    _ObjectId, _ObjectIdPydanticAnnotation,
    PlainSerializer(decode_object_id, return_type=str)
]

```

Collectives™ on Stack Overflow

FastAPI issues with MongoDB - TypeError: 'ObjectId' object is not iterable

3 Answers 3

Solution 1

Solution 2

Solution 3

Solution 4

Solution 5

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Solution 1

Solution 2

Solution 3

Solution 4

Solution 5

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related