Summary Running Django migrations against our MongoDB database does not create MongoDB collections or indexes as defined in our app. The command completes without errors, but no collections or indexes are provisioned in MongoDB.
Environment
- Django: 5.2.5
- django-mongodb-backend: 5.2.2
- Python: 3.11.14
- Database setup: PostgreSQL as default, MongoDB as secondary via django-mongodb-backend
Steps to Reproduce
- Configure DATABASES with a mongodb alias (see snippet below).
- Implement models that should live in MongoDB and include indexes/constraints.
- Implement a database router that routes models with
use_db = "mongodb"to the mongodb DB. Run: - python manage.py makemigrations mailbot_search_agent
- python manage.py migrate mailbot_search_agent --database=mongodb
Expected
- MongoDB collections are created for the models that declare
use_db = "mongodb". - Declared indexes and unique constraints are created.
- If supported by backend, custom Atlas Search/Vector index definitions are applied. Actual migrate --database=mongodb completes, but:
- Collections are not created (or get created only after first write).
- Indexes defined in migrations (0002) and in model Meta/indexes are not present in MongoDB.
- Atlas Search/Vector indexes (declared via backend-provided Index classes) are not created.
DATABASES Configuration (snippets)
MONGO_CONNECTION_STRING = os.environ.get("MONGO_CONNECTION_STRING")
MONGO_DB_NAME = os.environ.get("MONGO_DB_NAME", "execfn")
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "execfn",
"USER": "execfn_user",
"PASSWORD": os.environ.get("DJANGO_DB_PASSWORD"),
"HOST": "localhost",
"PORT": "5432",
},
"mongodb": {
"ENGINE": "django_mongodb_backend",
"NAME": "execfn",
"HOST": os.environ.get("MONGO_CONNECTION_STRING"),
},
}
Router (snippets)
class DatabaseRouter:
MONGODB = "mongodb"
DEFAULT = "default"
def _get_model_db(self, model):
# Class-level attribute `use_db` drives routing
use_db = getattr(model, "use_db", None)
return use_db or self.DEFAULT
def db_for_read(self, model, **hints):
return self._get_model_db(model)
def db_for_write(self, model, **hints):
return self._get_model_db(model)
def allow_relation(self, obj1, obj2, **hints):
db1 = self._get_model_db(obj1._meta.model if hasattr(obj1, "_meta") else type(obj1))
db2 = self._get_model_db(obj2._meta.model if hasattr(obj2, "_meta") else type(obj2))
return db1 == db2
def allow_migrate(self, db, app_label, model_name=None, **hints):
if "model" in hints:
model = hints["model"]
return db == self._get_model_db(model)
if model_name is not None:
try:
from django.apps import apps
model = apps.get_model(app_label, model_name)
return db == self._get_model_db(model)
except Exception:
pass
if db == self.DEFAULT:
return True
if db == self.MONGODB:
return False
return False
Model Definitions (snippets)
# mailbot_search_agent/models.py
from django.db import models
from django_mongodb_backend.fields import ArrayField, ObjectIdAutoField
from django_mongodb_backend.indexes import SearchIndex, VectorSearchIndex
from execfn.common.models import MongoDBModel # abstract, sets use_db="mongodb"
class Email(MongoDBModel):
use_db = "mongodb"
_id = ObjectIdAutoField(primary_key=True)
message_id = models.CharField(max_length=255)
thread_id = models.CharField(max_length=255)
user_id = models.IntegerField()
subject = models.CharField(max_length=512)
snippet = models.TextField(null=True)
received_at = models.DateTimeField()
labels = models.JSONField(default=list)
content = models.TextField()
from_email = models.CharField(max_length=320)
from_name = models.CharField(max_length=255)
to_emails = ArrayField(models.CharField(max_length=320), default=list)
to_names = ArrayField(models.CharField(max_length=255), default=list)
# cc_/bcc_ omitted for brevity
class Meta:
db_table = "emails"
constraints = [
models.UniqueConstraint(fields=["message_id"], name="message_id_unique"),
]
indexes = [
SearchIndex(
field_mappings={
"subject": {"type": "string"},
"content": {"type": "string"},
"from_email": {"type": "string"},
"from_name": {"type": "string"},
},
name="emails_text_search_index",
),
models.Index(fields=["user_id"]),
]
class EmailEmbedding(MongoDBModel):
use_db = "mongodb"
_id = ObjectIdAutoField(primary_key=True)
gmail_message_id = models.CharField(max_length=255)
user_id = models.IntegerField()
subject = models.CharField(max_length=512)
content = models.TextField()
received_at = models.DateTimeField()
labels = ArrayField(models.CharField(max_length=255), default=list)
chunk_start = models.IntegerField()
chunk_end = models.IntegerField()
embedding = ArrayField(models.FloatField(), size=1536)
from_email = models.CharField(max_length=320)
from_name = models.CharField(max_length=255)
class Meta:
db_table = "email_embeddings"
constraints = [
models.UniqueConstraint(
fields=["gmail_message_id", "chunk_start", "chunk_end"],
name="embedding_chunk_unique",
)
]
indexes = [
models.Index(fields=["user_id"]),
SearchIndex(
field_mappings={
"subject": {"type": "string"},
"content": {"type": "string"},
"from_email": {"type": "string"},
"from_name": {"type": "string"},
},
name="embeddings_text_search_index",
),
VectorSearchIndex(
fields=["embedding", "user_id", "gmail_message_id", "received_at", "from_email", "from_name"],
similarities="cosine",
name="embeddings_vector_search_index",
),
]
Migrations
- 001: legacy relational models (no Mongo collections)
- 002: contains index/collection creation for Mongo (our expectation) Command used: python manage.py migrate mailbot_search_agent --database=mongodb
Problem Details
- Despite the router routing these models to mongodb, applying 0002 on --database=mongodb does not create the declared collections or indexes.
- Unique constraints and Django models.Index definitions appear not to materialize in Mongo.
- Custom indexes declared via django_mongodb_backend.indexes.SearchIndex and VectorSearchIndex also do not materialize.
Help needed for these queries
- Are
models.Index,UniqueConstraint, and the backend’sSearchIndex/VectorSearchIndexexpected to be emitted into actual MongoDB indices via Django migrations withENGINE = "django_mongodb_backend"? - If yes, what migration operations are supported and how should we structure migrations to ensure collections and indexes are created on migrate --database=mongodb?
- If no, what is the recommended approach for provisioning:
- Collections (with validators if supported)
- Unique indexes
- Atlas Search and Vector Search indexes
- via Django migration operations (e.g., custom operations, RunPython) using this backend?
- Is there a known limitation where collections are only created upon first write, and indexes must be created programmatically instead of via migrations?
