1,665 questions
-4
votes
0
answers
19
views
Did not find local replica set configuration document at startup shard pod kubernetes replicaset is not initialized [closed]
{"t":{"$date":"2025-11-13T16:42:03.966+00:00"},"s":"I", "c":"STORAGE", "id":4795901, "ctx":"...
1
vote
1
answer
119
views
Sharding a large parquet file in Polars makes the system crash
I have been trying to make shards from .parquet file of size 20GB and my code tends to crash the system by the time it reaches the last shard but sometimes also works. I am quite new to working with ...
0
votes
1
answer
88
views
Why does my MongoDB sharded cluster have only 10 chunks after importing 716 million documents?
I have a MongoDB (v8.0.9) sharded cluster running in Kubernetes with the following setup:
3 shards, each with 2 replicas
An empty collection was created with a hashed shard key (over a UUID field)
...
- 1
-1
votes
1
answer
45
views
How to manage shard mechanism into the tables?
I have a database with 100 million records and 10 tables. Out of these, I’ve applied sharding (based on geo-location) to only 3 tables. The remaining tables are not sharded.
Now, if I want to run a ...
1
vote
0
answers
41
views
Best Approach to Handle Unsharded Collections in a MongoDB Global Write Cluster?
Thanks for taking the time to look into my issue.
I currently have a MongoDB Atlas global write cluster with three region-specific shards in UK, US, and HK. The unsharded collections are located ...
0
votes
1
answer
54
views
Per EntityType Active Entity Limit in Cluster Sharding Passivation
I have a clustered system that has a number of different entity types with different memory and computation complexity characteristics and would like to use active-entity-limit in my Passivation ...
- 14.5k
0
votes
1
answer
344
views
Replication vs Sharding (for read scalability vs write scalability)
Can someone explain why they say db replication is more ideal for read scalability while sharding is more ideal for write scalability?
From my current understanding:
replication allows read traffic ...
0
votes
0
answers
149
views
Create read-only shards from a postgres DB
I have an application that consists of a master application+DB and a bunch of edge servers. Each edge server syncs a subset of the master data via custom API calls. I would like to simplify this ...
- 142
0
votes
1
answer
178
views
Cron job scheduled in spring app to distribute data with all instances
I have a design question
Cron job scheduled in spring app using @scheduled, I have 4 instances and I want the job to run in all the instances by distributing data. Say I need to process 1000 data, ...
0
votes
1
answer
179
views
How failures and restore operations in sharding (consistent hashing)
In consistent hashing, suppose we are using username as for hashing
hashFunction(username) = nodeA
Now from what I understand, if there is any failure or a node is removed requests will be directed to ...
- 47
-3
votes
2
answers
123
views
Horizontal scaling strategy with 10,000 shards [closed]
My app has a User collection. Each document in the collection averages about .04 MB. At worst case, a document may slightly exceed .1 MB. Needless to say, these are small documents. However, each ...
3
votes
1
answer
3k
views
Why does Elastic search limit the maximum shard number to 1k per node?
"The cluster shard limits prevent creation of more than 1000 non-frozen shards per node, and 3000 frozen shards per dedicated frozen node. Make sure you have enough nodes of each type in your ...
- 317
0
votes
0
answers
95
views
MongoDB Shard Cluster
Good day people .Am having some trouble adding initiating replicas using the rs.initiate().
below is my docker-compose file
version: '3'
services:
configs1:
container_name: configs1
image: ...
- 28
0
votes
1
answer
88
views
Make shard processes use the same pool
I want to create a pool in my shard manager (server.js) and pass it to shard processes (bot.js). Here is my sharding manager (server.js):
var clientMysqlEvent = require('./database/botpool.js')....
- 41
0
votes
1
answer
181
views
Sharing large partition key in Cassandra: how to keep a fixed shard size?
I read this post on how to deal with large partitions and partitioning hotspots, their solution is to add a sharding key as part of the partition key, and keep the shard size at a fixed size, say 1000....
- 7,578
0
votes
1
answer
1k
views
How to fix TransientTransactionError in Shard Mongo DB with spring boot?
Scenario:
I am using Mongo DB 6+ version with spring boot.
@Bean
MongoTransactionManager transactionManager(MongoDatabaseFactory dbFactory) {
return new MongoTransactionManager(...
- 709
-1
votes
1
answer
303
views
Sharding multiple tables with no common column
I wanted to understand sharding in case of multiple tables which might be used for QnA websites like Quora/SO. Let's assume that users can ask questions, give answers and comment on both questions and ...
- 1
0
votes
1
answer
913
views
Upgraded Kibana and Elasticsearch to v8.9.2, Kibana is not starting, Error: Not enough active copies to meet shard count of [ALL] (have 1, needed 2)
I recently upgraded Elasticsearch and Kibana to v8.9.2 using Bitnami Helm chart on my Kubernetes AWS EKS cluster. Elasticsearch is running fine with 3 nodes but Kibana is restarting again and again as ...
- 5,920
0
votes
0
answers
15
views
Can't able to add clustered node IP in Mongos
I'm a beginner at Mongodb. so I trying to configure the sharding in MongoDB. so I have 2 separate server with Linux centos OS and installed MongoDB 4.2.24 version in both servers so initially there is ...
1
vote
0
answers
109
views
Question on sharding using planetscale (keeping data within a country using sharding)
I'm building a niche social media DB on planetscale that spans users living in multiple countries. Is there a way I can shard my social media user data per country and have that data physically ...
- 23
0
votes
0
answers
31
views
MongoDB sharding cluster show wrong size
I created a mongodb cluster according to digitalocean's instructions. I sharded my database with the following command:
sh.shardCollection("database.Collection", {"userId": 1})
...
1
vote
0
answers
86
views
How many shards is facebook's user table partitioned across?
"Shard Manager manages tens of millions of shards hosted on hundreds of thousands of servers across hundreds of applications in production."
https://engineering.fb.com/2020/08/24/production-...
1
vote
0
answers
340
views
Why does elasticsearch not use index.routing_partition_size as specified? (custom routing)
I'm using custom routing on elasticsearch 8.7.1, and it seems like the routing_partition_size is not being implemented as expected unless I manually specify number_of_routing_shards.
For example, if I ...
0
votes
0
answers
146
views
Why relocation of shards doesn't happens when unassigned shards are present in a cluster?
I was reading about relocation of shards in elastic search and allocation of unassigned shards. Came upon this issue - https://github.com/elastic/elasticsearch/issues/12273.
Here it is mentioned that ...
0
votes
0
answers
397
views
Apache SOLR from Version 8.6.0 - Joining between Multiple Collections and Multiple Shards in each collection
Please consider SOLR version greater than 8.6.0 for this query. There are many questions regarding this issue but all are before version 8.6.0 and at that time SOLR was not supporting Join between ...
- 363