2,858 questions
0
votes
1
answer
38
views
Upsert! Operation Throws "A table can't contain duplicate column names" Error
I have a base table A and a result table B in DolphinDB. Table B was initially empty and is used to store calculated results based on table A. When trying to insert the calculated results into table B,...
- 1
3
votes
1
answer
132
views
In Apache Ignite the Replication mode and Partition mode does not work all together
I’m working with Apache Ignite 2.17.0. I load database tables into Ignite caches and run SQL queries using the SQLFieldsQuery API.
Recently, I modified the cache configuration for some tables to use ...
- 799
0
votes
0
answers
62
views
Get two different nodes to access and distribute the same SQL table in Apache spark?
I have the following code to test. I created a table on worker 1. Then I tried to read the table on worker 2 and it got TABLE_OR_VIEW_NOT_FOUND. Worker 2 is in the some computer as Master.
I ran the ...
0
votes
0
answers
50
views
How to best partition my data with a 32 core EMR instance and make sure I max out the parallelize feature?
I’m optimizing a PySpark pipeline that processes records with a heavily skewed categorical column (category). The data has:
A few high-frequency categories (e.g., 90% of records fall into 2-3 ...
- 27
1
vote
1
answer
114
views
Distributed REST API Calls using SPARK with maintaining consistency
I have a Spark DataFrame created from a Delta table, with one column of type STRUCT(JSON). For each row in this DataFrame, I need to make a REST API call using the JSON payload in the column. ...
- 53
0
votes
0
answers
327
views
PyTorch DDP Multi-Node Training: ncclInternalError: Internal check failed. Bootstrap : no socket interface found
I am trying to run a multi-node training job using PyTorch's DistributedDataParallel (DDP) following this guide. However, when I launch the job with torchrun, I encounter the following NCCL error on ...
- 1
1
vote
0
answers
91
views
Segmentation Fault During Validation with MirroredStrategy on Multiple GPUs
I am training a model using TensorFlow 2.18.0 with the tf.distribute.MirroredStrategy across two GPUs. The training works fine on a single GPU, but when I try to run it on two GPUs, it ends with a ...
- 56
0
votes
0
answers
77
views
Vertex AI Reduction Server returning 500 Internal Error
I am looking to finetune a pre-trained deberta model on Vertex AI with pytorch. I'm attempting to run a distributed job, making use of the Vertex AI reduction server.
I'm following this notebook: ...
1
vote
0
answers
35
views
Distributed training with Trainer and ConstantLengthDataset classes
I have a custom ConstantLengthDataset class:
class ConstantLengthDataset(IterableDataset):
def __init__(
self,
tokenizer,
dataset,
infinite=False,
...
- 11
0
votes
0
answers
78
views
How to properly handle multi-GPU execution failing on one GPU due to OOM
I'm working with multiple GPUs handling large amaounts of data.
I want to create an out-of-memory (OOM) catch system that skips the current batch on all GPUs if any are out of memory.
However, for ...
- 534
0
votes
1
answer
477
views
Writing to a delta table spark 3.5.3 delta lake 3.2.0
It seems I'm unable to write using the delta format from my spark job, but I'm not sure what I'm missing. I'm using spark 3.5.3 and deltalake 3.2.0.
My error:
Exception in thread "main" org....
- 141
0
votes
0
answers
374
views
Cuda failure ‘named symbol not found’ when run on 4 L4 GPUs
Can someone help me with the following error. The code works fine on the 2 T4 GPUs. But fails when run on the 4 L4 GPUs. I am extending the Gemma 2B model for a multi-label multi-class classification ...
1
vote
0
answers
20
views
Why am I getting "TXN_REQUEST_IGNORED ERROR 10906" in GridDB due to an unknown event during cluster operations?
I’m using GridDB for a distributed database setup and recently encountered the following error while performing operations across nodes in the cluster:
from griddb_python import StoreFactory, ...
2
votes
2
answers
222
views
Client request failure in raft
Imagine a 3 node raft cluster. Each node is in sync has log [1,2,3] and entry 3 is committed by the leader.
Now leader receives an entry 4 but fails to commit it because of unreliable network and ...
- 129
0
votes
1
answer
133
views
Apache ignite compute broadcast
I'm trying apache ignite and must say ignite documentation is incomplete. Anyway, I've setup two node cluster using docker images 2.14.0-arm14 and exposed all Ports for both ignite containers, however ...
- 196
2
votes
0
answers
416
views
How to Implement Distributed AutoFAISS with PySpark for Large-Scale Vector Indexing?
I’m working on a project that involves creating a vector search index for a massive dataset consisting of 1.3 trillion tokens. I want to use AutoFAISS in a distributed environment to handle the scale ...
- 2,759
0
votes
1
answer
103
views
Do I need Apache Spark to execute my Airflow DAG tasks?
I have a workflow with multiple DAGs. Every DAG has multiple tasks.
These tasks are simple ETL tasks. It involves geo data in the form of kmls, csvs.
An example task:
We have meta data of road ...
0
votes
0
answers
35
views
How to securely conduct lottery-like draws with guaranteed randomness without auditing?
Is there an existing algorithm or method to conduct lottery-like draws that ensures secure and truly random results without the need for auditing?
There are any lib to do this?
I search on the web ...
- 233
1
vote
2
answers
925
views
How to reliably implement fan out write pattern?
I'm trying to RELIABLY implement that pattern.
For practical purposes, assume we have something similar to a twitter clone (in cassandra and nodejs).
So, user A has 500k followers. When user A posts a ...
2
votes
0
answers
25
views
Why am I getting a "LM_WRITE_LOG_FAILED ERROR 80000" in GridDB when writing to the log file?
I'm using GridDB for managing a distributed database system and recently encountered the following error while trying to perform operations:
80000 LM_WRITE_LOG_FAILED ERROR
Writing to log file failed. ...
- 91
2
votes
0
answers
42
views
Why am I getting a "SYNC_CREATE_CONTEXT_FAILED ERROR 20037" during data synchronization in my GridDB cluster?
I'm working on a distributed system where I need to synchronize data across a cluster of nodes. However, I'm encountering an error during the synchronization process. The error message I get is:
20037 ...
- 91
1
vote
0
answers
98
views
How to properly clean up non-serializable states associated with a Ray object?
Suppose I have a Ray actor that can create a Ray object that associates with some non-serializable states. In the following example, the non-serializable state is a temporary directory.
class MyObject:...
- 3,773
2
votes
0
answers
99
views
How does XGBoost aggregate models being trained in a distributed fashion across n machines?
I am trying to understand how XGBoost distributed training works. The best explanation I've found so far is in this paper: https://ml-pai-learn.oss-cn-beijing.aliyuncs.com/%E6%9C%BA%E5%99%A8%E5%AD%A6%...
- 359
0
votes
1
answer
162
views
Micrometer & Prometheus with Java subprocesses that can't expose HTTP
I have a complex product that runs like this.
A parent Java process which expose an HTTP service.
The parent process starts worker subprocesses (new JVM) and manage the lifecycle of them.
Worker ...
- 510
1
vote
1
answer
2k
views
How to debug ValueError: `FlatParameter` requires uniform dtype but got torch.float32 and torch.bfloat16?
I'm trying to do Pytorch Lightning Fabric distributed FSDP training with Huggingface PEFT LORA fine tuning on LLAMA 2 but my code ends up failing with:
`FlatParameter` requires uniform dtype but got ...
- 2,376