531 questions
0
votes
1
answer
53
views
GNN Link Prediction (GraphSAGE/PyG) - Validation AUC Consistently Below 0.5 Despite Overfitting Control
I'm working on a task dependency prediction problem using Graph Neural Networks with PyTorch Geometric. The goal is to predict directed precedence links (A -> B) between tasks within specific sets (...
- 101
1
vote
1
answer
75
views
Correctly invoke TorchEval AUC calculation for two tensors
I am new to torcheval and trying to measure the AUC of my binary classifier (doc).
I notice that while classifier accuracy is decent, the AUC metric evaluates to below 0.5, which is incorrect (given ...
- 19.2k
-2
votes
1
answer
62
views
geom_area() in ggplot2 giving off AUC shadings [closed]
I'm trying to use geom_area() in a different way, e.g. AUC. I refer to the example in https://www.sthda.com/english/wiki/ggplot2-area-plot-quick-start-guide-r-software-and-data-visualization
set.seed(...
- 1
0
votes
0
answers
50
views
AUC-ROC Comparison and Intepretation if FPR < 1
I am comparing two unsupervised models that predict a binary class, each with a different threshold parameter (p for model A and g for model B, ranging from 0 to 1).
You can see that Model B has no ...
1
vote
1
answer
67
views
How to find the number of samples that are picked in each boostrap of stratified bootstrap in pROC?
Question is regarding the roc function of pROC package.
Package link: https://www.rdocumentation.org/packages/pROC/versions/1.18.5/topics/roc. Paper link https://www.ncbi.nlm.nih.gov/pmc/articles/...
0
votes
0
answers
48
views
TypeError: Singleton array array(1) cannot be considered a valid collection
I have a dataset where my target variable is a number between 1 and 8.
Now I am going to implement Cubic SVM.
import numpy as np
import pandas as pd
from sklearn.model_selection import ...
-1
votes
1
answer
39
views
Can AUC range be other than between 0 and 1?
In this question and answer regarding AUC: Applying a function for calculating AUC for each subject
Why is the AUC not between 0 and 1? shouldn't it be?
Thank you so much in advance.
I have tried ...
1
vote
0
answers
91
views
I am trying to find brier score of survival data using risk regression pacakge, but its giving an error
xy = Score(list(cox_ph), formula=Surv(time,status)~1,data=cox.train, metrics=c("brier","auc"), null.model=FALSE,times=time.int,debug = TRUE)
Extracted test set and prepared output ...
- 11
0
votes
1
answer
518
views
Pycaret 3.3.0 compare_models() show zeros for all models AUC
During evaluation of the model using compare_model(). All AUCs are zero.
This output of Pycaret 3.3.0 is weird. what's the reason for that?
[1]: https://i.sstatic.net/qm2ZT.png
- 375
1
vote
1
answer
195
views
Implementation of Cobb-Douglas Utility Function to calculate Receiver Operator Curve & AUC
this is not a hard problem from my understanding, however, I have minimal experience with different R functions and problem solving within it. The text is long to provide a thorough understanding ...
- 23
0
votes
1
answer
356
views
How to Plot Cross-Validated AUROC and Find the Optimal Threshold? [closed]
While evaluating my machine learning model with cross-validation, I encountered an issue. I knew how to plot AUROC and the corresponding threshold for each fold in cross-validation, but I was unsure ...
0
votes
1
answer
251
views
How does sklearn calculate AUC for random forest and why it is different when using different functions?
I start with the example given for ROC Curve with Visualization API:
import matplotlib.pyplot as plt
from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from ...
- 803
0
votes
0
answers
234
views
Is there any way to plot a calibration curve and calculate a Brier score from a XGBoost model in R?
I have trained a XGBoost model in R, tuned the hyperparameters and plotted the ROC curve.
Is there any easy way to plot a calibration curve and calculate Brier score, calibration intercept and ...
- 1
1
vote
2
answers
549
views
How do I go from logistic regression to ROC curve and analysis?
I have a dataset that looks like this, where "1" represents if a host is infected and "0" represents if a host is uninfected at that specified dose. However, the ROC function needs ...
- 49
0
votes
1
answer
183
views
TypeError: auc() got an unexpected keyword argument 'reorder'
Kindly let me know why this error is coming?
def roc_aupr_score(y_true, y_score, average="macro"):
def _binary_roc_aupr_score(y_true, y_score):
precision, recall, pr_thresholds = ...
0
votes
1
answer
206
views
Why the in the ROC curve is not curve and how I can fix
I plot a ROC curve but I did't obtain a smooth line(curved line) why and how I can fix this
enter image description here
What I'm trying to do is detecting anomaly from Satellite dataset from (http://...
0
votes
1
answer
43
views
Why is the mean roc score from GridSearchCV using only 1 cv split, different from roc calculated with grid_search.score method or roc_auc_score func?
I was experimenting with sklearn's GridSearchCV, and I don't understand why the mean roc scores I get when using a single split defined with an iterable, are different than what I get running the ...
2
votes
1
answer
219
views
How do I reshape data to calculate ROC and AUC for binary text classification?
I'm very new to python and need to calculate the ROC and AUC of two binary classification models using NLP data. I can't seem to get my head around sparse vs dense arrays (I mean, I get that sparse ...
- 23
-1
votes
1
answer
430
views
How to find AUC value of Decision Tree?
Decision Tree
I have found Misclassification rates for all the leaf nodes.
samples = 3635 + 1101 = 4736, class = Cash, misclassification rate = 1101 / 4736 = 0.232.
samples = 47436 + 44556 = 91992, ...
1
vote
1
answer
1k
views
How to calculate area under curve in R?
I have data like in the following image and I want to calculate the area under the curve between the blue lines x = 5.75 and x = 6.45:
I have tried some of the answers given here and here. pracma::...
- 8,296
0
votes
1
answer
526
views
Make a beautiful ROC curve with 4 models on it
I've trained 4 different xgboost ML models, I tested them and got the prediction scores with this line of code: (The fist column is the target feature so I exclude it when predicting):
predict(model, ...
- 375
0
votes
1
answer
1k
views
Why a roc auc score from a regular cross-validation is very different from a roc auc score after an hyperparameter tuning?
I'm evaluating a XGBoost classifier. I split the dataset into train and validation sets, perform a cross-validation with the model default implementation using the train set and compute the ROC AUC:
...
1
vote
1
answer
92
views
Integrating under a curve in R
I apologise if this is a duplicate; I've read answers to similar questions to no avail.
I'm trying to integrate under a curve, given a specific formula (below) for said integration.
As a toy example, ...
- 109
0
votes
2
answers
2k
views
Confidence interval AUC with the bootstrap method
today I attempted to make a bootstrap to obtain the interval confidence of various different ML algorithm AUC.
I used my personal medical dataset with 61 features formatted liked this :
Age
Female
65
...
0
votes
2
answers
818
views
Calculate AUC by different segments in python
I have a dataset which contains id, datetime, model features, ground truth labels and the predicted probability.
id datetime feature1 feature2 feature3 ... label probability
001 ...
- 719