I have some software in Golang that runs 500-3000 tasks. Every task does little compute and a lot of IO (http requests). Every task needs to use its own unique transport (may occasionally change their transport) and cookie jar as shown below for the http client. Every task must use its own API responses and cannot share them with other tasks. My goal is to optimize for speed (every task runs as concurrently and fast as possible). I know the bottle neck is mainly network latency.
My current approach just launches every task in a goroutine and each makes its own http client. How can I optimize my setup? Do I use fasthttp package, is there a feasible method to speed up the io?
transport := &http.Transport{
Proxy: http.ProxyURL(&url.URL{
Scheme: "http",
User: url.UserPassword(task.Proxy.Username, task.Proxy.Password),
Host: task.Proxy.IP + ":" + strconv.Itoa(task.Proxy.Port),
}),
}
client := &http.Client{Transport: transport,}
