Questions tagged [pipelining]
The pipelining tag has no summary.
17 questions
0
votes
2
answers
239
views
Does Automated Pipeline Mean CI/CD Pipeline?
I am not sure if I have been using the term wrongly (and including it in my CV), so some inputs from the community will be appreciated.
I am not a DevOps, but a noob machine learning engineer. So I ...
- 169
1
vote
0
answers
172
views
Design patterns for versioning steps across data/workflow pipelines?
I'm sure this has been touched upon by a number of questions, but I'm struggling on drawing the boundaries between code, data and configuration versions when working with a large DAG (think airflow or ...
- 343
1
vote
1
answer
659
views
Should I use different Buildspec files for branch builds and deployment builds?
Very recently a discussion came up regarding the usage of different buildspec.yaml files, one for branch builds and the other for deployment builds and I was wondering, since after some research I ...
- 113
0
votes
3
answers
614
views
Is microservices architecture a good candidate for a pipeline?
I have a monolithic application which can be divided into multiple steps and different steps have variable scaling requirement, however the input of the next step is the output of the current step. ...
- 269
5
votes
2
answers
1k
views
What are the distinction and relation between batch processing and stream processing systems?
Design Data Intensive Applications says
Batch processing systems (offline systems) Chapter 10
A batch processing system takes a large amount of input data, runs a
job to pro‐ cess it, and ...
- 5,555
0
votes
1
answer
90
views
How could I pipeline two sequential workflows where there is room for overlap in the processes?
I'll break this post into two parts, because I'm trying to abstract the concept, but will explain my implementation at the end.
I have two workflows, Workflow A and Workflow B. Part of Workflow B ...
- 181
2
votes
0
answers
260
views
Data pipeline with fallbacks and callbacks
I'm refactoring our current design for how we download static data. It's a mess of deep class hierarchy and callback hell and I want to convert it to a more elegant straightforward design.
Here are ...
- 121
1
vote
1
answer
924
views
pipeline step with two outputs which will be used by different later steps
I am creating a java package which offers an API based on a pipeline pattern. That is I have a series of steps which can be plugged together in any combination provided their inputs match the Output ...
- 183
0
votes
1
answer
222
views
Unix way of filtering/mapping/mapcatting records via the pipeline?
At home I use Linux and have for more than a decade. At work we use Windows and so I find PowerShell more familiar. I am a Linux/Unix fanboy at heart, but I've had greater success grokking the ...
- 1,738