Some combinations of pipeline and deployment configurations can cause problems, even if they are not necessarily incorrect. A recurring case concerns the use of the pipeline executor connector with synchronous operation, that is, when the main flow triggers another and waits for it to conclude and respond before proceeding to the following steps.
This case requires special attention as there may be a "mismatch" of communication between the two pipelines depending on the deployment configuration of both, especially when there is a high volume of executions, leading to the following error:
The message could not be sent to the specified pipeline. Error org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 20000 millis due reply message with correlationID: Camel-…-… not received on destination: amq.gen-…. Exchange[…-…]
(Pipeline Executor timeout)
Even though the target pipeline executes in a timely manner, it returns this error to the main pipeline, as if the pipeline executor component had timed out. This happens when there is a much higher throughput of requests than the target pipeline could consume.
Identifying scenarios and how to resolve
When facing the error above, check whether the flows affected fit into one of the following scenarios:
Deployment of the main pipeline has more concurrent executions than that of the target pipeline;
Target pipeline is triggered by several others at the same time, resulting in more requests than the concurrent executions configured.
In order to fix the issue, increase the number of concurrent executions of the target pipeline to provide greater processing capacity (this may also require adding more replicas). If the operation does not necessarily need to be synchronous, then you can change it to asynchronous or even consider turning it into an event instead, allowing the queue manager to control the traffic in the target pipeline.