The Digibee platform was designed to ensure integrations run uninterrupted through high availability strategies. However, it is necessary to apply appropriate configurations to each deployment to prevent that requests above the pipeline's capacity from making it unavailable even for a brief moment.

In this article we will cover how to proceed if your flow is showing the error below in the Monitor => Completed Executions tab:

This error mainly occurs when there is an excessive memory and/or CPU request. It should be borne in mind that even the large size has a limited capacity that it can handle. So, to solve it, you will need to follow the steps below:

1. Open the Monitor => Metrics tab and filter by the period of the error in question. Note: the shorter the period selected, the more accurate the values on the graphs will be (article).

2. Check if there has been a memory or CPU overflow. Note: the memory limit is 100%, but the CPU limit is 20% for small, 40% for medium and 80% for large (article).

3. Check whether the size of the input or output payloads has reached 5 MB or the limit configured in the trigger.

4. Open the tab Monitor => Completed Executions or Pipeline Logs and check the last steps before the interruption.

5. Open the pipeline in Build and check whether it fits into any of the following scenarios:

- Large file being loaded (e.g. over 7 MB or over the limit configured in the file connector)

- Loading a lot of data via DB connector

- REST or SOAP which are costly to process

- Large number of connectors with external interaction (REST/SOAP/DB/Script/file)

- Loop blocks containing external connectors and running in parallel or sequence

- Parallel Execution followed by multiple external connectors

- Excessive amount of or very extensive logs

6. Solution: once there is a diagnosis, apply one or more of the following strategies:

- Increase the pipeline deployment size

- Reduce the number of concurrent executions (NOTE: The memory allocated to the deployment is shared between the simultaneous executions. For example, in a large deployment with 40 executions, the memory will be divided by 40. With 30 executions, the same amount of memory will be divided by 30, resulting in greater processing capacity for each execution, since the memory is divided between a smaller number of concurrent processes.)

- Reduce the amount of data at the input or output of the pipeline

- Control the data load in the pipeline by adding limits, filters and pagination

- Increase the pipeline frequency to make processing lighter

- Remove or shorten logs with very large payloads

- Clear the memory of session management components with the DELETE operation

- Prefer sequential executions to parallel ones

- Split processing into multiple pipelines

For an optimized and error-free deployment configuration, you must first publish the pipeline and monitor its performance to determine if it will need refinement.

In addition to the appropriate allocation of computational resources (vertical scale), the number of simultaneous executions (horizontal scale) must be adjusted in a balanced manner, providing a regular pace to the processing.

Other factors to consider are the tolerated queue size and the desired throughput rate, which is the relationship between executions per second (EPS) and pipeline response time.