Source Blocker

If a source server fails completely, high traffic on the FIT Engine might cause all processes to wait and block the FIT Server entirely. To prevent this, the FIT request system incorporates a source blocker. If a failure is detected, the source is blocked so that for a certain time no further requests are submitted to it. As a result, the FIT Server is not blocked and at the same time the load on the source system is reduced. The blockade lasts between half a minute and five minutes, depending on the timeout for that source. The source server is then tentatively enabled for one request and, depending on the result, the blockade is removed or extended.

To detect failures, aborted requests are logged in a volatile memory. The length of the timeout is incorporated into the calculation. The shorter the timeout, the more requests must fail before the threshold value is reached.

The threshold value must be adapted for the source blocker according to the installation. With the setting FIT_MAX_TIMEOUT_PROCESSES threshold you can define how many PHP-FPM processes are allowed to wait for slow sources at any one time.

The lower the threshold value setting, the earlier slow sources will be blocked. A suitable default setting is a value not exceeding one third of the number of permitted PHP-FPM processes (pm.max_children). The number of sources should be taken into account as well as the number of available processes. If a site uses just a single source, the value can be increased, since failure of that source renders the site unusable anyway.

The evaluation takes the hostnames and ports of sources into account, but not their protocols. If a server with different hostnames is addressed – for example example.com and www.example.com – it may take up to twice as long for the threshold to be reached.

Monitoring

When a source is blocked or released, a message is written to the log file fit_engine.log. You can easily monitor these messages:

Source Blocker: adding example.com:80 to blacklist for 60 s
Source Blocker: removing example.com:80 from blacklist