General request options that are suitable for all requests to a host or certain paths on a host may be set in conf/sources.xml
. They apply to all requests that FIT issues, be it user defined request actions or system requests (e.g. image scaling requests). The sources
document element may contain options for multiple sources.
The following example includes all available options and parameters.
<sources>
<source host="…" port="…" path="…">
<!-- Request options -->
<timeout request="5" connect="1" low-speed-time="3" low-speed-limit="1" />
<headers send-via="true" send-xff="true" pass="User-Agent, Accept-Language" />
<response-headers pass="WWW-Authenticate, Access-Control-*" remove="Set-Cookie, X-Foo" />
<ssl version="TLSv1" check-certificate="true" client-certificate="fit://site/conf/mycert.pem" certificate-password="pa5sw0Rd" ciphers="RC4" />
<transport protocol="http" server="127.0.0.1" port="8080" />
<proxy server="proxy.example.com:3128" cache="true" credentials="user:s3CreT" auth-method="basic" />
<cache enable="true" interval="3600" vary-headers="Accept-Language" allow-invalidation="false" />
<redirects enable="false" max="10" />
<cookies send="true" accept="true" mode="transparent" />
<auth method="basic" credentials="user:0P3n-5es4Me" />
<history enable="false" />
<parser value="json" />
<encoding value="utf-8" />
<normalize-urls enable="true" />
<query-encoding raw="*$!" />
<connection reuse="true" />
<!-- Request params -->
<response-header name="Cache-Control" value="public, max-age=3600, s-maxage=3600" replace="true" />
<header name="User-Agent" value="FIT User-Agent" />
<query name="X" value="Y" />
<query name="W" value="Z" />
<post name="A" value="B" />
<post name="C" value="D" />
<upload name="myFile" filename="file.txt" size="123" type="text/plain" src="fit://site/public/file.txt" />
</source>
</sources>
Options are defined within rules, that is source
elements. These elements are associated with specific backend servers by their host
, port
and path
attributes.
Each source
rule is evaluated depending on how well its host
, port
and path
specification matches the current request. For example, a host
with a port
is more specific than a host
without a port
.
<sources>
<source host="example.com" path="/checkout/billing">
<!-- Options for requests to example.com/checkout/billing and below (any port) -->
</source>
<source host="example.com" port="443" path="/">
<!-- Options for requests to example.com:443/ and below. -->
</source>
<source host="example.com" port="443">
<!-- The same as above (path="/"). -->
</source>
<source host="example.com">
<!-- Options for requests to example.com on any port. -->
</source>
<source>
<!-- Supplementary options for all requests -->
</source>
</sources>
The following illustration represents the resolving order for the options and parameters (see request actions, too) in a request. The smaller the number (1-n), the higher the priority of the options and parameter values:
<requests>
<option (2) /> <!-- Dynamic option (see description below in the section "Dynamic Parameters and Options") -->
<param (1) /> <!-- Dynamic parameter (see description below in the section "Dynamic Parameters and Options") -->
<request url="..." content="..." method="...">
<option (1) />
<param (n) />
</request>
</requests>
<sources>
<!-- Most specific source -->
<source host="..." port="..." path="...">
<option (3) />
<param (2) />
</source>
<!-- Less specific source -->
<source host="...">
<option (4) />
<param (3) />
</source>
<!-- Least specific source -->
<source>
<option (n) />
<param (4) />
</source>
</sources>
The better a rule matches the request, the earlier its options will be set. The options of low-priority rules may be added, but they never override previous definitions. The main difference between options and parameters is that the options defined in the request rule are prioritized higher than options set in any source. However, the parameters defined in a request rule have the lowest priority.
The option <headers pass="..." />
should be regarded as placeholder for multiple header
params in the respective rule. This is the single option, which values are evaluated in all matched rules.
For example, FIT would interpret an incoming request with the URL https://example.com:443/shop/?a=1&b=&foo
like:
<request url="https://example.com:443/shop/?a=1&b=&foo" ...>
<header name="User-Agent" value="Client User-Agent" />
<header name="X-Foo" value="Foo" />
<query name="a" value="1" />
<query name="b" value="" />
<query name="foo" />
</request>
Hence, the default-request
action would make the following “FIT request” corresponding to the incoming client request above:
<request url="https://example.com:443/shop/?a=1&b=&foo" ...>
<redirects enable="false" />
<query name="a" value="1" />
<query name="b" value="" />
<query name="foo" />
</request>
Let’s assume we have the following sources.xml
:
<sources>
<!-- Most specific source for URLs used in this example -->
<source host="example.com" port="443" path="/shop">
<!-- The following option is translated to: <header name="User-Agent" value="Client User-Agent" /> A header with the name "X-Not-Exists" will not be set, because here is no header with that name in the incoming request. -->
<headers pass="User-Agent,X-Not-Exists" />
<!-- Requests to https://example.com:443/shop with all optional sub-paths always override the X-Foo header with value "Bar" -->
<header name="X-Foo" value="Bar" />
<!-- Requests to https://example.com:443/shop with all optional sub-paths always replace the X-Blocked header with value "Allowed in shop" -->
<header name="X-Blocked" value="Allowed in shop" />
<!-- Requests to https://example.com:443/shop with all optional sub-paths always overwrite the query parameter "b" with value "2" -->
<query name="b" value="2" />
</source>
<!-- Less specific source for URLs used in this example -->
<source host="example.com" port="443">
<!-- The following option is translated to: <header name="X-Foo" value="Foo" /> -->
<headers pass="X-Foo" />
<!-- Removes/blocks the X-Blocked header (would be set in "Least specific source") definition for all requests to https://example.com:443 excepting requests to https://example.com:443/shop with all optional sub-paths -->
<header name="X-Blocked" value="" />
<!-- Replaces the User-Agent header for all requests to https://example.com:443 excepting requests to https://example.com:443/shop with all optional sub-paths -->
<header name="User-Agent" value="Example User-Agent" />
</source>
<!-- Least specific source for URLs used in this example -->
<source>
<redirects enable="true" />
<header name="User-Agent" value="FIT User-Agent" />
<header name="X-Blocked" value="Default" />
<header name="X-Ever" value="Ever" />
<query name="x" value="y" />
<query name="x" value="z" />
</source>
</sources>
On an incoming request to the URL http://example.com/?a=1&b=&foo
:
http://example.com/?a=1&b=&foo&x=y&x=z
query
parameters:a
: 1
(from the FIT request)b
: =EMPTY
(from the FIT request)foo
: EMPTY
(from the FIT request)x
: y
(set by the “Least specific source”)x
: z
(set by the “Least specific source”)header
parameters:X-Foo
: is removed by the default-request
actionUser-Agent
: FIT User-Agent
(set by the “Least specific source”)X-Blocked
: Default
(set by the “Least specific source”)X-Ever
: Ever
(set by the “Least specific source”)On an incoming request to the URL https://example.com:443/?a=1&b=&foo
:
https://example.com:443/?a=1&b=&foo&x=y&x=z
query
parameters:a
: 1
(from the FIT request)b
: =EMPTY
(from the FIT request)foo
: EMPTY
(from the FIT request)x
: y
(set by the “Least specific source”)x
: z
(set by the “Least specific source”)header
parameters:X-Foo
: Foo
(from the incoming request, passed by the “Less specific source”)User-Agent
: Example User-Agent
(set by the “Less specific source”)X-Blocked
: is removed by the “Less specific source”X-Ever
: Ever
(set by the “Least specific source”)On an incoming request to the URL https://example.com:443/shop/?a=1&b=&foo
:
https://example.com:443/shop/?a=1&foo&b=2&x=y&x=z
query
parameters:a
: 1
(from the FIT request)b
: 2
(from the FIT request, overwritten by the “Most specific source”)foo
: EMPTY
(from the FIT request)x
: y
(set by the “Least specific source”)x
: z
(set by the “Least specific source”)header
parameters:X-Foo
: Bar
(set by the “Most specific source”)User-Agent
: Client User-Agent
(from the incoming request, passed by the “Most specific source”)X-Blocked
: Allowed in shop
(set by the “Most specific source”)X-Ever
: Ever
(set by the “Least specific source”)In this example the X-Ever
header with the value Ever
and the query parameters “x=y&x=z” will always be set. However, the option <redirects enable="true" />
(defined in the “Least specific source”) can never be set.
Note, that the options set in the flow
(e.g. in a request
action) have the highest priority and will be evaluated first. See above for the difference between options and parameters, too.
As described above in the section Resolving Order, the request parameter values have the lowest priority (<param (n) />
). In some cases it could be useful to manipulate (set/unset/replace) the parameter values dynamically, e.g. in a requests
action. You can define your own parameters within the requests
element, but outside of each request
element:
<requests>
<param (1) /> <!-- This is a dynamic parameter -->
<option (2) /> <!-- This is a dynamic option -->
<request url="..." content="..." method="...">
<option (1) />
<param (n) />
</request>
...
</requests>
In summary, the Dynamic Parameters and Options behave as if they were children of the most specific <source>
element for any URL.
Note, that the dynamic parameter values have the highest priority and are applied for every request.
Note, that the dynamic options have a higher priority over any equivalent options in the sources.xml
.
Let’s assume we have the following simple sources.xml
:
<sources>
<source>
<query name="output" value="x" />
</source>
</sources>
and a requests
action created dynamically by the following XSLT:
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:template match="/">
<requests>
<query name="output" value="z" />
<request content="main" url="http://example.com/?output=y" />
</requests>
</xsl:template>
</xsl:stylesheet>
or alternative defined statically in a requests
action:
<requests>
<query name="output" value="z" />
<request content="main" url="http://example.com/?output=y" />
</requests>
In this case the request URL will always be http://example.com/?output=z
, because the dynamic parameter value has the highest priority. If you remove the dynamic parameter, the request URL will be resolved as http://example.com/?output=x
due to the next higher priority level in sources.xml
.
Sometimes it may be desirable to set the request options based on the Delivery Context properties. For example, you may want to send specific headers to a source depending on the device type. You can accomplish this by using the Dynamic Configuration mechanisms in your sources.xml
.
<sources>
<source host="...">
<choose>
<when test="client/hw/type = 'tablet'">
<header name="hwtype" value="tablet" />
</when>
<when test="client/hw/type = 'desktop'">
<header name="hwtype" value="desktop" />
</when>
<when test="client/hw/type = 'mobile'">
<header name="hwtype" value="mobile" />
</when>
<otherwise>
<header name="hwtype" value="unspecified" />
</otherwise>
</choose>
</source>
</sources>
The conf/sources.xml
is loaded and filtered before the Flow is started.
<sources>
<source ...>
<cache enable="true|false" ...>
</source>
...
</sources>
enable
: Defaults to true
. If enabled, FIT caches and reuses HTTP responses from sources servers. Interval Caching has priority over the automatic source caching. If enable="false"
, the Interval Caching is disabled, too.For caching and reusing HTTP responses from the source servers, the following requirements must be met.
Cacheability is determined as follows:
Vary
header fields contain *
values, the response won’t be cached.Cache-Control
headers are present, they may contain the public
, must-revalidate
, proxy-revalidate
, no-cache
, pre-check
, post-check
, max-age
or s-maxage
directives. If the max-age
or s-maxage
directives occur multiple times, only the first one will be considered.Cache-Control
headers as described above, all Pragma
headers will be ignored.Cache-Control
headers and a Pragma
header contains the no-cache
directive, the response won’t be cached.WWW-Authenticate
header is present, the response won’t be cached.Authorization
header is set, the response won’t be cached (HTTP Cache only. If enabled, the interval cache would still work).Note If a sources server uses an authentication method to communicate with the FIT Server, it is not recommended to pass the Authorization
header from the incoming request. In this case it is strongly recommended to use the auth
option to set the authentication settings.
If a resource is cacheable, its cache lifetime will be determined as follows:
Cache-Control
header contains a max-age
or s-maxage
directive, their values will be used to determine the cache lifetime. Any Expires
headers will be ignored.max-age
or s-maxage
directive, the value of the Expires
header will be used to calculate the cache lifetime. If there are multiple Expires
headers, the resource won’t be cached.max-age
or s-maxage
directive, nor an Expires
header is present, the cache lifetime will be based on the Last-Modified
or Age
header, if possible.Cache-Control
public
directive) or the status code defined as cacheable by default (see RFC 7231).Last-Modified
headers are considered invalid and will prevent caching.If a client sends any of the following cache headers, the cache will be bypassed, but the response data will be stored and possibly reused:
The documents are stored on disk. In a cluster, every FIT server maintains its own cache. Stale documents are removed periodically by a cron job.
history | |||||
Enables the recording of intermediate response information on redirects. The information is available in fit://request/content/<ID>/history | |||||
enable | boolean | Enables or disables redirect history recording (default false ). | |||
header | |||||
name | string | Name of the HTTP header field | |||
value | string | Value of the HTTP header. An empty value suppresses the respective HTTP header field. | |||
response-header | |||||
The HTTP header field with name and value as specified is set, removed or replaced immediately after the remove statements for response-headers are processed. | |||||
name | string | Name of the HTTP header field | |||
value | string | Value of the HTTP header (optional). If omitted or empty, all HTTP header fields matching the name will be removed. | |||
replace | boolean | Defines, whether the response HTTP header field shall be replaced (default: true ) | |||
headers | |||||
HTTP headers to be sent to the backend system. | |||||
pass | string | Comma-separated list of incoming request headers to be passed to the backend. _auto_ will pass Accept , Accept-Language , MSISDN , Profile , Referer , User-Agent and all X-… headers and can be combined with other headers, e.g. <headers pass="_auto_, Authorization" /> . Via and X-Forwarded-For headers may be sent modified using send-via and send-xff (see below). | |||
send-via | boolean | Whether to send or extend a Via header (default false ). | |||
send-xff | boolean | Whether to include the client IP address in a X-Forwarded-For header (default false ). | |||
response-headers | |||||
Management of the HTTP headers received from a backend system. | |||||
pass | string | Comma-separated list of backend response headers to be passed to the client. The header list may contain wildcards like * or ? .
Note: This option does not work for side requests (i.e. extra requests that use an ID other than | |||
remove | string | Comma-separated list of backend response headers to be removed immediately after receiving them from a backend system. Wildcards like * or ? are allowed. | |||
ssl | |||||
TLS/SSL settings for backend communication | |||||
check-certificate | boolean | Enables or disables the verification of certificates on SSL connections (default true ). | |||
client-certificate | string | Path to the client certificate (.pem ) | |||
certificate-password | string | The certificate password | |||
version | string | The SSL protocol version:
| |||
ciphers | string | Ciphers to use for TLS connections. Note that the cipher names and the format of this option depend on whether the system is backed by OpenSSL (SLES/openSUSE/Ubuntu) or NSS (RHEL/CentOS). | |||
proxy | |||||
Proxy server for communication with backend systems | |||||
server | string | Proxy server hostname with port (e.g. proxy.example.com:3128 ) | |||
cache | boolean | Whether to use proxy caching (default true ). | |||
credentials | string | Authentication data for the proxy server (e.g. username:password ). | |||
auth-method | string | Authentication method on selected proxy server: basic (default), ntlm . | |||
cache | |||||
Caching | |||||
enable | bool | Whether to enable HTTP source caching. (default true ). | |||
interval | integer | The cache interval in seconds (0 …86400 ). 0 disables caching (default). | |||
vary-headers | string | Comma-separated list of headers that should be considered to lookup the correct cache entry. Example: User-Agent, Accept-Language | |||
allow-invalidation | boolean | Allows to invalidate the cache by means of Cache-Control: no-cache or Pragma: no-cache respectively (default false ). | |||
timeout | |||||
Request, stream and connection timeouts and limits | |||||
request | float | Maximum time in seconds for processing a request. When the maximum time is reached before the response has been received, the processing will be terminated. Decimals are allowed, e.g. 1.5 (default: FIT_FETCH_DEFAULT_TIMEOUT ). | |||
connect | float | Maximum time in seconds for establishing a connection to a server (default: FIT_FETCH_DEFAULT_CONNECT_TIMEOUT ). | |||
low-speed-limit | int | If the average transfer speed in bytes per second of a transfer drops below low-speed-limit for low-speed-time seconds, FIT will abort the transfer with an error (default 1 ). This option has no effect unless the low-speed-time option is set to a value greater than 0. | |||
low-speed-time | int | If the average transfer speed in bytes per second of a transfer drops below low-speed-limit for low-speed-time seconds, FIT will abort the transfer with an error (default: FIT_FETCH_DEFAULT_LOW_SPEED_TIMEOUT ). This option has no effect unless the low-speed-limit option is set to a value greater than 0. low-speed-time may be useful to abort transfers in case the Time to First Byte is excessive, while still maintaining long timeouts for slow but consistent downloads. | |||
redirects | |||||
Configure the redirects | |||||
enable | bool | Whether to automatically follow redirects (default false ). | |||
max | integer | Maximum number of consecutive redirects to follow. | |||
cookies | |||||
Cookies | |||||
send | bool | Whether to send all matching cookies to the backend (default true ). If caching is enabled, the default is false . | |||
accept | bool | Whether to accept cookies received from the backend (default true ). If caching is enabled, the default is false . | |||
mode | string | By default, FIT encodes cookies received from source servers to ensure that FIT passes the cookies only to the correct servers (and paths) in subsequent requests. This option can be used to enable the transparent mode for the transfer of cookies to the client: encoded (default), transparent | |||
auth | |||||
Authentication settings for the backend system | |||||
method | string | Authentication method on backend system: basic (default), safe , any , digest , ntlm | |||
credentials | string | Authentication data for backend system (e.g. username:password ) | |||
parser | |||||
Normally, the Content-Type header is used to select the correct parser. This option allows select the parser directly. | |||||
value | string | The parser to use: xml , html , json , js , css , false | |||
encoding | |||||
This option can be used to set the encoding used for the URI including the query string and POST form data. | |||||
value | string | The name of the encoding, (default UTF-8 ). Other common values are ISO-8859-1 , ISO-8859-15 and Windows-1251 . For a list of all supported encodings, please refer to the PHP documentation. | |||
normalize-urls | |||||
If enabled, URLs within the document are normalized based on the source URL, i.e. the document does not contain any relative URLs anymore. | |||||
enable | bool | Whether to enable URL normalization (default true ). | |||
query-encoding | |||||
Normally, certain characters in the query string are URL encoded. This option can be used to set a list of characters which are to remain unencoded. | |||||
raw | string | Characters in the query string to exclude from the URL encoding. | |||
query | |||||
name | string | Name of the query (GET) parameter | |||
value | string | Value of the query (GET) parameter | |||
post | |||||
name | string | Name of the POST parameter | |||
value | string | Value of the POST parameter | |||
upload | |||||
name | string | Name (key) of the file to be sent | |||
filename | string | File name of the file to be sent | |||
size | integer | File size in bytes | |||
type | string | MIME type | |||
src | string | FIDJ resource to upload. | |||
transport | |||||
This option allows to alter protocol, host or port of an outgoing request e.g. to use HTTP for backend requests while using HTTPS with clients. To actually use this option it has to be allowed by the FIT_ALLOW_TRANSPORT_MAPPING setting in the fit.ini configuration file. In addition, the system ACL (/opt/sevenval/fit14/conf/acl.xml ) must permit access to the backend system. | |||||
protocol | string | Protocol for communication with the backend system. If not set, the original protocol won’t be changed. (http or https only) | |||
port | integer | Port for communication with the backend system. If not set, the original port won’t be changed. | |||
server | string | Backend server to communicate with. If this option is not set, the server won’t be changed. | |||
connection | |||||
By default, FIT uses HTTP Keepalive to minimize the TCP connection overhead and improve request performance. In the unlikely case that this causes problems, HTTP persistent connections can be disabled for individual sources using this option, or globally via the FIT_USE_PERSISTENT_CONNECTIONS setting in the fit.ini configuration file. | |||||
reuse | bool | Set to false to disable HTTP persistent connections. |