Sources (conf/sources.xml)

General request options that are suitable for all requests to a host or certain paths on a host may be set in conf/sources.xml. They apply to all requests that FIT issues, be it user defined request actions or system requests (e.g. image scaling requests). The sources document element may contain options for multiple sources.

Example

The following example includes all available options and parameters.

<sources>
  <source host="…" port="…" path="…">
    <!-- Request options -->
    <timeout request="5" connect="1" low-speed-time="3" low-speed-limit="1" />
    <headers send-via="true" send-xff="true" pass="User-Agent, Accept-Language" />
    <response-headers pass="WWW-Authenticate, Access-Control-*" remove="Set-Cookie, X-Foo" />
    <ssl version="TLSv1" check-certificate="true" client-certificate="fit://site/conf/mycert.pem" certificate-password="pa5sw0Rd" ciphers="RC4" />
    <transport protocol="http" server="127.0.0.1" port="8080" />
    <proxy server="proxy.example.com:3128" cache="true" credentials="user:s3CreT" auth-method="basic" />
    <cache enable="true" interval="3600" vary-headers="Accept-Language" allow-invalidation="false" />
    <redirects enable="false" max="10" />
    <cookies send="true" accept="true" mode="transparent" />
    <auth method="basic" credentials="user:0P3n-5es4Me" />
    <history enable="false" />
    <parser value="json" />
    <encoding value="utf-8" />
    <normalize-urls enable="true" />
    <query-encoding raw="*$!" />
    <connection reuse="true" />

    <!-- Request params -->
    <response-header name="Cache-Control" value="public, max-age=3600, s-maxage=3600" replace="true" />
    <header name="User-Agent" value="FIT User-Agent" />
    <query name="X" value="Y" />
    <query name="W" value="Z" />
    <post name="A" value="B" />
    <post name="C" value="D" />
    <upload name="myFile" filename="file.txt" size="123" type="text/plain" src="fit://site/public/file.txt" />
  </source>
</sources>

Resolving Order

Options are defined within rules, that is source elements. These elements are associated with specific backend servers by their host, port and path attributes.

Each source rule is evaluated depending on how well its host, port and path specification matches the current request. For example, a host with a port is more specific than a host without a port.

<sources>
  <source host="example.com" path="/checkout/billing">
    <!-- Options for requests to example.com/checkout/billing and below (any port) -->
  </source>

  <source host="example.com" port="443" path="/">
    <!-- Options for requests to example.com:443/ and below. -->
  </source>

  <source host="example.com" port="443">
    <!-- The same as above (path="/"). -->
  </source>

  <source host="example.com">
    <!-- Options for requests to example.com on any port. -->
  </source>

  <source>
    <!-- Supplementary options for all requests -->
  </source>
</sources>

The following illustration represents the resolving order for the options and parameters (see request actions, too) in a request. The smaller the number (1-n), the higher the priority of the options and parameter values:

<requests>
  <option (2) /> <!-- Dynamic option (see description below in the section "Dynamic Parameters and Options") -->
  <param (1) /> <!-- Dynamic parameter (see description below in the section "Dynamic Parameters and Options") -->

  <request url="..." content="..." method="...">
    <option (1) />
    <param (n) />
  </request>
</requests>

<sources>
  <!-- Most specific source -->
  <source host="..." port="..." path="...">
    <option (3) />
    <param (2) />
  </source>

  <!-- Less specific source -->
  <source host="...">
    <option (4) />
    <param (3) />
  </source>

  <!-- Least specific source -->
  <source>
    <option (n) />
    <param (4) />
  </source>
</sources>

The better a rule matches the request, the earlier its options will be set. The options of low-priority rules may be added, but they never override previous definitions. The main difference between options and parameters is that the options defined in the request rule are prioritized higher than options set in any source. However, the parameters defined in a request rule have the lowest priority.

The option <headers pass="..." /> should be regarded as placeholder for multiple header params in the respective rule. This is the single option, which values are evaluated in all matched rules.

Example

For example, FIT would interpret an incoming request with the URL https://example.com:443/shop/?a=1&b=&foo like:

<request url="https://example.com:443/shop/?a=1&amp;b=&amp;foo" ...>
  <header name="User-Agent" value="Client User-Agent" />
  <header name="X-Foo" value="Foo" />

  <query name="a" value="1" />
  <query name="b" value="" />
  <query name="foo" />
</request>

Hence, the default-request action would make the following “FIT request” corresponding to the incoming client request above:

<request url="https://example.com:443/shop/?a=1&amp;b=&amp;foo" ...>
  <redirects enable="false" />

  <query name="a" value="1" />
  <query name="b" value="" />
  <query name="foo" />
</request>

Let’s assume we have the following sources.xml:

<sources>
  <!-- Most specific source for URLs used in this example -->
  <source host="example.com" port="443" path="/shop">
    <!-- The following option is translated to: <header name="User-Agent" value="Client User-Agent" /> A header with the name "X-Not-Exists" will not be set, because here is no header with that name in the incoming request. -->
    <headers pass="User-Agent,X-Not-Exists" />

    <!-- Requests to https://example.com:443/shop with all optional sub-paths always override the X-Foo header with value "Bar" -->
    <header name="X-Foo" value="Bar" />

    <!-- Requests to https://example.com:443/shop with all optional sub-paths always replace the X-Blocked header with value "Allowed in shop" -->
    <header name="X-Blocked" value="Allowed in shop" />

    <!-- Requests to https://example.com:443/shop with all optional sub-paths always overwrite the query parameter "b" with value "2" -->
    <query name="b" value="2" />
  </source>

  <!-- Less specific source for URLs used in this example -->
  <source host="example.com" port="443">
    <!-- The following option is translated to: <header name="X-Foo" value="Foo" /> -->
    <headers pass="X-Foo" />

    <!-- Removes/blocks the X-Blocked header (would be set in "Least specific source") definition for all requests to https://example.com:443 excepting requests to https://example.com:443/shop with all optional sub-paths -->
    <header name="X-Blocked" value="" />

    <!-- Replaces the User-Agent header for all requests to https://example.com:443 excepting requests to https://example.com:443/shop with all optional sub-paths -->
    <header name="User-Agent" value="Example User-Agent" />
  </source>

  <!-- Least specific source for URLs used in this example -->
  <source>
    <redirects enable="true" />

    <header name="User-Agent" value="FIT User-Agent" />
    <header name="X-Blocked" value="Default" />
    <header name="X-Ever" value="Ever" />

    <query name="x" value="y" />
    <query name="x" value="z" />
  </source>
</sources>

On an incoming request to the URL http://example.com/?a=1&b=&foo:

  • The resulting URL: http://example.com/?a=1&b=&foo&x=y&x=z
  • The resulting query parameters:
    • a: 1 (from the FIT request)
    • b: =EMPTY (from the FIT request)
    • foo: EMPTY (from the FIT request)
    • x: y (set by the “Least specific source”)
    • x: z (set by the “Least specific source”)
  • The result header parameters:
    • X-Foo: is removed by the default-request action
    • User-Agent: FIT User-Agent (set by the “Least specific source”)
    • X-Blocked: Default (set by the “Least specific source”)
    • X-Ever: Ever (set by the “Least specific source”)

On an incoming request to the URL https://example.com:443/?a=1&b=&foo:

  • The resulting URL: https://example.com:443/?a=1&b=&foo&x=y&x=z
  • The resulting query parameters:
    • a: 1 (from the FIT request)
    • b: =EMPTY (from the FIT request)
    • foo: EMPTY (from the FIT request)
    • x: y (set by the “Least specific source”)
    • x: z (set by the “Least specific source”)
  • The header parameters:
    • X-Foo: Foo (from the incoming request, passed by the “Less specific source”)
    • User-Agent: Example User-Agent (set by the “Less specific source”)
    • X-Blocked: is removed by the “Less specific source”
    • X-Ever: Ever (set by the “Least specific source”)

On an incoming request to the URL https://example.com:443/shop/?a=1&b=&foo:

  • The resulting URL: https://example.com:443/shop/?a=1&foo&b=2&x=y&x=z
  • The resulting query parameters:
    • a: 1 (from the FIT request)
    • b: 2 (from the FIT request, overwritten by the “Most specific source”)
    • foo: EMPTY (from the FIT request)
    • x: y (set by the “Least specific source”)
    • x: z (set by the “Least specific source”)
  • The header parameters:
    • X-Foo: Bar (set by the “Most specific source”)
    • User-Agent: Client User-Agent (from the incoming request, passed by the “Most specific source”)
    • X-Blocked: Allowed in shop (set by the “Most specific source”)
    • X-Ever: Ever (set by the “Least specific source”)

In this example the X-Ever header with the value Ever and the query parameters “x=y&x=z” will always be set. However, the option <redirects enable="true" /> (defined in the “Least specific source”) can never be set.

Note, that the options set in the flow (e.g. in a request action) have the highest priority and will be evaluated first. See above for the difference between options and parameters, too.

Dynamic Parameters and Options

As described above in the section Resolving Order, the request parameter values have the lowest priority (<param (n) />). In some cases it could be useful to manipulate (set/unset/replace) the parameter values dynamically, e.g. in a requests action. You can define your own parameters within the requests element, but outside of each request element:


<requests>
  <param (1) /> <!-- This is a dynamic parameter -->
  <option (2) /> <!-- This is a dynamic option -->

  <request url="..." content="..." method="...">
    <option (1) />
    <param (n) />
  </request>

  ...
</requests>

In summary, the Dynamic Parameters and Options behave as if they were children of the most specific <source> element for any URL.

Note, that the dynamic parameter values have the highest priority and are applied for every request.

Note, that the dynamic options have a higher priority over any equivalent options in the sources.xml.

Example

Let’s assume we have the following simple sources.xml:


<sources>
  <source>
    <query name="output" value="x" />
  </source>
</sources>

and a requests action created dynamically by the following XSLT:


<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:template match="/">
    <requests>
      <query name="output" value="z" />

      <request content="main" url="http://example.com/?output=y" />
    </requests>
  </xsl:template>
</xsl:stylesheet>

or alternative defined statically in a requests action:


<requests>
  <query name="output" value="z" />

  <request content="main" url="http://example.com/?output=y" />
</requests>

In this case the request URL will always be http://example.com/?output=z, because the dynamic parameter value has the highest priority. If you remove the dynamic parameter, the request URL will be resolved as http://example.com/?output=x due to the next higher priority level in sources.xml.

Filters

Sometimes it may be desirable to set the request options based on the Delivery Context properties. For example, you may want to send specific headers to a source depending on the device type. You can accomplish this by using the Dynamic Configuration mechanisms in your sources.xml.

<sources>
  <source host="...">
    <choose>
      <when test="client/hw/type = 'tablet'">
        <header name="hwtype" value="tablet" />
      </when>
      <when test="client/hw/type = 'desktop'">
        <header name="hwtype" value="desktop" />
      </when>
      <when test="client/hw/type = 'mobile'">
        <header name="hwtype" value="mobile" />
      </when>
      <otherwise>
        <header name="hwtype" value="unspecified" />
      </otherwise>
    </choose>
  </source>
</sources>

The conf/sources.xml is loaded and filtered before the Flow is started.

HTTP Cache


<sources>
  <source ...>
    <cache enable="true|false" ...>
  </source>
  ...
</sources>
  • enable: Defaults to true. If enabled, FIT caches and reuses HTTP responses from sources servers. Interval Caching has priority over the automatic source caching. If enable="false", the Interval Caching is disabled, too.

For caching and reusing HTTP responses from the source servers, the following requirements must be met.

Cacheability is determined as follows:

  • If any Vary header fields contain * values, the response won’t be cached.
  • If any Cache-Control headers are present, they may contain the public, must-revalidate, proxy-revalidate, no-cache, pre-check, post-check, max-age or s-maxage directives. If the max-age or s-maxage directives occur multiple times, only the first one will be considered.
  • If there are any Cache-Control headers as described above, all Pragma headers will be ignored.
  • If there are no Cache-Control headers and a Pragma header contains the no-cache directive, the response won’t be cached.
  • If a WWW-Authenticate header is present, the response won’t be cached.
  • If a Authorization header is set, the response won’t be cached (HTTP Cache only. If enabled, the interval cache would still work).

Note If a sources server uses an authentication method to communicate with the FIT Server, it is not recommended to pass the Authorization header from the incoming request. In this case it is strongly recommended to use the auth option to set the authentication settings.

If a resource is cacheable, its cache lifetime will be determined as follows:

  • If a Cache-Control header contains a max-age or s-maxage directive, their values will be used to determine the cache lifetime. Any Expires headers will be ignored.
  • Without a max-age or s-maxage directive, the value of the Expires header will be used to calculate the cache lifetime. If there are multiple Expires headers, the resource won’t be cached.
  • If neither a max-age or s-maxage directive, nor an Expires header is present, the cache lifetime will be based on the Last-Modified or Age header, if possible.
    • This requires the response be marked as explicitly cacheable (e.g., with a Cache-Control public directive) or the status code defined as cacheable by default (see RFC 7231).
    • Multiple Last-Modified headers are considered invalid and will prevent caching.

If a client sends any of the following cache headers, the cache will be bypassed, but the response data will be stored and possibly reused:

  • Cache-Control: no-cache
  • Pragma: no-cache

The documents are stored on disk. In a cluster, every FIT server maintains its own cache. Stale documents are removed periodically by a cron job.

Options and Parameters

history
Enables the recording of intermediate response information on redirects. The information is available in fit://request/content/<ID>/history
enable boolean Enables or disables redirect history recording (default false).
header
name string Name of the HTTP header field
value string Value of the HTTP header. An empty value suppresses the respective HTTP header field.
response-header
The HTTP header field with name and value as specified is set, removed or replaced immediately after the remove statements for response-headers are processed.
name string Name of the HTTP header field
value string Value of the HTTP header (optional). If omitted or empty, all HTTP header fields matching the name will be removed.
replace boolean Defines, whether the response HTTP header field shall be replaced (default: true)
headers
HTTP headers to be sent to the backend system.
pass string Comma-separated list of incoming request headers to be passed to the backend. _auto_ will pass Accept, Accept-Language, MSISDN, Profile, Referer, User-Agent and all X-… headers and can be combined with other headers, e.g. <headers pass="_auto_, Authorization" />. Via and X-Forwarded-For headers may be sent modified using send-via and send-xff (see below).
send-via boolean Whether to send or extend a Via header (default false).
send-xff boolean Whether to include the client IP address in a X-Forwarded-For header (default false).
response-headers
Management of the HTTP headers received from a backend system.
pass string Comma-separated list of backend response headers to be passed to the client. The header list may contain wildcards like * or ?.

X-…-headers and Content-Disposition are always passed; Content-Type, Content-Length, Cache-Control (and friends) and Set-Cookie headers are handled automatically by FIT and should usually not be passed explicitly. All caching headers are also affected by the pass-cache-headers option.

Note: This option does not work for side requests (i.e. extra requests that use an ID other than main).

remove string Comma-separated list of backend response headers to be removed immediately after receiving them from a backend system. Wildcards like * or ? are allowed.
ssl
TLS/SSL settings for backend communication
check-certificate boolean Enables or disables the verification of certificates on SSL connections (default true).
client-certificate string Path to the client certificate (.pem)
certificate-password string The certificate password
version string The SSL protocol version:
  • auto: Automatic protocol negotiation (default)
  • TLSv1: TLS ≥ 1.0 required
  • TLSv1.0: TLS 1.0 required
  • TLSv1.1: TLS 1.1 required
  • TLSv1.2: TLS 1.2 required
  • SSLv3: SSL 3 required (deprecated, insecure)
ciphers string Ciphers to use for TLS connections. Note that the cipher names and the format of this option depend on whether the system is backed by OpenSSL (SLES/openSUSE/Ubuntu) or NSS (RHEL/CentOS).
proxy
Proxy server for communication with backend systems
server string Proxy server hostname with port (e.g. proxy.example.com:3128)
cache boolean Whether to use proxy caching (default true).
credentials string Authentication data for the proxy server (e.g. username:password).
auth-method string Authentication method on selected proxy server: basic (default), ntlm.
cache
Caching
enable bool Whether to enable HTTP source caching. (default true).
interval integer The cache interval in seconds (086400). 0 disables caching (default).
vary-headers string Comma-separated list of headers that should be considered to lookup the correct cache entry. Example: User-Agent, Accept-Language
allow-invalidation boolean Allows to invalidate the cache by means of Cache-Control: no-cache or Pragma: no-cache respectively (default false).
timeout
Request, stream and connection timeouts and limits
request float Maximum time in seconds for processing a request. When the maximum time is reached before the response has been received, the processing will be terminated. Decimals are allowed, e.g. 1.5 (default: FIT_FETCH_DEFAULT_TIMEOUT).
connect float Maximum time in seconds for establishing a connection to a server (default: FIT_FETCH_DEFAULT_CONNECT_TIMEOUT).
low-speed-limit int If the average transfer speed in bytes per second of a transfer drops below low-speed-limit for low-speed-time seconds, FIT will abort the transfer with an error (default 1). This option has no effect unless the low-speed-time option is set to a value greater than 0.
low-speed-time int If the average transfer speed in bytes per second of a transfer drops below low-speed-limit for low-speed-time seconds, FIT will abort the transfer with an error (default: FIT_FETCH_DEFAULT_LOW_SPEED_TIMEOUT). This option has no effect unless the low-speed-limit option is set to a value greater than 0. low-speed-time may be useful to abort transfers in case the Time to First Byte is excessive, while still maintaining long timeouts for slow but consistent downloads.
redirects
Configure the redirects
enable bool Whether to automatically follow redirects (default false).
max integer Maximum number of consecutive redirects to follow.
cookies
Cookies
send bool Whether to send all matching cookies to the backend (default true). If caching is enabled, the default is false.
accept bool Whether to accept cookies received from the backend (default true). If caching is enabled, the default is false.
mode string By default, FIT encodes cookies received from source servers to ensure that FIT passes the cookies only to the correct servers (and paths) in subsequent requests. This option can be used to enable the transparent mode for the transfer of cookies to the client: encoded (default), transparent
auth
Authentication settings for the backend system
method string Authentication method on backend system: basic (default), safe, any, digest, ntlm
credentials string Authentication data for backend system (e.g. username:password)
parser
Normally, the Content-Type header is used to select the correct parser. This option allows select the parser directly.
value string The parser to use: xml, html, json, js, css, false
encoding
This option can be used to set the encoding used for the URI including the query string and POST form data.
value string The name of the encoding, (default UTF-8). Other common values are ISO-8859-1, ISO-8859-15 and Windows-1251. For a list of all supported encodings, please refer to the PHP documentation.
normalize-urls
If enabled, URLs within the document are normalized based on the source URL, i.e. the document does not contain any relative URLs anymore.
enable bool Whether to enable URL normalization (default true).
query-encoding
Normally, certain characters in the query string are URL encoded. This option can be used to set a list of characters which are to remain unencoded.
raw string Characters in the query string to exclude from the URL encoding.
query
name string Name of the query (GET) parameter
value string Value of the query (GET) parameter
post
name string Name of the POST parameter
value string Value of the POST parameter
upload
name string Name (key) of the file to be sent
filename string File name of the file to be sent
size integer File size in bytes
type string MIME type
src string FIDJ resource to upload.
transport
This option allows to alter protocol, host or port of an outgoing request e.g. to use HTTP for backend requests while using HTTPS with clients. To actually use this option it has to be allowed by the FIT_ALLOW_TRANSPORT_MAPPING setting in the fit.ini configuration file. In addition, the system ACL (/opt/sevenval/fit14/conf/acl.xml) must permit access to the backend system.
protocol string Protocol for communication with the backend system. If not set, the original protocol won’t be changed. (http or https only)
port integer Port for communication with the backend system. If not set, the original port won’t be changed.
server string Backend server to communicate with. If this option is not set, the server won’t be changed.
connection
By default, FIT uses HTTP Keepalive to minimize the TCP connection overhead and improve request performance. In the unlikely case that this causes problems, HTTP persistent connections can be disabled for individual sources using this option, or globally via the FIT_USE_PERSISTENT_CONNECTIONS setting in the fit.ini configuration file.
reuse bool Set to false to disable HTTP persistent connections.