Sources (conf/sources.xml)

General request options that are suitable for all requests to a host or certain paths on a host may be set in conf/sources.xml. They apply to all requests that FIT issues, be it user defined request actions or system requests (e.g. image scaling requests). The sources document element may contain options for multiple sources.

Example

The following example includes all available options and parameters.

<sources>
  <source host="…" port="…" path="…">
    <!-- Request options -->
    <timeout request="5" connect="1" low-speed-time="3" low-speed-limit="1" />
    <headers send-via="true" send-xff="true" pass="User-Agent, Accept-Language" />
    <response-headers pass="WWW-Authenticate, Access-Control-*" remove="Set-Cookie, X-Foo" />
    <ssl version="TLSv1" check-certificate="true" client-certificate="fit://site/conf/mycert.pem" certificate-password="pa5sw0Rd" ciphers="RC4" />
    <transport protocol="http" server="127.0.0.1" port="8080" />
    <proxy server="proxy.example.com:3128" cache="true" credentials="user:s3CreT" auth-method="basic" />
    <cache enable="true" interval="3600" vary-headers="Accept-Language" allow-invalidation="false" />
    <redirects enable="false" max="10" />
    <cookies send="true" accept="true" mode="transparent" />
    <auth method="basic" credentials="user:0P3n-5es4Me" />
    <history enable="false" />
    <parser value="json" />
    <encoding value="utf-8" />
    <normalize-urls enable="true" />
    <query-encoding raw="*$!" />
    <connection reuse="true" />

    <!-- Request params -->
    <response-header name="Cache-Control" value="public, max-age=3600, s-maxage=3600" replace="true" />
    <header name="User-Agent" value="FIT User-Agent" />
    <query name="X" value="Y" />
    <query name="W" value="Z" />
    <post name="A" value="B" />
    <post name="C" value="D" />
    <upload name="myFile" filename="file.txt" size="123" type="text/plain" src="fit://site/public/file.txt" />
  </source>
</sources>

Resolving Order

Options are defined within rules, that is source elements. These elements are associated with specific backend servers by their host, port and path attributes.

Each source rule is evaluated depending on how well its host, port and path specification matches the current request. For example, a host with a port is more specific than a host without a port.

<sources>
  <source host="example.com" path="/checkout/billing">
    <!-- Options for requests to example.com/checkout/billing and below (any port) -->
  </source>

  <source host="example.com" port="443" path="/">
    <!-- Options for requests to example.com:443/ and below. -->
  </source>

  <source host="example.com" port="443">
    <!-- The same as above (path="/"). -->
  </source>

  <source host="example.com">
    <!-- Options for requests to example.com on any port. -->
  </source>

  <source>
    <!-- Supplementary options for all requests -->
  </source>
</sources>

The following illustration represents the resolving order for the options and parameters (see request actions, too) in a request. The smaller the number (1-n), the higher the priority of the options and parameter values:

<requests>
  <option (2) /> <!-- Dynamic option (see description below in the section "Dynamic Parameters and Options") -->
  <param (1) /> <!-- Dynamic parameter (see description below in the section "Dynamic Parameters and Options") -->

  <request url="..." content="..." method="...">
    <option (1) />
    <param (n) />
  </request>
</requests>

<sources>
  <!-- Most specific source -->
  <source host="..." port="..." path="...">
    <option (3) />
    <param (2) />
  </source>

  <!-- Less specific source -->
  <source host="...">
    <option (4) />
    <param (3) />
  </source>

  <!-- Least specific source -->
  <source>
    <option (n) />
    <param (4) />
  </source>
</sources>

The better a rule matches the request, the earlier its options will be set. The options of low-priority rules may be added, but they never override previous definitions. The main difference between options and parameters is that the options defined in the request rule are prioritized higher than options set in any source. However, the parameters defined in a request rule have the lowest priority.

The option <headers pass="..." /> should be regarded as placeholder for multiple header params in the respective rule. This is the single option, which values are evaluated in all matched rules.

Example

For example, FIT would interpret an incoming request with the URL https://example.com:443/shop/?a=1&b=&foo like:

<request url="https://example.com:443/shop/?a=1&amp;b=&amp;foo" ...>
  <header name="User-Agent" value="Client User-Agent" />
  <header name="X-Foo" value="Foo" />

  <query name="a" value="1" />
  <query name="b" value="" />
  <query name="foo" />
</request>

Hence, the default-request action would make the following “FIT request” corresponding to the incoming client request above:

<request url="https://example.com:443/shop/?a=1&amp;b=&amp;foo" ...>
  <redirects enable="false" />

  <query name="a" value="1" />
  <query name="b" value="" />
  <query name="foo" />
</request>

Let’s assume we have the following sources.xml:

<sources>
  <!-- Most specific source for URLs used in this example -->
  <source host="example.com" port="443" path="/shop">
    <!-- The following option is translated to: <header name="User-Agent" value="Client User-Agent" /> A header with the name "X-Not-Exists" will not be set, because here is no header with that name in the incoming request. -->
    <headers pass="User-Agent,X-Not-Exists" />

    <!-- Requests to https://example.com:443/shop with all optional sub-paths always override the X-Foo header with value "Bar" -->
    <header name="X-Foo" value="Bar" />

    <!-- Requests to https://example.com:443/shop with all optional sub-paths always replace the X-Blocked header with value "Allowed in shop" -->
    <header name="X-Blocked" value="Allowed in shop" />

    <!-- Requests to https://example.com:443/shop with all optional sub-paths always overwrite the query parameter "b" with value "2" -->
    <query name="b" value="2" />
  </source>

  <!-- Less specific source for URLs used in this example -->
  <source host="example.com" port="443">
    <!-- The following option is translated to: <header name="X-Foo" value="Foo" /> -->
    <headers pass="X-Foo" />

    <!-- Removes/blocks the X-Blocked header (would be set in "Least specific source") definition for all requests to https://example.com:443 excepting requests to https://example.com:443/shop with all optional sub-paths -->
    <header name="X-Blocked" value="" />

    <!-- Replaces the User-Agent header for all requests to https://example.com:443 excepting requests to https://example.com:443/shop with all optional sub-paths -->
    <header name="User-Agent" value="Example User-Agent" />
  </source>

  <!-- Least specific source for URLs used in this example -->
  <source>
    <redirects enable="true" />

    <header name="User-Agent" value="FIT User-Agent" />
    <header name="X-Blocked" value="Default" />
    <header name="X-Ever" value="Ever" />

    <query name="x" value="y" />
    <query name="x" value="z" />
  </source>
</sources>

On an incoming request to the URL http://example.com/?a=1&b=&foo:

• The resulting URL: http://example.com/?a=1&b=&foo&x=y&x=z
• The resulting query parameters:
  • a: 1 (from the FIT request)
  • b: =EMPTY (from the FIT request)
  • foo: EMPTY (from the FIT request)
  • x: y (set by the “Least specific source”)
  • x: z (set by the “Least specific source”)
• The result header parameters:
  • X-Foo: is removed by the default-request action
  • User-Agent: FIT User-Agent (set by the “Least specific source”)
  • X-Blocked: Default (set by the “Least specific source”)
  • X-Ever: Ever (set by the “Least specific source”)

On an incoming request to the URL https://example.com:443/?a=1&b=&foo:

• The resulting URL: https://example.com:443/?a=1&b=&foo&x=y&x=z
• The resulting query parameters:
  • a: 1 (from the FIT request)
  • b: =EMPTY (from the FIT request)
  • foo: EMPTY (from the FIT request)
  • x: y (set by the “Least specific source”)
  • x: z (set by the “Least specific source”)
• The header parameters:
  • X-Foo: Foo (from the incoming request, passed by the “Less specific source”)
  • User-Agent: Example User-Agent (set by the “Less specific source”)
  • X-Blocked: is removed by the “Less specific source”
  • X-Ever: Ever (set by the “Least specific source”)

On an incoming request to the URL https://example.com:443/shop/?a=1&b=&foo:

• The resulting URL: https://example.com:443/shop/?a=1&foo&b=2&x=y&x=z
• The resulting query parameters:
  • a: 1 (from the FIT request)
  • b: 2 (from the FIT request, overwritten by the “Most specific source”)
  • foo: EMPTY (from the FIT request)
  • x: y (set by the “Least specific source”)
  • x: z (set by the “Least specific source”)
• The header parameters:
  • X-Foo: Bar (set by the “Most specific source”)
  • User-Agent: Client User-Agent (from the incoming request, passed by the “Most specific source”)
  • X-Blocked: Allowed in shop (set by the “Most specific source”)
  • X-Ever: Ever (set by the “Least specific source”)

In this example the X-Ever header with the value Ever and the query parameters “x=y&x=z” will always be set. However, the option <redirects enable="true" /> (defined in the “Least specific source”) can never be set.

Dynamic Parameters and Options

As described above in the section Resolving Order, the request parameter values have the lowest priority (<param (n) />). In some cases it could be useful to manipulate (set/unset/replace) the parameter values dynamically, e.g. in a requests action. You can define your own parameters within the requests element, but outside of each request element:

<requests>
  <param (1) /> <!-- This is a dynamic parameter -->
  <option (2) /> <!-- This is a dynamic option -->

  <request url="..." content="..." method="...">
    <option (1) />
    <param (n) />
  </request>

  ...
</requests>

In summary, the Dynamic Parameters and Options behave as if they were children of the most specific <source> element for any URL.

Example

Let’s assume we have the following simple sources.xml:

<sources>
  <source>
    <query name="output" value="x" />
  </source>
</sources>

and a requests action created dynamically by the following XSLT:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:template match="/">
    <requests>
      <query name="output" value="z" />

      <request content="main" url="http://example.com/?output=y" />
    </requests>
  </xsl:template>
</xsl:stylesheet>

or alternative defined statically in a requests action:

<requests>
  <query name="output" value="z" />

  <request content="main" url="http://example.com/?output=y" />
</requests>

In this case the request URL will always be http://example.com/?output=z, because the dynamic parameter value has the highest priority. If you remove the dynamic parameter, the request URL will be resolved as http://example.com/?output=x due to the next higher priority level in sources.xml.

Filters

Sometimes it may be desirable to set the request options based on the Delivery Context properties. For example, you may want to send specific headers to a source depending on the device type. You can accomplish this by using the Dynamic Configuration mechanisms in your sources.xml.

<sources>
  <source host="...">
    <choose>
      <when test="client/hw/type = 'tablet'">
        <header name="hwtype" value="tablet" />
      </when>
      <when test="client/hw/type = 'desktop'">
        <header name="hwtype" value="desktop" />
      </when>
      <when test="client/hw/type = 'mobile'">
        <header name="hwtype" value="mobile" />
      </when>
      <otherwise>
        <header name="hwtype" value="unspecified" />
      </otherwise>
    </choose>
  </source>
</sources>

The conf/sources.xml is loaded and filtered before the Flow is started.

HTTP Cache

<sources>
  <source ...>
    <cache enable="true|false" ...>
  </source>
  ...
</sources>

• enable: Defaults to true. If enabled, FIT caches and reuses HTTP responses from sources servers. Interval Caching has priority over the automatic source caching. If enable="false", the Interval Caching is disabled, too.

For caching and reusing HTTP responses from the source servers, the following requirements must be met.

Cacheability is determined as follows:

• If any Vary header fields contain * values, the response won’t be cached.
• If any Cache-Control headers are present, they may contain the public, must-revalidate, proxy-revalidate, no-cache, pre-check, post-check, max-age or s-maxage directives. If the max-age or s-maxage directives occur multiple times, only the first one will be considered.
• If there are any Cache-Control headers as described above, all Pragma headers will be ignored.
• If there are no Cache-Control headers and a Pragma header contains the no-cache directive, the response won’t be cached.
• If a WWW-Authenticate header is present, the response won’t be cached.
• If a Authorization header is set, the response won’t be cached (HTTP Cache only. If enabled, the interval cache would still work).

If a resource is cacheable, its cache lifetime will be determined as follows:

• If a Cache-Control header contains a max-age or s-maxage directive, their values will be used to determine the cache lifetime. Any Expires headers will be ignored.
• Without a max-age or s-maxage directive, the value of the Expires header will be used to calculate the cache lifetime. If there are multiple Expires headers, the resource won’t be cached.
• If neither a max-age or s-maxage directive, nor an Expires header is present, the cache lifetime will be based on the Last-Modified or Age header, if possible.
  • This requires the response be marked as explicitly cacheable (e.g., with a Cache-Control public directive) or the status code defined as cacheable by default (see RFC 7231).
  • Multiple Last-Modified headers are considered invalid and will prevent caching.

If a client sends any of the following cache headers, the cache will be bypassed, but the response data will be stored and possibly reused:

• Cache-Control: no-cache
• Pragma: no-cache

The documents are stored on disk. In a cluster, every FIT server maintains its own cache. Stale documents are removed periodically by a cron job.

Options and Parameters