Difference between revisions of "AutomatedTests"

From Request Tracker Wiki
Jump to navigation Jump to search
m (4 revisions imported)
(add docs)
 
Line 1: Line 1:
= <span style="font-size:13px;line-height:21px;">RT has a full test suite for ensuring functionality remains functional between changes. Best Practical smoke tests every commit on a variety of configurations, but it's good practice to run tests yourself if you're making changes to RT when writing a patch you plan to submit (or even one you don't!).  The test suite is quite large, and will take some time to run.</span> =
== Test Prerequisites ==
<span style="font-size:13px;line-height:21px;">Instructions for running the tests are in RT's official documentation: </span>http://bestpractical.com/rt/docs/latest/hacking#Test-suite
 
The best way to set up RT for running tests (both its own and extensions') is by running the following from a source checkout:
 
<pre>./configure.ac --with-db-type=TYPE --enable-layout=inplace [other options ...]
mkdir -p var</pre>
 
Then make sure all your dependencies are in place by running <code>make testdeps</code>. (If you need help with that part, check out [[ManualInstallation]].) If you're at that point and just want to get running as fast as possible, skip down to "Environment Variables to Control Tests." The next couple of sections give more background about what's going on.
 
=== Why a real database? ===
 
RT does include database-specific tests to check that various integrations work as expected. There's no good way to abstract these, so the test suite selects and runs the tests appropriate for your database.
 
=== Why an inplace layout? ===
 
The RT tests read the configuration file <code>RT_Config.pm</code> located in the directory named at <code>$RT::Generated::EtcPath</code>. If you use the default relative layout, or another layout intended for production like FHS, the tests will try to read your production configuration. That's probably not what you want.
 
Note that the tests '''only''' read <code>RT_Config.pm</code>. They do '''not''' read <code>RT_SiteConfig.pm</code> or files under <code>RT_SiteConfig.d</code>. You cannot change the behavior of tests by editing configuration there.
 
(This is another reason you need to specify a database type on the configure line: so it gets put into <code>RT_Config.pm</code>. There are environment variables to control most database parameters you need, but not the database type.)
 
== Environment Variables to Control Tests ==
 
=== Test RT Configuration ===
 
Except on SQLite, the tests need permission to create new databases; to create new users in the database; and to grant rights to those users. To run the tests at all, you '''must''' set the environment variables <code>RT_DBA_USER</code> and <code>RT_DBA_PASSWORD</code> with the username and password, respectively, of a database account that has all those rights.
 
The tests write a small <code>RT_SiteConfig.pm</code> it uses to configure itself. You can control some of the settings with environment variables. Refer to [https://docs.bestpractical.com/rt/latest/RT_Config.html the RT_Config documentation] for full details.
 
{| class="wikitable" style="width: 100%;"
! Environment variable
! Corresponding RT_SiteConfig setting
|-
|<code>RT_TEST_DB_SID</code>
|<code>$DatabaseName</code> (intended for Oracle only)
|-
|<code>RT_TEST_DB_HOST</code>
|<code>$DatabaseHost</code>
|-
|<code>RT_TEST_RT_HOST</code>
|<code>$DatabaseRTHost</code>
|-
|<code>RT_TEST_DEVEL</code>
|<code>$DevelMode</code>
|-
|<code>RT_TEST_PLUGINS</code>
|<code>@Plugins</code> (split on spaces)
|}
 
=== Test Selection ===
 
A few tests are skipped unless you do/n't set an environment variable.
 
{| class="wikitable" style="width: 100%;"
! Environment variable
! Controls tests
|-
|<code>RT_TEST_HEAVY</code>
|Set a string here to run tests that require relatively expensive database queries.
|-
|<code>RT_TEST_SMIME_REVOCATION</code>
|Set a string here to run SMIME revocation tests. Doing so downloads a certificate revocation list (CRL) from the public Internet.
|-
|<code>SKIP_GPG_TESTS</code>
|Set a string here to skip tests that run GnuPG
|}
 
=== Test Run Environment ===
 
{| class="wikitable" style="width: 100%;"
! Environment variable
! Controls tests
|-
|<code>RTHOME</code>
|Path to the RT source code checkout you use for testing. The core RT tests don't use this, but extensions usually do.
|-
|<code>RT_TEST_PARALLEL</code>
|Set a string here to notify the test suite that you're running tests in parallel. This changes some of the logic around database creation and elsewhere to avoid contention. You can set this even if you only have a single test runner.
You may also find some references to <code>RT_TEST_PARALLEL_NUM</code>. This is used by RT's Makefile to control how many test runners run in parallel. If you don't go through RT's Makefile, then nothing else uses that variable.
|-
|<code>RT_TEST_APACHE</code>
|Path to the <code>httpd</code> executable on your system
|-
|<code>RT_TEST_APXS</code>
|Path to the <code>apxs</code> executable on your system
|-
|<code>RT_TEST_APACHE_MODULES</code>
|Path to the directory where your httpd module <code>.so</code> files live
|}
 
== Running the Tests ==
 
Once you've set and exported all the environment variables you need, run the tests with <code>prove</code> from your source directory:
 
<pre>RT_TEST_PARALLEL=1 prove -w --lib --recurse --jobs=NUMBER t</pre>
 
{| class="wikitable" style="width: 100%;"
! Option
! Purpose
|-
|<code>-w</code>
|Enable warnings
|-
|<code>--lib</code>
|Load libraries from the <code>lib</code> directory (i.e., your source checkout)
|-
|<code>--recurse</code>
|Recurse through the path argument(s) to find and run all test <code>.t</code> files
|-
|<code>--jobs=NUMBER</code>
|Run <code>NUMBER</code> test runners in parallel
|-
|<code>t</code>
|Run tests under this path. You can specify other directories (some extensions have tests in <code>xt</code>), specific subdirectories, or even individual files.
|}

Latest revision as of 18:05, 17 December 2021

Test Prerequisites

The best way to set up RT for running tests (both its own and extensions') is by running the following from a source checkout:

./configure.ac --with-db-type=TYPE --enable-layout=inplace [other options ...]
mkdir -p var

Then make sure all your dependencies are in place by running make testdeps. (If you need help with that part, check out ManualInstallation.) If you're at that point and just want to get running as fast as possible, skip down to "Environment Variables to Control Tests." The next couple of sections give more background about what's going on.

Why a real database?

RT does include database-specific tests to check that various integrations work as expected. There's no good way to abstract these, so the test suite selects and runs the tests appropriate for your database.

Why an inplace layout?

The RT tests read the configuration file RT_Config.pm located in the directory named at $RT::Generated::EtcPath. If you use the default relative layout, or another layout intended for production like FHS, the tests will try to read your production configuration. That's probably not what you want.

Note that the tests only read RT_Config.pm. They do not read RT_SiteConfig.pm or files under RT_SiteConfig.d. You cannot change the behavior of tests by editing configuration there.

(This is another reason you need to specify a database type on the configure line: so it gets put into RT_Config.pm. There are environment variables to control most database parameters you need, but not the database type.)

Environment Variables to Control Tests

Test RT Configuration

Except on SQLite, the tests need permission to create new databases; to create new users in the database; and to grant rights to those users. To run the tests at all, you must set the environment variables RT_DBA_USER and RT_DBA_PASSWORD with the username and password, respectively, of a database account that has all those rights.

The tests write a small RT_SiteConfig.pm it uses to configure itself. You can control some of the settings with environment variables. Refer to the RT_Config documentation for full details.

Environment variable Corresponding RT_SiteConfig setting
RT_TEST_DB_SID $DatabaseName (intended for Oracle only)
RT_TEST_DB_HOST $DatabaseHost
RT_TEST_RT_HOST $DatabaseRTHost
RT_TEST_DEVEL $DevelMode
RT_TEST_PLUGINS @Plugins (split on spaces)

Test Selection

A few tests are skipped unless you do/n't set an environment variable.

Environment variable Controls tests
RT_TEST_HEAVY Set a string here to run tests that require relatively expensive database queries.
RT_TEST_SMIME_REVOCATION Set a string here to run SMIME revocation tests. Doing so downloads a certificate revocation list (CRL) from the public Internet.
SKIP_GPG_TESTS Set a string here to skip tests that run GnuPG

Test Run Environment

Environment variable Controls tests
RTHOME Path to the RT source code checkout you use for testing. The core RT tests don't use this, but extensions usually do.
RT_TEST_PARALLEL Set a string here to notify the test suite that you're running tests in parallel. This changes some of the logic around database creation and elsewhere to avoid contention. You can set this even if you only have a single test runner.

You may also find some references to RT_TEST_PARALLEL_NUM. This is used by RT's Makefile to control how many test runners run in parallel. If you don't go through RT's Makefile, then nothing else uses that variable.

RT_TEST_APACHE Path to the httpd executable on your system
RT_TEST_APXS Path to the apxs executable on your system
RT_TEST_APACHE_MODULES Path to the directory where your httpd module .so files live

Running the Tests

Once you've set and exported all the environment variables you need, run the tests with prove from your source directory:

RT_TEST_PARALLEL=1 prove -w --lib --recurse --jobs=NUMBER t
Option Purpose
-w Enable warnings
--lib Load libraries from the lib directory (i.e., your source checkout)
--recurse Recurse through the path argument(s) to find and run all test .t files
--jobs=NUMBER Run NUMBER test runners in parallel
t Run tests under this path. You can specify other directories (some extensions have tests in xt), specific subdirectories, or even individual files.