- Update Docopt
- Switch toin-house Picofeed branch for now
- Update composer-bin
- Update php-cs-fixer
Daux has been left as-is for now even though we're using an old version
The Feed class sets dynamic properties on Picofeed classes; this will
need to be completely rewritten. Version 1.2 of the Laminas XML class
also uses a deprecated function, but upgrading it to 1.3 will require
PHP 7.3.
For now this only adds convenience wrappers around Guzzle to somewhat
emulate Diactoros (albeit with a different API). Code and tests will be
adjusted in due course.
Daux uses Cmmonmark, which does not support indention, required for
proper formatting of manual pages. Consequently, the manul page will
instead be standalone.
@ -8,7 +8,7 @@ Information on how to install and use the software can be found in [the manual](
The main repository for The Arsse can be found at [code.mensbeam.com](https://code.mensbeam.com/MensBeam/arsse/), with a mirror also available [at GitHub](https://github.com/mensbeam/arsse/). The GitHub mirror does not accept bug reports, but the two should otherwise be equivalent.
[Composer](https://getcomposer.org/) is required to manage PHP dependencies. After cloning the repository or downloading a source code tarball, running `composer install` will download all the required dependencies, and will advise if any PHP extensions need to be installed. If not installing as a programming environment, running `composer install --no-dev` is recommended.
[Composer](https://getcomposer.org/) is required to manage PHP dependencies. After cloning the repository or downloading a source code tarball, running `composer install` will download all the required dependencies, and will advise if any PHP extensions need to be installed. If not installing as a programming environment, running `composer install -o --no-dev --no-scripts` is recommended.
# Repository structure
@ -34,13 +34,15 @@ Also necessary to the functioning of the application is the `/vendor/` directory
The `/locale/` and `/sql/` directories contain human-language files and database schemata, both of which are occasionally used by the application in the course of execution. The `/www/` directory serves as a document root for a few static files to be made available to users by a Web server.
The `/dist/` directory, on the other hand, contains samples of configuration for Web servers and init systems. These are not used by The Arsse itself, but are merely distributed with it for reference.
The `/dist/` directory, on the other hand, contains general and system-specific build files, manual pages, and samples of configuration for Web servers and other system integration. These are not used by The Arsse itself, but are used during the process of preparing new releases for supported operating systems.
## Documentation
The source text for The Arsse's manual can be found in `/docs/`, with pages written in [Markdown](https://spec.commonmark.org/current/) and converted to HTML [with Daux](#building-the-manual). If a static manual is generated its files will appear under `/manual/`.
In addition to the manual the files `/README.md` (this file), `/CHANGELOG`, `/UPGRADING`, `/LICENSE`, and `/AUTHORS` also document various things about the software, rather than the software itself.
The Arsse also has a UNIX manual page written in [mdoc](https://man.archlinux.org/man/extra/mandoc/mandoc_mdoc.7.en) format, which can be found under `/dist/man/`.
In addition to the manuals the files `/README.md` (this file), `/CHANGELOG`, `/UPGRADING`, `/LICENSE`, and `/AUTHORS` also document various things about the software, rather than the software itself.
## Tests
@ -50,7 +52,7 @@ The `/tests/` directory contains everything related to automated testing. It is
@ -63,18 +65,18 @@ PHPUnit's configuration can be customized by copying its configuration file to `
The `/vendor-bin/` directory houses the files needed for the tools used in The Arsse's programming environment. These are managed by the Composer ["bin" plugin](https://github.com/bamarni/composer-bin-plugin) and are not used by The Arsse itself. The following files are also related to various programming tools:
| `/.gitattributes` | Git settings for handling files |
| `/.gitignore` | Git file exclusion patterns |
| `/.php_cs.dist` | Configuration for [php-cs-fixer](https://cs.symfony.com) |
| `/.php_cs.cache` | Cache for php-cs-fixer |
| `/.php-cs-fixer.dist.php` | Configuration for [php-cs-fixer](https://cs.symfony.com) |
| `/.php-cs-fixer.cache` | Cache for php-cs-fixer |
| `/composer.json` | Configuration for Composer |
| `/composer.lock` | Version synchronization data for Composer |
| `/RoboFile.php` | Task definitions for [Robo](https://robo.li/) |
| `/robo` | Simple wrapper for executing Robo on POSIX systems |
| `/robo.bat` | Simple wrapper for executing Robo on Windows |
In addition the files `/package.json`, `/yarn.lock`, and `/postcss.config.js` as well as the `/node_modules/` directory are used by [Yarn](https://yarnpkg.com/) and [PostCSS](https://postcss.org/) when modifying the stylesheet for The Arsse's manual.
In addition the files `/package.json` and `/postcss.config.js` as well as the `/node_modules/` directory are used by [Yarn](https://yarnpkg.com/) and [PostCSS](https://postcss.org/) when modifying the stylesheet for The Arsse's manual.
# Common tasks
@ -88,11 +90,11 @@ There is also a `test:quick` Robo task which excludes slower tests, and a `test:
### Test coverage
Computing the coverage of tests can be done by running `./robo coverage`, after which an HTML-format coverage report will be written to `/tests/coverage/`. Either [PCOV](https://github.com/krakjoe/pcov), [Xdebug](https://xdebug.org), or [phpdbg](https://php.net/manual/en/book.phpdbg.php) is required for this. PCOV is generally recommended as it is faster than Xdebug; phpdbg is faster still, but less accurate. If using either PCOV or Xdebug, the extension need not be enabled globally; PHPUnit will enable it when needed.
Computing the coverage of tests can be done by running `./robo coverage`, after which an HTML-format coverage report will be written to `/tests/coverage/`. Either [PCOV](https://github.com/krakjoe/pcov) or [Xdebug](https://xdebug.org) is required for this. PCOV is generally recommended as it is faster than Xdebug. Neither extension need be enabled globally; Robo will enable it when needed.
## Enforcing coding style
The [php-cs-fixer](https://cs.symfony.com) tool, executed via `./robo clean`, can be used to rewrite code to adhere to The Arsse's coding style. The style largely follows [PSR-2](https://www.php-fig.org/psr/psr-2/) with some exceptions:
The [php-cs-fixer](https://cs.symfony.com) tool, executed via `./robo clean`, can be used to rewrite code to adhere to The Arsse's coding style. The style largely follows [PSR-12](https://www.php-fig.org/psr/psr-12/) with some exceptions:
- Classes, methods, and functions should have their opening brace on the same line as the signature
- Anonymous functions should have no space before the parameter list
@ -107,13 +109,11 @@ The manual employs a custom theme derived from the standard Daux theme. If the s
## Packaging a release
Producing a release package is done by running `./robo package`. This performs the following operations:
Producing release packages is done by running `./robo package`. This performs the following operations:
- Duplicates a working tree with the commit (usually a release tag) to package
- Generates the manual
- Duplicates a [Git](https://git-scm.com/) working tree with the commit (usually a release tag) to package
- Generates the HTML manual
- Installs runtime Composer dependencies with an optimized autoloader
- Deletes numerous unneeded files
- Exports the default configuration of The Arsse to a file
- Compresses the remaining files into a tarball
Due to the first step, [Git](https://git-scm.com/) is required to package a release.
/** Returns the necessary shell arguments to print error output or all output to the bitbucket
*
* @param bool $all Whether all output (true) or only error output (false) should be suppressed
*/
protected function blackhole(bool $all = false): string {
$hole = IS_WIN ? "nul" : "/dev/null";
return $all ? ">$hole 2>&1" : "2>$hole";
}
/** Executes PHPUnit, used by the test and coverage tasks.
*
* This also executes the built-in PHP Web server, which is required to fetch some newsfeeds during tests
*
* @param string $executor The path to the PHP binary to execute with any required extra arguments. Normally this is either "php" or the result of findCoverageEngine()
* @param string $set The set of tests to run, either "typical" (excludes redundant tests), "quick" (excludes redundant and slow tests), "coverage" (excludes tests not needed for coverage), or "full" (all tests)
* @param array $args Extra arguments passed by Robo from the command line
*/
protected function runTests(string $executor, string $set, array $args): Result {
switch ($set) {
case "typical":
@ -145,9 +149,38 @@ class RoboFile extends \Robo\Tasks {
@ -314,6 +454,15 @@ class RoboFile extends \Robo\Tasks {
return $t->run();
}
/** Parses the contents of the CHANGELOG file into an array structure
*
* This is done line-by-line and tends to be quite strict.
* The parsed output can be used to generate changelogs in other formats,
* such as a Debian changelog or RPM changelog.
*
* @param string $text The text of the CHANGELOG file
* @param string $targetVersion The x.y.z version number of the latest release. This is used to check that version numbers and dates have been updated when preparing a release
*/
protected function changelogParse(string $text, string $targetVersion): array {
$lines = preg_split('/\r?\n/', $text);
$version = "";
@ -323,9 +472,9 @@ class RoboFile extends \Robo\Tasks {
$expected = ["version"];
for ($a = 0; $a <sizeof($lines);){
$l = rtrim($lines[$a++]);
if (in_array("version", $expected) && preg_match('/^Version (\d+(?:\.\d+)*) \(([\d\?]{4}-[\d\?]{2}-[\d\?]{2})\)\s*$/', $l, $m)) {
if (in_array("version", $expected) && preg_match('/^Version ([\d\?]+(?:\.[\d\?]+)*) \(([\d\?]{4}-[\d\?]{2}-[\d\?]{2})\)\s*$/D', $l, $m)) {
$version = $m[1];
if (!preg_match('/^\d{4}-\d{2}-\d{2}$/', $m[2])) {
if (!preg_match('/^\d{4}-\d{2}-\d{2}$/D', $m[2])) {
// uncertain dates are allowed only for the top version, and only if it does not match the target version (otherwise we have forgotten to set the correct date before tagging)
if (!$out && $targetVersion !== $version) {
// use today's date; local time is fine
@ -365,10 +514,10 @@ class RoboFile extends \Robo\Tasks {
$out .= "\n -- The Arsse team <no-contact@code.mensbeam.com> ".\DateTimeImmutable::createFromFormat("Y-m-d", $entry['date'], new \DateTimeZone("UTC"))->format("D, d M Y")." 00:00:00 +0000\n\n";
$out .= "\n -- $authorName <$authorMail> ".\DateTimeImmutable::createFromFormat("Y-m-d", $entry['date'], new \DateTimeZone("UTC"))->format("D, d M Y")." 00:00:00 +0000\n\n";
}
return $out;
}
/** Produces a Debian "source control" file from various bits of data
*
* As with a Debian changelog, the output is of a very exacting format,
* and this function should be modified with care.
*
* @param string $dir The path to Debian-specific files, with trailing slash
* @param string $version The Debian version string, in the format x.y.z-a
* @param array $tarballs An array of paths to the "orig" and "debian" tarball files
*/
protected function generateDebianSourceControl(string $dir, string $version, array $tarballs): string {
// read in control file
if (!$control = @file_get_contents($dir."control")) {
throw new \Exception("Unable to read Debian control file");
}
// read the format
if (!$format = @file_get_contents($dir."source/format")) {
throw new \Exception("Unable to read source format in Debian files");
}
// read the binary packages from the control file
if (preg_match_all('/^Package:\s*(\S+)/m', $control, $m)) {
$binary = [];
foreach ($m[1] as $pkg) {
$binary[] = $pkg;
}
} else {
throw new \Exception("No packages defined in Debian control file");
}
// read the package architectures from the control file
if (preg_match_all('/^Architecture:\s*(\S+)/m', $control, $m) || sizeof($m[1]) != sizeof($binary)) {
$architecture = [];
foreach ($m[1] as $pkg) {
$architecture[] = preg_replace('/\s/', "", $pkg);
}
} else {
throw new \Exception("Number of architectures defined in Debian control file does not match number of packages");
}
// read the package sections from the control file
if (preg_match_all('/^Section:\s*(\S+)/m', $control, $m) || sizeof($m[1]) != sizeof($binary)) {
$section = [];
foreach ($m[1] as $pkg) {
$section[] = $pkg;
}
} else {
throw new \Exception("Number of sections defined in Debian control file does not match number of packages");
}
// read the package priorities from the control file
if (preg_match_all('/^Priority:\s*(\S+)/m', $control, $m) || sizeof($m[1]) != sizeof($binary)) {
$priority = [];
foreach ($m[1] as $pkg) {
$priority[] = $pkg;
}
} else {
throw new \Exception("Number of priorities defined in Debian control file does not match number of packages");
}
// read simple metadata from the control file
$metadata = [];
foreach (["Source", "Maintainer", "Homepage", "Standards-Version", "Vcs-Browser", "Vcs-Git"] as $meta) {
if (preg_match('/^'.$meta.':\s*(.+)/m', $control, $m)) {
$metadata[$meta] = $m[1];
} else {
throw new \Exception("$meta is not defined in Debian control file");
}
}
// read build dependencies from control file
if (preg_match('/(?:^|\n)Build-Depends:\s*((?:[^\n]|\n(?= ))+)/s', $control, $m)) {
$buildDepends = preg_replace('/\s/', "", $m[1]);
} else {
$buildDepends = "";
}
// trim format
$format = trim($format);
// consolidate binaries and package list
$packageList = [];
for ($a = 0; $a <sizeof($binary);$a++){
$packageList[] = "$binary[$a] deb $section[$a] $priority[$a] arch=$architecture[$a]";
# Adapt the systemd service for Debian: this involves using only the "arsse-fetch" unit (renamed to "arsse"), removing the "PartOf" directive, and changing the user and group to "www-data"
Note that the above is the most concise process, not necessarily the recommended one. In particular [it is recommended](https://wiki.archlinux.org/title/PHP#Extensions) to use `/etc/php/conf.d/` to enable PHP extensions rather than editing `php.ini` as done above.
The PHP extensions listed in [the requirements](/en/Getting_Started/index) not mentioned above are compiled into Arch's PHP binaries and thus always enabled.
# Web server configuration
Sample configuration for both Nginx and Apache HTTP Server can be found in `/etc/webapps/arsse/nginx/` and `/etc/webapps/arsse/apache/`, respectively. The `example.conf` files are basic virtual host examples; the other files they include should normally be usable without modification, but may be modified if desired.
The above instructions assume you will be using the `php` package as your PHP interpreter. If you wish to use `php-legacy` (which is always one feature version behind, for compatibility) a few configuration tweaks are required. The follwoing commands are a short summary:
```sh
# Enable the necessary PHP extensions; curl is optional but recommended; pdo_sqlite may be used instead of sqlite3, but this is not recommended
sudo sed -i -e 's/^;\(extension=\(curl\|iconv\|intl\|sqlite3\)\)$/\1/' /etc/php-legacy/php.ini
# Modify the system service's environment
sudo sed -i -e 's/^ARSSE_PHP=.*/ARSSE_PHP=\/usr\/bin\/php-legacy/' /etc/webapps/arsse/systemd-environment
# Modify the PAM environment for the administrative CLI
echo "export ARSSE_PHP=/usr/bin/php-legacy" | sudo tee -a /etc/profile.d/arsse >/dev/null
# Modify the Nginx and Apache HTTPD configurations
sudo sed -i -se 's/\/run\/php-fpm\//\/run\/php-fpm-legacy\//' /etc/webapps/arsse/apache/arsse-fcgi.conf /etc/webapps/arsse/nginx/arsse-fcgi.conf
```
The above procedure can also be applied to use another PHP version from AUR if so desired.
# Next steps
If using a database other than SQLite, you will likely want to [set it up](/en/Getting_Started/Database_Setup) before doing anything else.
@ -47,7 +62,7 @@ You may also want to review the `config.defaults.php` file included in `/etc/web
# Upgrading
Upgrading The Arsse is done as like any other package. By default The Arsse will perform any required database schema upgrades when the new version is executed, so the service does need to be restarted:
Upgrading The Arsse is done like any other package. By default The Arsse will perform any required database schema upgrades when the new version is executed, so the service does need to be restarted:
Since version 0.10.0 pre-built Debian packages for The Arsse are available from the [OpenSUSE Build Service](https://build.opensuse.org/) (OBS) under [the author's personal project repository](https://build.opensuse.org/package/show/home:JKingWeb/arsse). This is the preferred method for instaling the software and is the means documented below.
Generic release tarballs may also be downloaded [from our Web site](https://thearsse.com), and a Debian package built manually. Installing directly from the generic release tarball without producing a Debian package is not recommended as the Debian packages make the set-up process on Debian systems significantly simpler.
# Adding the repository
In order to install The Arsse, the OBS repository must first be configured along with its signing key:
echo "deb [signed-by=/usr/share/keyrings/arsse-obs-keyring.gpg] https://download.opensuse.org/repositories/home:/JKingWeb/Debian_Unstable/ ./" | sudo tee "/etc/apt/sources.list.d/arsse-obs.list" >/dev/null
# Update APT's database
sudo apt update -qq
```
Please note that the "Unstable" qualifier in the repository URL is a reference to Debian's "sid" release and is not a reflection on The Arsse's stability. The repository should be suitable for any Debian version or derivative which includes a sufficiently recent version of PHP.
# Installation
Once the OBS repository is configured, installing The Arsse is achieved with a single command:
```sh
sudo apt install arsse
```
After installation is complete The Arsse will be started automatically.
During the installation process you will be prompted whether to allow `dbconfig-common` to configure The Arsse's database automatically. The default `sqlite3` (SQLite) option is a good choice, but `pgsql` (PostgreSQL) and `mysql` (MySQL) are possible alternatives. If you wish to [use a database other than SQLite](/en/Getting_Started/Database_Setup/index), you should install it before installing The Arsse:
```sh
# Install PostgreSQL
sudo apt install postgresql php-pgsql
# Install MySQL
sudo apt install mysql-server php-mysql
# Install SQLite explicitly
sudo apt install php-sqlite3
```
If you wish to change the database backend after having installed The Arsse, running `dpkg-reconfigure` after installing the database server can be used to achieve this:
```sh
sudo dpkg-reconfigure arsse
```
# Web server configuration
Sample configuration for both Nginx and Apache HTTP Server can be found in `/etc/arsse/nginx/` and `/etc/arsse/apache/`, respectively. The `example.conf` files are basic virtual host examples; the other files they include should normally be usable without modification, but may be modified if needed or desired. In particularly users of systems older than Debian 11 (Bullseye) or Ubuntu 20.04 (Focal Fossa) or derivatives will need to change the PHP-FPM socket path in Nginx or Apache's `arsse.conf`.
In order to use Apache HTTP Server the FastCGI proxy module must be enabled and the server restarted:
```sh
sudo a2enmod proxy proxy_fcgi
sudo systemctl restart apache2
```
No additional set-up is required for Nginx.
# Next steps
In order for The Arsse to serve users, those users [must be created](/en/Using_The_Arsse/Managing_Users).
You may also want to review the `config.defaults.php` file included in the download package and create [a configuration file](/en/Getting_Started/Configuration), though The Arsse can function even without using a configuration file.
# Upgrading
Upgrading The Arsse is done like any other package. Occasionally changes to Web server configuration have been required, such as when new protocols become supported; these changes are always explicit in the `UPGRADING` file.
The latest version of The Arsse can be downloaded [from our Web site](https://thearsse.com/). If installing an older release from our archives, the attachments named _arsse-x.x.x.tar.gz_ should be used rather than those marked "Source Code".
Installation from source code is also possible, but the release packages are recommended.
# Installation
Presently installing The Arsse on Debian systems is a manual process. The first step is to install its dependencies:
```sh
# Install PHP; this assumes the FastCGI process manager will be used
sudo apt install php-cli php-fpm
# Install the needed PHP extensions; php-curl is optional
Finally, services must be restarted to apply the new configurations, and The Arsse's service also started:
```sh
sudo systemctl restart systemd-sysusers
sudo systemd-tmpfiles --create
sudo systemctl restart php$php_ver-fpm
sudo systemctl reenable arsse
sudo systemctl restart arsse
```
# Web server configuration
Sample configuration for both Nginx and Apache HTTP Server can be found in `/etc/arsse/nginx/` and `/etc/arsse/apache/`, respectively. The `example.conf` files are basic virtual host examples; the other files they include should normally be usable without modification, but may be modified if desired.
In order to use Apache HTTP Server the FastCGI proxy module must be enabled and the server restarted:
```sh
sudo a2enmod proxy proxy_fcgi
sudo systemctl restart apache2
```
No additional set-up is required for Nginx.
# Next steps
If using a database other than SQLite, you will likely want to [set it up](/en/Getting_Started/Database_Setup) before doing anything else.
In order for The Arsse to serve users, those users [must be created](/en/Using_The_Arsse/Managing_Users).
You may also want to review the `config.defaults.php` file included in the download package and create [a configuration file](/en/Getting_Started/Configuration), though The Arsse can function even without using a configuration file.
# Upgrading
Upgrading The Arsse is simple:
1. Download the latest release
2. Check the `UPGRADING` file for any special notes
3. Stop the newsfeed refreshing service if it is running
4. Install the new version per the process above
6. Start the newsfeed refreshing service
By default The Arsse will perform any required database schema upgrades when the new version is executed. Occasionally changes to Web server configuration have been required, such as when new protocols become supported; these changes are always explicit in the `UPGRADING` file.
The Arsse should run on any operating system for which PHP and a Web server are available, but only the combination of Linux, Systemd, Nginx, and PHP-FPM has been extensively tested.
Below are very generic instructions and suggestions for installing The Arsse on systems for which pre-built packages are not available.
# Requirements
The Arsse has the following requirements:
- A Web server such as:
- [Nginx](https://nginx.org)
- [Apache HTTP server](https://httpd.apache.org) 2.4 or later
- PHP 7.3.0 or later with the following extensions:
- [intl](https://php.net/manual/en/book.intl.php), [json](https://php.net/manual/en/book.json.php), [hash](https://php.net/manual/en/book.hash.php), [filter](https://php.net/manual/en/book.filter.php), and [dom](https://php.net/manual/en/book.dom.php)
- [simplexml](https://php.net/manual/en/book.simplexml.php), and [iconv](https://php.net/manual/en/book.iconv.php)
- One of:
- [sqlite3](https://php.net/manual/en/book.sqlite3.php) or [pdo_sqlite](https://php.net/manual/en/ref.pdo-sqlite.php) for SQLite databases
- [pgsql](https://php.net/manual/en/book.pgsql.php) or [pdo_pgsql](https://php.net/manual/en/ref.pdo-pgsql.php) for PostgreSQL 10 or later databases
- [mysqli](https://php.net/manual/en/book.mysqli.php) or [pdo_mysql](https://php.net/manual/en/ref.pdo-mysql.php) for MySQL/Percona 8.0.11 or later databases
- [posix](https://php.net/manual/en/book.posix.php) and [pcntl](https://php.net/manual/en/book.pcntl.php) (both optional)
- An interface between PHP and the Web server, such as [PHP-FPM](https://php.net/manual/en/install.fpm.php)
- Privileges either to create and run system services, or to run cron jobs
# Installation
1. Download [the latest release](https://thearsse.com/releases/current) and extract it somewhere, such as `/usr/share/arsse/`
2. [Set up your database](/en/Getting_Started/Database_Setup)
3. Create [a configuration file](/en/Getting_Started/Configuration) if needed
4. Consult the files under `dist/nginx` and `dist/apache` for sample Web server configuration
5. Consult `dist/arsse` for a sample executable script which drops privileges on POSIX systems
6. Start the newsfeed fetching service:
- Sample Systemd service files are available under `dist/systemd`
- A sample System V init script is available in `dist/init.sh`
- A persistent process can be started by running `php arsse.php daemon`
- It is also possible [to use cron](/en/Using_The_Arsse/Other_Topics.html#page_Refreshing_newsfeeds_with_a_cron_job) or a similar task-scheduling tool
7. [Create users](/en/Using_The_Arsse/Managing_Users) to grant them access
# Upgrading
Upgrading The Arsse is usually simple:
1. Download the latest release
2. Check the `UPGRADING` file for any special notes
3. Stop the newsfeed refreshing service if it is running
4. Back up your configurationm and database
5. Extract the new version on top of the old one
6. Restart the newsfeed refreshing service
By default The Arsse will perform any required database schema upgrades when the new version is executed.
Occasionally changes to Web server configuration have been required, when new protocols become supported; such changes are always explicit in the `UPGRADING` file
We currently provide a few pre-built installation packages for the following operating systems:
- [Arch Linux and derivatives](On_Arch_Linux)
- [Debian and derivatives](On_Debian_and_Derivatives)
These packages significantly simplify installation, though a bit of manual effort may still be required. Updating The Arsse using these packages should require no manual intervention.
# Installing manually
For other systems The Arsse must currently be installed manually. As each operating system is different, we can only provide very general instructions:
- [Installing on other systems](On_Other_Systems)
We hope to support more operating systems in the future.
SQLite requires very little set-up. By default the database will be created at the root of The Arsse's program directory (e.g. `/usr/share/arsse/arsse.db`), but this can be changed with the [`dbSQLite3File` setting](/en/Getting_Started/Configuration#page_dbSQLite3File).
SQLite requires very little set-up. By default the database will be created at one of the following locations depending on installation method:
Regardless of the location chosen, The Arsse **must** be able to both read from and write to the database file, as well as create files in its directory. This is because SQLite also creates a write-ahead log file and a shared-memory file during operation.
| Manual installation | `arsse.db` in the The Arsse's installation directory |
This path can be changed with the [`dbSQLite3File` setting](/en/Getting_Started/Configuration#page_dbSQLite3File).
Regardless of the location used, The Arsse **must** be able to both read from and write to the database file, as well as create files in its directory. This is because SQLite also creates a write-ahead log file and a shared-memory file during operation.
The Arsse looks for configuration in a file named `config.php` in the directory where it is installed. For example, if The Arsse is installed at `/usr/share/arsse`, it will look for configuration in the file `/usr/share/arsse/config.php`. It is not an error for this file not to exist or to be empty: The Arsse will function with no configuration whatsoever, provided other conditions allow.
Depending on how The Arsse was installed, it will look for configuration in the following places:
| Arch Linux package | `/etc/webapps/arsse/config.php` |
| Debian package | `/etc/arsse/config.php` |
| Manual installation | `config.php` in the The Arsse's installation directory |
It is not an error for this file not to exist or to be empty: The Arsse will function with no configuration whatsoever, provided other conditions allow.
The configuration file is a PHP script which returns an associative array with keys and values for one or more settings. Any settings which are not specified in the configuration file will be set to their defaults. Invalid values will cause an error on start-up, while unknown keys are ignored. A basic configuration file might look like this:
@ -12,10 +20,10 @@ The configuration file is a PHP script which returns an associative array with k
];
```
The `config.defaults.php` file included with copies of The Arsse contains an annotated listing of every configuration setting with its default value. The settings are also documented in more detail below.
# List of all settings
The `config.defaults.php` file included with copies of The Arsse contains an annotated listing of every configuration setting with its default value. The settings are also documented in more detail below.
## General settings
### lang
@ -355,7 +363,7 @@ The default value is equal to two megabytes.
|---------|---------|
| boolean | `true` |
Whether to allow the possibility of fetching full article contents from an article's source, if a newsfeed only provides excerpts. Whether fetching will actually happen is governed by a per-newsfeed toggle (defaulting to `false`) which currently can only be changed by manually editing the database.
Whether to allow the possibility of fetching full article contents from an article's source, if a newsfeed only provides excerpts. Whether fetching will actually happen is governed by a per-newsfeed toggle (defaulting to `false`) which currently can only be changed via the Miniflux protocol.
Presently installing and setting up The Arsse involves some manual labour. We have packages for Arch Linux and hope to have installation packages available for other operating systems eventually, but for now the pages in this section should help get you up and running on Arch Linux or Debian-based systems, with Nginx or Apache HTTP Server.
It is also be possible to run The Arsse on other operating systems (including Windows) and with other Web servers, but the configuration required to do so is not documented in this manual.
# Requirements
For reference, The Arsse has the following requirements:
- A Linux server running Nginx or Apache 2.4
- PHP 7.1.0 or later with the following extensions:
- [intl](http://php.net/manual/en/book.intl.php), [json](http://php.net/manual/en/book.json.php), [hash](http://php.net/manual/en/book.hash.php), and [dom](http://php.net/manual/en/book.dom.php)
- [simplexml](http://php.net/manual/en/book.simplexml.php), and [iconv](http://php.net/manual/en/book.iconv.php)
- One of:
- [sqlite3](http://php.net/manual/en/book.sqlite3.php) or [pdo_sqlite](http://php.net/manual/en/ref.pdo-sqlite.php) for SQLite databases
- [pgsql](http://php.net/manual/en/book.pgsql.php) or [pdo_pgsql](http://php.net/manual/en/ref.pdo-pgsql.php) for PostgreSQL 10 or later databases
- [mysqli](http://php.net/manual/en/book.mysqli.php) or [pdo_mysql](http://php.net/manual/en/ref.pdo-mysql.php) for MySQL/Percona 8.0.11 or later databases
This section describes in brief some CLI commands. Please read [the general notes on the command line interface](index) before continuing.
# Refreshing newsfeeds with a cron job
Normally The Arsse has a systemd service which checks newsfeeds for updates and processes them into its database for the user. If for whatever reason this is not practical a [cron](https://en.wikipedia.org/wiki/Cron) job may be used instead.
This section details a few administrative tasks which may need to be performed after installing The Arsse. As no Web-based administrative interface is included, these tasks are generally performed via command line interface.
Though this section describes some commands briefly, complete documentation of The Arsse's command line interface is not included in this manual. Documentation for CLI commands can instead be viewed with the CLI itself by executing `arsse --help`.
Though this section describes some commands briefly, complete documentation of The Arsse's command line interface is not included in this manual. Documentation for CLI commands can instead be viewed using the system manual service by executing `man arsse`.
The Arsse supports not only the Tiny Tiny RSS protocol, but also extensions required by the FeedReader client and the more commonly supported `getCompactHeadlines` extension.
@ -37,7 +37,7 @@ The Arsse does not currently support the entire protocol. Notably missing featur
- Processing of the `search` parameter of the `getHeadlines` operation differs in the following ways:
- Values other than `"true"` or `"false"` for the `unread`, `star`, and `pub` special keywords treat the entire token as a search term rather than as `"false"`
- Invalid dates are ignored rather than assumed to be `"1970-01-01"`
- Only a single negative date is allowed (this is a known bug rather than intentional)
- Specifying multiple non-negative dates usually returns no results as articles must match all specified dates simultaneously; The Arsse instead returns articles matching any of the specified dates
- Dates are always relative to UTC
- Full-text search is not yet employed with any database, including PostgreSQL
- Article hashes are normally SHA1; The Arsse uses SHA256 hashes
/** Class for loading, saving, and querying configuration
*
* The Conf class serves both as a means of importing and querying configuration information, as well as a source for default parameters when a configuration file does not specify a value.
* All public properties are configuration parameters that may be set by the server administrator. */
#[AllowDynamicProperties]
class Conf {
/** @var string Default language to use for logging and errors */
select id from arsse_folders where owner = ? and coalesce(parent,0) = ? union all select arsse_folders.id from arsse_folders join folders on arsse_folders.parent=folders.id
)
select
id,
name,
arsse_folders.parent as parent,
coalesce(children,0) as children,
coalesce(feeds,0) as feeds
FROM arsse_folders
left join (SELECT parent,count(id) as children from arsse_folders group by parent) as child_stats on child_stats.parent = arsse_folders.id
left join (SELECT folder,count(id) as feeds from arsse_subscriptions group by folder) as sub_stats on sub_stats.folder = arsse_folders.id"
from arsse_folders
left join (select parent,count(id) as children from arsse_folders group by parent) as child_stats on child_stats.parent = arsse_folders.id
left join (select folder,count(id) as feeds from arsse_subscriptions group by folder) as sub_stats on sub_stats.folder = arsse_folders.id",
$q->setCTE("folders", "SELECT id from arsse_folders where owner = ? and coalesce(parent,0) = ? union all select arsse_folders.id from arsse_folders join folders on arsse_folders.parent=folders.id", ["str", "strict int"], [$user, $parent]);
$q->setCTE("topmost(f_id,top)", "SELECT id,id from arsse_folders where owner = ? and parent is null union all select id,top from arsse_folders join topmost on parent=f_id", ["str"], [$user]);
if ($id) {
// if an ID is specified, add a suitable WHERE condition and bindings
// this condition facilitates the implementation of subscriptionPropertiesGet, which would otherwise have to duplicate the complex query; it takes precedence over a specified folder
$q->setWhere("s.id = ?", "int", $id);
} elseif ($folder && $recursive) {
// if a folder is specified and we're listing recursively, add a common table expression to list it and its children so that we select from the entire subtree
$q->setCTE("folders(folder)", "SELECT ? union all select id from arsse_folders join folders on parent = folder", "int", $folder);
// add a suitable WHERE condition
// if a folder is specified and we're listing recursively, add a suitable WHERE condition
$q->setWhere("folder in (select folder from folders)");
} elseif (!$recursive) {
// if we're not listing recursively, match against only the specified folder (even if it is null)
// prepare the output column list; the column definitions are also used for ordering
$colDefs = $this->articleColumns();
if (!$cols) {
// if no columns are specified return a count; don't borther with sorting
@ -1534,7 +1549,23 @@ class Database {
assert(strlen($outColumns) > 0, new \Exception("No input columns matched whitelist"));
// define the basic query, to which we add lots of stuff where necessary
$q = new Query(
"SELECT
"WITH RECURSIVE
folders(id,req) as (
select 0, 0 union all select f.id, f.id from arsse_folders as f where owner = ? union all select f1.id, req from arsse_folders as f1 join folders on coalesce(parent,0)=folders.id
),
folders_top(id,top) as (
select f.id, f.id from arsse_folders as f where owner = ? and parent is null union all select f.id, top from arsse_folders as f join folders_top as t on parent=t.id
),
folder_data(id,name,top,top_name) as (
select f1.id, f1.name, top, f2.name from arsse_folders as f1 join folders_top as f0 on f1.id = f0.id join arsse_folders as f2 on f2.id = top
),
labelled(article,label_id,label_name) as (
select m.article, l.id, l.name from arsse_label_members as m join arsse_labels as l on l.id = m.label where l.owner = ? and m.assigned = 1
),
tagged(subscription,tag_id,tag_name) as (
select m.subscription, t.id, t.name from arsse_tag_members as m join arsse_tags as t on t.id = m.tag where t.owner = ? and m.assigned = 1
)
select
$outColumns
from arsse_articles
join arsse_subscriptions on arsse_subscriptions.feed = arsse_articles.feed and arsse_subscriptions.owner = ?
@ -1543,68 +1574,133 @@ class Database {
left join arsse_marks on arsse_marks.subscription = arsse_subscriptions.id and arsse_marks.article = arsse_articles.id
left join arsse_enclosures on arsse_enclosures.article = arsse_articles.id
join (
SELECT article, max(id) as edition from arsse_editions group by article
select article, max(id) as edition from arsse_editions group by article
) as latest_editions on arsse_articles.id = latest_editions.article
left join (
SELECT arsse_label_members.article, max(arsse_label_members.modified) as modified, sum(arsse_label_members.assigned) as assigned from arsse_label_members join arsse_labels on arsse_labels.id = arsse_label_members.label where arsse_labels.owner = ? group by arsse_label_members.article
select arsse_label_members.article, max(arsse_label_members.modified) as modified, sum(arsse_label_members.assigned) as assigned from arsse_label_members join arsse_labels on arsse_labels.id = arsse_label_members.label where arsse_labels.owner = ? group by arsse_label_members.article
) as label_stats on label_stats.article = arsse_articles.id",
["str", "str"],
[$user, $user]
["str", "str", "str", "str", "str", "str"],
[$user, $user, $user, $user, $user, $user]
);
$q->setCTE("topmost(f_id,top)", "SELECT id,id from arsse_folders where owner = ? and parent is null union all select id,top from arsse_folders join topmost on parent=f_id", ["str"], [$user]);
$q->setCTE("folder_data(id,name,top,top_name)", "SELECT f1.id, f1.name, top, f2.name from arsse_folders as f1 join topmost on f1.id = f_id join arsse_folders as f2 on f2.id = top");
$q->setLimit($context->limit, $context->offset);
if ($context instanceof UnionContext) {
// if the context is a union context, we compute each context in turn
$q->setWhereRestrictive(false);
foreach ($context as $c) {
$q->setWhereGroup($this->articleFilter($c));
}
} else {
// if the context is not a union, first validate input to catch 404s and the like
// next compute the context, supplying the query to manipulate directly
$this->articleFilter($context, $q);
}
// return the query
return $q;
}
protected function articleFilter(Context $context, QueryFilter $q = null) {
$q = $q ?? new QueryFilter;
$colDefs = $this->articleColumns();
// handle the simple context options
$options = [
// each context array consists of a column identifier (see $colDefs above), a comparison operator, a data type, and an option to pair with for BETWEEN evaluation
// each context array consists of a column identifier (see $colDefs above), a comparison operator, and a data type; the "between" operator has special handling
'cte_body' => "SELECT m.article, l.id, l.name from arsse_label_members as m join arsse_labels as l on l.id = m.label where l.owner = ? and m.assigned = 1",
'cte_body' => "SELECT m.subscription, t.id, t.name from arsse_tag_members as m join arsse_tags as t on t.id = m.tag where t.owner = ? and m.assigned = 1",
// each context array consists of a common table expression to select from, the column to match in the main join, the column to match in the CTE, the column to select in the CTE, an operator, and a type for the match in the CTE
// add a common table expression to list the folders and their children so that we select from the entire subtree
$q->setCTE("folders_multi(folder)", "SELECT id as folder from (select id from (select 0 as id union all select id from arsse_folders where owner = ?) as f where id in ($inClause)) as folders_multi union select id from arsse_folders join folders_multi on coalesce(parent,0) = folder", ["str", $inTypes], [$user, $inValues]);
// limit subscriptions to the listed folders
$q->setWhere("coalesce(arsse_subscriptions.folder,0) in (select folder from folders_multi)");
}
if ($context->not->folder()) {
// add a common table expression to list the folder and its children so that we exclude from the entire subtree
$q->setCTE("folders_excluded(folder)", "SELECT ? union all select id from arsse_folders join folders_excluded on coalesce(parent,0) = folder", "int", $context->not->folder);
// excluded any subscriptions in the listed folders
$q->setWhereNot("coalesce(arsse_subscriptions.folder,0) in (select folder from folders_excluded)");
// add a common table expression to list the folders and their children so that we select from the entire subtree
$q->setCTE("folders_multi_excluded(folder)", "SELECT id as folder from (select id from (select 0 as id union all select id from arsse_folders where owner = ?) as f where id in ($inClause)) as folders_multi_excluded union select id from arsse_folders join folders_multi_excluded on coalesce(parent,0) = folder", ["str", $inTypes], [$user, $inValues]);
// limit subscriptions to the listed folders
$q->setWhereNot("coalesce(arsse_subscriptions.folder,0) in (select folder from folders_multi_excluded)");
}
// handle text-matching context options
$options = [
@ -1741,29 +1758,56 @@ class Database {
"annotationTerms" => ["note"],
];
foreach ($options as $m => $columns) {
if (!$context->$m()) {
continue;
} elseif (!$context->$m) {
throw new Db\ExceptionInput("tooShort", ['field' => $m, 'action' => $this->caller(), 'min' => 1]); // must have at least one array element
}
$columns = array_map(function($c) use ($colDefs) {
assert(isset($colDefs[$c]), new Exception("constantUnknown", $c));
* @param string $user The user who owns the articles to be modified
* @param array $data An associative array of properties to modify. Anything not specified will remain unchanged
* @param Context $context The query context to match articles against
* @param RootContext $context The query context to match articles against
* @param bool $updateTimestamp Whether to also update the timestamp. This should only be false if a mark is changed as a result of an automated action not taken by the user
*/
public function articleMark(string $user, array $data, Context $context = null, bool $updateTimestamp = true): int {
public function articleMark(string $user, array $data, RootContext $context = null, bool $updateTimestamp = true): int {
$data = [
'read' => $data['read'] ?? null,
'starred' => $data['starred'] ?? null,
@ -1865,10 +1909,23 @@ class Database {
// marking as read is ignored if the edition is not the latest, but the same is not true of the other two marks
$this->db->query("UPDATE arsse_marks set touched = 0 where touched <> 0");
$q->setBody("UPDATE arsse_marks set \"read\" = ?, touched = 1 where article in(select article from target_articles) and subscription in(select distinct subscription from target_articles)", "bool", $data['read']);
$q->setBody("UPDATE arsse_marks set touched = 1, $set where article in(select article from target_articles) and subscription in(select distinct subscription from target_articles)", $setTypes, $setValues);
$q->setBody("UPDATE arsse_marks set $set where article in(select article from target_articles) and subscription in(select distinct subscription from target_articles)", $setTypes, $setValues);