|
|
|
@ -1,208 +1,119 @@ |
|
|
|
# The Advanced RSS Environment |
|
|
|
|
|
|
|
The Arsse is a news aggregator server which implements multiple synchronization protocols, including [version 1.2][NCNv1] of [NextCloud News][NCN]' protocol and the [Tiny Tiny RSS][TTRSS] protocol (details below). Unlike most other aggregator servers, The Arsse does not include a Web front-end (though one is planned as a separate project), and it relies on existing protocols to maximize compatibility with existing clients. |
|
|
|
The Arsse is a news aggregator server, written in PHP, which implements multiple synchronization protocols. Unlike most other aggregator servers, The Arsse does not include a Web front-end (though one is planned as a separate project), and it relies on existing protocols to maximize compatibility with existing clients. |
|
|
|
|
|
|
|
At present the software should be considered in an "alpha" state: though its core subsystems are covered by unit tests and should be free of major bugs, not everything has been rigorously tested. Additionally, many features one would expect from other similar software have yet to be implemented. Areas of future work include: |
|
|
|
Information on how to install and use the software can be found in [the manual](https://thearsse.com/manual/), which is available online as well as with every copy of the software. This readme file instead focuses on how to set up a programming environment to modify the source code. |
|
|
|
|
|
|
|
- Providing more sync protocols (Google Reader, Fever, others) |
|
|
|
- Better packaging and configuration samples |
|
|
|
- A user manual |
|
|
|
# Installing from source |
|
|
|
|
|
|
|
## Requirements |
|
|
|
The main repository for The Arsse can be found at [code.mensbeam.com](https://code.mensbeam.com/MensBeam/arsse/), with a mirror also available [at GitHub](https://github.com/mensbeam/arsse/). The GitHub mirror does not accept bug reports, but the two should otherwise be equivalent. |
|
|
|
|
|
|
|
The Arsse has the following requirements: |
|
|
|
[Composer](https://getcomposer.org/) is required to manage PHP dependencies. After cloning the repository or downloading a source code tarball, running `composer install` will download all the required dependencies, and will advise if any PHP extensions need to be installed. If not installing as a programming environment, running `composer install --no-dev` is recommended. |
|
|
|
|
|
|
|
- A Linux server utilizing systemd and Nginx (tested on Ubuntu 16.04) |
|
|
|
- PHP 7.0.7 or later with the following extensions: |
|
|
|
- [intl](http://php.net/manual/en/book.intl.php), [json](http://php.net/manual/en/book.json.php), [hash](http://php.net/manual/en/book.hash.php), and [pcre](http://php.net/manual/en/book.pcre.php) |
|
|
|
- [dom](http://php.net/manual/en/book.dom.php), [simplexml](http://php.net/manual/en/book.simplexml.php), and [iconv](http://php.net/manual/en/book.iconv.php) (for picoFeed) |
|
|
|
- One of: |
|
|
|
- [sqlite3](http://php.net/manual/en/book.sqlite3.php) or [pdo_sqlite](http://php.net/manual/en/ref.pdo-sqlite.php) for SQLite databases |
|
|
|
- [pgsql](http://php.net/manual/en/book.pgsql.php) or [pdo_pgsql](http://php.net/manual/en/ref.pdo-pgsql.php) for PostgreSQL 10 or later databases |
|
|
|
- [mysqli](http://php.net/manual/en/book.mysqli.php) or [pdo_mysql](http://php.net/manual/en/ref.pdo-mysql.php) for MySQL 8.0.7 or later databases |
|
|
|
- Privileges to create and run daemon processes on the server |
|
|
|
# Repository structure |
|
|
|
|
|
|
|
## Installation |
|
|
|
## Library code |
|
|
|
|
|
|
|
At present, installation of The Arsse is rather manual. We hope to improve this in the future, but for now the steps below should help get you started. The instructions and configuration samples assume you will be using Ubuntu 16.04 (or equivalent Debian) and Nginx; we hope to expand official support for different configurations in the future as well. |
|
|
|
The code which runs The Arsse, contained in `/arsse.php`, is only a short stub: the application itself is composed of the classes found under `/lib/`, with the main ones being: |
|
|
|
|
|
|
|
### Initial setup |
|
|
|
| Path | Description | |
|
|
|
|----------------|---------------------------------------------------------| |
|
|
|
| `CLI.php` | The command-line interface, including its documentation | |
|
|
|
| `Conf.php` | Configuration handling | |
|
|
|
| `Database.php` | High-level database interface | |
|
|
|
| `Db/` | Low-level database abstraction layer | |
|
|
|
| `REST/` | Protocol implementations | |
|
|
|
| `REST.php` | General protocol handler for CORS, HTTP auth, etc. | |
|
|
|
| `Arsse.php` | Singleton glueing the parts together | |
|
|
|
|
|
|
|
1. Extract the tar archive to `/usr/share` |
|
|
|
2. If desired, create `/usr/share/arsse/config.php` using `config.defaults.php` as a guide. The file you create only needs to contain non-default settings. The `userPreAuth` setting may be of particular interest |
|
|
|
3. Copy `/usr/share/arsse/dist/arsse.service` to `/lib/systemd/system` |
|
|
|
4. In a terminal, execute the following to start the feed fetching service: |
|
|
|
The `/lib/Database.php` file is the heart of the application, performing queries on behalf of protocol implementations or the command-line interface. |
|
|
|
|
|
|
|
``` sh |
|
|
|
sudo systemctl enable arsse |
|
|
|
sudo systemctl start arsse |
|
|
|
``` |
|
|
|
Also necessary to the functioning of the application is the `/vendor/` directory, which contains PHP libraries which The Arsse depends upon. These are managed by Composer. |
|
|
|
|
|
|
|
### Configuring the Web server and PHP |
|
|
|
## Supporting data files |
|
|
|
|
|
|
|
Sample configuration parameters for Nginx can be found in `arsse/dist/nginx.conf` and `arsse/dist/nginx-fcgi.conf`; the samples assume [a server group](http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream) has already been defined for PHP. How to configure an Nginx service to use PHP and install the required PHP extensions is beyond the scope of this document, however. |
|
|
|
The `/locale/` and `/sql/` directories contain human-language files and database schemata, both of which are occasionally used by the application in the course of execution. The `/www/` directory serves as a document root for a few static files to be made available to users by a Web server. |
|
|
|
|
|
|
|
### Adding users |
|
|
|
The `/dist/` directory, on the other hand, contains samples of configuration for Web servers and init systems. These are not used by The Arsse itself, but are merely distributed with it for reference. |
|
|
|
|
|
|
|
The Arsse includes a `user add <username> [<password>]` console command to add users to the database; for example running `php arsse.php user add admin password` will add the user `admin` with the password `pasword` to the database. Other commands for managing users are also available. |
|
|
|
## Documentation |
|
|
|
|
|
|
|
Alternatively, if the Web server is configured to handle authentication, you may set the configuration option `userPreAuth` to `true` and The Arsse will defer to the Web server and automatically add any missing users as it encounters them. |
|
|
|
The source text for The Arsse's manual can be found in `/docs/`, with pages written in [Markdown](https://spec.commonmark.org/current/) and converted to HTML [with Daux](#building-the-manual). If a static manual is generated its files will appear under `/manual/`. |
|
|
|
|
|
|
|
## Installation from source |
|
|
|
In addition to the manual the files `/README.md` (this file), `/CHANGELOG`, `/UPGRADING`, `/LICENSE`, and `/AUTHORS` also document various things about the software, rather than the software itself. |
|
|
|
|
|
|
|
If installing from the Git repository rather than a download package, you will need to follow extra steps before the instructions in the section above. |
|
|
|
## Tests |
|
|
|
|
|
|
|
First, you must install [Composer] to fetch required PHP libraries. Once Composer is installed, dependencies may be downloaded with the following command: |
|
|
|
The `/tests/` directory contains everything related to automated testing. It is itself organized as follows: |
|
|
|
|
|
|
|
``` sh |
|
|
|
php composer.phar install -o --no-dev --no-scripts |
|
|
|
``` |
|
|
|
| Path | Description | |
|
|
|
|--------------------|------------------------------------------------------------------------------------| |
|
|
|
| `cases/` | The test cases themselves, organized in roughly the same structure as the code | |
|
|
|
| `coverage/` | (optional) Generated code coverage reports | |
|
|
|
| `docroot/` | Sample documents used in some tests, to be returned by the PHP's basic HTTP server | |
|
|
|
| `lib/` | Supporting classes which do not contain test cases | |
|
|
|
| `bootstrap.php` | Bootstrap script, equivalent to `/arsse.php`, but for tests | |
|
|
|
| `phpunit.dist.xml` | PHPUnit configuration file | |
|
|
|
| `server.php` | Simple driver for the PHP HTTP server used during testing | |
|
|
|
|
|
|
|
Second, you may wish to create an example configuration file using the following command: |
|
|
|
PHPUnit's configuration can be customized by copying its configuration file to `/tests/phpunit.xml` and modifying the copy accordingly. |
|
|
|
|
|
|
|
``` sh |
|
|
|
php ./arsse.php conf save-defaults "./config.defaults.php" |
|
|
|
``` |
|
|
|
## Tooling |
|
|
|
|
|
|
|
## License |
|
|
|
The `/vendor-bin/` directory houses the files needed for the tools used in The Arsse's programming environment. These are managed by the Composer ["bin" plugin](https://github.com/bamarni/composer-bin-plugin) and are not used by The Arsse itself. The following files are also related to various programming tools: |
|
|
|
|
|
|
|
The Arsse is made available under the permissive MIT license. See the `LICENSE` and `AUTHORS` files included with the distribution or source code for exact legal text and copyright holders. Dependencies included in the distribution may be governed by other licenses. |
|
|
|
| Path | Description | |
|
|
|
|-------------------|----------------------------------------------------------| |
|
|
|
| `/.gitattributes` | Git settings for handling files | |
|
|
|
| `/.gitignore` | Git file exclusion patterns | |
|
|
|
| `/.php_cs.dist` | Configuration for [php-cs-fixer](https://cs.symfony.com) | |
|
|
|
| `/.php_cs.cache` | Cache for php-cs-fixer | |
|
|
|
| `/composer.json` | Configuration for Composer | |
|
|
|
| `/composer.lock` | Version synchronization data for Composer | |
|
|
|
| `/RoboFile.php` | Task definitions for [Robo](https://robo.li/) | |
|
|
|
| `/robo` | Simple wrapper for executing Robo on POSIX systems | |
|
|
|
| `/robo.bat` | Simple wrapper for executing Robo on Windows | |
|
|
|
|
|
|
|
## Contributing |
|
|
|
In addition the files `/package.json`, `/yarn.lock`, and `/postcss.config.js` as well as the `/node_modules/` directory are used by [Yarn](https://yarnpkg.com/) and [PostCSS](https://postcss.org/) when modifying the stylesheet for The Arsse's manual. |
|
|
|
|
|
|
|
Please refer to `CONTRIBUTING.md` for guidelines on contributing code to The Arsse. |
|
|
|
# Common tasks |
|
|
|
|
|
|
|
## Database compatibility notes |
|
|
|
We use a tool called [Robo](https://robo.li/) to simplify the execution of common tasks. It is installed with The Arsse's other dependencies, and its configured tasks can be listed by executing `./robo` without arguments. |
|
|
|
|
|
|
|
Functionally there is no reason to prefer either SQLite or PostgreSQL over the other. SQLite is significantly simpler to set up in most cases, requiring only read and write access to a containing directory in order to function; PostgreSQL may perform better than SQLite when serving hundreds of users or more, though this has not been tested. |
|
|
|
## Running tests |
|
|
|
|
|
|
|
MySQL, on the other hand, is **not recommended** due to its relatively constrained index prefix limits which may cause some newsfeeds which would otherwise work to be rejected. If using MySQL, special care should also be taken when performing schema upgrades, as errors during the process can leave the database in a half-upgraded state which The Arsse cannot itself recover from. |
|
|
|
The Arsse has an extensive [PHPUnit](https://phpunit.de/) test suite; tests can be run by executing `./robo test`, which can be supplemented with any arguments understoof by PHPUnit. For example, to test only the Tiny Tiny RSS protocol, one could run `./robo test --testsuite TTRSS`. |
|
|
|
|
|
|
|
Note that MariaDB is not compatible with The Arsse: its support for common table expressions is, as of this writing, not sufficient for our needs. |
|
|
|
There is also a `test:quick` Robo task which excludes slower tests, and a `test:full` task which includes redundant tests in addition to the standard test suite |
|
|
|
|
|
|
|
## Protocol compatibility notes |
|
|
|
### Test coverage |
|
|
|
|
|
|
|
### General |
|
|
|
Computing the coverage of tests can be done by running `./robo coverage`, after which an HTML-format coverage report will be written to `/tests/coverage/`. Either [PCOV](https://github.com/krakjoe/pcov), [Xdebug](https://xdebug.org), or [phpdbg](https://php.net/manual/en/book.phpdbg.php) is required for this. PCOV is generally recommended as it is faster than Xdebug; phpdbg is faster still, but less accurate. If using either PCOV or Xdebug, the extension need not be enabled globally; PHPUnit will enable it when needed. |
|
|
|
|
|
|
|
#### Type casting |
|
|
|
## Enforcing coding style |
|
|
|
|
|
|
|
The Arsse does not guarantee it will handle type casting of input in the same way as reference implementations for its supported protocols. As a general rule, clients should endeavour to send only correct input. |
|
|
|
The [php-cs-fixer](https://cs.symfony.com) tool, executed via `./robo clean`, can be used to rewrite code to adhere to The Arsse's coding style. The style largely follows [PSR-2](https://www.php-fig.org/psr/psr-2/) with some exceptions: |
|
|
|
|
|
|
|
The Arsse does, however, guarantee _output_ to be of the same type. If it is not, this is [a bug][newIssue] and should be reported. |
|
|
|
- Classes, methods, and functions should have their opening brace on the same line as the signature |
|
|
|
- Anonymous functions should have no space before the parameter list |
|
|
|
|
|
|
|
#### Content sanitization |
|
|
|
## Building the manual |
|
|
|
|
|
|
|
The Arsse makes use of the [picoFeed] newsfeed parsing library to sanitize article content. The exact sanitization parameters may differ from those of reference implementations for protocols The Arsse supports. |
|
|
|
The Arsse's user manual, made using [Daux](https://daux.io/), can be compiled by running `./robo manual`, which will output files to `/manual/`. It is also possible to serve the manual from a test HTTP server on port 8085 by running `./robo manual:live`. |
|
|
|
|
|
|
|
### NextCloud News v1.2 |
|
|
|
### Rebuilding the manual theme |
|
|
|
|
|
|
|
As a general rule, The Arsse should yield the same output as the reference implementation for all valid inputs (otherwise you've found [a bug][newIssue]), but there are exception, either because the NextCloud News (hereafter "NCN") [protocol description][NCNv1] is at times ambiguous or incomplete, or because implementation details necessitate it differ; this section along with the General section above detail these differences. |
|
|
|
The manual employs a custom theme derived from the standard Daux theme. If the standard Daux theme receives improvements, the custom theme can be rebuilt by running `./robo manual:theme`. This requires that [NodeJS](https://nodejs.org) and [Yarn](https://yarnpkg.com/) be installed, but JavaScript tools are not required to modify The Arsse itself, nor the content of the manual. |
|
|
|
|
|
|
|
#### Differences |
|
|
|
## Packaging a release |
|
|
|
|
|
|
|
- Article GUID hashes are not hashes like in NCN; they are integers rendered as strings |
|
|
|
- Article fingerprints are a combination of hashes rather than a single hash |
|
|
|
- When marking articles as starred the feed ID is ignored, as they are not needed to establish uniqueness |
|
|
|
- The feed updater ignores the `userId` parameter: feeds in The Arsse are deduplicated, and have no owner |
|
|
|
- The `/feeds/all` route lists only feeds which should be checked for updates, and it also returns all `userId` attributes as empty strings: feeds in The Arsse are deduplicated, and have no owner |
|
|
|
- The API's "updater" routes do not require administrator priviledges as The Arsse has no concept of user classes |
|
|
|
- The "updater" console commands mentioned in the protocol specification are not implemented, as The Arsse does not implement the required NextCloud subsystems |
|
|
|
- The `lastLoginTimestamp` attribute of the user metadata is always the current time: The Arsse's implementation of the protocol is fully stateless |
|
|
|
- Syntactically invalid JSON input will yield a `400 Bad Request` response instead of falling back to GET parameters |
|
|
|
- Folder names consisting only of whitespace are rejected along with the empty string |
|
|
|
- Feed titles consisting only of whitespace or the empty string are rejected with a `422 Unprocessable Entity` reponse instead of being accepted |
|
|
|
- Bulk-marking operations without a `newestItemId` argument result in a `422 Unprocessable Entity` reponse instead of silently failing |
|
|
|
- Creating a feed in a folder which does not exist places the feed in the root folder rather than suppressing the feed |
|
|
|
- Moving a feed to a folder which does not exist results in a `422 Unprocessable Entity` reponse rather than suppressing the feed |
|
|
|
Producing a release package is done by running `./robo package`. This performs the following operations: |
|
|
|
|
|
|
|
### Tiny Tiny RSS |
|
|
|
- Duplicates a working tree with the commit (usually a release tag) to package |
|
|
|
- Generates the manual |
|
|
|
- Installs runtime Composer dependencies with an optimized autoloader |
|
|
|
- Deletes numerous unneeded files |
|
|
|
- Exports the default configuration of The Arsse to a file |
|
|
|
- Compresses the remaining files into a tarball |
|
|
|
|
|
|
|
As a general rule, The Arsse should yield the same output as the reference implementation for all valid inputs (otherwise you've found [a bug][newIssue]), but there are exception, either because the Tiny Tiny RSS (hereafter "TTRSS") [protocol description][TTRSS] is incomplete, erroneous, or out of date, or because TTRSS itself is buggy, or because implementation details necessitate The Arsse differ; this section along with the General section above detail these differences. |
|
|
|
|
|
|
|
#### Extended functionality |
|
|
|
|
|
|
|
The Arsse supports both [the set of extensions][ext-feedreader] to the TTRSS protocol defined by [FeedReader], as well as [the `getCompactHeadlines` operation][ext-newsplus] defined by [News+]. |
|
|
|
|
|
|
|
We are not aware of any other extensions to the TTRSS protocol. If you know of any more, please [let us know][newIssue]. |
|
|
|
|
|
|
|
#### Missing features |
|
|
|
|
|
|
|
- The `getPref` operation is not implemented; it returns `UNKNOWN_METHOD` |
|
|
|
- The `shareToPublished` operation is not implemented; it returns `UNKNOWN_METHOD` |
|
|
|
- Setting an article's "published" flag with the `updateArticle` operation is not implemented and will gracefully fail |
|
|
|
- The `sanitize`, `force_update`, and `has_sandbox` parameters of the `getHeadlines` operation are ignored |
|
|
|
- String `feed_id` values for the `getCompactHeadlines` operation are not supported and will yield an `INCORRECT_USAGE` error |
|
|
|
- Articles are limited to a single attachment rather than multiple attachments |
|
|
|
|
|
|
|
#### Differences |
|
|
|
|
|
|
|
- Input that cannot be parsed as JSON normally returns a `NOT_LOGGED_IN` error; The Arsse returns a non-standard `MALFORMED_INPUT` error instead |
|
|
|
- Feed, category, and label names are normally unrestricted; The Arsse rejects empty strings, as well as strings composed solely of whitespace |
|
|
|
- Discovering multiple feeds during `subscribeToFeed` processing normally produces an error; The Arsse instead chooses the first feed it finds |
|
|
|
- Providing the `setArticleLabel` operation with an invalid label normally silently fails; The Arsse returns an `INVALID_USAGE` error instead |
|
|
|
- Processing of the `search` parameter of the `getHeadlines` operation differs in the following ways: |
|
|
|
- Values other than `"true"` or `"false"` for the `unread`, `star`, and `pub` special keywords treat the entire token as a search term rather than as `"false"` |
|
|
|
- Invalid dates are ignored rather than assumed to be `"1970-01-01"` |
