Commit graph

323 commits

Author SHA1 Message Date
de92fb514b Implement TTRSS opera getArticle; fixes #84 2017-11-15 15:38:49 -05:00
ea08bbb87b Implement TTRSS feed icons; fixes #121
This introduces a data model function of unusual privilege: it can retrieve favicon URLs for any subscription, regardless of user ID. This is a single-purpose hack and its use should be avoided if at all possible.
2017-11-10 12:02:59 -05:00
ea986f5032 Implement TTRSS operation updateArticle; fixes #83
This required adding the "notes" column to the arsse_marks table and adding same as a target value in Database::articleMark()

The Context class was also adjusted to remove the possibility of false positives in some tests
2017-11-09 14:21:12 -05:00
42a5ccb96c Handle request splitting in data model rather than controllers
Queries for multiple specific articles are limited in size because of limits on  the number of bound query parameters.

Currently this limit is somewhat arbitrarily set at 50, but it may increase.

Historically controllers would be responsible for chunking input, but this will present problems when the expected output is a result set, and of course the maintenance burden increases as the number of controllers increases.

This commit transfers the burden to the data model, and consequently introduces a ResultAggregate class which collects chunked result sets (currently only for articleList).

In the course of making these changes the mock Result class was also largely rewritten, fixing many bugs with it.

This commit does not modify the controllers nor their tests; this will be done in a subsequent commit.
2017-11-06 23:32:29 -05:00
cbe82c57cd Allow subscriptions to be listed non-recursively
This is used in multiple TTRSS operations (at least catchupFeed and getFeeds, and so is a useful optimization
2017-10-31 18:09:16 -04:00
579551f5fd Implement TTRSS operation getFeedTree; fixes #94 2017-10-30 13:11:27 -04:00
5c4772d95a CS fixes 2017-10-20 19:02:42 -04:00
d05aaf688f Preliminary TTRSS setArticleLabel implementation 2017-10-20 18:17:47 -04:00
fbbf751214 Implement the TTRSS getLabels operation; fixes #89 2017-10-13 17:05:06 -04:00
2e395f3cec Complete backend support for labels 2017-10-13 00:04:26 -04:00
20ff08a431 Implement TTTRSS getCounters operation; fixes #79 2017-10-11 12:55:50 -04:00
7a2de95c70 Consolidate article context handling into articleQuery function
Also consolidated article star counting into a generic articleCount function which accepts a context.

This may lead to slight efficiency losses in either listing or marking (and more significant ones in counting starred), but the advantages of centralized context handling are significant with the future addition of labels and the need to count articles under various future contexts in TTRSS.
2017-10-06 20:26:22 -04:00
26f6922b25 Partially implement labels
- Backend functions for adding, listing, removing, and editing (renaming) labels currently implemented
- TTRSS functions for adding (fixes #96), removing (fixes #97), and renaming (fixes #98) labels currently implemented
2017-10-05 17:42:12 -04:00
69b34a4e5a Implement TTRSS feed updating; fixes #86 2017-10-03 16:14:37 -04:00
03d5d554a9 Imple TTRSS functions getUnread and getConfig
- Fixes #78
- Fixes #85
2017-10-03 12:43:46 -04:00
bd6f23692c Implement TTRSS feed subscription; fixes #92 2017-10-03 10:43:09 -04:00
97b0134e56 Merge master 2017-10-02 15:42:15 -04:00
a80e283abc Perform feed discovery correctly; fixes #118 2017-10-02 11:53:52 -04:00
0a0aabe4ed Merge master 2017-09-30 12:52:05 -04:00
3482a35e54 Implement feed discovery; fixes #110 2017-09-30 11:43:43 -04:00
5488b994f7 Merged master; CS fixes 2017-09-28 10:16:24 -04:00
96ebf936e4 CS fixes 2017-09-28 09:01:43 -04:00
d365529493 Multiple fixes to input sanitization
- Database functions now accept any input, but throw typeViolation exceptions where appropriate instead of idMissing or subjectMissing
- Added unit tests for the new Misc\ValueInfo static class
- Added ValueInfo::id() method to centrally validate database IDs, and made use of it consistently
- Made use of PHP's filter_var() function where appropriate when validating or sanitizing input
- Made the NCN protocol handler reject most invalid IDs before handing off to method handlers
- Made NCN's feedUpdate and subscriptionMove methods return 422 on invalid input
- Adjusted several tests to handler type violations
2017-09-27 22:25:45 -04:00
7b121c74ec Merge fixes from master 2017-09-26 20:25:11 -04:00
e74a3ae3cb Fix numerous bugs when adding or changing folders
- Specifying a non-integer parent no longer silently casts to 0 or 1
- Specifying a folder ID of 0 now always converts to null automatically
- Performing both a rename and move to root in the same operation no longer results in potential duplicates
- Calling folderSetProperties with an empty data array no peforms an update; it now returns false before the update call
- Modification timestamps are now actually updated when a folder is modified
- Constraint violation exceptions triggered by code (rather than the database) now print a message
- Renaming a folder or subscription to a non-string value (e.g. an array) throws an exception rather than silently casting
- Added tests to better cover all the above
- Centralized the normalization of integers and title strings into a new ValueInfo static class
2017-09-26 16:45:41 -04:00
91432d4e16 Tests for sessions 2017-09-24 12:45:07 -04:00
b7ac63b9de Resuming a session from the database should have no side effects 2017-09-24 11:22:32 -04:00
1069447925 Make session cleanup more sophisticated 2017-09-24 10:09:36 -04:00
474d32e54f Preliminary implementation of TTRSS sessions (needs tests; may not work) 2017-09-16 19:57:33 -04:00
f7e50fe95d Passed code through linter 2017-08-29 10:50:31 -04:00
14951e2e4f Chnage "retain" settings to mean "purge; remove unused database settings 2017-08-20 15:46:35 -04:00
52104fb647 Implement article cleanup; fixes #28 2017-08-17 22:36:15 -04:00
eeb834fe4c Record the size of feeds upon update
Used to retain at least N articles when cleaning up, so that very infrequently changed feeds do not have their most recent articles purged
2017-08-14 20:07:31 -04:00
67203679aa Changed foreign key on marks table
Previously arsse_marks had a reference to arsse_users along with arsse_articles; the former has now changed to arsse_subscriptions.

Consequently deleting a subscription now deletes marks. Tests have been simplified as a consequence.

This change was understaken to simplify counting valid marks on articles for the purposes of article cleanup; now all marks are valid marks
2017-08-14 17:18:18 -04:00
3b018c89d1 Implemented cleanup of orphaned feeds; fixes #25 2017-08-02 18:27:04 -04:00
4066bba05e Full code coverage for Database class
- Fixes #49
2017-07-22 23:08:08 -04:00
da80a8ffa6 Complete tests for database driver
- Fixes #62
2017-07-22 15:29:12 -04:00
d3bca6eb47 More code coverage accommodation 2017-07-20 22:40:09 -04:00
3a26c75044 Tests for Database::meta*() functions
Improves #49
2017-07-18 16:38:23 -04:00
aaa4d1e988 Basic support for PicoFeed content scraping
- At the moment this is a completely manual setting: feed deduplication makes the setting very hard to handle for multiple users
- Improves #60
2017-07-17 14:56:50 -04:00
da092d5f8c Changed Misc\DateFormatter to a collection of static methods and renamed it to Date; renamed Data to Arsse to avoid confusion and better reflect its centrality 2017-07-17 07:47:57 -04:00
9f90756ede Improvements to Service 2017-07-16 14:55:37 -04:00
d4674c61b2 More bug fixes
- use DateFormatter throughout the Feed class
- Ensure dates have TEXT affinity in SQLite, in case it matters
- Add a userPreAuth setting for when relying on the Web server to do authentication
2017-07-15 13:33:17 -04:00
15285dba1e Add PHPHDoc type hints for certain class properties
This should aid autocompletion in IDEs
2017-07-14 10:16:16 -04:00
17ec6cf669 Move date formatting out of SQL and standardize on the DateFormatter trait
Fixes #56
2017-07-07 15:25:47 -04:00
7e7b204d85 Cleanup
- Revamped design of Query class to be more consistent and predictable, and generally suck less
- Removed special case for Query class in Statement class
- Cleaned up database schema somewhat
2017-07-07 11:49:54 -04:00
761c1054f3 Added context options to mark multiple specific articles or editions
This required significant rewrites of queries because of stale editions
2017-07-06 22:53:17 -04:00
c51ee594aa Last of the article series of tests 2017-07-05 10:59:13 -04:00
d78318bed7 Last of marking context tests 2017-07-05 09:09:38 -04:00
5742d949cc More article marking tests, some context tests
Various fixes as a result
2017-06-30 13:53:19 -04:00
c2f60bc5d2 More article tests
- Test the values returned by articleList
- Test paged and reversed listing
- Test marking, un/read, un/starred, and both
- Nearly complete rewrite of marking queries; previous efforts simply did not work, or overzealously updated or inserted
- Testing of marking contexts still required
2017-06-29 23:56:43 -04:00
ee9d8c1ceb First tests for articleList
Fixed many errors in queries:
- Prepared statement wasn't running at all :)
- Returning feed ID rather than subscription ID
- latestEdition wasn't matching
2017-06-22 13:07:56 -04:00
483874e21d Implemented query contexts
- Fixes #55
- Included test for Context
- Adjusted Database::editionLatest() to use Context
- Adjusted NCN handler and tests accordingly
- Also refined experimental Database::articleList() method and added experimental Database::articlePropertiesSet() method
2017-06-18 10:23:37 -04:00
21db3d0f52 Tweaks 2017-06-10 13:29:46 -04:00
c37d476ba0 Changes to Database internals
- Driver->prepare() can now take a new Database\Query object as the first parameter
- The Query class allows for building a complex query out of a main body, common table expressions, WHERE coniditions, ordering conditions, a limit, and offset
- The subscriptionList method is the prototypical example of usage
- The experimental articleList method will in time be adapted to use this as well
- A generic means of specifying a selection context is forthcoming; this would remove subscriptionList's $folder and $id parameters, for instance
2017-06-04 18:00:18 -04:00
2332cd52a3 Cleanup 2017-06-04 08:15:10 -04:00
dd24246f49 Finished last of feedUpdate tests; implemented NCN API functions
- Fixes #26
- Fixes #27
2017-06-03 17:34:37 -04:00
0514321a76 Save enclosures and categories properly in feedUpdate 2017-06-03 14:08:33 -04:00
c6b7e653c5 Tests for feedUpdate and feedListStale
The update process neglects to handle enclosures; this will have to be fixed
2017-06-02 19:16:36 -04:00
81419452b5 Added topmost folder to output of subscriptionList. Fixes #54
- Adjusted NCN handler to use the topmost folder for folderId; suppressed the actual folder so as not to pollute output with non-standard data
- Updated tests
2017-06-01 18:12:08 -04:00
ba6b5dee85 Simplify in-database settings
The previous complexity is really not required. Also modified how settingSet() works to avoid a REPLACE query, which is not compatible with PostgreSQL.
2017-06-01 16:24:11 -04:00
5b8f23441c Feed tests: article matching
These should be the last required tests for the Feed class
2017-05-30 20:18:04 -04:00
7656de716b Feed tests: fetching and parsing 2017-05-27 18:15:52 -04:00
88c0f4986c Implement NCN API v1-2 feed handling: tests
- Fixes #8
- Fixes #9
- Fixes #10
- Fixes #11
- Fixes #12

- Differentiated between a missing ID which is the subject of an action (e.g. the subscription when moving a subscription) and other missing IDs (e.g. the folder when moving a subscription)
- Enforced same rules for subscription titles as for folder names, save that null is valid for subscriptions
2017-05-21 10:10:36 -04:00
4a816f827b NCNv1 feed calls and other changes
- Implemented all but one feed-related function (it's more ofan item function)
- Fixed time conversion for input into SQL; dates in PM were previously wrong
- Added miscellaneous tentative functions to Database to help with peculiarities of NCNv1; these may change
- Tests to come soon
2017-05-18 23:03:33 -04:00
0bc2841837 Last of subscription tests
- Also tweaked SQL dateformat() function to output proper HTTP dates
- Also introduced method to set a default output date format
2017-05-18 13:21:17 -04:00
ef3cb4fa54 Test unread counts in subscriptionList 2017-05-16 22:19:40 -04:00
f635155687 Basic tests for subscription list
- Need entries and read marks to fully test
- Added assertion type to test result subsets
- Fixed custom subscription titles
2017-05-14 23:03:48 -04:00
98c950de58 Removed most calls to userExists()
- functions not related to user management now have the existence of the affected user checked in the authorizer, when the affected user differs from the actor
- User::authorizationEnabled() now nests: disabling twice and then enabling once leaves the authorizer disabled
- Disabling of the authorizer is now tested
- User tests now use a partial mock instead of relying on User::authorizationEnabled()
- Added authorizer tests against a missing user
- Removed folder tests related to missing users
- Also added more subscription tests
2017-05-11 18:00:35 -04:00
2083c6e397 Change transactions to auto-rollback on exceptions 2017-05-06 12:02:27 -04:00
db132c0c24 Various changes to Database
- Introduced "strict" binding types for use when inserting into NOT NULL columns: any null value supplied is always cast to the supplied type rather than passing through
- Fixed feed updating further
- Filled out full complement of subscription manipulation functions
- made folderPropertiesSet possibly return false
2017-05-04 19:12:33 -04:00
9c7f4710aa Fix feed fetching
There are almost certainly other bugs; proper tests forthcoming
2017-05-04 14:42:40 -04:00
9dccdb32a7 Fold feedAdd into subscriptionAdd
A feed will enver been added independently of a subscription
2017-05-04 11:13:24 -04:00
717e74fb2e Refactored special case for testing 2017-05-03 20:00:29 -04:00
8376cc9cb9 Mark articles as unread when they are updated 2017-05-01 18:29:07 -04:00
7546e3b11b Add the ability to throw exceptions in a feed update if desired
This is useful when adding a new feed
2017-04-30 18:36:31 -04:00
43f73b5d71 Make better use of prepared statements when updating feeds 2017-04-30 18:14:11 -04:00
3bc9082cfa Integrated adaptive update intervals Fixes #51 2017-04-30 17:54:29 -04:00
2e64e60f2e Implement part of adaptive update interval; improves #51
Implements part of algorithm used when a feed has not been updated; this is much simpler than when a feed has been modified
2017-04-24 21:51:56 -04:00
93c010d3d5 Move feed item change detection to Feed class 2017-04-22 23:40:57 -04:00
b1b96bb806 Completed refactoring of feed updates
- In-feed deduplication still to do
- Content scraping still to do
- Lots and lots of testing still required
2017-04-20 21:59:12 -04:00
119d42907e More feed update refactoring
Still very much incomplete: in its present form it would yield many false duplicates
2017-04-15 22:07:22 -04:00
4e57e56ca7 Start on refactoring of feed processing 2017-04-13 22:17:53 -04:00
557d17ef5d Add rollbacks in error cases 2017-04-09 18:15:00 -04:00
a67fe30408 Fix whitespace
Also fixed my editor so tabs won't happen again!
2017-04-06 21:41:21 -04:00
818707b067 Fix adding a subscription (again)
Also ensured HTTP last modified date is parsed properly.
2017-04-06 20:50:47 -04:00
e6f2f22888 use list() instead of extract()
Improves clarity where Database::generateSet() is actually used at the expense of clarity within the method itself
2017-04-06 13:29:39 -04:00
929d763981 Simplify the creation of arbitrary UPDATEs
The type parameters of Db\Driver::prepare() and the parameters of Db\Statement::run() can now be arrays, which will be iterated over recursively to bind scalar values to the SQL statement.

This simplifies the construction of arbitrary UPDATE statements (the WHERE clause no longer needs to be taken into account) and should make it clearer what is happening in these cases.

It should also simplify the creation of IN() clauses down the road if they become necessary.
2017-04-06 11:02:47 -04:00
25aeaf4801 Grab favicon from site's URL, not the feed's
• Changed Feed to where the favicon is gotten when parsing so the
feed’s site URL is used instead of the feed’s for favicons. Some feeds
don’t have the same domain.
• Fixed a bug in Database.
2017-04-02 21:23:15 -05:00
dc9f5e545e Fix adding a feed
Also start on handling the v1-2 API
2017-04-01 15:42:10 -04:00
1e1b848c62 Remove root field from folders table
The field is no longer required with the use of recursive common table expressions, and  presents a possible loss of referential integrity
2017-04-01 14:49:31 -04:00
a111bcc231 Folder get/set prop funcs and other changes
- Simplified folder removal; now properly relies on foreign keys for dependency resolution
- simplified *propertiesSet() methods by offloading input validation and query building to a generic function
- Implemented function to get the properties of a single folder (useful for internal use)
- Implemented a function to set the properties of a folder
2017-04-01 10:27:26 -04:00
2344911f78 Tests for removing folders 2017-03-31 18:48:24 -04:00
458eaac5c6 Tests for listing folders
- Fixed checking for existence of parent before listing
- Changed Result->getAll() to return an empty array rather than null for an empty result set
2017-03-31 17:42:28 -04:00
b68cea1188 Tests for adding folders 2017-03-31 15:27:59 -04:00
bc863ae935 Updated Database->updateFeeds()
• Added category updating in Database->updateFeeds()
• Made Database->updateFeeds() fail quietly when exceptions are
occurred, logging them in the database instead of failing.
• Changed the categories table name from arsse_tags to arsse_categories.
2017-03-30 09:42:37 -05:00
dbd735f1f3 Full set of tests for user-related Database funcs
- Fixed a few inconsistencies and bugs along the way
- User manager tests need some equivalent fleshing out
2017-03-29 23:41:05 -04:00
f902346b6c Eliminated passing of RuntimeData instances
- RuntimeData has now been replaced by a single static Data class
- The Data class has a load() method which fills the same role as the constructor of RuntimeData
- The static Lang class is now an instantiable class and is a member of Data
- All tests have been adjusted and pass
- The Exception tests no longer require convoluted workarounds: a simple mock  for Data::$l suffices; Lang tests also use a mock to prevent loops now instead of using a workaround
2017-03-28 18:50:00 -04:00
7236020b8b Changed "NewsSync" to "Arsse" 2017-03-27 23:12:12 -05:00
b8f9b6070e Experimental method to remove a folder 2017-03-27 08:39:24 -04:00
ce0584e7f8 Added Feed Updating
• Started implementing feed updating (Database->updateFeeds())
• Moved hashing to the Feed object, now done when parsing
• Moved adding of articles to the database to its own method
(Database->articleAdd())
2017-03-26 15:16:15 -05:00
8c76c22d74 Implement retrieving folder list in Database 2017-03-24 22:39:18 -04:00
254ffe7b48 Don't enforce 100-character limit on folder names 2017-03-24 13:26:13 -04:00
3f61921b97 Added picoFeed wrapper
• Implemented a simple wrapper for picoFeed which fixes the id problems
and keeps error handling within its own class
• Updated Database.php to use the new class
• Replaced mentions of ownCloud with NextCloud in the schema
• Added hashes to schema for identification and change detection;
removed NextCloud hash and fingerprint; removed enclosure and category
hashes
2017-03-18 11:01:23 -05:00
8c268581fb Properly handle adding a folder
Also added test file for now
2017-03-09 22:41:11 -05:00
689c1c77d0 Test for advisory locking 2017-03-09 17:14:26 -05:00
485400df2b Rename Result->get() to Result->getRow
It should be slightly clearer what it does
2017-03-09 15:01:18 -05:00
bdf3182305 Add result->lastId method; cleanup
- Result->lastId() added as MySQL and PostgreSQL have equivalent functionality
- Adjusted tests accordingly
- Cleaned up Database class to make use of this and getAll()
2017-03-09 14:48:42 -05:00
7687109132 Reorganize Db namespace; alter User ns to match 2017-03-07 18:01:13 -05:00
37dad63dee Impelemented Result->getAll()
- Fixes #45
- Renamed getSingle to getValue to avoid possible confusion
- Added test to ensure getValue() always returns the first datum of each row rather than going column-to-column
2017-03-06 16:34:38 -05:00
eeb1818bb5 Still more database changes
- Restructured tests
- Localized driver name for SQLite driver (fixes #37)
- Ensured that binding type definitions are required
2017-03-03 13:20:26 -05:00
2b7a236147 More database fixes
Now able to add feeds to the database
2017-03-02 22:43:59 -05:00
007e3747ef Various database fixes
Authentication appears to be broken with a real database
2017-03-02 20:47:00 -05:00
0c410fcf50 More binding tests and related changes
- Introduced abstract Statement class to hold common methods
- Common methods currently consist of a date formatter and type caster
- Moved binding tests to a trait for reuse with future drivers
2017-03-02 18:42:19 -05:00
337b2cf90c Remove last vestiges of PDO accomodation 2017-03-02 09:04:04 -05:00
8a0d021622 More User tests and resultant fixes 2017-02-20 19:04:08 -05:00
7785eb072b Complete rewrite of User class and other changes
- User-related database methods will now throw User\Exception upon errors
- Internal userAdd method can now generate random passwords
- Pursuant to above, dependency on password genrator has been added, and password-related methods now return strings instead of booleans
- User class methods now all explicitly follow different branches for internal/external/missing implementations
- various User class methods now perform auto-provisioning of the internal database when external implementations report success on users not in the database
- Tests have been adjusted to account for the above changes
- Lots is probably still broken
2017-02-20 17:04:13 -05:00
26989facfe More PicoFeed Integration Bug fixes
• Fixed a bug where the feed object’s variable was being reused as the
id for adding feeds to subscriptions
• Modified inline documentation
• Added a TODO for populating the articles table when adding a new feed
2017-02-20 11:58:26 -06:00
cb664eab03 Minor fixes to PicoFeed integration 2017-02-19 18:33:03 -06:00
e5d825d360 Started integration of PicoFeed
• Integrated PicoFeed into Database->subscriptionAdd
• Added exception handling for feeds
• Added static method for formatting SQL dates into Db/Common
2017-02-19 16:02:03 -06:00
e6feb8de8d First basic test for User class
As the User class depends on the database and this has yet to be tested (though I'm fairly certain it works), the mock driver also acts as a mock of the required database functions, with both instances sharing a common storage structure. Later test series should separate the two.
2017-02-19 00:22:16 -05:00
c5fac33398 Converted all hard tabs to soft tabs 2017-02-16 14:29:42 -06:00
c213b5928d move code out of vendor tree into lib 2017-02-11 14:48:48 -05:00
Renamed from vendor/JKingWeb/NewsSync/Database.php (Browse further)