Migrate to PostgreSQL #39

Merged
Miroslavsckaya merged 16 commits from migrate_to_postgresql into master 2022-07-13 22:53:54 +00:00

Migrate to PostgreSQL from sqlite
Closes #15
Closes #32

Migrate to PostgreSQL from sqlite Closes #15 Closes #32
Miroslavsckaya added 1 commit 2022-07-10 21:31:35 +00:00
WIP: migrate to PostgreSQL
Some checks failed
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is failing
1bdb00e8e7
skobkin reviewed 2022-07-12 19:28:31 +00:00
bot.py Outdated
@ -11,2 +11,2 @@
token = os.getenv('TELEGRAM_TOKEN')
db_path = os.getenv('DATABASE_PATH', './bot.db')
token = os.getenv('RSSBOT_TG_TOKEN')
db_dsn = os.getenv('RSSBOT_DSN',)
Collaborator

DSN literally means "Data Source Name". So if we have no another data source, we don't need to have prefix here.

It doesn't mean that it's wrong and you should delete it right now though.

DSN literally means "[Data Source Name](https://en.wikipedia.org/wiki/Data_source_name)". So if we have no another data source, we don't need to have prefix here. It doesn't mean that it's wrong and you should delete it right now though.
skobkin marked this conversation as resolved
database.py Outdated
@ -18,2 +16,2 @@
self.conn.row_factory = sqlite3.Row
self.cur = self.conn.cursor()
self.log.debug('Database.__init__(DSN=\'%s\')', dsn)
self.conn = psycopg2.connect(dsn)
Collaborator

Let's not forget about type hints.

Let's not forget about type hints.
skobkin marked this conversation as resolved
database.py Outdated
@ -26,2 +24,3 @@
self.cur.execute('INSERT INTO users (telegram_id) VALUES (%s)', [telegram_id])
self.conn.commit()
return self.cur.lastrowid
return self.find_user(telegram_id)
Collaborator
WAT - https://www.psycopg.org/docs/cursor.html#cursor.lastrowid or - https://www.postgresql.org/docs/current/sql-insert.html (`RETURNING`)
skobkin marked this conversation as resolved
database.py Outdated
@ -42,2 +40,3 @@
self.cur.execute('INSERT INTO feeds (url) VALUES (%s)', [url])
self.conn.commit()
return self.cur.lastrowid
return self.find_feed_by_url(url)
Collaborator

WAT

WAT
skobkin marked this conversation as resolved
database.py Outdated
@ -122,3 +121,3 @@
return list(map(lambda x: x['telegram_id'], subscribers))
def find_feeds(self) -> list[sqlite3.Row]:
def find_feeds(self) -> list[psycopg2.extras.DictRow]:
Collaborator

You can import DictRow.

You can import `DictRow`.
skobkin marked this conversation as resolved
database.py Outdated
@ -179,3 +177,3 @@
'CREATE TABLE IF NOT EXISTS feeds_last_items ('
' feed_id INTEGER,'
' url TEXT NOT NULL UNIQUE,'
' url TEXT NOT NULL,'
Collaborator

Do not forget to use GUID as the base for comparison. URL's are not reliable in our case.

Do not forget to use GUID as the base for comparison. URL's are not reliable in our case.
skobkin marked this conversation as resolved
telegram.py Outdated
@ -157,3 +158,3 @@
if self.sent_counter >= self.BATCH_LIMIT:
self.log.debug('Requests limit exceeded, sleeping for a second')
time.sleep(1)
time.sleep(2)
Collaborator

Why?

Why?
skobkin marked this conversation as resolved
skobkin added the
enhancement
refactoring
labels 2022-07-13 15:17:26 +00:00
skobkin added this to the MVP 0.1 milestone 2022-07-13 15:17:29 +00:00
Miroslavsckaya was assigned by skobkin 2022-07-13 15:17:37 +00:00
skobkin requested review from skobkin 2022-07-13 15:17:49 +00:00
Miroslavsckaya added 1 commit 2022-07-13 16:19:57 +00:00
fix removal feed, refactor return id and add type hints
Some checks failed
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is failing
f11b0b91af
skobkin reviewed 2022-07-13 17:02:53 +00:00
database.py Outdated
@ -128,3 +128,3 @@
return self.cur.fetchall()
def find_user_feeds(self, user_id: int) -> list[sqlite3.Row]:
def find_user_feeds(self, user_id: int) -> list[DictRow]:
Collaborator

I forgot to tell about that earlier, but it's better not to expose specifics of database implementation in public methods.
We can fix that by returning non-library-specific structure. For example list[dict] or list[collections.OrderedDict] instead of list[psycopg2.extras.DictRow].

I forgot to tell about that earlier, but it's better not to expose specifics of database implementation in public methods. We can fix that by returning non-library-specific structure. For example `list[dict]` or [`list[collections.OrderedDict]`](https://realpython.com/python-ordereddict/) instead of [`list[psycopg2.extras.DictRow]`](https://github.com/psycopg/psycopg2/blob/1d3a89a0bba621dc1cc9b32db6d241bd2da85ad1/lib/extras.py#L160-L211).
skobkin marked this conversation as resolved
Miroslavsckaya added 2 commits 2022-07-13 17:39:20 +00:00
add __convert_to_list_of_dicts method
Some checks failed
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is failing
feef909c61
skobkin reviewed 2022-07-13 17:42:07 +00:00
database.py Outdated
@ -186,1 +182,4 @@
)
@staticmethod
def __convert_to_list_of_dicts(rows: list[DictRow]) -> list[dict]:
Collaborator

Convert what? Maybe __dictrow_to_dict_list() или __dictrow_to_list_of_dicts()?

Convert what? Maybe `__dictrow_to_dict_list()` или `__dictrow_to_list_of_dicts()`?
Author
Owner

ok

ok
skobkin marked this conversation as resolved
Miroslavsckaya added 1 commit 2022-07-13 17:50:05 +00:00
rename __convert_to_list_of_dicts method to __dictrow_to_dict_list
Some checks failed
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is failing
50ac344095
Miroslavsckaya added 3 commits 2022-07-13 19:39:58 +00:00
fix update method in UpdateManager
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
ae31be33b6
Miroslavsckaya added 1 commit 2022-07-13 20:05:29 +00:00
refuse to unpack the dict in UpdateManager
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
6535055268
Miroslavsckaya added 1 commit 2022-07-13 21:42:10 +00:00
bug fix. add comparison by guid instead of url
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
c91549a311
Miroslavsckaya added 1 commit 2022-07-13 22:12:19 +00:00
not saving description in the database anymore
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
037272a167
Miroslavsckaya added 1 commit 2022-07-13 22:21:41 +00:00
not saving feed items title in the database anymore
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
61a2d8d215
Miroslavsckaya added 1 commit 2022-07-13 22:29:39 +00:00
little refactor in update_feed_items method
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
292a7a3c88
Miroslavsckaya added 1 commit 2022-07-13 22:37:19 +00:00
change time.sleep to 1
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2206c6299d
Miroslavsckaya added 1 commit 2022-07-13 22:42:00 +00:00
rename db_dsn to dsn variable in bot.py and update.py
Some checks failed
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is failing
ab77d44ba3
Miroslavsckaya added 1 commit 2022-07-13 22:48:20 +00:00
little fix in bot.by and update.py
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2bb167fef5
skobkin approved these changes 2022-07-13 22:52:15 +00:00
skobkin left a comment
Collaborator

Let's merge it!

Slap Shit Together and Deploy!

Let's merge it! ![Slap Shit Together and Deploy!](https://i.skobk.in/i/slap_shit_together_and_deploy.jpg)
Miroslavsckaya changed title from WIP: Migrate to PostgreSQL to Migrate to PostgreSQL 2022-07-13 22:52:45 +00:00
Miroslavsckaya merged commit e0992b2351 into master 2022-07-13 22:53:54 +00:00
Miroslavsckaya deleted branch migrate_to_postgresql 2022-07-13 22:53:54 +00:00
Miroslavsckaya referenced this pull request from a commit 2022-07-13 22:53:54 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Miroslavsckaya/tg_rss_bot#39
No description provided.