Migrate to PostgreSQL #39

Merged
Miroslavsckaya merged 16 commits from migrate_to_postgresql into master 2022-07-13 22:53:54 +00:00

Migrate to PostgreSQL from sqlite
Closes
Closes

Migrate to PostgreSQL from sqlite Closes #15 Closes #32
Miroslavsckaya added 1 commit 2022-07-10 21:31:35 +00:00
skobkin reviewed 2022-07-12 19:28:31 +00:00
bot.py Outdated
@ -11,2 +11,2 @@
token = os.getenv('TELEGRAM_TOKEN')
db_path = os.getenv('DATABASE_PATH', './bot.db')
token = os.getenv('RSSBOT_TG_TOKEN')
db_dsn = os.getenv('RSSBOT_DSN',)
Collaborator

DSN literally means "Data Source Name". So if we have no another data source, we don't need to have prefix here.

It doesn't mean that it's wrong and you should delete it right now though.

DSN literally means "[Data Source Name](https://en.wikipedia.org/wiki/Data_source_name)". So if we have no another data source, we don't need to have prefix here. It doesn't mean that it's wrong and you should delete it right now though.
skobkin marked this conversation as resolved
database.py Outdated
@ -18,2 +16,2 @@
self.conn.row_factory = sqlite3.Row
self.cur = self.conn.cursor()
self.log.debug('Database.__init__(DSN=\'%s\')', dsn)
self.conn = psycopg2.connect(dsn)
Collaborator

Let's not forget about type hints.

Let's not forget about type hints.
skobkin marked this conversation as resolved
database.py Outdated
@ -26,2 +24,3 @@
self.cur.execute('INSERT INTO users (telegram_id) VALUES (%s)', [telegram_id])
self.conn.commit()
return self.cur.lastrowid
return self.find_user(telegram_id)
Collaborator
WAT - https://www.psycopg.org/docs/cursor.html#cursor.lastrowid or - https://www.postgresql.org/docs/current/sql-insert.html (`RETURNING`)
skobkin marked this conversation as resolved
database.py Outdated
@ -42,2 +40,3 @@
self.cur.execute('INSERT INTO feeds (url) VALUES (%s)', [url])
self.conn.commit()
return self.cur.lastrowid
return self.find_feed_by_url(url)
Collaborator

WAT

WAT
skobkin marked this conversation as resolved
database.py Outdated
@ -122,3 +121,3 @@
return list(map(lambda x: x['telegram_id'], subscribers))
def find_feeds(self) -> list[sqlite3.Row]:
def find_feeds(self) -> list[psycopg2.extras.DictRow]:
Collaborator

You can import DictRow.

You can import `DictRow`.
skobkin marked this conversation as resolved
database.py Outdated
@ -179,3 +177,3 @@
'CREATE TABLE IF NOT EXISTS feeds_last_items ('
' feed_id INTEGER,'
' url TEXT NOT NULL UNIQUE,'
' url TEXT NOT NULL,'
Collaborator

Do not forget to use GUID as the base for comparison. URL's are not reliable in our case.

Do not forget to use GUID as the base for comparison. URL's are not reliable in our case.
skobkin marked this conversation as resolved
telegram.py Outdated
@ -157,3 +158,3 @@
if self.sent_counter >= self.BATCH_LIMIT:
self.log.debug('Requests limit exceeded, sleeping for a second')
time.sleep(1)
time.sleep(2)
Collaborator

Why?

Why?
skobkin marked this conversation as resolved
skobkin added the
enhancement
refactoring
labels 2022-07-13 15:17:26 +00:00
skobkin added this to the MVP 0.1 milestone 2022-07-13 15:17:29 +00:00
Miroslavsckaya was assigned by skobkin 2022-07-13 15:17:37 +00:00
skobkin requested review from skobkin 2022-07-13 15:17:49 +00:00
Miroslavsckaya added 1 commit 2022-07-13 16:19:57 +00:00
skobkin reviewed 2022-07-13 17:02:53 +00:00
database.py Outdated
@ -128,3 +128,3 @@
return self.cur.fetchall()
def find_user_feeds(self, user_id: int) -> list[sqlite3.Row]:
def find_user_feeds(self, user_id: int) -> list[DictRow]:
Collaborator

I forgot to tell about that earlier, but it's better not to expose specifics of database implementation in public methods.
We can fix that by returning non-library-specific structure. For example list[dict] or list[collections.OrderedDict] instead of list[psycopg2.extras.DictRow].

I forgot to tell about that earlier, but it's better not to expose specifics of database implementation in public methods. We can fix that by returning non-library-specific structure. For example `list[dict]` or [`list[collections.OrderedDict]`](https://realpython.com/python-ordereddict/) instead of [`list[psycopg2.extras.DictRow]`](https://github.com/psycopg/psycopg2/blob/1d3a89a0bba621dc1cc9b32db6d241bd2da85ad1/lib/extras.py#L160-L211).
skobkin marked this conversation as resolved
Miroslavsckaya added 2 commits 2022-07-13 17:39:20 +00:00
skobkin reviewed 2022-07-13 17:42:07 +00:00
database.py Outdated
@ -186,1 +182,4 @@
)
@staticmethod
def __convert_to_list_of_dicts(rows: list[DictRow]) -> list[dict]:
Collaborator

Convert what? Maybe __dictrow_to_dict_list() или __dictrow_to_list_of_dicts()?

Convert what? Maybe `__dictrow_to_dict_list()` или `__dictrow_to_list_of_dicts()`?
Author
Owner

ok

ok
skobkin marked this conversation as resolved
Miroslavsckaya added 1 commit 2022-07-13 17:50:05 +00:00
Miroslavsckaya added 3 commits 2022-07-13 19:39:58 +00:00
Miroslavsckaya added 1 commit 2022-07-13 20:05:29 +00:00
Miroslavsckaya added 1 commit 2022-07-13 21:42:10 +00:00
Miroslavsckaya added 1 commit 2022-07-13 22:12:19 +00:00
Miroslavsckaya added 1 commit 2022-07-13 22:21:41 +00:00
Miroslavsckaya added 1 commit 2022-07-13 22:29:39 +00:00
Miroslavsckaya added 1 commit 2022-07-13 22:37:19 +00:00
Miroslavsckaya added 1 commit 2022-07-13 22:42:00 +00:00
Miroslavsckaya added 1 commit 2022-07-13 22:48:20 +00:00
skobkin approved these changes 2022-07-13 22:52:15 +00:00
skobkin left a comment
Collaborator

Let's merge it!

Slap Shit Together and Deploy!

Let's merge it! ![Slap Shit Together and Deploy!](https://i.skobk.in/i/slap_shit_together_and_deploy.jpg)
Miroslavsckaya changed title from WIP: Migrate to PostgreSQL to Migrate to PostgreSQL 2022-07-13 22:52:45 +00:00
Miroslavsckaya merged commit e0992b2351 into master 2022-07-13 22:53:54 +00:00
Miroslavsckaya deleted branch migrate_to_postgresql 2022-07-13 22:53:54 +00:00
Miroslavsckaya referenced this pull request from a commit 2022-07-13 22:53:54 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Miroslavsckaya/tg_rss_bot#39
No description provided.