Migrate to PostgreSQL #39

Merged
Miroslavsckaya merged 16 commits from migrate_to_postgresql into master 2022-07-13 22:53:54 +00:00

Migrate to PostgreSQL from sqlite
Closes #15
Closes #32

Migrate to PostgreSQL from sqlite Closes #15 Closes #32
Miroslavsckaya added 1 commit 2022-07-10 21:31:35 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
1bdb00e8e7
WIP: migrate to PostgreSQL
skobkin reviewed 2022-07-12 19:28:31 +00:00
bot.py Outdated
@ -11,2 +11,2 @@
token = os.getenv('TELEGRAM_TOKEN')
db_path = os.getenv('DATABASE_PATH', './bot.db')
token = os.getenv('RSSBOT_TG_TOKEN')
db_dsn = os.getenv('RSSBOT_DSN',)
Collaborator

DSN literally means "Data Source Name". So if we have no another data source, we don't need to have prefix here.

It doesn't mean that it's wrong and you should delete it right now though.

DSN literally means "[Data Source Name](https://en.wikipedia.org/wiki/Data_source_name)". So if we have no another data source, we don't need to have prefix here. It doesn't mean that it's wrong and you should delete it right now though.
skobkin marked this conversation as resolved
database.py Outdated
@ -18,2 +16,2 @@
self.conn.row_factory = sqlite3.Row
self.cur = self.conn.cursor()
self.log.debug('Database.__init__(DSN=\'%s\')', dsn)
self.conn = psycopg2.connect(dsn)
Collaborator

Let's not forget about type hints.

Let's not forget about type hints.
skobkin marked this conversation as resolved
database.py Outdated
@ -26,2 +24,3 @@
self.cur.execute('INSERT INTO users (telegram_id) VALUES (%s)', [telegram_id])
self.conn.commit()
return self.cur.lastrowid
return self.find_user(telegram_id)
Collaborator
WAT - https://www.psycopg.org/docs/cursor.html#cursor.lastrowid or - https://www.postgresql.org/docs/current/sql-insert.html (`RETURNING`)
skobkin marked this conversation as resolved
database.py Outdated
@ -42,2 +40,3 @@
self.cur.execute('INSERT INTO feeds (url) VALUES (%s)', [url])
self.conn.commit()
return self.cur.lastrowid
return self.find_feed_by_url(url)
Collaborator

WAT

WAT
skobkin marked this conversation as resolved
database.py Outdated
@ -122,3 +121,3 @@
return list(map(lambda x: x['telegram_id'], subscribers))
def find_feeds(self) -> list[sqlite3.Row]:
def find_feeds(self) -> list[psycopg2.extras.DictRow]:
Collaborator

You can import DictRow.

You can import `DictRow`.
skobkin marked this conversation as resolved
database.py Outdated
@ -179,3 +177,3 @@
'CREATE TABLE IF NOT EXISTS feeds_last_items ('
' feed_id INTEGER,'
' url TEXT NOT NULL UNIQUE,'
' url TEXT NOT NULL,'
Collaborator

Do not forget to use GUID as the base for comparison. URL's are not reliable in our case.

Do not forget to use GUID as the base for comparison. URL's are not reliable in our case.
skobkin marked this conversation as resolved
telegram.py Outdated
@ -157,3 +158,3 @@
if self.sent_counter >= self.BATCH_LIMIT:
self.log.debug('Requests limit exceeded, sleeping for a second')
time.sleep(1)
time.sleep(2)
Collaborator

Why?

Why?
skobkin marked this conversation as resolved
skobkin added the
enhancement
refactoring
labels 2022-07-13 15:17:26 +00:00
skobkin added this to the MVP 0.1 milestone 2022-07-13 15:17:29 +00:00
Miroslavsckaya was assigned by skobkin 2022-07-13 15:17:37 +00:00
skobkin requested review from skobkin 2022-07-13 15:17:49 +00:00
Miroslavsckaya added 1 commit 2022-07-13 16:19:57 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
f11b0b91af
fix removal feed, refactor return id and add type hints
skobkin reviewed 2022-07-13 17:02:53 +00:00
database.py Outdated
@ -128,3 +128,3 @@
return self.cur.fetchall()
def find_user_feeds(self, user_id: int) -> list[sqlite3.Row]:
def find_user_feeds(self, user_id: int) -> list[DictRow]:
Collaborator

I forgot to tell about that earlier, but it's better not to expose specifics of database implementation in public methods.
We can fix that by returning non-library-specific structure. For example list[dict] or list[collections.OrderedDict] instead of list[psycopg2.extras.DictRow].

I forgot to tell about that earlier, but it's better not to expose specifics of database implementation in public methods. We can fix that by returning non-library-specific structure. For example `list[dict]` or [`list[collections.OrderedDict]`](https://realpython.com/python-ordereddict/) instead of [`list[psycopg2.extras.DictRow]`](https://github.com/psycopg/psycopg2/blob/1d3a89a0bba621dc1cc9b32db6d241bd2da85ad1/lib/extras.py#L160-L211).
skobkin marked this conversation as resolved
Miroslavsckaya added 2 commits 2022-07-13 17:39:20 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
feef909c61
add __convert_to_list_of_dicts method
skobkin reviewed 2022-07-13 17:42:07 +00:00
database.py Outdated
@ -186,1 +182,4 @@
)
@staticmethod
def __convert_to_list_of_dicts(rows: list[DictRow]) -> list[dict]:
Collaborator

Convert what? Maybe __dictrow_to_dict_list() или __dictrow_to_list_of_dicts()?

Convert what? Maybe `__dictrow_to_dict_list()` или `__dictrow_to_list_of_dicts()`?
Author
Owner

ok

ok
skobkin marked this conversation as resolved
Miroslavsckaya added 1 commit 2022-07-13 17:50:05 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
50ac344095
rename __convert_to_list_of_dicts method to __dictrow_to_dict_list
Miroslavsckaya added 3 commits 2022-07-13 19:39:58 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
ae31be33b6
fix update method in UpdateManager
Miroslavsckaya added 1 commit 2022-07-13 20:05:29 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
6535055268
refuse to unpack the dict in UpdateManager
Miroslavsckaya added 1 commit 2022-07-13 21:42:10 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
c91549a311
bug fix. add comparison by guid instead of url
Miroslavsckaya added 1 commit 2022-07-13 22:12:19 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
037272a167
not saving description in the database anymore
Miroslavsckaya added 1 commit 2022-07-13 22:21:41 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
61a2d8d215
not saving feed items title in the database anymore
Miroslavsckaya added 1 commit 2022-07-13 22:29:39 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
292a7a3c88
little refactor in update_feed_items method
Miroslavsckaya added 1 commit 2022-07-13 22:37:19 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
2206c6299d
change time.sleep to 1
Miroslavsckaya added 1 commit 2022-07-13 22:42:00 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
ab77d44ba3
rename db_dsn to dsn variable in bot.py and update.py
Miroslavsckaya added 1 commit 2022-07-13 22:48:20 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
2bb167fef5
little fix in bot.by and update.py
skobkin approved these changes 2022-07-13 22:52:15 +00:00
skobkin left a comment
Collaborator

Let's merge it!

Slap Shit Together and Deploy!

Let's merge it! ![Slap Shit Together and Deploy!](https://i.skobk.in/i/slap_shit_together_and_deploy.jpg)
Miroslavsckaya changed title from WIP: Migrate to PostgreSQL to Migrate to PostgreSQL 2022-07-13 22:52:45 +00:00
Miroslavsckaya merged commit e0992b2351 into master 2022-07-13 22:53:54 +00:00
Miroslavsckaya deleted branch migrate_to_postgresql 2022-07-13 22:53:54 +00:00
Sign in to join this conversation.
No reviewers
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Miroslavsckaya/tg_rss_bot#39
No description provided.