Migrate to PostgreSQL #39

Merged
Miroslavsckaya merged 16 commits from migrate_to_postgresql into master 2022-07-14 01:53:54 +03:00

Migrate to PostgreSQL from sqlite
Closes #15
Closes #32

Migrate to PostgreSQL from sqlite Closes #15 Closes #32
WIP: migrate to PostgreSQL
Some checks failed
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is failing
1bdb00e8e7
bot.py Outdated
@ -11,2 +11,2 @@
token = os.getenv('TELEGRAM_TOKEN')
db_path = os.getenv('DATABASE_PATH', './bot.db')
token = os.getenv('RSSBOT_TG_TOKEN')
db_dsn = os.getenv('RSSBOT_DSN',)
Collaborator

DSN literally means "Data Source Name". So if we have no another data source, we don't need to have prefix here.

It doesn't mean that it's wrong and you should delete it right now though.

DSN literally means "[Data Source Name](https://en.wikipedia.org/wiki/Data_source_name)". So if we have no another data source, we don't need to have prefix here. It doesn't mean that it's wrong and you should delete it right now though.
skobkin marked this conversation as resolved
database.py Outdated
@ -18,2 +16,2 @@
self.conn.row_factory = sqlite3.Row
self.cur = self.conn.cursor()
self.log.debug('Database.__init__(DSN=\'%s\')', dsn)
self.conn = psycopg2.connect(dsn)
Collaborator

Let's not forget about type hints.

Let's not forget about type hints.
skobkin marked this conversation as resolved
database.py Outdated
@ -26,2 +24,3 @@
self.cur.execute('INSERT INTO users (telegram_id) VALUES (%s)', [telegram_id])
self.conn.commit()
return self.cur.lastrowid
return self.find_user(telegram_id)
Collaborator
WAT - https://www.psycopg.org/docs/cursor.html#cursor.lastrowid or - https://www.postgresql.org/docs/current/sql-insert.html (`RETURNING`)
skobkin marked this conversation as resolved
database.py Outdated
@ -42,2 +40,3 @@
self.cur.execute('INSERT INTO feeds (url) VALUES (%s)', [url])
self.conn.commit()
return self.cur.lastrowid
return self.find_feed_by_url(url)
Collaborator

WAT

WAT
skobkin marked this conversation as resolved
database.py Outdated
@ -122,3 +121,3 @@
return list(map(lambda x: x['telegram_id'], subscribers))
def find_feeds(self) -> list[sqlite3.Row]:
def find_feeds(self) -> list[psycopg2.extras.DictRow]:
Collaborator

You can import DictRow.

You can import `DictRow`.
skobkin marked this conversation as resolved
database.py Outdated
@ -179,3 +177,3 @@
'CREATE TABLE IF NOT EXISTS feeds_last_items ('
' feed_id INTEGER,'
' url TEXT NOT NULL UNIQUE,'
' url TEXT NOT NULL,'
Collaborator

Do not forget to use GUID as the base for comparison. URL's are not reliable in our case.

Do not forget to use GUID as the base for comparison. URL's are not reliable in our case.
skobkin marked this conversation as resolved
telegram.py Outdated
@ -157,3 +158,3 @@
if self.sent_counter >= self.BATCH_LIMIT:
self.log.debug('Requests limit exceeded, sleeping for a second')
time.sleep(1)
time.sleep(2)
Collaborator

Why?

Why?
skobkin marked this conversation as resolved
skobkin added this to the MVP 0.1 milestone 2022-07-13 18:17:29 +03:00
fix removal feed, refactor return id and add type hints
Some checks failed
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is failing
f11b0b91af
database.py Outdated
@ -128,3 +128,3 @@
return self.cur.fetchall()
def find_user_feeds(self, user_id: int) -> list[sqlite3.Row]:
def find_user_feeds(self, user_id: int) -> list[DictRow]:
Collaborator

I forgot to tell about that earlier, but it's better not to expose specifics of database implementation in public methods.
We can fix that by returning non-library-specific structure. For example list[dict] or list[collections.OrderedDict] instead of list[psycopg2.extras.DictRow].

I forgot to tell about that earlier, but it's better not to expose specifics of database implementation in public methods. We can fix that by returning non-library-specific structure. For example `list[dict]` or [`list[collections.OrderedDict]`](https://realpython.com/python-ordereddict/) instead of [`list[psycopg2.extras.DictRow]`](https://github.com/psycopg/psycopg2/blob/1d3a89a0bba621dc1cc9b32db6d241bd2da85ad1/lib/extras.py#L160-L211).
skobkin marked this conversation as resolved
add __convert_to_list_of_dicts method
Some checks failed
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is failing
feef909c61
database.py Outdated
@ -186,1 +182,4 @@
)
@staticmethod
def __convert_to_list_of_dicts(rows: list[DictRow]) -> list[dict]:
Collaborator

Convert what? Maybe __dictrow_to_dict_list() или __dictrow_to_list_of_dicts()?

Convert what? Maybe `__dictrow_to_dict_list()` или `__dictrow_to_list_of_dicts()`?
Author
Owner

ok

ok
skobkin marked this conversation as resolved
rename __convert_to_list_of_dicts method to __dictrow_to_dict_list
Some checks failed
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is failing
50ac344095
fix update method in UpdateManager
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
ae31be33b6
refuse to unpack the dict in UpdateManager
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
6535055268
bug fix. add comparison by guid instead of url
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
c91549a311
not saving description in the database anymore
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
037272a167
not saving feed items title in the database anymore
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
61a2d8d215
little refactor in update_feed_items method
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
292a7a3c88
change time.sleep to 1
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2206c6299d
rename db_dsn to dsn variable in bot.py and update.py
Some checks failed
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is failing
ab77d44ba3
little fix in bot.by and update.py
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2bb167fef5
skobkin approved these changes 2022-07-14 01:52:15 +03:00
skobkin left a comment

Let's merge it!

Slap Shit Together and Deploy!

Let's merge it! ![Slap Shit Together and Deploy!](https://i.skobk.in/i/slap_shit_together_and_deploy.jpg)
Miroslavsckaya changed title from WIP: Migrate to PostgreSQL to Migrate to PostgreSQL 2022-07-14 01:52:45 +03:00
Miroslavsckaya deleted branch migrate_to_postgresql 2022-07-14 01:53:54 +03:00
Miroslavsckaya referenced this pull request from a commit 2022-07-14 01:53:54 +03:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Miroslavsckaya/tg_rss_bot!39
No description provided.