Improvements part 1 #89

New issue

Closed

opened 2025-10-16 23:26:55 +03:00 by skobkin · 0 comments

skobkin commented

2025-10-16 23:26:55 +03:00

Owner

This may be useful.

Add a dedicated “request processing” timeout in configuration (e.g. keep LLM_REQUEST_TIMEOUT, but expose structured subfields for chat, summary, image recognition) and centralise context creation: derive from reqCtx.Context() when available, wrap it with the configured deadline, and hand that context down to every helper (LLM calls, Telegram API, downloads).

Change the LLM connector to wrap underlying errors (return "", nil, errors.Join(ErrLlmBackendRequestFailed, err)) so handlers can detect errors.Is(err, context.DeadlineExceeded) and emit a specific “Timed out after Xs” reply. Hook this path into stats so we can measure how often timeouts occur.

Make image recognition lazy or throttled: either cache descriptions in MessageHistory, skip re-processing when MessageData.Image is already present, or only trigger recognition when we actually need to send history to the LLM. At minimum, reuse the per-request context for GetFile and replace tu.DownloadFile with a context-aware downloader to allow cancellation.

Wrap handlers in a helper such as runWithTimeout(chatID, func(ctx context.Context) error) that starts/stops typing indicators, executes the work, and handles timeout reporting in one place. This keeps command handlers consistent and makes it harder to forget user notifications.

Verify whether Ollama stops GPU work on HTTP cancellation; if it doesn’t, we may need to call its explicit cancel API (the /api/generate stop endpoint) once our context times out. That would require the connector to track request
identifiers.

Related items: #88, #71, #90

This may be useful. > - Add a dedicated “request processing” timeout in configuration (e.g. keep LLM_REQUEST_TIMEOUT, but expose structured subfields for chat, summary, image recognition) and centralise context creation: derive from reqCtx.Context() when available, wrap it with the configured deadline, and hand that context down to every helper (LLM calls, Telegram API, downloads). > - Change the LLM connector to wrap underlying errors (return "", nil, errors.Join(ErrLlmBackendRequestFailed, err)) so handlers can detect errors.Is(err, context.DeadlineExceeded) and emit a specific “Timed out after Xs” reply. Hook this path into stats so we can measure how often timeouts occur. > - Make image recognition lazy or throttled: either cache descriptions in MessageHistory, skip re-processing when MessageData.Image is already present, or only trigger recognition when we actually need to send history to the LLM. At minimum, reuse the per-request context for GetFile and replace tu.DownloadFile with a context-aware downloader to allow cancellation. > - Wrap handlers in a helper such as runWithTimeout(chatID, func(ctx context.Context) error) that starts/stops typing indicators, executes the work, and handles timeout reporting in one place. This keeps command handlers consistent and makes it harder to forget user notifications. > - Verify whether Ollama stops GPU work on HTTP cancellation; if it doesn’t, we may need to call its explicit cancel API (the /api/generate stop endpoint) once our context times out. That would require the connector to track request identifiers. Related items: #88, #71, #90

skobkin added the

enhancement

label

2025-10-16 23:26:55 +03:00

skobkin self-assigned this

2025-10-16 23:26:55 +03:00

skobkin referenced this issue from a pull request that will close it,

2025-10-17 01:59:11 +03:00

A bunch of improvements. #91

skobkin closed this issue

2025-10-17 01:59:45 +03:00