Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OperationalError: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac #60

Open
sentry-io bot opened this issue Dec 14, 2020 · 8 comments
Labels
Backend bug Something isn't working URLHaus

Comments

@sentry-io
Copy link

sentry-io bot commented Dec 14, 2020

Sentry Issue: CHECKMATE-6

OperationalError: SSL error: decryption failed or bad record mac

  File "sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
  File "newrelic/hooks/database_psycopg2.py", line 65, in execute
    **kwargs)
  File "newrelic/hooks/database_dbapi2.py", line 39, in execute
    *args, **kwargs)

OperationalError: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac

[SQL: INSERT INTO urlhaus_rule (id, hash, rule) VALUES (%(id_m0)s, %(hash_m0)s, %(rule_m0)s), (%(id_m1)s, %(hash_m1)s, %(rule_m1)s), (%(id_m2)s, %(hash_m2)s, %(rule_m2)s), (%(id_m3)s, %(hash_m3)s, %(rule_m3)s), (%(id_m4)s, %(hash_m4)s, %(rule_m4)s), (%(id_m5)s, %(hash_m5)s, %(rule_m5)s), (%(id_m6)s, %(hash_m6)s, %(rule_m6)s), (%(id_m7)s, %(hash_m7)s, %(rule_m7)s), (%(id_m8)s, %(hash_m8)s, %(rule_m8)s), (%(id_m9)s, %(hash_m9)s, %(...
(11 additional frame(s) were not displayed)
...
  File "sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
  File "newrelic/hooks/database_psycopg2.py", line 65, in execute
    **kwargs)
  File "newrelic/hooks/database_dbapi2.py", line 39, in execute
    *args, **kwargs)
@seanh seanh added this to the 1 - Implement a checking service milestone Dec 14, 2020
@jon-betts
Copy link
Contributor

Some possible thoughts on this: https://medium.com/@philamersune/fixing-ssl-error-decryption-failed-or-bad-record-mac-d668e71a5409

This seems to be to do with starting a connection in one thread, and using it in another.

@jon-betts jon-betts added the bug Something isn't working label Dec 14, 2020
@jon-betts
Copy link
Contributor

So looking at the trace above, this isn't coming from our code at all, but from the New Relic introspection of our tasks. This is where the other thread is. So my guess is if we get a reporting period from New Relic while we are trying to update, it explodes.

We'll probably have to do a minor version of what we did in the WebSocket to turn off transaction tracing:

https://docs.newrelic.com/docs/agents/python-agent/configuration/python-agent-configuration#txn-tracer-settings

@jon-betts
Copy link
Contributor

I've tried disabling New Relic transaction monitoring, but we're going to have to wait and see if anything happens.

@jon-betts
Copy link
Contributor

This is still working with the transaction monitoring disabled, but New Relic is still monkey patching Postgres, so we're going to try disabling it completely and see what happens.

@jon-betts jon-betts modified the milestones: 1 - Implement a checking service, 2A - Checkmate follow up work Jan 6, 2021
@seanh seanh modified the milestones: 2A - Checkmate follow up work, Add a blocklist to both public Via and LMS's Via Jan 14, 2021
@seanh seanh closed this as completed Jan 14, 2021
@jon-betts
Copy link
Contributor

Currently deploying the PR which removes New Relic, just to rule it out of being the source. Unfortunately we aren't going to know for a while / ever, because this is very intermittent. We can go a whole week without seeing this sometimes.

@jon-betts
Copy link
Contributor

I've deployed this now and closed the Sentry issue, but we're going to have to check on this for at least a week if it doesn't just show up immediately.

@seanh
Copy link
Contributor

seanh commented Jan 22, 2021

@jon-betts Re-opening this because it happened again: https://sentry.io/organizations/hypothesis/issues/2086421688/ (latest event is today just after midnight, 12:01:08.

It looks like New Relic is innocent here, and we should revert #87 and #83?

@seanh seanh reopened this Jan 22, 2021
@jon-betts
Copy link
Contributor

Yep

@seanh seanh self-assigned this Feb 16, 2021
@seanh seanh removed this from the 1. Add a blocklist to public Via and LMS's Via milestone Feb 16, 2021
@seanh seanh removed their assignment Feb 16, 2021
@seanh seanh added the URLHaus label Mar 5, 2021
robertknight added a commit that referenced this issue Apr 14, 2021
This was disabled in an attempt to fix #60
but the issue still occurred even when New Relic was disabled. This
re-enables the setting per #60 (comment)
and #296 (review)
seanh added a commit that referenced this issue Apr 14, 2021
seanh added a commit that referenced this issue Apr 14, 2021
seanh added a commit that referenced this issue Apr 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Backend bug Something isn't working URLHaus
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants