Added experimental background job queue #20985
Conversation
Added config flags for the job queue to make it switchable config options are in services:jobs:queue and are 'enabled,reportStats,reportInterval,maxWorkers,logLevel(info/debug),pollMinInterval,pollMaxInterval,queueCapacity,fetchCount.
|
It looks like this PR contains a migration 👀 General requirements
Schema changes
Data changes
|
github-actions
bot
added
the
migration
| @@ -985,7 +985,9 @@ module.exports = { | |||
| started_at: {type: 'dateTime', nullable: true}, | |||
| finished_at: {type: 'dateTime', nullable: true}, | |||
| created_at: {type: 'dateTime', nullable: false}, | |||
| updated_at: {type: 'dateTime', nullable: true} | |||
| updated_at: {type: 'dateTime', nullable: true}, | |||
| metadata: {type: 'text', maxlength: 1000000000, fieldtype: 'long', nullable: true}, | |||
There was a problem hiding this comment.
Do you have a sense of how big we'd expect this field to actually be? I wonder if allowing a maxlength this large is potentially encouraging misuse of this field?
There was a problem hiding this comment.
I originally had it like other text fields at 2000 chars. I think that ought to be sufficient, but this is why I was holding off on the migration. We could start with 2000 and go from there?
There was a problem hiding this comment.
Yeah 2000 seems more sane to me — in most cases this should basically be a resource ID and maybe a couple small pieces of metadata (e.g. a timestamp or two, maybe a url pointing to a file or storage bucket, etc.)
There was a problem hiding this comment.
Cool, I'll bump back. Basically, since this is a 'JSON' field I had copied what we had elsewhere. Agree it seems to be overkill and it's easier to bump up than down.
There was a problem hiding this comment.
Adjusted. If I mark this resolved, does it also resolve it for you? I forget how GH handles that.
| @@ -42,7 +42,7 @@ const initTestMode = () => { | |||
| }, 5000); | |||
| }; | |||
|
|
|||
| const jobManager = new JobManager({errorHandler, workerMessageHandler, JobModel: models.Job, domainEvents}); | |||
| const jobManager = new JobManager({errorHandler, workerMessageHandler, JobModel: models.Job, domainEvents, isDuplicate: true}); | |||
There was a problem hiding this comment.
Not a blocker but just a note for future: it would be good to get to the bottom of why we need two instances of the job system, and ideally fix that so we don't need this anymore
There was a problem hiding this comment.
Agreed! I've got a note to look into this because this is rather clunky way of handling it that I'm also really not a fan of.
| * | ||
| * @returns {Promise<void>} | ||
| */ | ||
| async startQueueProcessor() { |
There was a problem hiding this comment.
Just for the sake of readability and testability I think it might be good to break this bad boy up a bit, it's a little hard for me to follow in its current form.
There was a problem hiding this comment.
I did a pretty significant refactor of that entire class to break it up a lot, and expose more so testing is easier.
This is slightly less efficient - potentially - because we do a select before inserting instead of ignoring the insert conflict. It's likely less efficient because I don't anticipate a ton of duplicates, although the analytics job can certainly result in that, which is where I'd expect the knex implementation to slightly win out. Regardless, testing is a fucking nightmare with knex directly as we have to spin up a db and use a schema for the table. Let's go down that path later if we need the performance improvements.
|
Need to add a couple integration tests then we ought to be ok. Not sure what the failure for Ghost-CLI is. |
|
It looks like there's an error that's preventing the persisted logs from being printed 🙃 Here's the culprint:
Looking at the lockfile changes, I'm not sure if they're intentional - there are a lot of new dependencies, which I think might stem from an older version of |
- moved to a more appropriate location (integration tests from e2e-server)
|
Ok, clearly there's some kind of race condition with the config mocks and initializing the service. I will have to look into that when I can reprioritize this. |
ref ???