What problem does this PR solve?

Issue Number: close #56113

Problem Summary:

What changed and how does it work?

add storeBalancer

	 [prepareAndSendJob]---jobToWorkerCh->[storeBalancer(optional)]->[workers]
	                     ^                                             |
	                     |                                             v
	                     |                           [storeBalancer(optional)]
	                     |                                             |
	                     |                                     jobFromWorkerCh
	                     |                                             |
	                     |                                             v
	               [regionJobRetryer]<-------------[dispatchJobGoroutine]-->done

and refine the components workflow. Now the behaviour of workflow is described in code comment

	// Above is the happy path workflow of region jobs. A job is generated by
	// prepareAndSendJob and terminated to "done" state by dispatchJobGoroutine. We
	// maintain an invariant that the number of generated jobs (after job.ref())
	// minus the number of "done" jobs (after job.done()) equals to jobWg. So we can
	// use jobWg to wait for all jobs to be finished.
	...

Previously I don't want to write many select {case <- ctx.Done: ..., case ...} for every channel action, so in the old code I assume the output channel must not be blocked (which means its reader/receiver must not exit earlier than writer/sender). But after I come back to this function after some months, I found it's hard to remember the different exit behaviour of components. So I changed them to new behaviours like comment said.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None