Windows keyboard scan codes

[ProbablePrime]

I was having a few issues with your module and some applications mostly video games. These applications won't respond to keyboard events triggered from robot.

After tracing this down and examining several other keyboard emulators(robotjs,Java's Robot Classes, My own Handler) I've pinned this down to using virtual key codes over scan codes for windows keyboard events. Which DirectInput/XInput utilizing projects seem to listen to a lot more reliably.

I've edited the keyboard code for robot to use scan codes and then tested it with the problematic applications which now see events from Robot.

Other software designed to interface with keyboard events also tend to use scan codes Examples being FreePIE and AutoHotKey

I was going to open a PR to convert to scan codes but thought some discussion might be better to start as virtual key codes might be useful for some cases. I can't find anything on the internet to suggest this.

From AutoHotKey it looks like we might be able to send both Virtual and scan codes at the same time. I can update things to do that too.

Do you have any thoughts?

[dkrutsko]

Hi, Thanks for your bug report! Somehow I feel like I've seen that before, probably in the java.awt.robot source code. But I sorta discarded it because I thought it was legacy code. Unfortunately I don't know anything about scan codes and in my tests, never came across the problem. Perhaps somebody more knowledgeable will be able to chime in.

It might be good to know which applications and games are breaking so that I can have something to test against. This will obviously come in handy for #16 as well.

As for the fix, please feel free to open a PR. With this new knowledge I will be able to look at other technologies again and apply the improvements needed to get everything working correctly. There's no rush, if you plan to look into this problem further, take as much time as you need to get a good solution in place.

And thanks again for pointing this out!

[ProbablePrime]

Oh sure, I forgot to list them.

Pretty much any AAA game title won't see inputs however the following have been tested:

• The Division
• Grand Theft Auto V
• Magicka

Oh and the Dolphin Emulator.

I'm looking for something light weight that we can use for testing.

I'll do some additional poking around and testing but if I don't find anything I'll open a PR and move the discussion there.

[dkrutsko]

Yeah I can see where GTA V would become problematic. They have three options of input, raw input (the default) being one of them. I'm not sure if scan codes would actually fix this issue for that game. Does AutoIt or java.awt.robot work with it?

[ProbablePrime]

awt.robot works for it yes. Let me fire up gtav with my changes and see if scan codes work.

@dkrutsko GTAV works with scan codes.

const keyboard = robot.Keyboard();
setTimeout(()=> {
    keyboard.click(68);
    keyboard.click(68);
    keyboard.click(68);
    keyboard.click(68);
}, 2000)

Rotated the character right a bit :D

[dkrutsko]

Okay that's good to hear. You may want to try with multiple input modes but I suspect it'll work fine.

[ProbablePrime]

I'm not entirely sure what you mean by input modes. If you can alaborate a bit i can try some stuff out.

[dkrutsko]

In the settings of GTA V there's a way to select the "input mode" or whatever it's called. I know there's a "raw input" mode which makes the input more responsive, but you can change it to other modes.

[ProbablePrime]

Sorry for the delay here, I noticed activity on the repo and remembered I needed to follow up.

@dkrutsko The library changes I made responded to all available input modes within GTAV.

Based on this I would suggest that my changes would allow Robot to interface with pretty much everything on windows.

Shall I open a PR?

[dkrutsko]

Yes please, but only if you're finished. I don't really have a set schedule for PR integrations but I'll roll everything out in one release. Everyone will have plenty of time to test everything out though so no worries.

[p120ph37]

We should not drop virtual-key support. Virtual-keys allow keyboard-layout-independant mapping. For example, if you want to type the literal letter "Z", regardless of whether the keyboard is German or English, the virtual-key mechanism allows this directly, whereas if you did it via scancodes, you'd have to first somehow detect the keyboard layout and then send different scancodes depending on the layout. This is the reason the Windows API provides both methods, and if Robot-JS does add scancode support, it should be done without removing the existing virtual-key support, since both have valid uses.

[p120ph37]

Also related (might be worth a separate issue?), the Windows API does support a third way of sending keyboard input: as a series of unicode characters. This is the method used by virtual-keybards (e.g. Swype) and speech-recognition (e.g. Cortana) to inject literal text regardless of physical keyboard region or capslock state. This would be a nice additional option to expose in Robot.

[p120ph37]

So, I've actually run into a case where this lack of scancode in the keyboard events is biting me. I've implemented the AutoHotKey way of doing it where the virtual-key is sent as the primary identifier, but the scancode (obtained via MapVirtualKey) is also populated. Reading the docs in detail, it seems that MapVirtualKey is keyboard layout-aware, so it should produce the correct scancode for the current layout whenever possible (or 0 if not possible, which is the same as Robot transmits currently).

I've opened #116 for this, and also added it (as well as #115) to my robot-js prebuilt fork (as tag 2.0.0-scancode-prebuilt), so it can be previewed by @ProbablePrime or others by setting the NPM dependency to: "robot-js": "github:p120ph37/robot-js#prebuilt"