1. Computing

Automating with Robot


Robot is a class provided by AWT (the base windowing toolkit in Java) to automate mouse movements and keystrokes at the native level. This can be used from anywhere in your Java applications, whether they have visible windows displayed or not, so this can easily be used to send virtual keystrokes from your web application to the server's operating system.

The first question you may be asking is why you would want to do that. This web application is not meant to be run on a web server, it's meant to be run on a desktop computer, laptop or other Windows computer running fullscreen applications (games, for example). From a second computer you can just touch a few buttons and trigger actions on the desktop computer without having to remember complicated keystrokes or break from your fullscreen game. The primary use of this web application is to control media players, chat programs and the like while playing games and streaming on Twitch.tv.

JRuby Crash Course

If you're familiar with JRuby, feel free to skip this section. If you're familiar with JRuby, but are not sure why we're using it, see Why JRuby? JRuby is not only a Ruby interpreter written in Java, but it's also a dead simple bridge to any and all Java code. It's just as easy to access actual Java classes as it is to access Ruby classes.

To get started, use the java_import method. The java_import method imports a Java class into your Ruby program. From that point forward, you can use that Java class just as you would a Ruby class. So, we'll start by importing the two classes we'll need.

java_import 'java.awt.event.KeyEvent'
java_import 'java.awt.Robot'

From this point onward, we have a Robot class to instantiate Robot objects, as well as the KeyEvent class that holds constants we'll need to refer to the key names.

Using Robot

Using Robot is very simple. First, instantiate a Robot object.

robot = Robot.new

There are two versions of the Robot constructor. The first takes no arguments, just instantiate it and you're good. On systems with multiple graphics systems, you'll need the second variant. As this is almost never needed, including on Windows systems with multiple displays, it is not covered here.

Now that you have your Robot object, it's time to simulate some keypresses. Robot is a rather low-level interface, so you'll be required to call both a "key down" event as well as a "key up" event, just as the events are fired from your keyboard as you press and release keys. For most uses, there's no need to add a delay between the two calls. Most applications use an event-based system to scan for key presses, and they'll see both the key down and key up events no matter how quickly they happen. Some applications, on the other hand, get the state of the entire keyboard every so often and look for keys that are currently pressed. While no Windows applications do this to my knowledge, it used to be common practice in older games and some applications, so we'll use an optional delay between the two calls.

And of course we don't want to do this all the time, so we'll write a method to handle it for use. It takes a single KeyEvent object and an optional delay between the key up and key down events. For a list of keys that can be pressed using Robot, see the KeyEvent documentation. Also, we'll take advantage of Ruby's open classes and add our simulate_key method directly to the Robot class.

require 'java'
java_import 'java.awt.Robot'
java_import 'java.awt.event.KeyEvent'

class Robot
  def simulate_key(ev, delay=nil)
	sleep(delay) if delay

r = Robot.new

Note that since we're inside of the Robot class, the key_press and key_release methods refer to Robot#key_press and Robot#key_release methods respectively.

More Robotics

Robot can do other things as well. While Robot was really intended for automated testing of GUI applications, there are still some useful methods we can use here.

  • move_move, mouse_press and mouse_release - These methods can simulate mouse movements and mouse button presses. Though mouse movements are in absolute screen coordinates, it can still be used to automate some things, some trial and error and accurate window positioning may be needed though.

  • create_screen_capture - Here, you can grab the contents of the screen and send it back as an image. Very useful if you're trying to automate a machine remotely and need to see the screen. In just a few lines of code, you can hack up a quick poor-man's VNC.

  1. About.com
  2. Computing
  3. Ruby
  4. JRuby
  5. Automation of Windows using FFI
  6. Automating with Robot

©2014 About.com. All rights reserved.