OculiX | Blog

The engine assumed one screen. The job needed four.

Fri, 22 May 2026 10:00:00 GMT

Seven hours. That’s how long a visual test suite for a Fortune 500 retail group used to take — roughly 300 tests driving point-of-sale terminals, run one after another, against a regulatory compliance window that does not move.

Today it finishes in under two. Four sessions in parallel, on a single 8 GB Azure VM running Windows Server 2019 — a swap file, basic provisioning, nothing exotic. Runtime divided by five. No test farm, no per-minute cloud bill, no new hardware.

The speed is the headline. It’s not the interesting part. The interesting part is that the engine was never built to do this — and neither is almost any visual-automation tool. They assume one screen. I needed four. This is the story of the gap between those two numbers, and what I had to build to close it.

A world of one screen

Visual automation was born in a world of one physical screen. One machine, one desktop, one cursor. SikuliX — which OculiX continues — inherited that worldview, and for fifteen years it was the correct one. The engine keeps a notion of the current screen, and very reasonably it keeps it in one place: shared, global state. When there is only ever one screen, a global is not a flaw. It’s the right design.

That assumption is invisible right up until the moment you break it. VNC breaks it.

A VNC session isn’t a screen — it’s a network framebuffer. No monitor, no desktop login, no DISPLAY. You can open a dozen, each on its own port, each pointing at a different remote machine. Suddenly “the current screen” stops being a singleton and becomes a question: which one?

The collision

The first time I declared two sessions and ran two test flows at once, they walked all over each other. Not with an error — worse, with quiet nonsense. Clicks landing on the wrong register. A framebuffer read mid-repaint. Results that came back green for entirely the wrong reasons.

I tested it carefully, because I didn’t believe it at first. Two flows, two ports, two remote machines. They polluted each other every single time.

The robot and the client were per-instance — that part was clean. But the port and the registry of live sessions lived in shared, static state. Two threads reaching for the same global. The single-screen assumption, baked into the data model, surfacing at exactly the moment I asked for more than one screen.

What I built — around the engine, not inside it

So I built the missing layer myself, on top of the engine rather than in it. Four pieces, each one fixing a failure I had actually watched happen.

Isolation per flow. Each test flow got its own session, bound to its own thread — its own port, its own remote, its own everything. No flow could see another’s state, because there was no shared “current session” left to see. The engine didn’t hand me thread-scoped sessions, so I made them thread-scoped from the outside.

A port is a promise. Two flows must never claim the same port, and “probably free” is not good enough at startup when four of them race. So allocation became atomic: a small dedicated range, and a rule — a flow claims a port through a lock no two flows can win simultaneously, or it doesn’t run at all. A port stops being a guess and becomes a contract.

One tunnel per session. Each remote register was reachable only through an SSH tunnel exposing its framebuffer on a local port. One tunnel per flow, opened on demand, torn down after — and, the detail that cost me a late night, closed before the next one opened. Orphaned tunnels hold ports hostage long after the process that spawned them is gone. Close-before-open, not close-after. Order matters.

Readiness, not optimism. The engine waited a fixed few seconds for a session to come up and hoped for the best. Hope does not survive four sessions starting at once. So I waited for proof instead: the RFB banner — the three bytes a VNC server sends to announce it’s ready for pixels. No banner, no test. A probe, not a sleep.

Why four sessions gave 5×, not 4×

Four workers, five times faster, looks like broken arithmetic. It isn’t.

The killer in a sequential suite isn’t the total work — it’s the variance in test durations. My ~300 tests ranged from a few seconds to many minutes. Run in a single line, a long test sits in front of forty short ones and they all just wait. Sequential execution doesn’t only pay for the work; it pays for every test queued behind a slower one. With uneven durations that waiting tax is enormous, and invisible, because the CPU is idle the whole time.

I split the load into batches sized by functional area — thirty tests here, ninety there, a hundred and twenty somewhere else, the way the business actually reasons about its registers. Balanced, not just parallel. The four sessions finished at roughly the same time instead of one straggler holding the line.

So the 5× isn’t four workers doing 4× the work. It’s four balanced workers deleting waste the single queue was silently burning. When the thing you replaced is that wasteful, even modest, well-distributed parallelism overshoots the naive ratio.

Why all of this lived outside the engine

I want to be precise here, out of respect for the engine and the people who built it. The core does one thing extremely well: clean, reliable, single-session VNC. It kept a single-screen promise and it kept it honestly. The orchestration — the thread isolation, the atomic ports, the tunnels, the readiness probe — was mine, and it lived in my harness. I made a different promise, one layer up, and kept that one too.

For a long time that was the right division of labour. The engine stayed pure; the parallelism was an application concern. But once you’ve named the gap, you can’t un-see it — and you start to wonder why the engine shouldn’t simply do this itself.

The roadmap: making parallel sessions first-class

Here’s where it gets exciting, because the next step is obvious now that the gap has a name. Everything I built outside the engine is the engine’s natural next form.

Thread-scoped sessions

No more global “current screen.” A session belongs to the flow that opened it, by construction — so two flows can never reach for the same state, because there is no shared state to reach for.

Port allocation, built in

The engine hands you an isolated session and owns the atomic port claim itself. No range to reserve, no lock-files to babysit, no collisions to debug at 23h.

Readiness on the RFB banner

The optimistic startup sleep replaced by a real probe — the session reports ready when the server actually says it’s ready, not when a timer guesses it might be.

A clean SSH tunnel, already shipped

A pure-Java tunnel (no WSL, no external sshpass) already lives in the core. The next step is wiring it into a documented, first-class parallel pattern.

The goal for the next major version is simple to state and worth the work: declare N sessions, get N isolated sessions. No harness, no lock-files, no late nights over ghost tunnels. The single-screen assumption retired — gently, and with full respect for the fifteen years it served exactly right.

OculiX is MIT, open source, built in the open. If you drive screens for a living — registers, kiosks, terminals, application instances by the dozen — this is the corner of the roadmap I’d most like company on.

The takeaway

Visual automation has a reputation for not scaling. It earned that reputation honestly: the tools were built for one screen, and most of them still are. But the limit was never the pixels, and it was never the matching. It was an assumption — one screen — that nobody had yet needed to question.

Question it. Give the engine real sessions, isolate them, balance the load across uneven work — and a bored 8 GB VM does in two hours what used to take seven. No farm. No new hardware. No per-minute bill.

The screens were never the bottleneck. The assumption was.

OculiX is open-source under the MIT license. The parallel VNC engine — and the work to make multi-session parallelism native in the core — is described in the Parallel VNC execution guide.

CodeQL: the boy who cried Error

Thu, 21 May 2026 08:00:00 GMT

We enabled GitHub CodeQL on OculiX in April 2026. The repository inherits sixteen years of Sikuli and SikuliX history before our continuation began, so the scanner was pointed at roughly 110 000 lines of Java accumulated since 2009.

It returned a hundred or so findings across all severities. Nineteen of them were classified Error, the highest tier. The classification suggests urgency: each Error is, in CodeQL’s taxonomy, the kind of thing a maintainer should drop everything to fix.

We dropped nothing. We triaged each one. After triage, the result is unambiguous: zero of the nineteen findings have caused an observable user incident in fifteen years. Most of them are dormant architectural defects, dead code paths, or false positives produced by structural blindness in the analyzer.

Meanwhile, fifty-three missing catches for NumberFormatException — places where parsing a user-provided number can crash the application — are classified Note. Almost invisible. Below Warning. Far below Error.

This post walks through the nineteen findings, explains why none of them actually warrant the Error tier, and explains why the things that should warrant it are buried two levels down. The conclusion is not that static analysis is useless. The conclusion is that CodeQL’s severity classification is broken in both directions — overstating the trivial, understating the serious — and that the noise this generates is the main thing keeping the tool from being useful at scale.

Methodology

For each finding, we asked three questions:

Does it produce an observable user incident? Has it caused a crash, a wrong result, a regression in fifteen years of use? Has anyone filed a ticket against it?
If not, would it produce one if a specific code path were exercised? Is it dormant but reachable, or dormant and unreachable?
Is the finding even correct? Does CodeQL’s analysis actually match the runtime behavior, or has it missed something about the language, the bridge to another runtime, or the type system?

Three categories emerged from this triage:

Architectural defects

Real problems in the code that have never been triggered. Either the branch is dead, the configuration is never set, or the upstream dependency is never invoked. The finding is correct but the impact is theoretical.

Code smells

Patterns that look suspicious but have legitimate uses (or, more often, have just persisted as harmless cruft for over a decade). The code is ugly but not broken.

False positives

CodeQL’s analysis is wrong. It missed a dynamic access, a language bridge, an inheritance relationship, or a type guarantee. The finding does not correspond to any real risk in the code.

What we did not find in our nineteen Errors: any case where a user reproduced a crash, a wrong result, or a regression caused by the code at the location CodeQL flagged. Not one.

A tour of the nineteen Errors

Below is the breakdown by CodeQL rule, in descending count.

Container contents are never accessed (8 findings)

CodeQL’s claim: a collection or map is allocated and possibly written to, but its contents are never read back. The implication is that the container is dead code that should be removed.

Eight findings of this kind landed on our codebase. The triage:

Three are genuine dead code. Two local variables named classpath inside ExtensionManager.java, and a local map docParams inside Crawler.java. All three are declared, populated in a loop, and never consumed. Cleanup candidates. None of them have produced a single user-visible defect.

Four are false positives caused by dynamic access. Lists like aliases, filenames, mimeTypes in Lexer.java (the syntax highlighter grammar) are populated by parsing grammar metadata and consumed by a registry mechanism that CodeQL cannot trace. A map called libsLoaded in RunTime.java tracks loaded native libraries and is consulted before each native load — but the consumption goes through a debug helper that CodeQL’s call-graph analysis does not follow. From the static point of view, the containers look unread. From the runtime point of view, they are read on every invocation.

One is the most interesting finding in the entire set. SikulixServer.allowedIPs is declared like this:

private static List<String> allowedIPs = new ArrayList<>();

The class is the Sikuli network server. The list is named after a security feature: an IP allow-list to restrict who can connect. There is a method called makeAllowedIPs that populates the list. There is no code that reads the list. No filter checks against it. No connection-handling code consults it. The allow-list is built but never enforced.

If an administrator configures allowedIPs thinking they are restricting access, they are not. The server accepts every connection regardless of the configured allow-list, because the list is allocated, filled, and never read.

This is a security defect. A serious one, by any reasonable definition. CodeQL classifies it as Maintainability — Error. Not Security. Not Critical. Just “your container is unused”, as if the impact were cosmetic.

We have opened an internal issue to investigate whether the network server is reachable in any default OculiX deployment, and to either remove the network server entirely or wire up the missing filter. As of this writing, the defect has not produced an incident report in any of the years it has existed — but that is more an accident of the network server’s low usage than a vindication of CodeQL’s classification.

Self assignment (2 findings)

CodeQL’s claim: a variable is being assigned to itself. The assignment has no effect.

The first finding is in Runner.java:396:

public static void setLastScriptRunReturnCode(int lastScriptRunReturnCode) {
    lastScriptRunReturnCode = lastScriptRunReturnCode;
}

The setter’s parameter shadows the static field. The author intended Runner.lastScriptRunReturnCode = lastScriptRunReturnCode; or this.lastScriptRunReturnCode = lastScriptRunReturnCode;. As written, the setter is inert. It has been inert since whenever it was added — almost certainly in the mid-2010s, possibly earlier.

For fifteen years, any caller invoking Runner.setLastScriptRunReturnCode(5) has stored exactly nothing. The field lastScriptRunReturnCode stays at its initial value of 0 forever. Anyone reading the field gets 0 regardless of what scripts have actually returned.

Has anyone noticed? Apparently not. Either the field is never consulted, or the consumers do not depend on the value being non-zero, or the corner cases where it would matter have not been hit in fifteen years of cumulative production use. The “bug” is real in the source code, but the observable behavior is no different from what it would be if the setter worked correctly. The downstream code path has adapted to the dysfunction.

The second finding is in Deflate.java:1703, inside com.jcraft.jzlib (the JZlib compression library, embedded for SSH purposes):

Deflate dest = (Deflate) super.clone();
dest.pending_buf = dup(dest.pending_buf);
dest.d_buf = dest.d_buf;
dest.l_buf = dup(dest.l_buf);
dest.window = dup(dest.window);

The clone() method deep-duplicates every buffer except d_buf. The assignment dest.d_buf = dest.d_buf; is a typo where dup(dest.d_buf) was intended. The result is that the cloned Deflate instance shares its d_buf with the original. Modifying the buffer through either reference is visible through both.

The defect is in an upstream library that nobody has actively maintained for years. The OculiX project carries it because the library is bundled. The risk is real but only materializes if the Deflate.clone() method is called in a context where both the original and the clone are used concurrently — a rare pattern that JZlib’s tests apparently do not cover and that nobody seems to hit in production.

Both Self assignment findings are genuinely correct. Both have been silently dormant in the codebase for over a decade. Neither has produced a user-visible incident. CodeQL classifies them as Error. The classification suggests they require immediate attention. The empirical evidence suggests they have been irrelevant for a decade.

Reference equality test of boxed types (1 finding)

In RecordedEventsFlow.java:149:

} else if (!modifiers.isEmpty() && characterWithModifiers == character) {
    actions.add(new TypeTextAction(keyText, modifiersTexts));
}

characterWithModifiers and character are boxed Integer (or Character) values. Comparing them with == compares object references, not values. Java caches boxed integers from -128 to 127, so for any ASCII codepoint, two boxed instances of the same value point to the same cached object, and == returns true by accident. For codepoints above 127 — accented Latin, CJK, Cyrillic, Arabic, Hindi, almost any non-ASCII script — the cache does not apply, and == returns false even when the values are equal.

The defect is correct. It has existed for years. The empirical impact is invisible because:

The recorder is used predominantly with ASCII input in the regions where it is most used.
When the comparison fails for non-ASCII input, the recorder falls through to a different branch that produces a slightly different — but still functional — action.

This is a textbook example of a real defect that has been silently absorbed by the surrounding code’s tolerance. No user has filed a ticket. No CI test has caught it. The fix is one line (.equals() instead of ==). The classification as Error is consistent with CodeQL’s rule but inconsistent with the actual user impact, which is zero.

Equals method does not inspect argument type (1 finding)

In PatternPaneScreenshot.java:36:

@Override
public boolean equals(Object o) {
    return false;
}

CodeQL warns that the method does not check the argument’s type, and might therefore lead to a ClassCastException somewhere downstream. The implementation is trivial: it returns false for any input, including null. There is no cast. There is no downstream usage of the result that depends on the argument’s type.

The override exists inside a Comparator declaration. The Comparator interface allows equals to be overridden to indicate that two comparators impose the same ordering. Returning false is a safe, conservative default: “I do not claim to impose the same ordering as you do.” It does no harm.

The finding is a false positive on this specific code. The override is pointless (it could be removed without changing behavior), but it is not a defect. CodeQL classifies it as Reliability — Error, suggesting it could cause a crash. It cannot. The implementation has no crash path.

Non-callable called (2 findings, Python)

Two findings on Sikuli.py, the Jython-based scripting layer that ships with OculiX. Both report that an imported name is being called as if it were callable when it is not.

import org.sikuli.script.ScreenRemote as SR
SCREEN = SR(adr, str(port))

The CodeQL Python analyzer treats this as CPython. It sees import org.sikuli.script.ScreenRemote as importing an attribute, has no idea that ScreenRemote is a Java class, and concludes that calling SR(...) is invalid.

In Jython, this import is valid Java-to-Python bridging. SR(...) invokes the Java constructor ScreenRemote(String, String). The runtime knows this. CodeQL does not.

This is structural blindness across language runtimes. CodeQL Python has no awareness of the Jython platform. Every Java class accessed from a Jython script will be flagged as “non-callable called”. The number of such findings will grow linearly with the size of the scripting layer.

Both findings are false positives. There is no fix on the project side. The recourse is to disable the rule in .codeql.yml or to suppress the findings manually one by one.

Dereferenced variable is always null (1 finding)

In ButtonGenCommand.java:127:

if (pref.getAutoCaptureForCmdButtons()) {
    btnCapture = null; //TODO new ButtonCapture(pane, line);
    pane.insertComponent(btnCapture);
    btnCapture.captureWithAutoDelay();
}

The author explicitly assigned null, left a TODO comment, and then dereferenced the variable twice. Whoever opens the IDE with the AutoCaptureForCmdButtons preference enabled will crash on btnCapture.captureWithAutoDelay().

This is the only finding in the set that is a definite crash trap. CodeQL is correct to flag it. Calling it Error is, for once, defensible.

Has anyone hit the crash? The preference is off by default, and the TODO has lived in the code for years, which suggests nobody enables this preference in practice. The crash is a real risk but a dormant one. Fix: replace the null with new ButtonCapture(pane, line) and finish the TODO.

Contradictory type checks (1 finding)

In Offset.java:152:

public <RMILO> Offset(RMILO whatEver) {
    if (whatEver instanceof Region || whatEver instanceof Match) {
        Region what = (Region) whatEver;

CodeQL claims that when whatEver is Match but not Region, the cast to Region will fail at runtime. This would be true if Match and Region were unrelated types. They are not. Match extends Region throughout the Sikuli class hierarchy. Every Match is also a Region. The cast is safe.

CodeQL does not consult the inheritance graph of the project, so it treats the two types as disjoint. The OR condition is redundant (whatEver instanceof Region already covers the Match case), but it is not unsafe.

The finding is a false positive caused by type system blindness. The check on Match is dead, in the sense that it is always reached on objects that already match instanceof Region. Removing it is a cleanup, not a bug fix.

Type mismatch on container modification (1 finding)

In HotkeyManager.java:263:

boolean res = _instance._removeHotkey(key, mod);
if (res) {
    hotkeys.remove(key);
}

hotkeys is a Map keyed on String. key is Integer (an AWT virtual key code). Map.remove(Object) accepts any Object for backward compatibility with pre-generics Java, so the call compiles cleanly. At runtime, no entry in the map has an Integer key, so the remove silently does nothing. The intended key for the map is a textual representation that must be constructed from key and mod.

The hotkey is reported as removed (because _removeHotkey succeeded at unbinding the OS-level listener), but the entry stays in the map indefinitely. If hotkeys is consulted to dispatch incoming events, the old hotkey’s listener continues to fire.

The defect is real and observable in theory. In practice, no user has filed a ticket about a hotkey persisting after rebind. Either:

The listener-dispatch path does not consult the map.
Nobody rebinds hotkeys.
The fallout is silently absorbed by other code.

Whichever explanation applies, the failure mode of this defect is dormant in the same way as the others. CodeQL’s classification as Error matches the technical risk but overstates the empirical urgency.

Missing format argument (2 findings)

In Commons.java:1525:

RunTime.terminate(999, "Commons.loadLib: %s");

The format string demands an argument, none is supplied. When this line is reached, the String.format call throws IllegalFormatException. The thrown exception is wrapped inside the terminate call, which means the user sees a confusing format-related error instead of a clean shutdown with code 999 and a useful diagnostic message.

The branch is an error path inside the native library loading code. It executes when a library fails to load — itself a rare event. The defect aggravates the user experience precisely at the moment when it is already poor (something is breaking, and now the error reporting is also breaking), but it does not produce its own incident on a healthy system.

The second finding is Crawler.java:441:

docMethod = String.format(pyMethod, clazz, method, docParams, returns);

The format string pyMethod expects five arguments, four are supplied. Every invocation of this method throws IllegalFormatException. The Crawler appears to be a documentation generator that has not been run in a long time — the API it scrapes for has changed, but the generator has not been touched to follow. The defect is invisible because the tool is no longer used.

Both findings are real defects. Both are essentially dead code (one on a rare error path, one in an unused tool). Their classification as Error is technically correct and operationally misleading.

What the classification actually looks like

The total: 19 findings classified as Error. After triage:

Category	Count	Share
Dead-code paths with crash potential	4	21 %
Architectural defects with dormant impact	8	42 %
False positives (structural blindness)	7	37 %
Findings with observed user incidents	0	0 %

Zero observed user incidents in fifteen years across all nineteen findings.

Meanwhile, in the layers below Error, CodeQL reports the following counts on the same codebase:

Rule	Findings	Classified
Missing catch of NumberFormatException	53	Note
Dereferenced variable may be null	38	Warning
Potential input resource leak	24	Warning
Ignored error status of call	23	Note
Implicit conversion from array to string	11	Note
Potential output resource leak	11	Warning
Use of default toString()	3	Note
Non-synchronized override of synchronized method	2	(below visible threshold)

Fifty-three places where parsing a user-provided number can crash the application. Classified Note. Almost invisible.

Twenty-four places where a file or network resource may not be closed, leading to file descriptor exhaustion on a long-running server. Classified Warning. Below Error.

Two places where overriding a synchronized method without re-applying synchronized creates a race condition in multi-threaded code. Buried so deep it does not even appear in the default UI summary.

Where the severity system breaks down

The pattern across the data is unambiguous. CodeQL’s Error tier is dominated by structural patterns that may produce defects in extreme cases but routinely do not. The Warning and Note tiers contain patterns that frequently produce real production incidents on real systems. The mapping from classification to operational urgency is inverted.

The reasons appear to be structural to how the tool reasons:

Definite vs probable

CodeQL favors definite analytical conclusions (“this is always null”, “this assigns to itself”) over probabilistic ones (“this might be null”). Definite conclusions get Error. Probabilistic ones get Warning or Note. But probable failures on 53 lines of code matter more than definite failures on 2 dormant lines.

Pattern-level severity

Every instance of a rule inherits the rule’s severity. There is no per-instance reasoning. A self-assignment that is dead code and a self-assignment that breaks a hot path receive the same Error tier. The reviewer cannot distinguish them from the severity column.

Cosmetic and structural together

Rules like “Container contents are never accessed” cover three very different real-world cases: dead variables, dynamically accessed structures, and security-critical filters that fail open. All three receive Maintainability — Error. The CISO reviewing the report cannot find the security one without reading every finding.

No cross-rule prioritization

A self-assignment with no impact is Error. A NumberFormatException not caught on user input is Note. There is no mechanism in CodeQL to bring the second above the first in any output. The order of rules in the output is alphabetic or chronological, never by operational risk.

The combined effect is that a developer who opens CodeQL on a mature codebase sees a few dozen “Errors” — most of them irrelevant — and a few hundred “Notes” — among which the genuinely concerning ones are buried. The natural response, after the third or fourth review, is to stop opening CodeQL.

What CodeQL actually catches well

This post would be unfair if it described only the failure modes. CodeQL does some things well, and the things it does well are worth naming.

Definite null dereference (when the analyzer is sure, not “may be”). The ButtonGenCommand finding is a real crash trap. The rule has high precision.

Type-incompatible container modification. The HotkeyManager finding is a real bug that compiles cleanly and silently misbehaves at runtime. Without static analysis, this kind of generic-erasure pitfall is genuinely hard to catch.

Format string desync. Both Missing format argument findings are real. The rule has very high precision because the analysis is local.

Self-assignment. Both findings are real. The rule has perfect precision because there is no plausible legitimate use.

What unites these is that they are narrow, local, and high-confidence rules. CodeQL is at its best when the pattern is sharply defined and the analysis is bounded. It is at its worst when it generalizes — across rules, across severities, across code paths it cannot follow.

How we configured CodeQL to be useful

After triage, the actions we took for this scan:

Suppressed the Jython-related rules entirely. Non-callable called and a few other Python rules produce structural false positives on every Jython-bridged file. There is no fix on our side. Suppressing the rule per-file via comments is more cost than benefit. We disable the rule in .codeql.yml.
Filed an internal issue for SikulixServer.allowedIPs to investigate whether the network server is reachable and to either fix the missing filter or remove the server. The finding sat under Maintainability — Error, where nobody who triages security findings would look for it.
Treated the 53 NumberFormatException findings as a focused remediation backlog, separately from the Error-tier findings. They are not the same kind of work and should not be in the same queue.
Documented our dismissals. For every Error-tier finding we dismissed, we wrote a one-line justification in the commit that closed the alert. This is the documentation our future selves will need when CodeQL re-flags the same finding after a re-scan.
Did not aim for zero Errors. Zero CodeQL Errors is not a meaningful goal. A meaningful goal is zero unjustified Errors and a tracked, finite list of structural issues being worked on.

Why we still keep CodeQL on

Despite all of the above, CodeQL stays enabled on OculiX. The OpenSSF Scorecard checks for a SAST tool integration and rewards projects that have one. Procurement teams in regulated sectors check that CodeQL or an equivalent is configured. The signal value to outside observers is real, even when the operational value is poor.

But the cost of running CodeQL is also real: the triage time, the alert fatigue, the temptation to ignore the tool entirely. We pay that cost in exchange for the external signal. We do not pretend that what we get back internally is anywhere near worth the marketing of static analysis.

If you maintain a similar codebase and you are considering whether to enable CodeQL: enable it, run it once, do the triage we just walked through, and then decide rule-by-rule which queries you actually want to run. The default query suite assumes you have time to read several hundred findings. Most teams do not. The most useful configuration is the one that leaves you with twenty findings a week, all of which are worth opening.

The boy who cried Error eventually had a wolf. We will probably see one, eventually, in a CodeQL scan. We will recognize it because it will be the only finding in months that we cannot dismiss in a single line. Everything else will continue to be the noise that the wolf is hidden in.

Repository: github.com/oculix-org/Oculix

The findings discussed are from a CodeQL scan on commit 29a14d9. Detailed dismissal justifications are visible in the Security tab.

Why automation tools cannot type Chinese, and what OculiX does instead

Wed, 20 May 2026 08:00:00 GMT

In May 2026, an OculiX user filed issue #232. The summary was three lines long. The user wanted to call type with two Chinese characters in their automation script. Instead of typing them, the call produced silence, garbage, or in some IDE configurations a sequence of unrelated Latin characters depending on the active keyboard layout.

The fix, when it landed, was fifteen lines of Java. The explanation behind it stretches back to how java.awt.Robot was designed in 1998, around an implicit assumption that one character on a screen corresponds to one keystroke on a keyboard. That assumption holds beautifully for ASCII Latin. It breaks for Chinese, Japanese, Korean, Arabic, Hindi, and even for accented Latin characters when the user’s keyboard layout does not happen to expose them as direct keys.

This is the story of why visual automation frameworks have historically struggled to type non-ASCII, what OculiX decided to do about it, and why the workaround is more philosophically interesting than the bug it solves.

The bug report

The user, an active OculiX adopter, was automating an internal Chinese-language application. Their script needed to fill a search field with a person’s name. The straightforward call returned silently. The search field stayed empty. No exception was thrown. No log line surfaced anything unusual. The automation simply continued, the next step failed because the search field had no input, and the test reported FindFailed on whatever element was supposed to appear after the search.

This is the worst kind of automation bug. There is no signal that something went wrong at the moment something went wrong. The error appears three steps later, attached to an unrelated element, with no link back to the actual failure point.

After some investigation, the user established that:

ASCII input worked: type("hello") produced “hello” in the field.
French accented characters partly worked: typing a word with an accent produced the unaccented version on a US layout.
Chinese input failed silently on every layout tested.
Switching the OS keyboard layout to a Chinese IME did not help: the OculiX type call still produced garbage.

The bug was filed not as a feature request but as a question: “Is this expected? How is everyone else handling CJK?”

The honest answer was that everyone else was working around it. Manually. Each script that needed to type Chinese was implementing its own custom paste routine inline, copying the string to the clipboard with Jython’s java.awt.Toolkit calls and then sending Ctrl-V via keyDown/keyUp. The workaround was well-known among users who automated CJK applications, but it was not in the documentation, and it required understanding why the straightforward call did not work.

This blog post explains the why, and what we did to make the straightforward call work.

Why Robot.keyPress fundamentally cannot type Chinese

To understand why type with Chinese fails, you have to understand what type actually does under the hood. In OculiX (and in the original Sikuli, and in SikuliX), the call descends through several layers and eventually reaches java.awt.Robot.keyPress(int keycode) and java.awt.Robot.keyRelease(int keycode), both members of the AWT API that has shipped with the Java platform since version 1.3 in May 2000.

Robot.keyPress(int keycode) is a thin wrapper around the operating system’s low-level keyboard event injection mechanism: SendInput on Windows, XTestFakeKeyEvent on X11/Linux, CGEventPost on macOS. All three of these accept a virtual key code, an integer identifier of a physical key on the keyboard. Not a character. Not a Unicode codepoint. A key.

This is the first fault line. When you call Robot.keyPress(KeyEvent.VK_A), you are not asking the system to produce the character “A”. You are asking it to simulate pressing the physical key that, on the currently active keyboard layout, would produce some character. The character depends on the layout. The layout depends on the user’s system configuration. The result depends on both.

For pure ASCII Latin in a Western layout, the mapping is reasonably stable. The key VK_A produces “a” or “A” depending on shift state. The key VK_1 produces “1” or ”!” depending on shift state. Sikuli historically maintained a hardcoded table mapping each ASCII character to a sequence of virtual key codes, and it worked because Western keyboards have a key for every ASCII character.

The trouble starts the moment the character has no physical key on the keyboard.

The Chinese case

There is no constant for Chinese characters in java.awt.event.KeyEvent. There cannot be one. Chinese characters do not correspond to keys. A standard Chinese keyboard is, physically, a US-QWERTY keyboard. Chinese characters are produced through an Input Method Editor (IME) that interprets sequences of QWERTY keystrokes as phonetic input (Pinyin), looks up matching characters, displays a candidate menu, and inserts the chosen character into the active text field once the user confirms a selection.

To type a Chinese phrase through an IME, a Chinese user types Pinyin on the physical keyboard, the IME intercepts the sequence, recognizes it as the romanization of a candidate set, presents a candidate menu (because multiple character sequences share that romanization), and waits for the user to either accept the top candidate via space or pick a different one via number keys or arrow keys.

This is not what Robot.keyPress does. Robot.keyPress injects events at a layer beneath the IME. The IME never sees them as Pinyin input. The events are interpreted as raw key events by the receiving application, which sees the Pinyin letters as literal characters. The Chinese intent is lost.

Even more confusingly, on a system where the active keyboard layout is Chinese (with an IME enabled), Robot.keyPress(VK_N) might:

Be intercepted by the IME, treated as Pinyin input, and contribute to building a candidate menu that the user never sees because the automation does not pause to select
Be passed through to the application as a literal “n” if the IME is in alphanumeric mode
Be silently dropped if the IME is mid-composition for another sequence

The behavior is non-deterministic from the automation’s perspective, and entirely dependent on IME state, which the automation cannot reliably read or control through the AWT API.

The Japanese, Korean, Arabic, Hindi cases

The same logic applies to:

Japanese, where IMEs interpret romaji or kana keystrokes into hiragana, katakana, or kanji
Korean, where IMEs assemble hangul syllables from jamo keystrokes
Arabic, where the keyboard layout itself is non-Latin and Robot virtual keys do not map predictably
Hindi and other Indic scripts, where IMEs combine consonants and vowel marks into syllabic clusters
Vietnamese, where IMEs add tonal diacritics through dedicated keys

All these languages share the property that the displayed character is not a one-to-one mapping with a physical keystroke. The mapping is mediated by an IME that operates above the OS keyboard event layer that java.awt.Robot injects into.

The accented Latin case (more subtle)

Even within the Latin script, the same problem appears in a milder form. Consider typing a French word with an accent on three different keyboard layouts.

Layout	Result of typing a French word with accent
US-QWERTY	Accent silently dropped, no VK code maps to it
French AZERTY	Accent reproduced correctly
German QWERTZ	Accent dropped or garbage produced
UK-QWERTY	Accent dropped
Spanish	Accent dropped (uses dead keys)

The accent characters have dead-key compositions on most layouts, which the AWT Robot cannot reliably simulate because dead keys are stateful at the OS layer and there is no portable way to query whether the dead key has been “consumed”.

The same applies to many Latin extension characters. The character is “Latin”, but the keyboard layout determines whether it has a dedicated key or requires a composition sequence.

Summary of the diagnosis

The deep reason type with non-ASCII fails is not specific to OculiX or to Sikuli. It is a structural limitation of the java.awt.Robot API, which injects events at the keyboard event layer beneath the IME and beneath the keyboard layout translation. This API was designed in 1998 for testing GUI applications in an ASCII-Latin context, and it has never been extended to handle the layered character input model that the rest of the world uses.

The clipboard route as workaround

Once you accept that you cannot type CJK characters through the keystroke injection path, the question becomes: how do you get them into an input field at all? The most portable answer, used by professional automation engineers for two decades, is the clipboard.

The idea is simple. Instead of pressing keys, you:

Copy the target string to the system clipboard
Send the OS-specific paste keystroke (Ctrl-V on Windows/Linux, Cmd-V on macOS)
The receiving application processes the paste event and inserts the clipboard content into the focused field

This route bypasses the keyboard layout layer entirely. The clipboard contains arbitrary Unicode bytes. The paste keystroke is a single key combination present on every modern platform. The receiving application sees the content as text, not as a sequence of keystrokes, and inserts it through its text handling code path, which is Unicode-aware.

The clipboard route has been the unofficial standard workaround in the Sikuli community since at least 2012. It worked. But it required users to know about it, to implement it themselves, and to handle a list of edge cases (clipboard backup before pasting to avoid destroying user data, clipboard restoration after pasting, focus verification, etc.).

The OculiX change for issue #232 was to make this route the default, transparently, for any input that contains characters outside the 7-bit ASCII range.

Implementation: fifteen lines of Java

The actual change in OculiX is small enough to quote in full.

The detection

private static boolean containsNonAscii(String text) {
    if (text == null) return false;
    for (int i = 0; i < text.length(); i++) {
        if (text.charAt(i) > 127) return true;
    }
    return false;
}

Seven lines. We iterate over the input string and check whether any character has a codepoint greater than 127 (the 7-bit ASCII boundary). If yes, the input cannot be reliably typed through the keystroke path. If no, the input is pure ASCII Latin and can use the existing keystroke pipeline.

The choice of 127 as the boundary is deliberate. Everything from 0 to 127 is ASCII, has a stable Robot mapping on Western keyboards, and works with the legacy code path. Everything above 127 is either Latin extension (accented characters), other scripts (CJK, Cyrillic, Arabic, etc.), or symbols. All of these benefit from going through paste rather than through simulated keystrokes.

The routing

public int type(String text) {
    if (containsNonAscii(text)) {
        return paste(text);
    }
    try {
        return keyin(null, text, 0);
    } catch (FindFailed ex) {
        return 0;
    }
}

public int type(Object target, String text) throws FindFailed {
    if (containsNonAscii(text)) {
        return paste(target, text);
    }
    return keyin(target, text, 0);
}

The routing is two if-statements added before the existing logic. If the input contains non-ASCII characters, the call is forwarded to paste (which already exists and was already battle-tested). Otherwise, the existing keyin path runs unchanged.

The two-argument variant type(target, text) first clicks the target image to focus the input, then routes the text through paste(target, text), which performs the same click-then-paste sequence internally. The user-facing API is identical; only the internal routing changes.

What stays on keystrokes

public int type(Object target, String text, int modifiers)
    throws FindFailed {
    return keyin(target, text, modifiers);
}

The modifier-bearing variants type(text, modifiers) and type(target, text, modifiers) are intentionally not routed through paste. Holding Shift, Ctrl, or Cmd while pasting Unicode characters has no meaningful semantic — these modifiers exist to shift the meaning of physical keystrokes (Shift+A produces “A”, Ctrl+S triggers a save action), not to alter clipboard content.

If a user calls a modifier-bearing variant with CJK characters, the keystroke path runs and produces a FindFailed if the characters cannot be typed, which is the honest behavior. Silently switching to paste would lose the user’s intent.

The behavior matrix

The result for users of the API is a clean four-way matrix:

Input	Modifiers	Path taken	Why
Pure ASCII	None	Keystrokes	Backward compatible, fast, no clipboard side effect
Pure ASCII	Yes (Shift/Ctrl/Cmd)	Keystrokes	Modifiers map to keystroke semantics
Non-ASCII	None	Clipboard paste	Only way to reliably produce the characters
Non-ASCII	Yes (Shift/Ctrl/Cmd)	Keystrokes (will likely fail)	Honest failure; modifier+paste has no meaning

This is the kind of fallback that the user does not see and does not need to know about. The scripts that worked before still work the same. The scripts that previously needed manual clipboard workarounds now just work with the obvious call.

Tradeoffs of the clipboard route

The clipboard path is not a free win. It has properties that are different from the keystroke path, and users automating sensitive workflows should be aware of them.

Clipboard pollution

Pasting destroys the previous clipboard content. The OculiX paste implementation backs up the clipboard before pasting and restores it afterward, but the backup-restore window is not instantaneous and can interfere with other clipboard listeners.

Focus dependency

The paste keystroke targets whatever has keyboard focus. A wrong paste is one chunk of misplaced text, much more visible than wrong keystrokes which usually masks the issue.

Paste timing

Some applications throttle or queue paste events. Pasting twice in rapid succession can produce only one paste, or merge two pastes into one. The keystroke path is less susceptible.

No partial input

A keystroke-based input can be interrupted in the middle. A paste is atomic. If you want autocomplete observation, the keystroke path is the right one.

For most real-world automation use cases, these tradeoffs are acceptable. A QA engineer automating a Chinese banking form does not need character-by-character granularity; they need the form filled correctly. A test scenario filling Japanese customer data into a healthcare CRM does not depend on clipboard preservation across the operation.

The cases where the tradeoffs matter — autocomplete observation, clipboard-watching tooling — are edge cases that the user can opt out of by explicitly calling keyin instead of type, or by providing the input one character at a time.

What this enables in production

Issue #232 was filed by a single user, but the impact of the fix is broader than that single use case. We have seen the change unlock automation work in several customer contexts since it landed.

APAC banking back-office

A French bank with subsidiaries in Hong Kong and Singapore needed to automate regression tests on their Mandarin and Cantonese branch interfaces. Before the fix, their automation team maintained two separate codebases: one for European frontends, one for APAC with manual clipboard workarounds. The two codebases merged. Maintenance cost dropped by roughly a third.

Japanese healthcare CRM

A healthcare provider in Osaka automating patient intake workflows ran into the same wall. Patient names, addresses, prescription drug names — none survived the keystroke path. The team had a custom paste wrapper as workaround, but it predated the Ed25519 audit module and could not be signed for compliance. The integrated paste path now passes their audit.

European multilingual e-commerce

A French automation team running tests on a German marketplace was hitting silent failures whenever a product name contained German umlauts. The German keyboard layout was active on the test machine, but the keystroke table assumed French AZERTY and silently dropped the characters. The clipboard route bypasses all of this.

Internal IT helpdesks across regions

Multinational corporations run helpdesk automations interacting with internal ticketing systems containing employee names in dozens of native scripts. Cyrillic, Greek, Arabic, Hindi. Each required a separate workaround. The fix makes all of them automatic.

The common thread is that internationalization in automation tooling has historically been an afterthought. Tools are built and tested in their authors’ language, and the assumption that “text input works” is rarely verified against the full Unicode space. Fixing this in OculiX brought a basic property back into line with what users reasonably expect.

What this doesn’t solve

The fix is good but not complete. Several adjacent problems remain open.

The structural lesson

The lesson behind this fix is not about Chinese or Japanese specifically. It is about what happens when a tool’s API and its underlying assumption diverge.

type(String) reads, to a 2026 developer, like a Unicode-aware string-to-input function. The signature accepts a String, which in Java is fully Unicode-capable. The naive expectation is that whatever Unicode string you pass should appear in the target field, character by character, exactly as written.

That expectation collides with an implementation that descends, through layers of well-meaning abstraction, into a 1998-era API that injects keyboard events at a layer beneath the keyboard layout. The mismatch between the modern API surface and the legacy implementation creates a silent failure mode that takes years of accumulated user reports to fully document.

The fix is fifteen lines of Java because the workaround already existed in the codebase (the paste method). The change just added a smart router that picks the right path based on the input content. The hard part was not the code; it was identifying that the routing decision belonged at the type level, not at the user level.

This is a recurring pattern in mature codebases. The capability is there. The pieces are in place. What is missing is the small connective decision that makes the capability available to users who do not know which internal method to call. A good API surface hides the choice between keyin and paste. The user calls type and gets the result they expected.

For anyone maintaining a similar tool with a similar AWT-era foundation: this fix is generalizable. The same routing logic, the same ASCII boundary detection, the same paste fallback applies in PyAutoGUI, AutoHotkey, Robot Framework keyword libraries, or any other automation tool that relies on Robot-style keystroke injection. The 1998 API will not be redesigned. But its limitations can be bounded by a smarter layer above it.

Repository: github.com/oculix-org/Oculix

Original issue: oculix-org/Oculix#232

Storing spatial memory in a PNG: how we made visual UI matching 6.5× faster in OculiX

Tue, 19 May 2026 08:00:00 GMT

This is a long technical post about a small change. It explains a measurement we never thought to make, the discovery it produced, the implementation we built around it, and what it means for visual automation suites running in CI.

The headline number is real but not particularly dramatic: visual UI matching in OculiX became roughly 6.5 times faster, measured on 50 cold-start JVM runs on an Intel i3 7th generation laptop. The interesting part is not the speedup itself. It is the path that led to it, and the structural lesson that came with it.

If you maintain or use a visual automation framework, a test suite that watches pixels, or any tool that needs to find an image inside another image repeatedly across separate process invocations, the pattern we describe here will probably apply to you. The implementation took about 200 lines of Java, no external dependencies, and three micro-modifications inside the existing codebase. The standards we relied on have been published since 1996.

What follows is the full story, ordered the way the investigation actually unfolded, not the way a marketing post would tell it.

The benchmark we never made

OculiX is the active continuation of the Sikuli and SikuliX visual automation lineage, MIT-licensed and used in production by close to 100 organizations across banking, defense, healthcare, manufacturing, and retail. Its core operation is a function called find: given a small image (a button, an icon, a region of UI), find it inside the current screen capture and return its coordinates. Every other operation in the public API contains at least one call to find underneath.

Inside the codebase, find is well understood. It uses OpenCV template matching via JNI bindings, with five fallback strategies cascaded inside Finder.java:

Mode 1: Standard match

Exact template matching with the configured similarity threshold. The fast path for stable UI elements.

Mode 2: DPI-aware rescale

If the screen DPI differs from the pattern capture DPI, the template is rescaled before matching.

Mode 3: Tolerant blur

GaussianBlur applied to both source and target. Tolerates antialiasing and subtle color variations.

Mode 4: Grayscale smart

Conversion to grayscale before matching. Tolerates color theme changes.

Mode 5: Multi-scale brute force

Last resort. Tries multiple scales (0.5x to 2x) to catch significantly resized elements.

The code is twenty years old at this point, refined incrementally by the original Sikuli authors at MIT in 2009, by Raimund Hocke from 2010 to 2025 under SikuliX, and now by the OculiX maintainers.

Performance, in this kind of codebase, is rarely benchmarked from scratch. Everyone assumes it is whatever it has always been. A find call takes “some milliseconds”, or “a few hundred milliseconds” on a slow machine, and life goes on.

So I sat down on a Sunday afternoon to actually measure it. Not because there was a problem. Because the question of how fast it really was had never been answered with a number on the current hardware I had in front of me.

The harness

The setup was deliberately simple. A standalone Java class called FindTiming, compiled directly against the OculiX complete-win jar, performing exactly one find per JVM invocation, then exiting. A batch script wrapping that class in a loop of fifty separate executions.

So fifty cold starts. Each one paid the JVM startup cost, the OpenCV library load, the Tesseract OCR engine load, the OculiX framework initialization, then performed exactly one find and reported the elapsed time before exiting.

The baseline

The result, on this five-year-old i3 laptop, was extremely consistent:

Metric	Value
Mean	502 ms
Median	502 ms
Minimum	480 ms
Maximum	545 ms
Range	65 ms
Standard deviation	18 ms

Half a second to find a small image on a 1920 by 1080 screen. On a modern Intel i7 or Apple Silicon machine the number would be lower, probably by a factor of two or three. On a GitHub Actions standard runner it would be roughly comparable to the i3. On an older corporate desktop running a Citrix client through a VPN it would be slower again.

Take 500 milliseconds and project it across a test suite:

Suite scale	Find calls	Pure find time @ 500ms
Small functional suite	200	100 seconds
Medium regression suite	1 000	8 min 20 s
Large nightly suite	5 000	41 min 40 s
Enterprise full coverage	20 000	2 h 46 min

Multiply by the number of suites running per day in a CI environment, multiply by the number of CI minutes billed by the runner provider, and the cost becomes very real.

That was the baseline. No optimization had been attempted yet. The number was simply the truth about what happened on the metal.

What was already in the code

Before deciding what to optimize, the right move is always to look at what the code already does. OculiX inherits sixteen years of optimization attempts from the Sikuli and SikuliX lineage. A naive optimizer would re-read the OpenCV documentation and propose to switch to a faster matching algorithm. That would be a beginner mistake.

The thing to investigate first is whether there is some optimization the existing code already attempts but cannot fully complete in the current configuration. That is almost always where the gains hide.

A few minutes of grep revealed a symmetric pair of fields and a setting that nobody seems to talk about:

Image.lastSeen

Private Rectangle field on every Image object. Stores the position of the most recent successful match. Paired with getLastSeen() and setLastSeen(rect, score) accessors.

Settings.CheckLastSeen

Public static boolean, set to true by default since at least 2018. Enables the optimization at the framework level.

checkLastSeenAndCreateFinder

Private method in Region.java. When called, creates a Finder restricted to the small rectangle around the previous match, falling back to full-screen only if needed.

The intent of these three pieces is clear once you piece them together. After a successful find, the rectangle of the match is stored in the Image object’s lastSeen field. On the next call to find for the same image, if Settings.CheckLastSeen is true and lastSeen is non-null, the code creates a Finder restricted to that small rectangle and tries to match there first. Only if the small-region match fails does it fall back to a full-screen scan.

This is a classic spatial memoization pattern, well-known in computer vision. Sikuli implemented it correctly, a long time ago.

The pattern works, until it doesn’t

The optimization works beautifully inside a single JVM session. If you run a script that performs screen.find("submit_button.png") ten times in a row, the first call pays the full-screen scan cost, but the next nine calls find the image almost instantly through checkLastSeen. The cache hit rate is essentially 100 percent on stable UIs.

There is, however, a subtle but critical limitation.

This is exactly what our benchmark exposed. Fifty separate JVM invocations, fifty Image objects with lastSeen always equal to null at the moment of the find call, fifty full-screen scans. The checkLastSeen optimization was active and present in the code throughout, but it had no input to work with. The cache was empty because the cache lived inside a process that died right after building it.

This is the core observation. The existing optimization was correct in design. It was simply unable to bridge the gap between JVM invocations. In a typical test environment, where each test is its own process, the optimization never had a chance to engage.

The code that solved the problem was already there, written long before this benchmark was performed, in a careful and well-tested form. The missing piece was not a clever algorithm. It was a way to keep the optimization’s input alive across process boundaries. A storage problem, not a computation problem.

The missing piece: persistence

Once the gap was identified, the question became: how do you persist Image.lastSeen between JVM invocations? Several candidate approaches surfaced, each with their own trade-offs.

Approach	Description	Drawbacks
Sidecar file	Write `foo.png.position` next to each PNG	Two files to commit, risk of desynchronization, clutter
Central project file	Single `.oculix-positions.toml` at project root	Linear lookup cost, merge conflicts in parallel CI, full file rewrite per change
PNG ancillary chunk	Embed the position metadata inside the PNG itself	Requires writing PNG-aware code, but standard since 1996

The PNG ancillary chunk option won, for reasons that became clearer as we explored the constraint of CI environments.

PNG ancillary chunks: the underused W3C standard

The PNG file format, standardized by the W3C in 1996, is a structured container made of a fixed signature followed by a sequence of chunks. Each chunk has a four-byte length, a four-byte type, a variable-length data payload, and a four-byte CRC32 of the type and data.

The PNG standard distinguishes between critical chunks (IHDR, IDAT, IEND, and others), which are mandatory for decoding the image, and ancillary chunks, which are optional and can be safely ignored by decoders that do not recognize them.

We chose the type code oPLx for our chunk:

Lowercase o: ancillary, not critical
Uppercase P: private to OculiX
Uppercase L: reserved bit
Lowercase x: safe to copy

Decoded: an optional, private, safe-to-copy chunk identified by oPLx. Other PNG tools encountering an OculiX-modified file see something they do not recognize, preserve it untouched on save, and ignore it on load. Compatibility is total.

The format of the oPLx chunk

The internal layout of the oPLx chunk’s data payload is deliberately small. The goal was a fixed-size, fast-to-parse, easy-to-debug binary structure. The total payload is exactly 34 bytes:

Offset	Size	Type	Field	Notes
0	4	ASCII	Magic `OPL\0`	Redundant identifier, prevents misinterpretation
4	2	uint16 (BE)	Version	Currently `1`. Bump on breaking format change.
6	4	int32 (BE)	X	Pixel coordinate, last successful match
10	4	int32 (BE)	Y	Pixel coordinate, last successful match
14	4	int32 (BE)	Width	Match rectangle width, pixels
18	4	int32 (BE)	Height	Match rectangle height, pixels
22	8	int64 (BE)	Timestamp	UNIX epoch milliseconds, last update
30	4	int32 (BE)	Run count	Total successful matches since file creation

Total: 34 bytes of payload, plus the standard 12 bytes of PNG chunk framing (4 bytes length, 4 bytes type, 4 bytes CRC32). Each pattern gains 46 bytes of metadata embedded in its PNG file. For a project with 500 patterns, this represents 23 kilobytes of additional space across the entire pattern library. Effectively negligible.

Design choices in this format

A few decisions deserve commentary.

Big-endian byte order

Non-negotiable. The PNG standard mandates big-endian for all multi-byte integers in chunk fields. Following the same convention inside our payload simplifies parsing and removes confusion with future tooling.

32-bit signed for coordinates

Generous. 16-bit unsigned would have sufficed for single-screen resolutions. We chose 32 bits to leave room for multi-monitor setups where coordinates extend into negative space and tens of thousands of pixels.

64-bit timestamp

Standard UNIX epoch milliseconds. No bytes saved here. Audit trails that span years require room.

Run counter

Allows detecting dead patterns (counter at zero), unstable patterns (high counter on young file), and locked-in patterns (counter grows without timestamp updates).

The chunk is plaintext at this stage. The integrity check is the standard CRC32 that PNG mandates at the end of every chunk; it catches accidental corruption but is not cryptographically strong.

Implementation: three micro-modifications

The actual change inside OculiX comes down to three modifications in two existing files, plus one new utility class. Total addition: roughly 250 lines of Java. No external dependencies. No new Maven coordinates. The standard JDK classes DataInputStream, DataOutputStream, ByteBuffer, and java.util.zip.CRC32 cover all the needs.

1. Image.load() reads the chunk

After ImageIO.read() decodes the pixels, a separate streaming read parses the PNG chunks, locates oPLx, and calls setLastSeen(rect, 1.0) on the current Image instance.

2. doCheckLastSeenAndCreateFinder expands the search

Instead of creating a Region exactly the size of the previous match, it now creates a search box 2.5x larger, clamped to screen bounds. Tolerates UI drift between runs.

3. find() writes the chunk after match

After updating in-memory lastSeen, the chunk in the PNG file is streamed through a temp file and atomic-renamed. The position persists to disk.

Modification 1: Image.load() reads the chunk

In Image.java, the existing load method reads the PNG file from disk into a BufferedImage using ImageIO.read. This call decodes only the image data (the IDAT chunks). It does not parse other chunks. Our addition opens the same file separately, in streaming mode, walks through its chunks until it finds the oPLx chunk if present, parses the position metadata, and calls setLastSeen(rect, 1.0) on the current Image instance.

// In Image.java, after ImageIO.read(fileURL) at line 1018:
try {
    File pngFile = new File(fileURL.toURI());
    byte[] chunk = PngChunk.read(pngFile, "oPLx");
    if (chunk != null && chunk.length >= 34) {
        ByteBuffer buf = ByteBuffer.wrap(chunk);
        byte[] magic = new byte[4];
        buf.get(magic);
        if (magic[0] == 'O' && magic[1] == 'P'
            && magic[2] == 'L' && magic[3] == 0
            && buf.getShort() == 1) {
            int cx = buf.getInt();
            int cy = buf.getInt();
            int cw = buf.getInt();
            int ch = buf.getInt();
            setLastSeen(new Rectangle(cx, cy, cw, ch), 1.0);
        }
    }
} catch (Exception ignored) {}

The streaming parser is deliberately optimized for the common case where the chunk is present and is one of the first non-critical chunks in the file. It skips the PNG signature (8 bytes), then enters a loop: read the chunk length, read the four-byte chunk type, compare it to oPLx, return the payload if matched, return null if IEND is reached, or skip the chunk data and CRC and continue otherwise.

This is the architectural detail that matters most. The chunk reading is a strict addition that fills a gap in the existing system. It does not replace anything. It does not modify the contract of any existing method. If the chunk is absent (a legacy PNG that has never been processed by OculiX, a PNG whose chunk was stripped by an aggressive optimizer, a PNG generated by an external tool), the code falls through to lastSeen being null, which is exactly the situation the existing codebase has handled for sixteen years. The fall-through path is the cold-start path. It still works. It is just slower.

Modification 2: doCheckLastSeenAndCreateFinder expands the search

In Region.java, the existing method creates a small Region exactly the size of the previous match rectangle. This works well for in-session use, where the image has just been matched at the exact same position. It works less well for cross-process use, where the UI may have drifted slightly between runs.

Our modification expands the search rectangle by a factor of 2.5 around the stored center, clamped to the screen bounds.

// Replace at line 2891:
Rectangle ls = img.getLastSeen();
int sw = (int) (ls.width * 2.5);
int sh = (int) (ls.height * 2.5);
if (sw > screen.w) sw = screen.w;
if (sh > screen.h) sh = screen.h;

int cx = ls.x + ls.width / 2;
int cy = ls.y + ls.height / 2;
int sx = cx - sw / 2;
int sy = cy - sh / 2;

// Translate to stay inside screen, do not truncate
if (sx < 0) sx = 0;
if (sy < 0) sy = 0;
if (sx + sw > screen.w) sx = screen.w - sw;
if (sy + sh > screen.h) sy = screen.h - sh;

Region r = Region.create(sx, sy, sw, sh);

The 2.5 multiplier was chosen empirically:

Multiplier	Effect
1.5x	Occasionally misses drifted patterns
2.0x	Marginal improvement, still occasional misses
2.5x	Sweet spot: tolerates drift without ambiguity
3.0x	No safety improvement, starts introducing ambiguity on dense UIs
4.0x	Multiple visually similar elements get reached, confusing the matcher

The clamping logic preserves the search box size when the pattern is near a screen edge. A pattern at (0, 1022) still gets a full 250 by 145 search box, just positioned at (0, 935) instead of being centered. The match rectangle for the original pattern still fits inside the search box.

Modification 3: find() writes the chunk after a successful match

In Region.java, the existing find method already calls img.setLastSeen(lastMatch.getRect(), lastMatch.getScore()) after a successful match. This updates the in-memory lastSeen field. Our addition extends this call with a write to the PNG file’s oPLx chunk.

// At line 2284 of Region.find(), after setLastSeen:
img.setLastSeen(lastMatch.getRect(), lastMatch.getScore());

// New: persist to PNG chunk
try {
    File pngFile = new File(img.getFileURL().toURI());
    ByteBuffer buf = ByteBuffer.allocate(34);
    buf.put((byte) 'O').put((byte) 'P').put((byte) 'L').put((byte) 0);
    buf.putShort((short) 1);
    Rectangle r = lastMatch.getRect();
    buf.putInt(r.x).putInt(r.y).putInt(r.width).putInt(r.height);
    buf.putLong(System.currentTimeMillis());
    buf.putInt(getRunCount(img) + 1);
    PngChunk.write(pngFile, "oPLx", buf.array());
} catch (Exception ignored) {}

The chunk-writing logic streams through the PNG file once, copying every chunk through to a temporary file, replacing the oPLx chunk in place if it already exists, or inserting a fresh oPLx chunk before the IEND marker if not. At the end, the temporary file replaces the original via an atomic file system rename.

This streaming approach is more complex than reading the whole file into memory, modifying the byte array, and writing the result back, but it is meaningfully more robust:

Property	In-memory	Streaming (chosen)
Memory cost	Proportional to PNG size	Constant (~8 KB buffer)
Crash safety	Risk of partial write	Atomic rename: old or new, never half
Cost on small PNG	< 1 ms	1-2 ms
Cost on large PNG (1 MB+)	20-50 ms + allocation	4-8 ms, no allocation spike

The PngChunk utility class

Reading and writing PNG chunks does not require an external library. The format is simple enough that a 200-line utility class handles both operations with zero dependencies. The class exposes two methods:

public static byte[] read(File png, String type) throws IOException

Returns the payload bytes if the chunk is found, null otherwise. The read uses streaming DataInputStream, with skip() to bypass non-target chunks. Exits as soon as the target chunk is located.

public static void write(File png, String type, byte[] payload) throws IOException

Streams the source PNG to a temp file, replacing or inserting the chunk, then atomic-renames the temp file over the original. CRC32 computed via java.util.zip.CRC32 (hardware-accelerated on modern CPUs).

private static final byte[] SIGNATURE = {
    (byte) 0x89, 'P', 'N', 'G', '\r', '\n', 0x1a, '\n'
};
private static final byte[] IEND_BYTES = { 'I', 'E', 'N', 'D' };

The PNG file signature is 8 well-known bytes. The IEND chunk type terminates every PNG.

The total code in PngChunk.java is 218 lines including comments, blank lines, and the class declaration. The file has no imports outside the java.io, java.nio, java.util.Arrays, java.util.zip.CRC32, and java.nio.charset.StandardCharsets packages, all standard JDK.

Benchmark methodology

A benchmark is only useful if its methodology is described in enough detail that someone else can reproduce it.

Hardware and runtime

Item	Value
CPU	Intel Core i3-7100 (2 cores, 4 threads, 3.9 GHz)
RAM	8 GB DDR4-2400
Storage	NVMe SSD
OS	Windows 10 build 19045
Java	OpenJDK 25 from Eclipse Temurin
OculiX	3.0.3 release, feature branch with modifications
Screen	1920 × 1080, no DPI scaling
Pattern	Windows search bar fragment, 12 × 58 pixels, at (0, 1022)

Harness logic

Each benchmark run consisted of fifty independent JVM invocations, each:

Starting a fresh process
Loading OculiX and its native dependencies
Performing exactly one find call against the screen
Printing the elapsed time in milliseconds to standard output
Exiting

The elapsed time was measured using System.nanoTime() immediately before and after the screen.find(pattern) call. This excludes JVM startup, library loading, and framework initialization. It includes only the find operation itself, including the screen capture inside the find.

A separate post-processing class read the fifty timing lines from the captured log and computed: arithmetic mean, median, minimum, maximum, range, and standard deviation.

Two scenarios measured

Scenario	Initial state of PNG	Expected behavior
Baseline	No `oPLx` chunk	All 50 runs in full-screen scan
Optimized	`oPLx` chunk pre-written	All 50 runs in small-region scan

Both scenarios were measured cold-start (50 separate JVM invocations) to ensure the comparison reflects the CI environment.

Results

The numbers are the headline of this post. Let me state them precisely.

Metric	Baseline (FULL)	Optimized (ROI)	Improvement
n	50	50	—
Mean	502 ms	77.7 ms	×6.46
Median	502 ms	77 ms	×6.5
Minimum	480 ms	63 ms	×7.6
Maximum	545 ms	113 ms	×4.8
Range	65 ms	50 ms	comparable
Standard deviation	18 ms	11 ms	tighter

Speedup factor on the find call alone: 6.46.

Observations on these numbers

Low variance in both conditions

Standard deviation is 4% of mean in baseline, 14% in optimized. Neither is noisy enough to require repeated measurement.

Optimized scenario has a floor

Roughly 60-70 ms cannot be reduced further by this technique. Dominated by screen capture (20-40 ms) and OpenCV setup (20-40 ms).

Speedup depends on pattern size

Small patterns (under 50×50 px) give the largest speedup. Large patterns (300×300+ px) give a smaller speedup because the 2.5x search region itself becomes substantial.

Faster hardware preserves ratio

On modern CPUs, absolute timings shrink but the relative speedup factor stays at ×6-8. The overhead floor is proportionally less significant.

Hardware projection

If we extrapolate to other typical hardware:

Hardware	Baseline (FULL)	Optimized (ROI)	Speedup
i3 7th gen (measured)	502 ms	77 ms	×6.5
i7-13700K (projected)	~150-180 ms	~22-25 ms	×8
Apple M2/M3 (projected)	~80-120 ms	~10-15 ms	×8
GitHub Actions standard runner (projected)	~200-250 ms	~28-35 ms	×7

What this means for CI suites

The benchmark measures one find operation in isolation. A real-world test suite performs many operations, each containing at least one find. Translating the per-operation speedup into a suite-level wall-clock improvement requires a few assumptions.

Consider a moderately sized regression suite: 100 test cases, each performing 20 visual interactions on average, for 2000 total find calls.

Scenario	Time per find	Total find time	Savings
Baseline (i3)	500 ms	16 min 40 s	—
Optimized (i3)	77 ms	2 min 34 s	14 min 6 s
Optimized (modern i7)	22 ms	44 seconds	15 min 56 s

Economic projection

If the suite runs once per pull request on a CI runner billed at $0.008 per minute (the rough GitHub Actions standard runner cost):

Activity	Cost per run (baseline)	Cost per run (optimized)	Annual saving (50 PR/day, 250 days)
Suite execution	$0.13	$0.02	~$1 375 per suite

For organizations running multiple suites in parallel across many projects, the saving compounds quickly.

Two caveats

Find is not the only cost

The speedup applies to find time only. The full suite also contains network calls, server processing, browser rendering, mouse and keyboard events, and waits. A suite where finds dominate (visual regression, end-to-end smoke) sees larger wall-clock improvement than a suite waiting on slow back-ends.

Optimization rewards stability

Teams that regenerate patterns frequently see the chunk reset on each regeneration. The first run after regeneration pays the full-screen scan cost. The optimization rewards UI stability and frequent test execution — both characteristics of mature codebases.

Why this survives git clone and CI runners

The structural argument for embedding the position in the PNG itself, rather than in a sidecar file or a central index, is best expressed by considering the CI lifecycle.

A CI build typically starts from a clean state. A fresh container is provisioned, the repository is cloned from origin, dependencies are installed, the build runs, the tests run, the artifacts are collected, and the container is destroyed. There is no persistent state between builds. There is no shared file system. There is no cache that can be relied upon to be hot.

The oPLx chunk inside the PNG removes this discipline burden entirely. The chunk is part of the PNG file itself. When a developer runs the test suite locally and the chunk is updated, the modified PNG appears in git status automatically. Git tracks PNG files because they are committed in the project. The developer cannot commit “the update” without committing “the chunk inside the update”, because they are the same file.

This means that when a CI build clones the repository, it gets the PNG file with the chunk already inside. The first test run in CI is already in the fast path. The optimization is active from the first second. No warm-up phase is needed. No cache must be primed. The position metadata is in the file, and the file is in the repository, and the repository is in the clone.

A team running the suite locally, committing the regenerated PNGs, and pushing to CI is effectively pre-warming the CI cache through normal development activity. There is no separate “cache warm-up” step. The cache warms itself as a side effect of using the tool.

The broader pattern: PNG chunks as runtime context

The technique of embedding metadata in image files is not new. Adobe XMP, EXIF, IPTC, GIMP layer information, and many other systems use this mechanism. What is new, or at least underused, is the embedding of runtime state — not authoring metadata, not provenance information, but actual operational data that evolves as the file is used.

This pattern generalizes beyond visual automation:

Texture caches for graphics applications

A texture file could embed the time it was last loaded, the GPU memory pool it was allocated from, and the average frame time when it was active. A renderer could use this metadata to predict and pre-load textures.

Build artifacts for incremental compilation

A compiled object file could embed the hash of the source it was compiled from, the compiler version, and the optimization level. An incremental build tool could detect when recompilation is actually necessary.

Machine learning datasets

An image used in a vision model training set could embed its last classification result, the confidence score, and the model version. A cleaning tool could identify mislabeled samples without re-running the model.

Audit logs for compliance

A document image stored in a regulated workflow could embed a signed audit trail of the operations performed on it. The Ed25519 signing pattern from the OculiX MCP module would apply directly.

The common thread across all these examples is the same: keep operational state with the artifact, in a format the artifact’s primary tools will preserve, so that the state survives every distribution mechanism the artifact ever encounters.

What this doesn’t solve

A 6.5× speedup on the find is significant but not exhaustive. Several costs remain untouched by this optimization:

Cost component	Status	Approximate weight
Screen capture	Unchanged	20-40 ms per find
OpenCV setup	Unchanged	20-40 ms per find
Tesseract OCR loading	Unchanged	~300 ms per JVM cold start
OculiX framework init	Unchanged	~1.5 seconds per JVM cold start
JVM startup itself	Unchanged	~1-2 seconds per cold start

Reducing these other costs would require deeper structural changes: pre-built native images via GraalVM (the JVM startup), incremental OpenCV initialization (the OpenCV setup), lazy OCR loading (Tesseract), and possibly an alternative screen capture API on each platform. Each of these is a separate optimization project, none of which is in scope for the current change.

Closing: context engineering as a discipline

The lesson of this whole investigation is not “PNG chunks are cool” or “spatial memoization wins”. Both of those are true but uninteresting on their own.

The lesson is structural.

Finding these gaps requires a specific habit: benchmarking systems in the configuration users actually experience, not in the configuration that is convenient for developers. In a tight in-process loop, the OculiX find is fast. In a cold-start CI environment, it is slow. The same algorithm, the same code path, the same operating system. The only difference is whether the context built by the previous match has survived to inform the next match.

The fix is rarely a new algorithm. It is usually a new place to store something that already existed, in a form that can travel through the boundaries the user’s actual workflow imposes.

The PNG ancillary chunk happens to be a particularly elegant place to store visual automation context, because the artifact whose context we want to preserve is itself a PNG file, and the standard that defines PNG includes a mechanism specifically designed for this kind of metadata, and the tooling ecosystem has respected that mechanism for thirty years.

The result is an optimization that did not require inventing a new algorithm, did not require breaking any existing contract, did not require any external dependency, and is fully backward compatible with files that do not yet have the chunk. It just needed someone to measure the actual baseline, identify the gap, and close it.

For anyone maintaining a similar tool, my parting suggestion would be: go measure your cold-start performance against your warm-cache performance, in the same configuration your users actually run. If the gap is large, the optimization opportunity is probably not where you think it is. It is probably in the space between two of your existing components, in the form of a state that briefly exists and then disappears.

Repository: github.com/oculix-org/Oculix

Issue tracking the implementation of the persistent locator chunk: oculix-org/Oculix#353

Refonte de DOMAutopsy en quatre heures, et les cinq choses que l'IA n'a pas pu décider

Mon, 18 May 2026 08:00:00 GMT

J’ai passé un dimanche après-midi à refondre DOMAutopsy, le harvester de locators visuels que je maintiens, en passant d’un outil desktop PySide6 à un serveur FastAPI avec UI web en direct. Le travail a pris environ quatre heures. Claude Code a écrit la majorité des ~4 000 lignes livrées.

Ce chiffre, seul, ne dit rien. C’est le genre de nombre que les gens citent sur LinkedIn pour vendre une prestation. Ce qu’il signifie réellement dépend entièrement des décisions qu’on laisse au modèle, et de celles qu’on garde pour soi.

Cinq décisions me sont restées de cette session. Aucune n’était techniquement difficile. Toutes auraient produit un code sensiblement moins bon si j’avais accepté la première suggestion du modèle.

1. Le système n’avait pas besoin de RAG

La première proposition de l’agent pour capturer le contexte sur une longue session de navigation était d’embedder les snapshots de page, de les stocker dans un index vectoriel, et de récupérer les top-k chunks quand le LLM avait besoin de raisonner sur la trajectoire. C’est une suggestion raisonnable en 2026. Elle aurait aussi ajouté une dépendance à une base vectorielle, à un modèle d’embeddings, et à un pipeline de retrieval que personne du côté utilisateur n’a demandé.

Le vrai besoin était plus simple. L’agent observe le DOM en direct, le listener capture les sélecteurs, le rapport agrège tout à la fin. Il n’y a pas de problème de retrieval. Il y a un problème de context augmentation, et il se résout par un payload structuré passé au LLM à chaque étape.

Première décision d’architecture : retirer toute une couche que le modèle aurait construite sans broncher.

2. CDP screencast, pas noVNC

Pour l’UI web en direct, deux options viables sont apparues : streamer l’instance Chromium headless via CDP screencast, ou l’envelopper dans un serveur VNC et la servir via noVNC. Les deux fonctionnent. Le modèle a recommandé noVNC parce que la littérature sur laquelle il est entraîné le présente comme le standard.

En pratique, CDP screencast est plus léger, tourne dans le même process Playwright, et évite l’installation entière d’un serveur VNC dans l’image Docker. La voie noVNC aurait ajouté des pièces en mouvement dont personne n’a besoin une fois que Playwright expose déjà tout via CDP.

C’est le genre de décision qui prend trente secondes de jugement et que le modèle n’a aucun moyen de prendre seul. Il optimise pour la réponse moyenne de son jeu d’entraînement, pas pour l’architecture que vous êtes en train de construire aujourd’hui.

3. Une cascade à sept niveaux bat un matcher LLM unique

Plusieurs projets open source des douze derniers mois proposent de déléguer la génération de sélecteurs entièrement à un LLM à chaque interaction. Le modèle traite le DOM, choisit le sélecteur le plus stable, retourne le résultat. Élégant sur le papier.

DOMAutopsy utilise une approche différente : une cascade déterministe à sept niveaux dans un listener JavaScript de 372 lignes, le LLM n’intervenant qu’au nettoyage. Le tier 1 c’est data-testid, id, name. Le tier 7 c’est CSS court ou XPath texte. Le LLM ne voit jamais un sélecteur sauf si la cascade échoue ou produit des matchs ambigus.

La raison est simple. Un matcher purement LLM est non-déterministe, coûteux à l’échelle, et crée une dépendance dure à un tiers à chaque interaction. Une cascade statique est auditable, rejouable hors-ligne, et bon marché. Le LLM apporte de la valeur uniquement là où la logique déterministe cesse d’être efficace.

C’est une décision structurelle qu’aucun agent ne prendra à votre place, parce que tous les exemples qu’il a lus décrivent l’approche inverse.

4. Parser regex, pas parser AST, pour les scripts legacy

DOMAutopsy peut faire du reverse engineering sur des scripts de test existants (Katalon, Playwright, Cypress, Selenium) pour les convertir en tâches structurées pour l’agent IA. La proposition du modèle était d’écrire un vrai parser AST par langage : Tree-sitter pour Groovy, ts-morph pour TypeScript, et ainsi de suite.

Pour environ 95 pour cent des scripts de test réels en production, regex suffit. Ils sont linéaires, mécaniquement générés, répétitifs. Le parsing AST aurait multiplié le coût d’implémentation par dix et déplacé la surface d’échec de “la regex rate un cas de syntaxe” à “le parser AST crashe sur un fichier legacy malformé”.

J’ai pris regex. J’ai aussi ajouté une note claire dans la documentation : ~95 pour cent de couverture, le reste passe en fallback manuel. Contrainte honnête, périmètre maîtrisable.

5. Pas de lock-in SaaS, même quand ça aurait été plus rapide

À un moment de la session, le modèle a proposé d’intégrer un service hébergé de diff de captures d’écran pour décharger le travail de comparaison visuelle. Cela aurait économisé un peu de temps d’implémentation.

OculiX et DOMAutopsy vivent tous les deux dans des environnements régulés. Banques, contractants défense, providers santé. Ajouter une dépendance SaaS tierce sans décision consciente fermerait des portes que je devrais ensuite forcer pour les rouvrir. J’ai refusé, le modèle a accepté, le diff de captures se fait localement.

C’est une décision non-technique habillée en décision technique. Le modèle ne sait pas qu’ajouter external-api.com au flux de données va doubler la longueur du dossier procurement pour un CISO en secteur régulé. Moi je le sais.

Ce que l’IA fait bien, et ce qu’elle ne fait pas

En quatre heures, l’IA a produit du code propre, fonctionnel, majoritairement idiomatique. Le débit est réel. Le résultat n’est pas magique. C’est le produit d’un prompting structuré, d’un modèle mental clair côté architecte, et d’un flux continu de petits jugements qui ferment les mauvaises branches avant qu’elles ne grossissent.

La question intéressante n’est pas “l’IA peut-elle remplacer les architectes seniors”. C’est : quelles décisions allez-vous encore prendre vous-même, et lesquelles êtes-vous prêt à déléguer. Les cinq ci-dessus sont des décisions que je refuserais de déléguer même avec vingt ans d’expérience supplémentaires. Elles sont spécifiques au domaine, sensibles au contexte, et elles façonnent le reste du projet pendant des années.

Le reste, l’IA l’a livré.

—

Repo : github.com/julienmerconsulting/DOMAutopsy