Building a Handwriting Input Method from Scratch: Final Chapter

After compiling the npm package with WASM and model weights using wasm-pack, we can now apply it to a front-end project.

Here, we’ll focus on some key details.

Since model loading and inference are time-consuming operations, we need to handle these tasks in a Web Worker process.

Additionally, we need a singleton to ensure the model is only loaded once.

import init, { Model } from "ochw-wasm"
// Implement a singleton for the HandingWrite class
export class HandingWrite {
    private static instance: Model | undefined;
    private constructor() { }
    public static async getInstance(): Promise<Model> {
        if (!this.instance) {
            await init(wasmUrl)
            self.postMessage({ status: `loading model` });
            this.instance = Model.new()
            return this.instance;
        } else {
            return this.instance;
        }
    }
}

Next, we listen for events from the main thread, receive image data, and perform model inference:

// Listen for message events from the main thread
self.addEventListener("message", async (event: MessageEvent) => {
    try {
        // Get the singleton instance of the HandingWrite class to ensure the model is loaded
        const model = await HandingWrite.getInstance();

        // Extract uint8Array data from the message event for model prediction
        const { uint8Array } = event.data;

        // Use the model to predict the input uint8Array data
        const res = model.predict(uint8Array);

        // Parse the prediction result into JSON format and send it back to the main thread via postMessage
        self.postMessage({
            status: "complete", // Mark the task as complete
            output: JSON.parse(res), // Return the prediction result
        });
    } catch (e) {
        // If an error occurs, catch the exception and return the error message via postMessage
        self.postMessage({ error: `worker error: ${e}` });
    }
});

On the web page, initialize the worker:

const worker = new Worker(new URL("./worker.js", import.meta.url), {
  type: "module",
});

Every time the user writes, capture the image from the canvas and send it to the worker for inference:

	const stage = e.target.getStage();
    if (stage) {
      const blob: Blob = (await stage.toBlob()) as Blob;
      const arrayBuffer = await blob.arrayBuffer();
      const uint8Array = new Uint8Array(arrayBuffer);
      // console.log(uint8Array)
      worker.postMessage({
        uint8Array,
        width: stage.width(),
        height: stage.height(),
      });
      //  debug: Download the image
      // const dataURL = stage.toDataURL({
      //   pixelRatio: 1, // double resolution
      // });

      // create link to download
      // const link = document.createElement("a");
      // link.download = "stage.png";
      // link.href = dataURL;
      // document.body.appendChild(link);
      // link.click();
      // document.body.removeChild(link);
    }

We also need to retrieve the results from the Web Worker:

  const [candidateWords, setCandidateWords] = useState<CandidateWord[]>([]);

  useEffect(() => {
    worker.onmessage = (e) => {
      // console.log(e.data);
      if (e.data && e.data.status =='complete') {
        setCandidateWords(e.data.output);
      }
    };
  }, []);

Issues

There are still some issues with the current model.

For example, some characters with simple strokes are not recognized, indicating problems with dataset processing. More ablation experiments are needed to improve recognition accuracy.

If you are interested in the code for “Building a Handwriting Input Method from Scratch,” you can visit: https://github.com/ximeiorg/ochw. Please give it a star.

If you want to see the results, please visit: https://ochw.ximei.me.

If you notice that recognition performance degrades on a PC, it might be due to the canvas being distorted on PCs, causing aspect ratio differences and leading to input size issues that reduce accuracy.