Hu Hanyang

Hu Hanyang

Master's student in ECE at UC San Diego, specializing in Intelligent Systems, Robotics & Control.

Automatic Sentence Correction with BERT and NCM

View the code and examples on GitHub.

This was a group project on string edit distance. My contribution was a simplified implementation of BERT-based noisy-channel spelling correction. Although the method is deliberately small, it provides a useful setting in which to see how a language-model prior and an explicit error model complement one another.

Model

We choose a pre-trained BERT model provided by Hugging Face with the architecture illustrated below:

BERT Architecture

BERT is especially good at “understanding” the meaning of sentences: even if part of the sentence is wrong due to typos, we can mask them and make good predictions as long as the model captures the sentence overall.

For an observed word $x$ and a candidate correction $c$, I used the following noisy-channel score:

\[\begin{aligned} \log P(x\mid c) + \log Z = \begin{cases} \alpha & \text{if } c = x \\ -\gamma \cdot \log\left(d(c, x)\right) & \text{otherwise} \end{cases} \end{aligned}\]

Here, $d(c,x)$ is the Damerau-Levenshtein distance, and $Z$ is a normalizing constant. The parameter $\alpha$ controls the preference for retaining the observed word, while $\gamma$ controls how strongly the likelihood penalizes edit distance. Candidate generation also uses a maximum edit-distance threshold, denoted by $N$ in the implementation.

What the hyperparameters do

We could intuitively see the effect of these hyperparameters from the examples below:

  1. If $\alpha$ is too small, the model may overcorrect.

    (a) For example, when $\alpha = 1$ and $\gamma = 8$, we have

     > Input: Where should we meat tommorrow?
 > Output: Where should be met tomorrow?

    Modified to $\alpha = 6$, we have

     > Input: Where should we meat tommorrow?
 > Output: Where should we meet tomorrow?

    Raising $\alpha$ prevents the unnecessary replacement of “we.”

    (b) However, when $\alpha$ is too high, this approach has a tendency to preserve the original word.

    For example, when $\alpha = 7$, we have

     > Input: The plane tickets is expensive.
 > Output: The plane tickets is expensive.

    Modified to $\alpha = 3$, we have

     > Input: The plane tickets is expensive.
 > Output: The plane ticket is expensive.

    Thus, an excessively large $\alpha$ creates the opposite problem: the system becomes too conservative.

  2. A larger $\gamma$ gives stronger preference to candidates with smaller edit distance.

    For example, when $\alpha = 5$ and $\gamma = 1$, we have

     > Input: That is so god!
 > Output: That is so cool!

    Modified to $\gamma = 7$, we have

     > Input: That is so god!
 > Output: That is so good!

Limitations

The repository examples expose several limitations of this simple left-to-right system:

  1. Corrected output always has the same number of words as the input, so it cannot handle typos of missing or extra space.

     > Input: This isa dog.
 > Intended Output: This is a dog.
 > Actual Output: This is it.

  2. Unable to correct out-of-vocabulary words, such as domain specific words or slang.

     > Input: Backpropagation is important for machine learning.
 > Intended Output: [same as the original sentence since it is correct]
 > Actual Output: Propagation is important for machine learning.

  3. Our noisy channel model is primarily based on the edit distance, hence it lacks phonetic analysis, leading to errors with homophones.

     > Input：We are not aloud to do that.
 > Intended Output: We are not allowed to do that.
 > Actual Output: We are not about to do that.

  4. Words are corrected from left to right, so the model may commit to an early mistake.

     > Input: The cats is small.
 > Intended Output: The cats are small.
 > Actual Output: The cast is small.

The reference paper uses beam search to mitigate the final issue. The other failures suggest two natural extensions: a sequence-to-sequence model that can change the number of tokens, and a richer likelihood that incorporates phonetic similarity rather than relying only on edit distance. The main lesson from this project was not that a small noisy-channel model solves grammatical correction, but that an explicit error model makes the behavior—and the failure modes—much easier to interpret.