charck

Command line tool that finds the characters you cannot see in a text file, and removes or replaces them according to decisions you record once.

You cannot see a no-break space

charck checks text files for characters that are not plain ASCII, tells you exactly what and where they are, and removes or replaces them according to decisions you record once.

20 commits | 7,212 insertions(+) | 1,429 deletions(-)

Invisible characters get into markdown and configuration files by way of copy-paste from web pages, PDFs, word processors and, increasingly, LLM output. They survive review, because there is nothing to review. The file looks right. Then a no-break space defeats a grep for a plain space, or a byte order mark at offset 0 corrupts YAML front matter, and the failure turns up somewhere unhelpful.

Report first

Point it at a file or a directory. Nothing is changed and nothing is decided for you:

$ charck content/
content/notes.md
  line    1, col  11   U+00A0   NO-BREAK SPACE                 UNDECIDED      Ships on 5«U+00A0»March.
  line    2, col  17   U+200B   ZERO WIDTH SPACE               UNDECIDED      The plan is done«U+200B».
  line    3, col   3   U+2019   RIGHT SINGLE QUOTATION MARK    UNDECIDED      It«U+2019»s ready.

Summary
  U+00A0   NO-BREAK SPACE                      1   UNDECIDED   <- appended to config
  U+200B   ZERO WIDTH SPACE                    1   UNDECIDED   <- appended to config
  U+2019   RIGHT SINGLE QUOTATION MARK         1   UNDECIDED   <- appended to config

  created /tmp/site/.charck.toml
  3 characters need a decision - edit /tmp/site/.charck.toml
  3 occurrences in 1 file  (1 scanned, 0 skipped, 0 ignored)

Every character it has not seen before is appended to a TOML ledger, undecided, with a # suggested: comment where the handling is well understood. You write the action yourself:

[chars."U+00A0"]
name   = "NO-BREAK SPACE"
cat    = "Zs"
seen   = 1
action = "replace"
to     = " "
# suggested: action = "replace", to = " "  (deleting would glue the surrounding words together)

There are four actions. An empty one reports and never modifies, delete removes the character, replace substitutes to, and ignore keeps it and stops reporting it. Then --fix applies exactly what the report showed:

$ charck --fix content/
content/notes.md
  line    1, col  11   U+00A0   NO-BREAK SPACE                 replace " "    Ships on 5«U+00A0»March.
  line    2, col  17   U+200B   ZERO WIDTH SPACE               delete         The plan is done«U+200B».
  line    3, col   3   U+2019   RIGHT SINGLE QUOTATION MARK    replace "'"    It«U+2019»s ready.

Summary
  U+00A0   NO-BREAK SPACE                      1   replace " "
  U+200B   ZERO WIDTH SPACE                    1   delete
  U+2019   RIGHT SINGLE QUOTATION MARK         1   replace "'"

  3 characters fixed in 1 file
  3 occurrences in 1 file  (1 scanned, 0 skipped, 0 ignored)

Exit 0 means clean, 1 means there is something to look at or apply, and 2 is an operational error. That 1 is what makes the report usable as a build gate.

Detection is separate from policy

The tool reports, and the decisions are based on the configuration. A global configuration file in ~/.config/charck/ holds the decisions that apply everywhere, and a project’s .charck.toml overrides it only for the characters it names. A project that wants to override the global configuration says so here and inherits the rest. Paths the tool should leave alone go in the same file, and .gitignore itself is read and used as well.

Special characters that you actually might want to use in your language get an exemption, taken from your locale. cs_CZ.UTF-8 exempts the fifteen letters Czech writes with and no others.

Requirements

Python 3.11 or newer for tomllib, standard library only, no dependencies. make install goes through pipx and links the command into ~/.local/bin, or copy charck.py somewhere on your PATH and run it. It is one file.

I wrote it for my own files and my own habits. It does what I need it to do, and I have not tried it against anything much stranger than my own writing.