Strings and runes in Go: why a Kazakh letter is two bytes

Why this matters

You will need to measure the length of a title, trim a summary to a hundred characters, check that a name is not empty. In English all of that works the obvious way — and in Kazakh the obvious way gives the wrong answer.

len("шаңырақ")   // 14, and there are seven letters

This is not a quirk of Go. It is how text is stored in every programming language; Go simply declines to pretend the problem is not there.

The whole thing at once

A new folder, go mod init sabaq06, main.go:

package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	word := "Go тілі"

	fmt.Println("bytes:", len(word))
	fmt.Println("letters:", utf8.RuneCountInString(word))

	for i, r := range word {
		fmt.Printf("%d\t%c\n", i, r)
	}

	fmt.Printf("cut at five bytes: %q\n", word[:5])
}

Before you run it, and without looking below, say out loud what it will print. Then run it and compare: the difference is exactly where you do not know the language yet.

go run . prints:

bytes: 11
letters: 7
0	G
1	o
2	 
3	т
5	і
7	л
9	і
cut at five bytes: "Go т"

Look at the left column: 0, 1, 2, 3, 5, 7, 9. There is no 4 and no 6.

Taking it apart

A string is bytes

A string in Go is a run of bytes, not of letters. A Latin letter takes one byte, a Kazakh or Russian one takes two, an ideograph three, and an emoji can take four.

Picture it. A tape with letters of different widths stuck to it. len measures the tape in centimetres and answers honestly how many there are. A centimetre is simply not a letter, and never was.

This is UTF-8, a way of writing every alphabet in the world in one encoding. Latin letters cost one byte in it so that old texts would not swell; the other alphabets pay for that with two bytes and more.

len counts bytes, and it is right to

len("Go тілі") returns 11. Not a mistake: “how much room does it take” and “how many letters” are different questions, and len answers the first. When you are sizing a file or checking a storage limit, bytes are exactly what you want.

A rune is a character’s number

rune is a character’s number in Unicode, the world-wide list of characters. The rune type is an int32 — just a number.

Picture it. A census of every letter on earth. “қ” has its own number in that list the way a house has an address. A rune is not the letter itself but its number; to print it you turn it back into a string.

A single rune goes in single quotes: 'қ' is a rune, "қ" is a string of one letter. Different things.

range over a string

for i, r := range word {

range walks a string by Unicode code points, not by bytes, and hands you two things at each step: i, the byte offset from the start, and r, the rune itself.

That is why the indexes jumped: after т at position 3 the next letter landed at 5, because т took two bytes. The index is an address on the tape, not a letter’s place in the queue.

How to count letters

utf8.RuneCountInString(word)   // 7

The unicode/utf8 package from the standard library; nothing to install. This is what you want when the page says “120 characters left”.

One caveat, so it does not surprise you later. A rune is a number in Unicode, not a guaranteed single visible sign. For ordinary Kazakh and Russian text runes and letters line up, and counting them is safe. But an é written as e plus a separate mark is two runes for one visible character, and a single family emoji is seven. You will only notice this where such text actually turns up: in names, and in text written by readers.

The trap: cutting by bytes

Here is what the lesson is for:

word := "шаңырақ"
fmt.Println(word[:5])

You asked for five bytes and Go gave you exactly five. But the fifth byte is the first half of the letter “ң”, so what reaches the screen in its place is rubbish: ша\xd2.

Picture it. Cutting a word with scissors not between letters but through the middle of one. Both halves exist; the word does not.

This is how summaries break: “we will trim the description to 200 characters” turns into a stump on every second Kazakh article. The right way is to count letters:

r := []rune(word)
fmt.Println(string(r[:5]))   // шаңыр

[]rune(word) takes the string apart into runes. That is a separate pass over the whole string, so it does not belong inside a loop — but for one trim it is fine.

What to do about it in practice

Joining and comparing strings is safe: "Go" + " тілі" works and == compares whole strings, byte by byte. Bytes only surface where you take an index, cut, or measure a length. The rule is short: len for size, RuneCountInString for letters, range for walking.

The lesson map

Say it in your own words

Answer out loud or on paper without looking. The answers are at the end of the lesson.

1. Why did the range indexes run 0, 1, 2, 3, 5, 7, 9?
2. How does 'қ' differ from "қ"?
3. A reader typed the name “Әсел” and complains that the counter showed 8 characters instead of 4. What happened?

Exercise

Required. Write a function letters(s string) int returning the number of letters in a string, and a function bytes(s string) int returning the number of bytes. Call both for "шаңырақ", "Salem" and "Go тілі" and print the pairs.

Optional.

• A function cut(s string, n int) string that trims a string to n letters without breaking the last one. Test it on "шаңырақ" with n = 5.
• Walk "шаңырақ" with range and print each letter with its Unicode number: fmt.Printf("%c = %d\n", r, r).
• Compare len("Әсел") with len("Asel"). Why is the difference exactly that?

Where this goes in your blog

Estimating reading time, trimming a summary to the card’s eight lines, checking that a title is no longer than sixty characters — all of it counts letters, not bytes. Get it wrong and every article’s Kazakh version is cut mid-word while the English one carries on as though nothing were amiss. It is the kind of mistake somebody writing only in Latin never sees.

Build your own blog. The blog page can now count letters and reading time. The state after this lesson is step-2: a title, a letter count, readingTime and an if. Nothing you have not been taught yet.

Answers

Sources

• Strings, bytes, runes and characters in Go — the go.dev blog
• Package unicode/utf8 — documentation
• Go spec: For statements with range clause
• The UTF-8 standard — RFC 3629