Commit 874acb6f authored by Paolo Bonzini's avatar Paolo Bonzini
Browse files

checkpatch: detect doubly-encoded UTF-8 



Copy and pasting from Thunderbird's "view source" window results in double
encoding of multibyte UTF-8 sequences.  The appearance of those sequences is
very peculiar, so detect it and give an error despite the (low) possibility
of false positives.

As the major offender, I am also adding the same check to my applypatch-msg
and commit-msg hooks, but this will also cause patchew to croak loudly when
this mistake happens.

Signed-off-by: default avatarPaolo Bonzini <pbonzini@redhat.com>
Message-Id: <1558099140-53240-1-git-send-email-pbonzini@redhat.com>
Signed-off-by: default avatarPaolo Bonzini <pbonzini@redhat.com>
parent 19752e09

scripts/checkpatch.pl

+16 −0
Original line number Diff line number Diff line
@@ -262,6 +262,19 @@ our $UTF8 = qr{ 
	| $NON_ASCII_UTF8

}x;



# some readers default to ISO-8859-1 when showing email source. detect

# when UTF-8 is incorrectly interpreted as ISO-8859-1 and reencoded back.

# False positives are possible but very unlikely.

our $UTF8_MOJIBAKE = qr{

	\xC3[\x82-\x9F] \xC2[\x80-\xBF]                    # c2-df 80-bf

	| \xC3\xA0 \xC2[\xA0-\xBF] \xC2[\x80-\xBF]         # e0 a0-bf 80-bf

	| \xC3[\xA1-\xAC\xAE\xAF] (?: \xC2[\x80-\xBF]){2}  # e1-ec/ee/ef 80-bf 80-bf

	| \xC3\xAD \xC2[\x80-\x9F] \xC2[\x80-\xBF]         # ed 80-9f 80-bf

	| \xC3\xB0 \xC2[\x90-\xBF] (?: \xC2[\x80-\xBF]){2} # f0 90-bf 80-bf 80-bf

	| \xC3[\xB1-\xB3] (?: \xC2[\x80-\xBF]){3}          # f1-f3 80-bf 80-bf 80-bf

	| \xC3\xB4 \xC2[\x80-\x8F] (?: \xC2[\x80-\xBF]){2} # f4 80-b8 80-bf 80-bf

}x;



# There are still some false positives, but this catches most

# common cases.

our $typeTypedefs = qr{(?x:
@@ -1506,6 +1519,9 @@ sub process { 
			ERROR("Invalid UTF-8, patch and commit message should be encoded in UTF-8\n" . $hereptr);

		}



		if ($rawline =~ m/$UTF8_MOJIBAKE/) {

			ERROR("Doubly-encoded UTF-8\n" . $herecurr);

		}

# Check if it's the start of a commit log

# (not a header line and we haven't seen the patch filename)

		if ($in_header_lines && $realfile =~ /^$/ &&