From: Thomas Hochstein <thh@inter.net>
Date: Sat, 29 May 2021 08:17:00 +0000 (+0200)
Subject: Forcibly decode headers with unencoded 8bit chars.
X-Git-Url: https://code.th-h.de/?p=usenet%2Fnewsstats.git;a=commitdiff_plain;h=b99d41010ddffd330464c6a80da615c232b2832e

Forcibly decode headers with unencoded 8bit chars.

Just assume UTF-8 for the time being.
Fixes database errors with illegal characters
when writing parsed data.

Signed-off-by: Thomas Hochstein <thh@inter.net>
---

diff --git a/bin/parsedb.pl b/bin/parsedb.pl
index a5b2ce8..8004cb1 100755
--- a/bin/parsedb.pl
+++ b/bin/parsedb.pl
@@ -198,6 +198,11 @@ while (my $HeadersR = $DBQuery->fetchrow_hashref) {
         $Headers{$HeaderName.'_parsed'} = decode('MIME-Header',$Headers{$_})
           if (exists($LegalEncodings{$Encoding}));
       }
+      # forcibly modify headers with un-encoded 8bit data assuming utf-8
+      # TODO: try to guess correct enconding
+      elsif ($Headers{$_} =~ /[^\x00-\x7F]/) {
+        $Headers{$_} = decode('utf-8',$Headers{$_});
+      }
       # extract name(s) and mail(s) from From: / Sender: / Reply-To:
       # in parsed form, if available
       if ($_ ne 'subject') {