Try several encodings to read old WEO releases

Related to management#786 (closed)

weo_convert.py

@@ -60,6 +60,16 @@ def unit_slugify(unit_label):
     return slugify(lower_unit_label, separator="_", stopwords=["of"])
 
 
+def dataframe_from_csv(csv_file: Path, encoding_list):
+    """Try several encodings to read csv content."""
+    for encoding in encoding_list:
+        try:
+            return pd.read_csv(csv_file, delimiter="\t", encoding=encoding)
+        except UnicodeDecodeError:
+            pass
+    raise ValueError(f"Can't find an encoding to read {csv_file!r}")
+
+
 def process_csv_file(
     csv_file: Path, area_info: Dict[str, Any], year_month: Tuple[str, str]
 ):
@@ -73,7 +83,7 @@ def process_csv_file(
         "unit": {},
     }
 
-    df = pd.read_csv(csv_file, delimiter="\t")
+    df = dataframe_from_csv(csv_file, ["utf-8", "iso-8859-1"])
 
     # Remove unused lines
     df = df[df["WEO Subject Code"].notnull()]

[Christophe Benz]

I think it is not a good practice to catch all exceptions. Does Pandas raise a particular exception for encoding errors?

If you're unsure about the exact exceptions to catch, could you add a comment to state that explicitly?

[Christophe Benz]

I think it is not a good practice to raise an Exception, which is a base class, I would recommend raising a ValueError.