{"id":125,"date":"2020-03-28T18:48:17","date_gmt":"2020-03-28T16:48:17","guid":{"rendered":"https:\/\/guoyunhe.me\/en\/?p=125"},"modified":"2020-03-29T23:42:12","modified_gmt":"2020-03-29T20:42:12","slug":"integrate-crowdin-to-retroarch","status":"publish","type":"post","link":"https:\/\/guoyunhe.me\/en\/2020\/03\/28\/integrate-crowdin-to-retroarch\/","title":{"rendered":"Integrate Crowdin To RetroArch"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">RetroArch project uses *.h files to store translation strings. Here aren't any tools to make the translation process easier. When source strings changed, you have to manually review the changes, locate and update translation strings. It is a hard work. As a big fan of RetroArch, I was thinking if it can be improved with modern i18n platforms.<\/p>\n\n\n\n<!--more-->\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_87_1 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/guoyunhe.me\/en\/2020\/03\/28\/integrate-crowdin-to-retroarch\/#Why_crowdin\" >Why crowdin?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/guoyunhe.me\/en\/2020\/03\/28\/integrate-crowdin-to-retroarch\/#But_h_files_are_not_supported%E2%80%A6\" >But *.h files are not supported&#8230;<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/guoyunhe.me\/en\/2020\/03\/28\/integrate-crowdin-to-retroarch\/#Isnt_Bash_script_enough\" >Isn&#8217;t Bash script enough?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/guoyunhe.me\/en\/2020\/03\/28\/integrate-crowdin-to-retroarch\/#Crowdin_project_configuration\" >Crowdin project configuration<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/guoyunhe.me\/en\/2020\/03\/28\/integrate-crowdin-to-retroarch\/#Glue_up_the_whole_workflow\" >Glue up the whole workflow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/guoyunhe.me\/en\/2020\/03\/28\/integrate-crowdin-to-retroarch\/#Initial_upload\" >Initial upload<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/guoyunhe.me\/en\/2020\/03\/28\/integrate-crowdin-to-retroarch\/#README_and_PR\" >README and PR<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Why_crowdin\"><\/span>Why crowdin?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ol class=\"wp-block-list\"><li>Self-hosting solution is too expensive.<\/li><li>Crowdin has better editor than Weblate\/Transifex.<\/li><li>Crowdin provides very good email support.<\/li><\/ol>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"But_h_files_are_not_supported%E2%80%A6\"><\/span>But *.h files are not supported&#8230;<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Let's convert *.h files to *.json files. (JSON is well supported by Python)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I have two Python scripts:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/github.com\/libretro\/RetroArch\/blob\/master\/intl\/h2json.py\">h2json.py<\/a><\/li><li><a href=\"https:\/\/github.com\/libretro\/RetroArch\/blob\/master\/intl\/json2h.py\">json2h.py<\/a><\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">You can see they do two-way conversion.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Isnt_Bash_script_enough\"><\/span>Isn&#8217;t Bash script enough?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Bash scripts aren't supported by Windows. Many Unix utilities are not existing in different Linux distros and macOS.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Python (3) is more platform independent. And it has built-in JSON support.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Crowdin_project_configuration\"><\/span>Crowdin project configuration <span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The only thing to notice is the language mapping.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">First, I tried to use <code>language_mapping<\/code> in crowdin.yml file as below. But unluckily <strong>it doesn't work<\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-yaml\">\"files\":\n  [\n    {\n      \"source\": \"\/msg_hash_us.json\",\n      \"translation\": \"\/msg_hash_%locale%.json\",\n      \"translation_replace\": { \"_us\": \"\" },\n      \"languages_mapping\": { \"zh-CN\": \"zhs\", \"zh-TW\": \"zht\" },\n    },\n  ]<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The only working solution is to go to Crowdin website, open project settings and add \"<strong>Language Mapping<\/strong>\" in first tab page. (The button is not very visible)<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Glue_up_the_whole_workflow\"><\/span>Glue up the whole workflow<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ol class=\"wp-block-list\"><li>git pull<\/li><li>Convert source (en) *.h to *.json<\/li><li>Upload source (en) *.json to Crowdin<\/li><li>Download translation *.json<\/li><li>Convert translation *.json to *.h<\/li><li>git commit and push<\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">The only difficulty is that Crowdin CLI installation on Windows isn't easy. So in the Python script, I download the crowdin-cli.jar into the project directory and execute it by Java.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Check <a href=\"https:\/\/github.com\/guoyunhe\/RetroArch\/blob\/master\/intl\/crowdin_sync.py\">the full workflow script<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Initial_upload\"><\/span>Initial upload<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ol class=\"wp-block-list\"><li>Convert all *.h to *.json<\/li><li>Upload msg_hash_us.json as source<\/li><li>Upload other *.json as translations<\/li><li>Run the workflow script and compare the changes<\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Because the translation is managed as key-value pairs, some source string changes are not reflected in translation strings. They need to be reviewed in future.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this process, I found some issues of my convert scripts:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Strings shouldn't be trimmed<\/li><li>Replace should start from the right side<\/li><li>It doesn't escape <code>\"<\/code> correctly<\/li><li>It doesn't support multi-line strings<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Then I keep improving the scripts and repeat above process. The final result should be: you don't lose existing translations.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"README_and_PR\"><\/span>README and PR<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">For developers and translators, Crowdin might be a new thing. I write a <a href=\"https:\/\/github.com\/guoyunhe\/RetroArch\/blob\/master\/intl\/README.md\">README<\/a> to introduce how to translate on Crowdin and how to run the synchronization script on Linux\/macOS\/Windows.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, make a PR and explain how it can improve the i18n workflow.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>RetroArch project uses *.h files to store translation strings. Here aren't any tools to make the translation process easier. When source strings changed, you have to manually review the changes, locate and update translation strings. It is a hard work. As a big fan of RetroArch, I was thinking if it can be improved with [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[11,10,13,12],"class_list":["post-125","post","type-post","status-publish","format-standard","hentry","category-code","tag-crowdin","tag-i18n","tag-python","tag-retroarch"],"_links":{"self":[{"href":"https:\/\/guoyunhe.me\/en\/wp-json\/wp\/v2\/posts\/125","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/guoyunhe.me\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/guoyunhe.me\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/guoyunhe.me\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/guoyunhe.me\/en\/wp-json\/wp\/v2\/comments?post=125"}],"version-history":[{"count":4,"href":"https:\/\/guoyunhe.me\/en\/wp-json\/wp\/v2\/posts\/125\/revisions"}],"predecessor-version":[{"id":131,"href":"https:\/\/guoyunhe.me\/en\/wp-json\/wp\/v2\/posts\/125\/revisions\/131"}],"wp:attachment":[{"href":"https:\/\/guoyunhe.me\/en\/wp-json\/wp\/v2\/media?parent=125"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/guoyunhe.me\/en\/wp-json\/wp\/v2\/categories?post=125"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/guoyunhe.me\/en\/wp-json\/wp\/v2\/tags?post=125"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}