January 2019 - issue - lists.texjp.org

[issue:254] Re: regexpatch: error on (u)pTeX with Japanese char
by Hironobu Yamashita 17 Jan '19

17 Jan '19

Hi Bruno, > Do you know why there are two separate > primitives \catcode and \kcatcode? > It seems that their functionality > could have been combined into a single one. I don't know the birth of pTeX, but I guess they cannot be merged because they have some different meanings. [1] Although both of these primitives take an argument which denotes a character code, the meaning is different. * \catcode (allowed values: 0--15) is meant for individual char code. e.g. \catcode`A=13 affects only the character "41 (= A). * \kcatcode (allowed values: 16--18 for pTeX, 15--19 for upTeX) is meant for char code range. The code range is different between pTeX and upTeX, because of its internal encodings. - pTeX can handle JIS X 0208 characters. These characters are classified into "ku"-"ten" table, so the range is based on "ku" ("区") in JIS standard. - upTeX can handle Unicode characters, so the range is based on Unicode blocks. e.g. \kcatcode`あ=18 affects all of Hiragana characters. Please consider the following example: \showthe\kcatcode`あ % => 17 \showthe\kcatcode`い % => 17 \kcatcode`あ=18 \showthe\kcatcode`あ % => 18 \showthe\kcatcode`い % => 18 "あ" and "い" are Hiragana characters, which are located in JIS 4-ku (for pTeX) and Unicode block 0x3040--0x309F (for upTeX). The default \kcatcode for these ranges are 17, but if we change that of "あ" to 18, all of Hiragana characters have 18. [2] upTeX's \kcatcode has additional meaning, compared to pTeX's. When (u)pTeX is given an input, JP characters and non-JP characters are distinguished during tokenization. The criteria here is different between pTeX and upTeX: * With pTeX, the criteria is always fixed: - char code 0--255: non-JP - others: JP * With upTeX, the criteria can be flexibly changed, even inside a single document. For this purpose, upTeX extended the \kcatcode primitive as follows: - if \kcatcode = 15 is set, non-JP (precisely "non-CJK") - if \kcatcode = {16,17,18,19} is set, JP (precisely "CJK") A simple example: % plain upTeX \showthe\kcatcode`^^c0 % => default: 15 (it only means non-JP) \showthe\catcode`^^c0 % => 12 \kcatcode`^^c0=18 % (change this Unicode block to JP) \showthe\kcatcode`^^c0 % => 18 \showthe\catcode`^^c0 % => returns 12, but not actually used. After the distinction, JP char will be converted to JP char token accompanying \kcatcode; non-JP char will be converted to non-JP char token accompanying \catcode. > I may be missing some reason why \catcode and \kcatcode are separate. > Of course the documentation is hard to read for non-Japanese speakers. Currently upTeX is not well documented, but we hope to provide some in the future... And also, English manuals for both pTeX and upTeX should be there. Hironobu

1 0

[issue:253] Fwd: regexpatch: error on (u)pTeX with Japanese char
by Hironobu Yamashita 17 Jan '19

17 Jan '19

Bruno さんからこんな問い合わせが来ました。次からは issue ML も送信先 (cc:) に含めるようにします…。 -- Hironobu Yamashita ---------- Forwarded message ---------- From: Bruno Le Floch <blflatex(a)gmail.com> Date: Wed, 16 Jan 2019 23:57:47 +0100 Subject: Re: regexpatch: error on (u)pTeX with Japanese char To: Hironobu Yamashita <h.y.acetaminophen(a)gmail.com> Cc: latex-team(a)latex-project.org Hello Hironobu, Enrico forwarded some simple code of yours to determine if a character is jp or not in ptex/uptex. Do you know why there are two separate primitives \catcode and \kcatcode? It seems that their functionality could have been combined into a single one. We may decide to provide only one higher-level expl3 function that would call either of these as needed. I may be missing some reason why \catcode and \kcatcode are separate. Of course the documentation is hard to read for non-Japanese speakers. Best regards, Bruno

1 0

[issue:252] Fwd: regexpatch: error on (u)pTeX with Japanese char
by Hironobu Yamashita 17 Jan '19

17 Jan '19

前に転送した egreg さんのメールに対する私の返信です。 -- Hironobu Yamashita ---------- Forwarded message ---------- From: Hironobu Yamashita <h.y.acetaminophen(a)gmail.com> Date: Wed, 16 Jan 2019 23:25:36 +0900 Subject: Re: regexpatch: error on (u)pTeX with Japanese char To: Enrico Gregorio <enrico.gregorio(a)univr.it> Hi, The problem seems to be solved, but just for your information: > If there is a sane way to check whether a character token will > give a character code illegal with \catcode, then I guess a > further check can be added when the engine is epTeX. I think you can detect whether a character token is JP char or non-JP char by following code, for example. By design, the behavior of \kcatcode is different between pTeX and upTeX (the latter one allows flexible changing of JP / non-JP classification), so different code should be used here. \ifx\enablecjktoken\@undefined \def\@ifisjp#1{% pTeX version \ifnum`#1<255\relax \expandafter\@secondoftwo % non-JP \else \expandafter\@firstoftwo % JP \fi} \else \def\@ifisjp#1{% upTeX version \ifnum\kcatcode`#1=15\relax \expandafter\@secondoftwo % non-JP \else \expandafter\@firstoftwo % JP \fi} \fi The test case is shown at https://gist.github.com/aminophen/154cb16cfe989dd605b0983f3e3bd20b Best regards, Hironobu

1 0

[issue:251] Fwd: regexpatch: error on (u)pTeX with Japanese char
by Hironobu Yamashita 16 Jan '19

16 Jan '19

さらにもう一本転送します。 -- Hironobu Yamashita ---------- Forwarded message --------- From: Enrico Gregorio <enrico.gregorio(a)univr.it> Date: 2019年1月16日(水) 7:39 Subject: Re: regexpatch: error on (u)pTeX with Japanese char To: Hironobu Yamashita <h.y.acetaminophen(a)gmail.com> Good news! Bruno has found a workaround that needs no juggling with category codes. https://github.com/latex3/latex3/commit/df76fcaaa224fee558dda954c2c57d51b88… You’ll have to wait for the next update, though. The test file compiles with both platex and uplatex, With \xshowcmd\hoge before and after the patch: > \hoge=macro:->ああ，ほげとほげがぴよ。. <recently read> } l.8 \xshowcmd\hoge ? > \hoge=macro:->ああ，ふがとほげがぴよ。. <recently read> } l.12 \xshowcmd\hoge ? Ciao Enrico

1 0

[issue:250] Fwd: regexpatch: error on (u)pTeX with Japanese char
by Hironobu Yamashita 16 Jan '19

16 Jan '19

もう一本転送します。 -- Hironobu Yamashita ---------- Forwarded message --------- From: Enrico Gregorio <enrico.gregorio(a)univr.it> Date: 2019年1月16日(水) 4:14 Subject: Re: regexpatch: error on (u)pTeX with Japanese char To: Hironobu Yamashita <h.y.acetaminophen(a)gmail.com> Now I know better about the issue. Of course, e-TeX is required, but this you already were aware of. The main problem is that `\tl_set_rescan:Nnn needs to check whether the text to be rescanned starts with a space and to do this it looks at the first token. If the token is not a control sequence, the category code is checked, but \catcode doesn’t work with kanji or kana characters (section 10 in the pTeX manual, if I understand it correctly with the help of Google). The team command of Japanese is very scarce (big understatement). Maybe \kcatcode could help in this case, but the details are only in Japanese. If there is a sane way to check whether a character token will give a character code illegal with \catcode, then I guess a further check can be added when the engine is epTeX. Ciao Enrico

1 0

[issue:249] Fwd: regexpatch: error on (u)pTeX with Japanese char
by Hironobu Yamashita 16 Jan '19

16 Jan '19

転送します。 -- Hironobu Yamashita ---------- Forwarded message --------- From: Enrico Gregorio <enrico.gregorio(a)univr.it> Date: 2019年1月15日(火) 22:26 Subject: Re: regexpatch: error on (u)pTeX with Japanese char To: Hironobu Yamashita <h.y.acetaminophen(a)gmail.com> > On 15 Jan 2019, at 04:41, Hironobu Yamashita <h.y.acetaminophen(a)gmail.com> wrote: > > \documentclass{article} > \usepackage{regexpatch} > \def\hoge{ああ，ほげとほげがぴよ。} > \begin{document} > Before: \hoge > > \xpatchcmd*{\hoge}{ほげ}{ふが}{}{} > > After: \hoge > \end{document} This is a lower level problem: the same error is produced by \documentclass{article} \usepackage{expl3} \ExplSyntaxOn \tl_set_rescan:Nnn \l_tmpa_tl {} {あ} that reports Bad character code (12354) with uplatex and Bad character code (42146) with platex. I’ll mail the list. Ciao Enrico

1 0

[issue:248] regexpatch: error on (u)pTeX with Japanese char
by Hironobu Yamashita 15 Jan '19

15 Jan '19

Hi Enrico, The following source causes an error on pLaTeX and upLaTeX. > ! Bad character code (42146). > \char_value_catcode:n ...val:w #1\__int_eval_end: > > l.19 \xpatchcmd*{\hoge}{ほげ}{ふが}{}{} > ^^M > A character number must be between 0 and 255. > I changed this one to zero. % ====== \documentclass{article} \usepackage{regexpatch} \def\hoge{ああ，ほげとほげがぴよ。} \begin{document} Before: \hoge \xpatchcmd*{\hoge}{ほげ}{ふが}{}{} After: \hoge \end{document} % ====== (e-)(u)pTeX does not allow character code > 255 for \catcode, \sfcode, etc. For the time being, we can avoid the error by adding empty braces at the beggining like \def\hoge{{}ああ，ほげとほげがぴよ。} but I hope the issue can be resolved on the package. Best, Hironobu --- Hironobu YAMASHITA (GitHub: @aminophen)

1 0

[issue:247] Re: arydshln incompatible with new longtable.sty
by Hiroshi Nakashima 09 Jan '19

09 Jan '19

山下様ご連絡ありがとうございます。longtable の最新版を入手して \color@{begin|end}group が加えられていることを確認しました。なるべく早目に対応します。中島＠京大 -----Original Message----- From: Hironobu Yamashita Sent: Wednesday, January 09, 2019 7:43 AM To: Hiroshi Nakashima Cc: Japanese TeX development issues Subject: arydshln incompatible with new longtable.sty 中島先生山下です。あけましておめでとうございます。先日は，新しい array への対応ありがとうございました。今度はやはり，新しい longtable と arydshln が衝突するようです。前回の TeX Forum と同じソースでエラーが再現します。 https://oku.edu.mie-u.ac.jp/tex/mod/forum/discuss.php?d=2550#p14912 お時間のある時にご対応いただけると幸いです。宜しくお願い致します。 ---- 山下弘展 (Hironobu Yamashita) e-mail: h.y.acetaminophen(a)gmail.com

1 0

[issue:246] arydshln incompatible with new longtable.sty
by Hironobu Yamashita 09 Jan '19

09 Jan '19

中島先生山下です。あけましておめでとうございます。先日は，新しい array への対応ありがとうございました。今度はやはり，新しい longtable と arydshln が衝突するようです。前回の TeX Forum と同じソースでエラーが再現します。 https://oku.edu.mie-u.ac.jp/tex/mod/forum/discuss.php?d=2550#p14912 お時間のある時にご対応いただけると幸いです。宜しくお願い致します。 ---- 山下弘展 (Hironobu Yamashita) e-mail: h.y.acetaminophen(a)gmail.com

1 0