Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-11T05:06:59.971Z
Learning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust

Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-11T05:06:59.971Z
Learning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust

Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-11T05:06:59.971Z
Learning: Applies to Lib/**/*.py : Minimize modifications to CPython standard library files; only make minimal changes to work around RustPython limitations

Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-11T05:06:59.971Z
Learning: Applies to extra_tests/**/*.py : Use ruff to lint custom Python code

stdlib/src/tokenize.rs (4)

39-42: Don't swallow non-StopIteration errors from readline.

Catching every exception and treating it as EOF hides real failures (TypeError, UnicodeDecodeError, etc.). Only StopIteration should terminate the stream; all other errors must propagate.

Apply this diff to fix the error handling:

-            let raw_line = match self.readline.invoke((), vm) {
-                Ok(v) => v,
-                Err(_) => return Ok(String::new()),
-            };
+            let raw_line = match self.readline.invoke((), vm) {
+                Ok(v) => v,
+                Err(exc) => {
+                    if exc.fast_isinstance(vm.ctx.exceptions.stop_iteration) {
+                        return Ok(String::new());
+                    }
+                    return Err(exc);
+                }
+            };

---

116-138: End-of-file tuple needs to come from the real EOF token.

Synthesizing the EOF tuple here hardcodes (line, -1) and reuses the previous line text, so consumers see the wrong coordinates and line content. Instead, retrieve the parser's EndOfFile token, set it as prev_token, and let the existing emission path produce the correct tuple.

Apply this diff to use the parser's EOF token:

                 if nline.is_empty() {
                     state.eof = true;
-                    *zelf.state.write() = state.clone();
-
-                    let line_num = &state.start().0;
-                    let out = vm
-                        .ctx
-                        .new_tuple(vec![
-                            token_kind_value(TokenKind::EndOfFile).to_pyobject(vm),
-                            vm.ctx.new_str("").into(),
-                            vm.ctx
-                                .new_tuple(vec![line_num.to_pyobject(vm), (-1).to_pyobject(vm)])
-                                .into(),
-                            vm.ctx
-                                .new_tuple(vec![line_num.to_pyobject(vm), (-1).to_pyobject(vm)])
-                                .into(),
-                            vm.ctx.new_str(state.current_line()).into(),
-                        ])
-                        .into();
-                    return Ok(PyIterReturn::Return(out));
+                    if let Some(eof) = state
+                        .tokens
+                        .iter()
+                        .rev()
+                        .find(|token| token.kind() == TokenKind::EndOfFile)
+                    {
+                        state.prev_token = Some(*eof);
+                        break *eof;
+                    }
+                    *zelf.state.write() = state.clone();
+                    return Ok(PyIterReturn::StopIteration(None));
                 }

---

151-154: Don't trim token text when extra_tokens is false.

trim() removes indentation whitespace and other significant padding, causing INDENT tokens to become empty strings. CPython preserves the exact lexeme. Use the slice as-is without trimming.

Apply this diff to preserve the exact lexeme:

-            let mut token_repr = &state.source[state.range()];
-            if !zelf.extra_tokens {
-                token_repr = token_repr.trim();
-            }
+            let token_repr = &state.source[state.range()];

---

178-180: extra_tokens must default to False.

CPython's TokenizerIter allows calling TokenizerIter(readline) without naming the flag. Requiring the argument here breaks API compatibility.

Apply this diff to make extra_tokens optional with a default:

         #[pyarg(positional)]
         readline: ArgCallable,
-        #[pyarg(named)]
-        extra_tokens: bool,
+        #[pyarg(named, default = "false")]
+        extra_tokens: bool,
         #[pyarg(named, optional)]
         encoding: Option<PyStrRef>,

stdlib/src/tokenize.rs (2)

108-109: Reminder: Address the TODO comment for error handling.

The TODO indicates that error checking (e.g., raising SyntaxError for parse errors) is missing. The state.errors field is populated but never checked.

Do you want me to generate code to check state.errors and raise appropriate SyntaxError exceptions, or open a new issue to track this task?

---

200-207: Re-parsing the entire source on each line may be inefficient.

The push_line method re-parses the entire accumulated source text each time a new line is added. For large files, this could become a performance bottleneck with O(n²) behavior.

However, this approach may be necessary for correct tokenization since Python's grammar is context-sensitive (e.g., for indent/dedent tracking). Consider profiling to determine if incremental parsing or caching strategies would be beneficial.

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-11T05:06:59.971Z
Learning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust

Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-11T05:06:59.971Z
Learning: Applies to Lib/**/*.py : Minimize modifications to CPython standard library files; only make minimal changes to work around RustPython limitations

Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-11T05:06:59.971Z
Learning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust

stdlib/src/tokenize.rs (5)

121-143: EOF tuple must use the parser's actual EndOfFile token.

The code synthesizes the EOF tuple with hardcoded (-1) positions (lines 132, 135) and reuses state.current_line(), which produces incorrect coordinates and line content. Instead, retrieve the parser's EndOfFile token from state.tokens, set it as prev_token, and let the standard token-emission path (lines 145-175) produce the tuple with correct metadata.

Apply this approach:

                 let nline = zelf.readline(vm)?;
                 if nline.is_empty() {
                     state.eof = true;
-                    *zelf.state.write() = state.clone();
-
-                    let line_num = &state.start().0;
-                    let out = vm
-                        .ctx
-                        .new_tuple(vec![
-                            token_kind_value(TokenKind::EndOfFile).to_pyobject(vm),
-                            vm.ctx.new_str("").into(),
-                            vm.ctx
-                                .new_tuple(vec![line_num.to_pyobject(vm), (-1).to_pyobject(vm)])
-                                .into(),
-                            vm.ctx
-                                .new_tuple(vec![line_num.to_pyobject(vm), (-1).to_pyobject(vm)])
-                                .into(),
-                            vm.ctx.new_str(state.current_line()).into(),
-                        ])
-                        .into();
-                    return Ok(PyIterReturn::Return(out));
+                    if let Some(eof) = state
+                        .tokens
+                        .iter()
+                        .find(|token| token.kind() == TokenKind::EndOfFile)
+                    {
+                        state.prev_token = Some(*eof);
+                        break *eof;
+                    }
+                    *zelf.state.write() = state.clone();
+                    return Ok(PyIterReturn::StopIteration(None));
                 }

---

156-159: Remove trim() to preserve exact token lexemes.

Calling trim() when extra_tokens is false (line 158) strips significant whitespace, causing INDENT tokens to become empty strings. CPython's tokenizer preserves the exact lexeme. Use the slice as-is.

Apply this diff:

-            let mut token_repr = &state.source[state.range()];
-            if !zelf.extra_tokens {
-                token_repr = token_repr.trim();
-            }
+            let token_repr = &state.source[state.range()];

---

183-184: Make extra_tokens optional with default false.

CPython allows TokenizerIter(readline) without specifying extra_tokens. The current declaration requires the argument, breaking API compatibility.

Apply this diff:

-        #[pyarg(named)]
-        extra_tokens: bool,
+        #[pyarg(named, default = "false")]
+        extra_tokens: bool,

---

227-243: Token ordering skips tokens that overlap the current range.

When prev_token is None, self.range() returns TextRange::default() (0..0). The comparison range.ordering(self.range()) > Ordering::Greater (line 236) only matches tokens completely after the current range. Tokens starting at position 0 (e.g., 0..1) overlap with 0..0, returning Ordering::Equal, and are skipped. The first token in the source is never yielded.

Fix by comparing start positions:

-                if matches!(range.ordering(self.range()), Ordering::Greater) {
+                if range.start() > self.range().end() {
                     self.prev_token = Some(*token);
                     return self.prev_token;
                 }

---

286-391: Critical token mapping errors: duplicates and CPython conflicts.

Multiple critical issues remain:

1. Duplicate value 62 (lines 384-385): Both Comment and TStringStart map to 62
2. Incorrect Comment mapping (line 384): Maps to 62, but CPython 3.14 has COMMENT=61
3. FString conflicts (lines 381-383): Values 59, 60, 61 correspond to SOFT_KEYWORD, ERRORTOKEN, and COMMENT in CPython 3.14
4. Exclamation collision (line 380): Maps to 54, which is CPython's OP token
5. Misleading comments (lines 385-387): "3.14 compatible" claims are inaccurate

These mappings must be corrected to either:

• Match CPython 3.14 standard token values exactly, or
• Use reserved/unassigned values (e.g., 100+) to avoid collisions

Run this verification to confirm no remaining duplicates after fixes:

#!/bin/bash
# Extract all token value assignments and check for duplicates
rg -n "TokenKind::\w+ => \d+" stdlib/src/tokenize.rs | \
  gawk '{print $NF}' | sort | uniq -d | \
  xargs -I {} echo "Duplicate token value found: {}"

stdlib/src/tokenize.rs (2)

189-212: State structure is correct; consider performance implications.

The push_line method reparses the entire accumulated source on each call (line 208). While correct, this has O(n²) complexity for n lines. This is acceptable for now but may need optimization for large files.

---

113-118: TODO: Implement syntax error handling.

The TODO on line 114 indicates that parse errors from state.errors are not currently raised as SyntaxError. This means invalid Python syntax may be silently ignored. Consider implementing error checking in a follow-up to ensure proper error reporting.

Based on learnings.

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-11T05:06:59.971Z
Learning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust

Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-11T05:06:59.971Z
Learning: Applies to Lib/**/*.py : Minimize modifications to CPython standard library files; only make minimal changes to work around RustPython limitations

Learnt from: CR
Repo: RustPython/RustPython PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-09-11T05:06:59.971Z
Learning: Applies to {vm,stdlib}/**/*.rs : Use RustPython macros (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust