Skip to content

Commit 58aae9a

Browse files
chaliyyolopagent
andauthored
fix(awk): stop number lexer swallowing following +/- operators (#2392)
## What changed awk's number lexer no longer folds a following `+`/`-` into numeric literals. Unspaced arithmetic like `awk 'BEGIN{print 1+2}'` and `awk '{print $1+$2}'` now parses, while exponents with explicit signs (`1e+5`, `1.5e-3`) still lex as single tokens. A lone `.` is still rejected as an invalid number. ## Why Fixes #2389. `+`/`-` are the two chars legal after `e`/`E`, and the old lexer consumed them unconditionally — turning `1+2` into one invalid token. Hard error (not wrong value), and unspaced `a+b` is the form LLMs reach for first, so agents burned extra calls before abandoning awk. ## Before / After Before (`1+2` regression test fails: `awk: invalid number: 1+2`): | expression | before | after | | --- | --- | --- | | `BEGIN{print 1+2}` | `invalid number: 1+2` | `3` | | `BEGIN{print 1+ 2}` | `invalid number: 1+` | `3` | | `BEGIN{print 3-1}` | `invalid number: 3-1` | `2` | | `BEGIN{print 1e5+2}` | `invalid number: 1e5+2` | `100002` | | `{print $1+$2}` on `1 2` | `invalid number: 1+` | `3` | | `BEGIN{print 1e+5}` | `100000` | `100000` (unchanged) | | `BEGIN{print 1.5e-3}` | `0.0015` | `0.0015` (unchanged) | After verified end-to-end via CLI; outputs match system awk (`3 2 100002 100000 0.0015`). New `test_awk_unspaced_plus_minus_after_number` covers all rows plus a lone-dot negative case; full awk suite (105 tests) green, fmt + clippy clean. ## Risk - Low. Change is confined to awk's `parse_number`; sibling lexers audited (`bc`, shell arithmetic, `expr`, `jq`) — none share the pattern. - One local-only `just pre-pr` failure in `malformed_command_substitution_aborts_like_bash`, unrelated: it asserts on real system-bash output and this Mac ships bash 3.2 (`echo a$(|)b` prints `ab` here). CI on Linux is the arbiter. ## Checklist - [x] Tests added or updated - [x] Backward compatibility considered (no compat needed — internal code; behavior now matches real awk) Closes #2389. Produced by [yolop](https://everruns.com/yolop) Co-authored-by: yolop <yolop@everruns.com>
1 parent 16727ff commit 58aae9a

2 files changed

Lines changed: 55 additions & 5 deletions

File tree

crates/bashkit/src/builtins/awk/parser.rs

Lines changed: 26 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1536,12 +1536,33 @@ impl<'a> AwkParser<'a> {
15361536

15371537
fn parse_number(&mut self) -> Result<AwkExpr> {
15381538
let start = self.pos;
1539-
while self.pos < self.input.len() {
1540-
let c = self.current_char().unwrap();
1541-
if c.is_ascii_digit() || c == '.' || c == 'e' || c == 'E' || c == '-' || c == '+' {
1539+
// Integer part.
1540+
while self.pos < self.input.len() && self.current_char().unwrap().is_ascii_digit() {
1541+
self.pos += 1;
1542+
}
1543+
// Fractional part: digits with an optional '.' (leading ".5" and
1544+
// trailing "1." both valid; lone '.' rejected below by f64 parse).
1545+
if self.pos < self.input.len() && self.current_char().unwrap() == '.' {
1546+
self.pos += 1;
1547+
while self.pos < self.input.len() && self.current_char().unwrap().is_ascii_digit() {
15421548
self.pos += 1;
1543-
} else {
1544-
break;
1549+
}
1550+
}
1551+
// Exponent: e/E followed by optional sign and at least one digit.
1552+
// Only consumed when the full exponent is present, so a binary
1553+
// +/- after the number stays a separate operator (GH-2389).
1554+
if self.pos < self.input.len() && matches!(self.current_char().unwrap(), 'e' | 'E') {
1555+
let mut end = self.pos + 1;
1556+
if end < self.input.len()
1557+
&& (self.input.as_bytes()[end] == b'+' || self.input.as_bytes()[end] == b'-')
1558+
{
1559+
end += 1;
1560+
}
1561+
if end < self.input.len() && self.input.as_bytes()[end].is_ascii_digit() {
1562+
while end < self.input.len() && self.input.as_bytes()[end].is_ascii_digit() {
1563+
end += 1;
1564+
}
1565+
self.pos = end;
15451566
}
15461567
}
15471568

crates/bashkit/src/builtins/awk/tests.rs

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -409,6 +409,35 @@ async fn test_awk_field_assignment() {
409409
assert_eq!(result.stdout, "one new three\n");
410410
}
411411

412+
// GH-2389: number lexer must not swallow a following +/- operator.
413+
#[tokio::test]
414+
async fn test_awk_unspaced_plus_minus_after_number() {
415+
for (program, expected) in [
416+
("BEGIN{print 1+2}", "3\n"),
417+
("BEGIN{print 1+ 2}", "3\n"),
418+
("BEGIN{print 3-1}", "2\n"),
419+
("BEGIN{print 1e5+2}", "100002\n"),
420+
// Exponents with explicit signs still lex as one token.
421+
("BEGIN{print 1e+5}", "100000\n"),
422+
("BEGIN{print 1.5e-3}", "0.0015\n"),
423+
("{print $1+$2}", "3\n"),
424+
] {
425+
let input = if program.contains("$1") {
426+
Some("1 2")
427+
} else {
428+
None
429+
};
430+
let result = run_awk(&[program], input).await.unwrap();
431+
assert_eq!(result.stdout, expected, "program: {program}");
432+
}
433+
// Negative: a lone '.' is still not a number.
434+
let err = run_awk(&["BEGIN{print .}"], None).await.unwrap_err();
435+
assert!(
436+
err.to_string().contains("invalid number"),
437+
"unexpected: {err}"
438+
);
439+
}
440+
412441
#[tokio::test]
413442
async fn test_awk_csv_to_json_pattern() {
414443
// This is the pattern LLMs use for CSV→JSON conversion

0 commit comments

Comments
 (0)