Repository navigation
Conversation
Escape::html ran nine sequential str_replace passes over the full rendered output. The escape keys and token keys are disjoint and strtr never rescans replaced text, so one strtr pass gives identical output at about a third of the cost.
Twig tags and GDScript ran one preg_match_all per keyword (38 and 36 patterns). Every keyword is a whole word bounded by \b, so a single alternation yields exactly the same tokens with one regex pass.
…guages The pattern and injection caches were keyed by spl_object_id. Injections created a new language object for every match, so the caches missed on each nested parse, and a freed object's id could be reused by a different language, which then got the stale patterns. Store the caches in a WeakMap keyed by the language instance, and let the four injections that built a language per match reuse one instance.
…ting Patterns used (.|\n)*? to match across lines. PCRE handles the alternation group with one backtracking frame per character, so a block of roughly 50KB or more exhausts the JIT stack. preg_match_all then returns false and the block is silently left unhighlighted. [\s\S] matches the same characters as (.|\n) without the group, so the output is unchanged and the size limit is gone.
ArtMin96
marked this pull request as draft
October 20, 2026 08:53
…ctions Lets a pattern or injection declare substrings it needs, so the highlighter can skip it when the content contains none of them.
…-8 errors
Unclosed {- and similar markers made the marker injections rescan to the
end of the content for every opener. 100KB of them took ~650ms. Matching
now stops after the last valid closer, and Addition/Deletion build their
output in one pass instead of one substr_replace per marker.
A reused withGutter() highlighter kept line marks from the previous
parse, and content with invalid UTF-8 made the gutter throw a TypeError
because preg_split returned false. Marks are now cleared per parse, and
a byte-level split with the same line breaks is used as a fallback.
The Base markers also declare the substrings they need, so the
highlighter can skip them.
…bugs Lazy (\\"|.)*? string patterns and (.)* comment patterns exhausted the PCRE JIT stack on a single line of ~6KB, after which every later match of the same pattern was silently lost. The rewrites match the same text without per-character backtracking frames. Also: - PHP keywords: (?<!\$) in a double-quoted PHP string compiled to the end anchor (?<!$), so $new in function parameters got a keyword span. - Python: a missing | meant 'and' and '_' were never keywords. - StaticClassCallPattern was quadratic on long identifiers. - RubyLexer kept the last input in a static property for the life of the process; it is now held per language instance. - Comment patterns stop at an unclosed opener with (*COMMIT) instead of rescanning to the end (quadratic without JIT).
…mory - Built-in patterns that use IsPattern::match() get their final regex and token type resolved once per language instead of on every parse. - Patterns and injections marked with #[RequiresSubstring] are skipped when the content contains none of the substrings. - Tokens from one pattern are cloned from a prototype, and render and grouping loops do less work per token. - Tokens are sorted on one packed integer key instead of array_multisort over four arrays. A 2MB PHP file no longer runs out of memory at 128M. - RenderTokens only caches markup for enum token types, keyed by class and case. Custom class names grew the cache without bound, and a custom TokenType returning a built-in value got the wrong markup. Adds EdgeCaseBench for the pathological inputs.
Injections declare the literal every match needs, so the highlighter skips them on content that cannot match. PhpGenericTypeInjection now uses one constant pattern instead of building a regex per type name, which also stops type names from corrupting earlier markup. Injection regexes stop at an unclosed opener with (*COMMIT).
ArtMin96
changed the title
ArtMin96
marked this pull request as ready for review
October 20, 2026 11:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Highlighting is about 32% faster across the 35 bench fixtures (Twig 63%, Blade 40%). Output is byte-identical except where it was wrong. Profiling turned up a number of bugs, most of them also on
main, and they are fixed here with a failing test each.Performance
Escape::htmlis onestrtrpassstr_replacepasses over the whole output. The keys don't overlap, so one pass gives the same result.preg_match_allper keyword (38 and 36). Whole-word keywords give the same tokens from one alternation."/{$pattern}/"andgetPattern()were rebuilt on every call, including nested parses. CustomPatternclasses still go throughmatch().#[RequiresSubstring]attributearray_multisortover four arrays was the peak memory point.Bug fixes
mainspl_object_id{-markersmemory_limit=128MFoo::after a 50K-char identifierfunction f($new)$<keyword>new</keyword>(lookbehind compiled to an end anchor)and,_|)withGutter()highlighterTypeError{:class:...:}with many class namesRubyLexerPhpGenericTypeInjectionBenchmarks
phpbench with Xdebug off.
main, the first commit set and this branch ran alternately, and the faster of two rounds counts.mainEvery language got faster. On 50KB inputs Twig went from 15.0 to 4.8ms, Blade from 12.2 to 5.2ms and PHP from 7.1 to 4.8ms.
EdgeCaseBenchcovers the pathological inputs above.Behaviour changes
Gutter marks now reset after each parse. Marks added with
addIcon/addClassbefore callingparse()still apply to that parse. Ruby patterns accept an optional sharedRubyLexer, and the default stays as before.Verification
I snapshotted 8,388 outputs from
main: every fixture parsed as every language, three themes, gutter on and off, 30KB inputs and adversarial snippets. The only differences are the fixes above. Running the branch with skipping disabled gives identical output over that corpus and 180k fuzzed inputs. Every regex rewrite was fuzzed against the original with no differences, apart from the intendedPhpGenericTypeInjectionfix. There are 1,197 tests; phpstan and php-cs-fixer are clean, and each commit passes on its own.Left for later
Swift's recursive patterns and Markdown emphasis still fail on very long lines. Svelte and Vue highlight some content twice, Blade echo and multiple
<script>blocks match greedily, and Pythonf(a = 1)gets no argument token. Custom classes with many distinct names are still quadratic,PhpHeredocInjectionbuilds a regex per language name, and blocks of about 1MB hitpcre.backtrack_limit.