Discover spec directories without globbing every spec file

[connorshea]

AI Disclosure: This was generated with help from Claude Code, Opus 5/Fable 5. I have reviewed and tested this change manually, and written this PR description myself.

Fixes #2912.

Previously, the rspec_rails.code_statistics initializer was enumerating every file under every directory in spec with a *_spec.rb name just to find the spec directories that exist. This ran on every boot of the app, so it was done on all Rails 8+ apps whenever tests were run. It also meant that larger repositories would spend more time on this as they grew. It only ends up resulting in maybe 40ms of difference on my Mac when tested with our large production app at work, but that's a small boot penalty on every test run and CI runners are generally less powerful than my dev machine, so I think it's worth fixing.

This change gets the exact same results as the previous implementation, but it stops after the first _spec.rb file is found in each directory, and so we can avoid extra work in most cases. I have tested the rails stats command against mastodon, discourse, gitlab, canvas, my own small personal rails app, and our app at work, and gotten the exact same results before and after this PR on all of them, as well as some generated test apps to look for any risk of breakage.

Benchmarking, courtesy of Claude, 30 iterations each:

Tree	Before	After	Change
This repo (rspec-rails, 2 spec dirs)	0.56 ms	0.31 ms	1.8x faster
Mastodon (22 spec dirs, 1,089 spec files)	6.07 ms	1.71 ms	3.5x faster
Discourse (21 spec dirs, 1,691 spec files)	9.18 ms	3.51 ms	2.6x faster
Canvas (20 spec dirs, 2,680 spec files)	15.63 ms	2.38 ms	6.6x faster
GitLab (47 spec dirs, 12,091 spec files)	106.42 ms	37.87 ms	2.8x faster
Synthetic, issue scale (960 dirs, 1,890 spec files, 30 top-level)	24.10 ms	1.15 ms	21.0x faster
Large, specs directly in every top-level dir (3,750 dirs, 10,000 spec files)	113.19 ms	4.19 ms	27.0x faster
Large, specs only in leaf dirs (6,500 dirs, 10,000 spec files)	153.46 ms	96.90 ms	1.6x faster
Large, specs only in leaves + spec-free 600-dir spec/factories (7,101 dirs, 10,000 spec files)	161.90 ms	108.55 ms	1.5x faster
Adversarial (900-subdir spec/support with no specs; one spec buried 5 deep)	11.81 ms	12.26 ms	~1.0x — effectively the same

Benchmark script

#!/usr/bin/env ruby
# frozen_string_literal: true

# Benchmarks the spec directory discovery done by rspec-rails' `rspec_rails.code_statistics`
# initializer (Rails 8+), comparing the implementation before and after rspec/rspec-rails#2914.
#
# Only reads the filesystem: no Bundler, no Rails, and no need to `bundle install` the app.
#
#   ruby bench_code_statistics.rb /path/to/rails/app [iterations]

require 'benchmark'
require 'pathname'

root = Pathname(ARGV[0] || Dir.pwd).expand_path
iterations = Integer(ARGV[1] || 30)
abort "No spec directory found in #{root}" unless root.join('spec').directory?

# Before: rspec-rails 8.0.4 (lib/rspec-rails.rb)
before = lambda do
  dirs = Dir[root.join('spec', '**', '*_spec.rb').to_s]
           .map { |f| f.sub(%r{^#{Regexp.escape(root.to_s)}/(spec/\w+)/.*}, '\\1') }
           .uniq
           .select { |f| File.directory?(root.join(f)) }
  Hash[dirs.map { |d| [d.split('/').last, d] }].map { |type, dir| [type, root.join(dir).to_s] }.sort
end

# After: rspec/rspec-rails#2914
after = lambda do
  Dir[root.join('spec', '*').to_s].sort.filter_map do |dir|
    type = File.basename(dir)
    next unless type.match?(/\A\w+\z/) && File.directory?(dir) && !File.symlink?(dir)
    next if Dir.glob('*_spec.rb', base: dir).empty? && Dir.glob('**/*_spec.rb', base: dir).empty?

    [type, dir]
  end
end

before_result = before.call
after_result = after.call
unless before_result == after_result
  warn "Results differ!\n  before: #{before_result.map(&:first)}\n  after:  #{after_result.map(&:first)}"
  exit 1
end

measure = lambda do |impl|
  impl.call # warm up
  Benchmark.realtime { iterations.times { impl.call } } / iterations * 1000
end

before_ms = measure.call(before)
after_ms = measure.call(after)
spec_files = Dir.glob('spec/**/*_spec.rb', base: root).size.to_s.gsub(/(\d)(?=(\d{3})+$)/, '\1,')

puts "Discovered #{after_result.size} spec directories: #{after_result.map(&:first).join(', ')}"
puts "#{iterations} iterations, mean per run:"
puts
puts '| App | Before | After | Change |'
puts '|---|---|---|---|'
puts format('| %s (%d spec dirs, %s spec files) | %.2f ms | %.2f ms | **%.1fx faster** |',
            root.basename, after_result.size, spec_files, before_ms, after_ms, before_ms / after_ms)

The main reason the synthetics are significantly faster is because real apps have directories like fixtures and factories that never have spec files in them (so we have to read every file in them, we can't bail early), and every file in those directories has to be checked. We could potentially cut further time off by just excluding those two directories, but it technically comes with the risk of someone having actual tests in one of those two directories. I'm unsure whether Rails/rspec would actually work if you did that, but I don't know for sure and I'd rather not complicate this change.

We can also bench the memory allocations and see that a lot less memory allocation churn is needed now that we aren't using a regex or loading nearly as many individual file paths. It's only KB or MB at most and is transient, but still nice for reducing GC pressure a bit:

Repo	Before (allocations / memory)	After (allocations / memory)	Change
rspec-rails (2 spec dirs, 58 spec files)	1,248 objects / 128 KB	124 objects / 7 KB	10x fewer objects, 18x less memory
Mastodon (22 spec dirs, 1,089 spec files)	20,258 objects / 2,100 KB	821 objects / 62 KB	25x fewer objects, 34x less memory
Discourse (21 spec dirs, 1,691 spec files)	30,418 objects / 3,263 KB	1,430 objects / 116 KB	21x fewer objects, 28x less memory
Canvas LMS (20 spec dirs, 2,680 spec files)	47,162 objects / 5,441 KB	1,156 objects / 93 KB	41x fewer objects, 58x less memory
GitLab (47 spec dirs, 12,091 spec files)	209,157 objects / 22,578 KB	2,390 objects / 217 KB	88x fewer objects, 104x less memory

[pirj]

I don't have access to any large codebase now, @bquorning do you think you can test such a patch for performance improvements?

[bquorning]

On a repo with 28 spec folders and ~1700 spec files, I get these numbers:

ruby 3.4.9 (2026-03-11 revision 76cca827ab) +PRISM [arm64-darwin24]
Warming up --------------------------------------
             pr_2914    13.000 i/100ms
        sha_1fc6126e     2.000 i/100ms
Calculating -------------------------------------
             pr_2914    152.493 (±10.5%) i/s    (6.56 ms/i) -    754.000 in   5.020119s
        sha_1fc6126e     23.718 (± 8.4%) i/s   (42.16 ms/i) -    118.000 in   5.022581s

Comparison:
             pr_2914:      152.5 i/s
        sha_1fc6126e:       23.7 i/s - 6.43x  slower

[connorshea]

I've also tested it on my rails app at work to confirm:

old: 41.21 ms
new: 6.64 ms
speedup: 6.2x  (saves 34.58 ms per boot)

Best as I can tell, the spec failures in this PR are unrelated to this change and are present on main as well.

[pirj]

Thanks for the effort.
Big question is: is shaving off 40ms on a big project worth making such a change? Please correct me if I’m mistaken, but it may have edge cases, and brings in risks.

[connorshea]

Thanks for the effort. Big question is: is shaving off 40ms on a big project worth making such a change? Please correct me if I’m mistaken, but it may have edge cases, and brings in risks.

This is a valid concern, but I've tested it on a few real codebases and a bunch of synthetic codebases with whatever edge cases I could think of, and it has output the same exact data for every one of them before and after.

Even if it's only saving 40ms or 100ms on a large app, that's still significant when you're running 50 parallel CI rspec jobs on every PR, hundreds of times every day.

The worst downside if this has an edge case I haven't handled would be that it miscounts things in the rails stats, but I'm fairly confident that shouldn't happen. And I'm confident that there's not any case where this would be capable of causing a crash or anything like that.

I'm happy to pull a few more major open source rails apps to confirm the data output is identical if you want me to, though.

[pirj]

In my experience of improving CI wall clock time on large Rails projects, up to hundreds of hours if run sequentially, the effort never gets to improve things that are measured in microseconds. 50% of time is usually spent on factories, a significant lag before RSpec can even run anything - on preparing dependencies, starting up docker containers, running migrations. For browser tests major contributors are db truncation (with a solution that tends to break now and then) and long manual timeouts to wait for JS. Every single thing on this list is measured in seconds, not fractions of.

I'm proud and jelous if your project is at a point where you've squeezed all that to the point where there's nothing else left than to optimize RSpec itself, which, with a few exceptions, is the last one to blame for being a performance sink.

[connorshea]

In my experience of improving CI wall clock time on large Rails projects, up to hundreds of hours if run sequentially, the effort never gets to improve things that are measured in microseconds. 50% of time is usually spent on factories, a significant lag before RSpec can even run anything - on preparing dependencies, starting up docker containers, running migrations. For browser tests major contributors are db truncation (with a solution that tends to break now and then) and long manual timeouts to wait for JS. Every single thing on this list is measured in seconds, not fractions of.

I'm proud and jelous if your project is at a point where you've squeezed all that to the point where there's nothing else left than to optimize RSpec itself, which, with a few exceptions, is the last one to blame for being a performance sink.

I certainly don't disagree with any of this. This change isn't going to revolutionize anything, but I do think boot time especially is important to get down as low as possible, because it's a cost incurred on every boot, no matter how much you try to parallelize a massive test suite. And it continues to scale with the number of files you have in your suite.

I've updated the PR to be as minimal as possible and updated the benchmark timings in the PR description accordingly. This is now a much simpler change, at the cost of a bit of efficiency, but that seems fine to me.

[JonRowe]

👋 I'm away at the moment but have had a look at this, I'm inclined to merge this once I've had time to sanity check it has the same behaviour, however it will need to be squashed / rewritten to remove Claude from the commits. Its fine to use LLMs to assist you given you've declared it, however commits should be written by a human author who is responsible for the content.

[connorshea]

I've squashed this PR into one commit and removed the co-author attributions accordingly.

[connorshea]

Made one more tweak to make it a bit faster on various real codebases (going back to using globbing instead of Find, but still bailing early in most cases. This is better than the previous iteration because it doesn't cost as much when a Rails app has a large amount of files in support, fixtures, or factories that aren't spec files). I've also updated the PR description with benchmarks from various open source Rails apps.

[JonRowe]

The one thing that still stands out here is the change from relative paths to absolute, whilst I'm not sure if this affects anything it is a change in behaviour and not how the Rails internals work which makes me want to align with them

[connorshea]

The one thing that still stands out here is the change from relative paths to absolute, whilst I'm not sure if this affects anything it is a change in behaviour and not how the Rails internals work which makes me want to align with them

I've updated it to return the relative path now 👍