Benchmarking Skill Search for Engineering Firms: Lessons from Wade Trim’s WTSS

Wade Trim Skill Search Dataset (WTSS) puts keyword and semantic search to the test on engineering text and shows how to tune candidate lists and use LLM ranking with guardrails.

Staffing pressure does not end when an engineering firm fills an open role. It continues every time a scope shifts or a deadline moves up. Engineering leaders still have to locate the right experience across the firm, and that experience is often buried in project records and described in inconsistent terms.

The shortage compounds the problem. The United States faces an annual shortage of 18,000 engineers, according to the ACEC Research Institute’s report titled “The Workforce of the Future.” That makes existing staff — people versed in company culture and experience — more valuable and harder to replace. It also raises the cost of slow internal expertise discovery — something many companies attempt to solve using complex, hard-to-manage tools, and skills matrices.

In response to this challenge, Wade Trim built an internal engineering skill search platform using machine learning and natural language processing techniques. The platform supports flexible searching on free-form text entries that reflect how engineers describe their work. This reduces the need for excessive standardization and allows engineers and managers to describe the work they do in their own words.

This article draws from their technical whitepaper on WTSS, a synthetic benchmark dataset that tests how modern retrieval methods perform on engineering text. We’ll focus on what engineering leaders can take from the findings and act on.

Why Expertise Discovery Is Now a Staffing Advantage

As companies grow in size and teams become more disconnected geographically and technically siloed, project managers and practice leaders routinely need to locate specific experience that fits the project scope. When that expertise retrieval depends on personal memory and who happens to know whom, the process can become uneven, leaving junior or new managers without the relationships to quickly find the right person. Managers lose time searching back and forth, often ending up in an endless loop.

Robert Sinclair, advanced design technology practice lead at Wade Trim, describes the daily reality clearly: “In an engineering firm, staffing is a constant ‘expert-finding’ problem: project managers need to quickly locate people with the right experience for a scope, deadline, and clients.” Those delays compress decision time and increase the chance of a mismatch. When the wrong experience is assigned, the cost can surface later as rework, quality issues, or diminished client confidence. Mobilization also slows when teams cannot see qualified internal options quickly.

Wade Trim treated the engineering skill search as a workflow problem with practical targets. Engineering expertise doesn’t fit in a dropdown, and that is where most legacy skill systems break. The system is built to support expertise discovery within one minute, enable users to update skills within five minutes, and allow flexible searching on free-form text.

For leaders, “good” looks like timely and reliable discovery staff based on their skills. It means faster staffing of projects, less reliance on heroic memory, and more confidence that obvious internal matches will not be missed.

Why Engineering Expertise Doesn’t Fit in a Dropdown Menu

Many firms still try to represent expertise through fixed skill categories and self-ratings. Someone picks “stormwater” or “water resources,” assigns a rating from one to five, and moves on. That structure breaks down in engineering because expertise is contextual. It changes as projects change, and the details that matter rarely fit inside a single label. As Sinclair notes, “the signal is in the nuance of project descriptions, methods, constraints, codes/standards, and deliverables.”

Take a term like “stormwater.” In terms of the potential scope of work, it can mean modeling, design, permitting support, or construction-phase services. The same core capability can also look different under different client standards and deliverables. People may also use different terms to describe the same skill, which makes fixed categories even less reliable for finding expertise.

That nuance lives in unstructured text: the narrative record of what someone did, under which constraints, and for what outcome. This is also why performance on domain-specific technical content deserves its own evaluation. General-purpose search benchmarks do not reliably transfer to engineering text.

That is what an engineering skill search requires: a retrieval built and tested on the content it will actually query.

What a Modern Skill Search Pipeline Looks Like

Many skill search systems handle retrieval and ranking in separate steps so managers can move quickly without compromising relevance: retrieve candidates quickly, then rank for relevance. WTSS is the benchmark data set that Wade Trim built to evaluate these pipeline choices on engineering text. The image below shows the basic flow.

Step 1: Retrieval

Retrieval pulls an initial set of candidates quickly from a much larger pool. The goal is coverage, even if the first pass includes some noise. Some firms use semantic search here to match meaning when wording differs, but the objective stays the same: surface strong candidates.

Step 2: Ranking

This reorders those results so that the best matches rise to the top. This step can improve usability and may filter items that are not reasonably relevant, but it adds latency and requires consistent output behavior.

Two other choices shape how well this pipeline works in practice:

• Candidate set size (k): k is the number of candidates you pull before refinement.

• Generation (optional layer): Generation can add a chatbot-style summary on top of retrieved results. This layer may be removed to reduce runtime, which underscores that generation belongs only where it supports decisions without slowing the workflow.

With the pipeline in place, Wade Trim’s key move was to benchmark these choices on engineering text as opposed to assuming general-purpose results would hold.

Why Wade Trim Built WTSS and What the Benchmark Is

WTSS is the benchmark data set that Wade Trim built to evaluate expertise retrieval on engineering content. Search approaches are widely tested on general benchmarks, but performance on domain-specific technical text is still underexplored. WTSS was built to close that gap.

The benchmark is grounded in Wade Trim’s production-deployed enterprise skill search system, available to approximately 800 internal users. That deployment provided real examples to shape the benchmark.

WTSS is a synthetic data set. The team used few-shot prompting to generate resume-style expertise descriptions and natural language queries, grounded in real samples from the deployed system. The intent was a dataset that can be shared for evaluation while still reflecting how expertise is described inside an engineering firm.

At a high level, WTSS includes 20 queries (the searches a manager might type) and 911 expertise descriptions (short free-form write-ups of skills and experience). One query, for example, is “Looking for a structural engineer to check beam loads.” A matching record reads, “Experienced structural engineer focused on the design of reinforced concrete structures.” WTSS also includes a relevance map showing which descriptions match which queries.

WTSS uses N to describe the average number of true matches per query. In this benchmark, N is 46.45. That value gives a reference point for framing practical tuning questions, including how large the candidate list should be before refinement.

WTSS was designed to test two questions that drive expertise retrieval decisions:

• How large should the candidate list be to achieve high recall?

• How many candidates can a large language model rank in one pass before output format failures rise?

To answer them, the team compared BM25 (a keyword search method) with semantic retrieval and evaluated GPT-4o as a ranker where output format consistency is critical.

Finding #1: Semantic Search Beats Keyword Search on WTSS

When the system depends on perfect wording, an engineering skill search can become ineffective. A query about checking beam loads and a record focused on reinforced concrete design can point to the same expertise without sharing exact wording.

WTSS compares two ways to handle that mismatch:

• Keyword search (BM25): Finds matches when the query uses the same words as the profile text.

• Semantic search: Finds matches when the query and profile mean the same thing, even if the words differ.

As Wade Trim’s AI Engineer, Kevin Hou notes, semantic retrieval is built for exactly this: “Dense/semantic retrieval finds people when the query and profile mean the same thing, even if the words differ.”

On WTSS, when k = N, semantic search achieved 0.82 recall versus 0.66 for BM25, and 0.84 precision versus 0.68 for BM25.

Operationally, that means managers can search more naturally. They do not need to know the exact term that appears in a profile or project write-up, which reduces time lost to trial-and-error phrasing and scanning irrelevant results.

WTSS is a benchmark result on a specific dataset, not a universal guarantee for every firm’s data. The practical takeaway is the direction of the result, then validating it against your own engineering text before adopting it system-wide.

But even the best retrieval method has a practical tuning question: how many candidates do you pull before you refine the results?

Finding #2: Candidate Set Size Is a Tuning Lever – the 1.2N Rule of Thumb

WTSS puts a number behind the candidate list question: how large does the first-pass list need to be to avoid missing qualified candidates?

Hou summarizes the tradeoff this way: “Too small, you risk missing strong candidates. Too large, you swamp the ranker with noise and waste time and compute.”

In the benchmark, reaching 90 percent recall required k ≈ 55, which the paper describes as about 1.2N using dense retrieval.

As k rises, precision falls. The query has a fixed number of relevant matches, and a growing list fills the remaining slots with weaker results.

For engineering leaders, 1.2N is best treated as a benchmark-derived starting point. Calibrate k to your content, your speed targets, and how much noise managers will tolerate before results feel unreliable.

As candidate lists grow, an LLM ranker becomes the next logical step, but WTSS shows that move comes with a practical ceiling you have to design around.

Finding #3: LLM Ranking Helps, But Stability Caps List Size

Ranking is the second pass in the pipeline. It takes the candidates’ retrieval already surfaced and reorders them, so the best matches rise to the top.

Ranking can improve ordering quality, but it does not improve recall, because it cannot recover candidates that were never retrieved in the first place.

WTSS adds a practical constraint that leaders need to understand before using an LLM this way.

In the benchmark, GPT-4o performed stable list-wise ranking up to about k = 40. Anything beyond that, and failures rose quickly.

A “failure” here is a format problem (not a judgment call). The model must return a clean order of the same list. When it drops items, repeats indices, or changes the required output format, the rank output becomes unusable.

The operational takeaway from all this? Design around that ceiling.

Keep LLM rank lists below the stability threshold. Validate output format before using results. If the output is invalid, revert to the original ordering. A broken list should never drive a staffing decision.

This discipline is AI governance in practice: the checks that keep automated tools reliable when the decisions they support matter.

What Engineering Leaders Can Take From WTSS: Measurement, AI Governance, and a Practical Checklist

LLMs are not engines of engineering truth. Wade Trim’s posture is to treat them as decision support that helps teams find internal expertise quickly and verify it against real work.

From the findings, here are three principles to follow:

• Match meaning when wording differs. Retrieval should surface relevant experience even when teams describe it in different terms.

• Tune candidate lists to your goal. Candidate list size should reflect how many qualified matches you want to capture before refinement.

• Bound ranking for reliability. Validate ranking and keep within sizes that produce consistent, usable outputs.

Here is a practical checklist to make the engineering skill search dependable:

1. Get the inputs right: Start with unstructured skill and project text, plus a realistic set of internal queries.

2. Evaluate before rollout: Benchmark retrieval on your own engineering content so tuning decisions are evidence-led.

3. Tune candidate set size: Use benchmark-driven starting points, then calibrate to your firm’s speed targets and tolerance for noise.

4. Rank with guardrails: Constrain list sizes, validate output format, and use fallbacks when outputs are invalid.

5. Operate with AI governance: Document failure modes, monitor performance, and keep humans in the loop for consequential staffing decisions.

NIST’s AI risk management framing calls for governing, measuring, and managing AI systems throughout their use. For engineering firms, that translates directly: design for reliability, validate outputs, and build fallbacks before the workflow depends on them.

WTSS will not transfer directly to every firm’s data. What it does show is what rigorous engineering skill searches look like: measured on real content, tuned with evidence, and governed so it stays reliable.

Interested in staying at the forefront of technology in the engineering industry? Explore ACEC’s Technology Resources to access valuable information and discover how you can join the Technology Committee.