Experts go all in when CEBAF is in trouble

For decades, physicists and researchers from around the globe have flocked to the U.S. Department of Energy’s Thomas Jefferson National Accelerator Facility to unlock the subatomic mysteries of how the universe works.

Here, the world-class Continuous Electron Beam Accelerator Facility (CEBAF) is a marvel of science and technology, a DOE user facility and underground “racetrack” that blasts beams of electrons at nearly the speed of light at carefully chosen targets in up to four experimental halls simultaneously. These beams of electrons allow scientists to study the cascade of fleeting particles produced in the electron-target collisions for their research.

Beamtime is precious, so experiments are carefully vetted and booked years in advance.

But what happens when this sought-after marvel breaks?

The question isn’t academic. In April, sensors showed a vacuum seal had failed in a critical area of the injector – the chopper – where electrons are sorted by large copper cavities and directed for experiments, which run in distinct areas at the end of the accelerator. But without that vacuum seal, outside air we breathe will enter those cavities, contaminating the system and effectively crippling accelerator operations.

So an ad hoc team of experts from the accelerator and engineering divisions assembled to diagnose the situation and figure out how to fix it. Fast. Luckily, the machine was nearing the end of a scheduled three-month shutdown for maintenance, so experiments weren’t immediately affected.

“That was lucky timing,” said Joe Grames, head of the lab’s Center for Injectors and Sources, which builds and maintains all of the lab’s high-performance particle beam sources for injectors and accelerators.

The team had roughly a month left in the shutdown to make the CEBAF chopper ready to go before users would need beams of electrons again.

‘All hands on deck’

“It took a process to find the problem,” said Mark Wissmann, a staff radio frequency (RF) electrical engineer and core team member.

“It took sleuthing, not so different from a puzzle. It was both challenging and rewarding at the same time,” said Grames. “People with a lot of vacuum expertise and experience were contributing to this overall solution, putting their eyes on all their systems. It was sort of an all-hands-on-deck for the injector area.”

The nearly mile-long CEBAF accelerator is entirely under vacuum, at literally less than one-billionth of the air pressure we breathe, despite thousands of joints and welded parts. If a part fails or breaks, it can cause a small leak that can be tracked down and then replaced or quickly repaired in-house. This new leak wasn’t one of the usual suspects. So team members expanded their range from the usual.

It took a day or so to track down the problem to the beam “chopper” — a component to a unique system that was custom designed and built about 40 years earlier.

“Literally what we did is, we had a turbo pump connected to the vacuum system and we took a quarter inch piece of hose, and you could hear it go ‘Sssss,’” said Kevin Jordan, a staff engineer and lead for the Diagnostics Development group in the lab’s Center for Advanced Studies of Accelerators. “It was pretty obvious where the problem was.”

And it was also pretty obvious what was at stake: Without a beam chopper, the CEBAF can’t run its four experimental halls.

“That particular system, besides being critical to having multiple users, is unique in that we operate only one of them at CEBAF,” said Grames. “It’s like a beam switchyard in space and time. By having a chopper, we can deliver beam to four users at once, and that’s a big deal for our user community and an important metric for operations.”

The broken component was a chopper coupler, a roughly four-inch copper cylinder with multiple internal parts. To repair it, the team reached out to Jake Haimson, the esteemed accelerator physicist who helped design and custom-build the CEBAF chopper four decades earlier.

“I was surprised how quickly he remembered so many technical details about this part,” said Jordan.

“He was eager and excited to help us,” Grames said. “He’s very interested in helping accelerators do their business.”

Haimson is 95 today, president of the Haimson Research Corporation based in Santa Clara, Calif., and he is still active designing and maintaining accelerators in many countries.

When a breakdown to one of those facilities is serious enough that it can’t be handled in-house, he’s brought in. It’s critical work that keeps him trotting around the globe.

“You’re so busy you haven’t got the time to stand back and really feel well about it, because you’re hit again with another contract and you have to immediately go to another regime,” Haimson said. “It’s like having a lot of children that you sent out — they get matured after a while. So, of course, we feel an obligation, and if someone’s in trouble, of course you respond immediately.”

Rare Repairs on Tap

About 20 years ago, Haimson and his team were called in on another serious issue with the chopper and worked day and night to get the repair back to Jefferson Lab in under four days.

This time, the issue was a very small part among some 200 parts inside the big chopper cavity: The chopper RF power coupler.

“The insulator was cracked,” Haimson said. “Once it cracked, it lost the vacuum.”

Jefferson Lab shipped the chopper coupler to Haimson, who repaired and returned it in about two weeks.

“We were very lucky, because we had all the old fixtures and the jigs and the spares,” Haimson said. “More importantly, we had in stock a lot of the gold and silver alloy that is required during the brazing.”

At the same time, Jefferson Lab experts fervently pursued a parallel in-house approach: to design and fabricate a replacement coupler based on Haimson’s design, but more robust.

“I looked at the component and said, ‘Let’s just make one,’” said Jordan. “All we have to do is copy this. It’s a block of copper, we’ve got to drill some holes in it, we’ve got to braze some stuff, we’ve got to weld some stuff, we’ve got to do this and that.”

So Joe Gubeli, a Jefferson Lab engineer, helped to create a 3D model of the new part. The team asked Haimson to review the design, and then sent it off to the lab’s machine shop. When the technicians produced the coupler, Wissmann and his crew made RF measurements to confirm it was a reliable upgraded copy of the original, and then they installed it in early May. It worked perfectly!

“When I saw in the operations logbook an entry that said the chopper is working as it did before, that was a combination of relief and euphoria,” Wissmann said.

Meanwhile, Jefferson Lab is keeping Haimson’s repaired component as a spare.

“It’s our first spare, and we’re making more,” Grames said. “It’s a learning method.”

In-house expertise will become more important for CEBAF and other linear accelerators now that Haimson is in the process of dissolving his company after more than half a century.

“One of the major factors of our success has been that, when we’ve developed special machines, we’ve also gone to a great deal of trouble to train the personnel and make sure they’re self-sufficient,” Haimson said. “The homemade part seemed to work out quite well. And that’s nice, because that means they really got a grip on these things, and they’re following it and can handle it right away.”

As many as 20 Jefferson Lab physicists, engineers and technicians — with a core group of half a dozen — had a hand in getting the CEBAF chopper up and running in time, so that the larger CEBAF operations team could continue re-commissioning the rest of the machine and run beam to experiments again on schedule.

“It’s like rungs on a ladder,” said Grames. “Every rung is important. People have expertise and we need to put them all together to get where we need to be.”

By Tamara Dietrich

Further Reading:
Meet Our People: Kevin Jordan – Electrical Engineer

-end-

Jefferson Science Associates, LLC, manages and operates the Thomas Jefferson National Accelerator Facility, or Jefferson Lab, for the U.S. Department of Energy's Office of Science.

DOE’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our ?me. For more informa?on, visit https://energy.gov/science.