No Great Magic by Fritz Leiber.
It's a long short story - more of a novella. You can read it on Gutenberg here. It's one of his Change War stories about a war between factions called the Snakes and the Spiders, and follows on from his novel The Big Time. I don't think the story was ever published in the same anthology as Population Implosion though they were published in the same decade.
For example the scene you remember with Queen Elizabeth is:
"Martin," I said with difficulty, "I don't think I'm going to like what we're doing."
He turned on me, his short hair elfed, his fists planted high on his hips at the edge of his black tights, which now were all his clothes.
"You knew!" he said impatiently. "You knew you were signing up for more than acting when you said, 'Count me in the company.'"
Like a legged sapphire the blue fly walked across her upper lip and stopped by the thread of foam.
"But Martin ... changing the past ... dipping back and killing the real queen ... replacing her with a double—"
His dark brows shot up. "The real—You think this is the real Queen Elizabeth?" He grabbed a bottle of rubbing alcohol from the nearest table, gushed some on a towel stained with grease-paint and, holding the dead head by its red hair (no, wig—the real one wore a wig too) scrubbed the forehead.
The white cosmetic came away, showing sallow skin and on it a faint tattoo in the form of an "S" styled like a yin-yang symbol left a little open.
"Snake!" he hissed. "Destroyer! The arch-enemy, the eternal opponent! God knows how many times people like Queen Elizabeth have been dug out of the past, first by Snakes, then by Spiders, and kidnapped or killed and replaced in the course of our war. This is the first big operation I've been on, Greta. But I know that much."
My head began to ache. I asked, "If she's an enemy double, why didn't she know a performance of Macbeth in her lifetime was an anachronism?"
"Foxholed in the past, only trying to hold a position, they get dulled. They turn half zombie. Even the Snakes. Even our people. Besides, she almost did catch on, twice when she spoke to Leicester."
"Martin," I said dully, "if there've been all these replacements, first by them, then by us, what's happened to the real Elizabeth?"
He shrugged. "God knows."