I contend that we are both atheists. I just believe in one fewer god than you do. When you understand why you dismiss all the other possible gods, you will understand why I dismiss yours.
-- Stephen Roberts

As it happens, the accusation that Christians are actually atheists is almost as old as Christianity itself. The earliest version of this assertion, however, came not from the camp of what we would call atheists today, but rather from angry polytheists.

The Romans had an indigenous pantheon made up of gods and goddesses native to the Italian peninsula; these gods are usually called Etruscan. Later, some of these gods were assimilated with Greek gods, which is why so many schoolchildren learn that Zeus "equals" Jupiter, that Athena "equals" Minerva, and so forth.

However, the Romans did not stop there. They frequently "imported" gods from other cultures, especially from territories they conquered. For example, Isis, who is about as Egyptian a goddess as you can get, was very swiftly Romanized after the Roman conquest of Egypt, and her cult soon became very popular in Italy. In general, the Romans believed that the more blessings one received, the better, so they invited pretty much any god they encountered into their pantheon.

One of the reasons why Christians annoyed the Romans so much, then, is that Christians didn't let them import Jesus in that way. If an ordinary pagan innocently tried to add Jesus' name to the list of gods from whom he sought blessings, Christians would protest, indignantly asserting that Jesus was off-limits. Of course, not all Romans were interested in Jesus to begin with; a common Jewish criminal who had been executed for sedition hardly seemed to the average Roman as a figure worthy of honour. Still, evidence exists that there were plenty of Romans who tried to treat Jesus just like every other god they knew. (My favourite example is a prayer to "Jesus, Earth, Hecate, Jesus," quoted by Morton Smith in his classic study Jesus the Magician.)

What is worse, Christians denied the existence of all the other gods in the Empire, or relegated them to the status of demons. Not only did this aggravate the Romans, but it frightened them. How would the gods react to such an insult? It couldn't be good...

But the whole multitude, marvelling at the nobility of mind displayed by the devout and godly race of Christians, cried out, "Away with the Atheists!"
-- The Martyrdom of Polycarp 3.2

Christians were atheoi, "no-godders," and thus were considered a serious political and religious threat.

Now of course, Jews would also qualify as atheists from this perspective. However, they didn't get harassed by the Roman authorities nearly as much as Christians did, since Judaism was recognized as a very ancient religion, and the Romans had great respect for any culture older than their own. Though the Romans sometimes thought Judaism was strange, and though occasionally the Jews would be accused of getting special treatment from the government (e.g., Jews were not conscripted into the army, since they refused on principle to fight on the Sabbath), usually they were left alone. The historian Geoffrey E.M. de Ste. Croix describes them as "licensed atheists."

Philosophical monotheists existed among pagans, as well, as beleaguered Christians in the Roman Empire often complained. Why weren't they the target of the government's ire? The answer is quite simple: no matter how much a given philosopher might speculate about the irrationality of his own religion, he would still take part in the public rituals that bound his community together. He would never refuse to honour the genius -- that is to say, the guardian-spirit -- of the Emperor, and he would never turn down meat from a sacrifice. Thus he would be perceived neither as a threat to the social order nor as a target of divine anger.

And we confess that we are atheists, so far as gods of this sort are concerned, but not with respect to the most true God, the Father of righteousness and temperance and the other virtues, who is free from all impurity.
-- Justin Martyr, First Apology 6.1

So, just as Stephen Roberts suspected, Christians and atheists have something in common: they reject the existence of certain gods. Christians have always understood this. For centuries they have been coming up with explanations for why they did not reject the last god on Mr. Roberts' list.