The asterisk is used in two ways in
linguistics. In both it is placed in front of an expression that does not actually occur in a language.
1. In historical linguistics an asterisk is placed in front of a word that has been reconstructed: that is, there is no written example of it, but it is believed that speakers of the ancient language would actually have used it. Sometimes a double asterisk is used to indicate a form that linguists believe did not occur.
See asterisk words for more details. I'll add an example here: the Praenestine Fibula is an ancient brooch that was long believed to have the oldest Latin inscription, MANIOS MED FHEFHAKED NVMASIOI ('Manius made me for Numasius'). The word FHEFHAKED, representing the pronunciation fefaked, comes from pre-Latin *dhedhaked. The problem is that by regular sound change that should have become **fedaked. For this reason many scholars think the Fibula is a modern fake.
The double asterisk is also used to indicate reconstructions much deeper in time than the single asterisk. In practice the two notations don't interfere.
A single asterisk can also mark a merely unattested form such as the nominative singular of a word that's only known in the genitive in manuscripts.
2. In discussions of sentences, the asterisk in front means it's ungrammatical, that is the native speaker wouldn't actually say it unless there was something like a slip of the tongue. It's typically used for studying the grammar of related sentences:
Mary speaks French fluently.
* Mary speaks fluently French.
Mary speaks fluent French.
In addition to the asterisk, the
question mark and the
hash sign are used. The leading question mark indicates a sentence that's grammatically doubtful -- either one speaker can't decide whether or not they'd say it, or they would accept it but it doesn't sound as good as a fully
grammatical sentence, or some speakers accept it but others don't. A stronger doubt is expressed by two question marks: this sentence sounds very strange, not positively ungrammatical perhaps, but almost certainly can't be said. A question mark and an asterisk together probably means the
linguist writing it thinks it's ungrammatical but is open to the possibility that others will find it marginally
acceptable. These markings of course are subjective, but they are used precisely in complicated cases where it's hard for even
native speakers to tell whether they would or would not say something, and this is an interesting and fruitful
intuition in itself.
? This is the movie Mary fell asleep during.
?? John is a person who I don't really like his paintings much.
The hash indicates a sentence that is grammatical but can't be used for
pragmatic reasons, or is very unlikely to be used. The error is that it doesn't fit how we know the world works.
# The lamp-post collided with the car.
# It was rude of Mary to see John.
Brackets are used to indicate optional material.
I threw the vase out (of) the window.
Asterisks (and the other markings) are used with
brackets as follows. Brackets mean something is optional, so the asterisk outside brackets means being optional is unacceptable: the material has to be present. On the other hand an asterisk inside brackets means omitting the material is acceptable but having it present is not. So
I took the toy out *(of) the box.
means you can't omit the 'of', and is shorthand for the pair
* I took the toy out the box.
I took the toy out of the box.
whereas conversely
I threw the vase out (*of).
means the included material is ungrammatical but omitting it is grammatical, and this is shorthand for
I threw the vase out.
* I threw the vase out of.