I don't think I've ever encountered or observed a genuine apology in my life. So I have concluded the thing doesn't exist, and people don't like apologizing because they don't know what it actually means, since it doesn't really mean anything. It is a social form with no corresponding personal content.

So what's going on here?

Apology and forgiveness are not well-posed psychological notions. Unlike shame, regret, guilt, and remorse, they are not primary human emotions, but rather social forms that help resolve conflict, by mitigating malicious intent after the fact. It's like a tax deduction.

They are legalist notions really. They mitigate the intent of an action with hindsight, a necessary discounting, since intent governs punishment (the punishment for murder depends on whether it was first-degree, manslaughter, defensive or purely accidental).

It is easy to see why. Apology requires a nominally shared notion of right and wrong. That gives us 4 cases.

If the nominal notion of right and wrong is also the substantive one (eg. both are practicing Christians), then:

1a. Either... the offender knowingly did wrong, and therefore did it due to temptation or carelessness (eg. manslaughter due to drunk driving). But since the wrong is perceived to be wrong with reference to an abstract and external moral frame, the recompense due is to the legitimate authority behind that framework (eg. confession to a priest as a representative of the Christian god, or surrendering for punishment to the legal system). In other words, here apology/forgiveness, if genuinely felt, are not special emotions towards the counterparty, but emotions towards a parental or god figure, or an abstract value system. Here the personal content of apology/forgiveness is really a sort of act of reverence towards a third party. Or in Freudian terms, an apology is your superego beating up on your id. Forgiveness is your superego calming your angry id.

1b. Or... the offender did not know or intend the wrong while doing it, and therefore cannot be held responsible for malicious intent. Apologizing in this case is a matter of ritual form.

If the nominal notion of right and wrong is NOT the substantive one (eg. you are an atheist in a theocracy), then whether you intended the "wrong" or not (2a and 2b), you don't accept that it IS a wrong in the nominal sense, even if you publicly pretend to. So apology is merely an expedient tactic to minimize penalties. Whether to apologize or not becomes a rational calculation.

There is one really interesting case within the 2a. This is when you don't agree with the nominal right/wrong, but your internal private morality actually judges you to be deserving of more punishment than the nominal one metes out to you, you suffer guilt (often this is a result of moral miscalculation: after the fact, you realize you've done more damage than you intended).

But in this case, since apology is merely a social form there is no point in apologizing more than dictated by convention, since it doesn't help socially, and doesn't balance your private books. So instead of apology, under such circumstances, you typically try to balance your internal moral books through material compensation of some sort to the victim (perhaps anonymous), or if there is nothing you can do there, general acts of penitence.

Finally... the relationship of the core emotions (shame, regret, guilt, remorse) to apologies. When you are able to sincerely display those emotions (or fake them) alongside the words, you get what is known as a "heartfelt" apology. If you want the intention discount, apologies have to look heartfelt.

But even this notion of "heartfelt" apology is not well-posed. Social expectations of such demonstrations are no more than commands to produce emotion on demand. It's a "prove you REALLY feel sorry" thing. Conventions require this on-demand demonstration because actually verifying authentic emotion would be too expensive in general.

But when it is possible, simply exhibiting the emotions without the apologetic words does the trick. In fact, most people are more likely to buy the emotions without the words attached. Would you be more likely to believe a murderer tearfully apologizing to the victim's family, or the murderer getting deeply drunk and depressed by himself, as a sign of true remorse?

It is generally too expensive to get a detective to shadow the murderer and look for real signs of remorse, so we end up asking them to produce "heartfelt apologies."