1. Avoid stereotypes.
2. Inject some sympathetic aspect into even the most evil and despicable of your characters.
3. Force your characters, especially your protagonist, to change and grow throughout the tale.
Stereotypes are boring. Avoid them like the plague. The kind-hearted priest? Seen it. The hard drinking Irish cop? Seen that too. The pissed off police captain? Ditto.
A useful way to avoid stereotyping a character is to think of a type then present its opposite. Imagine a sheriff from the deep south who is not a bigot and a dimwit, but is bristling with intelligence and dignity, passionate about revealing the truth and delivering an even-handed justice. Or a nun who is a baseball fanatic and is a genius at game statistics.
Truly memorable characters start off in one place and end up in another. In Kramer vs. Kramer, Dustin Hoffman begins as an insensitive, selfish narcissist but ends up as a kind and wise father who puts the happiness of his child first. At the start of The Godfather, Michael Corleone is innocent, principled, moral. By the end he is heartless, bereaved and soulless—a power hungry murderer of many, including his own brother.
Not every character needs to change, of course. Patton stays the same throughout the movie of the same name, although his character is challenged and is explained in a way that reveals to us why he is the way he is—an inflexible but powerful warrior to the last.
Well rounded, complex and conflicted characters are more absorbing than facile, boring ones. But with the interest that comes from lying, scheming and conniving comes the danger of characters becoming unlikable. It is, therefore, important to ensure that some aspect remains sympathetic to the reader or audience. If we don’t like our characters, especially the protagonist, we won’t like her story.
Oedipus murders his father then performs incest with his mother: horrific actions for a protagonist to indulge in. The writer, Sophocles, ensures that Oedipus remains sympathetic to his audience firstly by showing that Oedipus is unaware of the true facts of his coupling, and, secondly, by having him show deep and genuine remorse upon learning the truth.
In a Bridge on the River Kwai the Japanese commander of the prison camp is a cruel tyrant whose humanity still manages to peep through, if even once. He violates international laws, holds his prisoners in hot boxes, tortures and humiliates them, yet the writer portrays him as an unfortunate wretch who is tapped in a harsh command structure by permitting us to see him weep.
Well drawn characters are an indispensable part of successful story telling. Avoiding stereotypes, injecting character growth and creating sympathy are some of the ways of creating engaging characters.