I'm on the editorial board of a journal in my field. I am often assigned manuscripts as managing editor. This means finding reviewers. Of late I've noticed a disturbing trend (ANECDOTE ALERT!!!). People I ask are taking an unreasonably long time to decide whether or not they will review a manuscript. Days. A week even. If this were just one or two people you could explain it away easily enough. They're traveling, for example. But it's not one or two. It's approaching 30-40%. Given that I'm managing two to four manuscripts at any given moment, that's a lot. And when they eventually get back to me (those that do...), they invariably say no they can't review the manuscript.
Why are you taking so long? Read the abstract (which we send in the email), think about what else you have to get done in the next couple of weeks, and decide whether or not this is a review you want to do. Then get back to me by reply email. Not a hard process. The longer you take to decline the invitation, the longer the whole process takes. Is that what you want to happen with your manuscripts? Didn't think so. So if you're going to decline, get off your rear end and say no quickly.