Do you need a valid visa to move out of the country?

There are probably three different questions in here

1. do I still need to hold a valid visa of country A?

Your presence in country A must be legal at all times. In this regard exiting the country is no different then going to the super market. As long your are physically present, you need legal authorization to do so. This may require a Visa or Esta or permit etc, or nothing at all depending on the countries involved.

As long as you are legally in the country, you can legally exit (unless there are some pending criminal charges, tax evasion, etc., i.e. there is a court ruling that requires you to stay).

2. Assuming I'm illegal, will I get caught when I try to exit

Depends a lot on the country, local policies and the way you exit. Some countries (e.g. US) don't have exit controls but they still may leverage airlines to check and record exits. Other countries have pretty strict exit controls (e.g. China). Some border crossing have no controls at all.

3. What will happen when I get caught

Depends a lot on the country and on the specifics of your infraction. Most countries consider illegal presence to be a serious crime. Consequences range from a slap on the wrist to jail time.

This is highly dependent on the country, but authorities will often want to check that you didn't overstay.

If you did, the immediate consequences could range from nothing at all (they're happy you're leaving) to a fine (often computed as an amount per day over the allowed stay) to jail time (though countries usually prefer to throw people out rather than feed them -- but it may be the consequence of not being able or willing to pay the fine).

There's also a big chance the overstayer will be banned from returning for a period that could range from months to forever. Even if not outright banned, it will usually make future visa applications more difficult, as you would be considered not to be trustworthy.

Note that some countries share such data with others, so even if you don't intend to ever return to said country (and never is a very long time), it could affect your prospects elsewhere.

Some countries do not have exit checks at all (the US and the UK, notably), so there's very little chance you could be fined or prevented from leaving (unless you are wanted for other reasons, of course), but they will know when you left and will be aware of any overstays with the same consequences as above in terms of possible bans or difficulty re-applying.

Exit visas exist:

Iraq, Kuwait, Lebanon, Oman, Saudi Arabia, and the United Arab Emirates all have an exit visa requirement for alien foreign workers. This is part of their kafala work visa sponsorship system. Consequently, at the end of a foreign worker's employment period, the worker must secure clearance from their employer stating that the worker has satisfactorily fulfilled the terms of their employment contract or that the worker's services are no longer needed. The exit visa can also be withheld if there are pending court charges that need to be settled or penalties that have to be meted out.

You should have a valid document which allows you to stay in country A on the day you are leaving it. It doesn't necessarily have to be a visa (you could get extensions, residence permits etc. while you reside in the country, it could even be the passport of the country B in certain conditions), but there must be one.

While some countries may have lax practices regarding visa checks upon departure, this is not something you should rely on. You can still be subject to a random security check, or get your papers checked for an unrelated reason, and you will be in trouble if you can't justify your presence in the country.