Open source software is software whose source code has been made publically available by its copyright holder. Under a true open source licence, the software is developed collaboratively, and other programmers can look at, modify, or use the code their own purposes. This "pure" open source model is often referred to as FOSS (free and open source software).
A variant of open source is "source available", which means that no permission is granted to modify or otherwise use the code, but that it is available for inspection. For security purposes this is just as good as true open source, so when I refer to "open source", I include code that is "source available".
Most software is written and developed by commercial companies. Understandably, these companies are keen not to have others stealing their hard work or trade secrets, so they hide their code away from prying eyes using encryption, and any attempt to use or modify the code without permission will result in lawsuits or worse.
As I say, this is all quite understandable, but when it comes to security it presents a major problem. If no-one can see the details of what a program does, how can we know that it is not doing something malicious? Basically we can't, so we simply have to trust the company involved, which is something us paranoid security types are loath to do (with good reason).
If code is open source then it can be independently examined and audited by anyone qualified to do so, in order to check that there are no backdoors, vulnerabilities, or other security issues. Open source is not a perfect solution (see below), but it is the only way to have to verify that software is only doing exactly what it is supposed to be doing.
Even if the code has not been audited, the very fact that it freely available to be audited provides a strong indication that it can be trusted, as it is unlikely that developers would include malicious code and then leave it open to be discovered by anyone who cares to look.
Unfortunately, there are a limited number of individuals with both the skills and time to audit open source software (usually for free), which means that the vast majority of open source programs have not been audited.
This problem is compounded by the fact that many open source programs are extremely complex, containing thousands upon thousands of lines of code, so even if they have been audited, it is entirely possible that the auditors missed a problem (especially if malicious code has been deliberately concealed).