Does this mean that if some hacker found out what my website was made of, he could easily find his way in? Am I making a site that is more easier to hack, as all source is free-ly avaiable for them to find security holes?
Don't believe in security through obscurity. Open Source may mean people find holes, but if that happens they are quickly fixed - and then they are no longer there. Much better than if they are there, only not (yet) known.