Is it ok to have fruit flies in my compost pile?

I started a compost pile in a large potter (I only have a potted garden on apartment porch) I understand the importance of green and brown, but I dont ever put brown in because I dont have anything (clippings, newspaper, etc). Well I just put my vegetable/fruit scraps in there w/ a little soil. I occasionally turn it, but man there are a lot of fruit flies!!! I am afraid they will get to my veggies I am growing and eat the wrong thing. Is this OK? Any suggestions?