Do they really have a case against me? I have just been served for grandparent visitation out of nowhere!?

My son is 2 and 1/2 years old and has visited my ex's parents frequently! I let them see them every Friday night for the most part. They were very overbearing and trying to control every aspect of my life and began to drive me crazy. They were almost haring me, they were calling me everyday and constantly complaining I dont send them enough pictures, complaining if his hair wasnt cut just hte way they wanted it to be...she would even introduce me to people as her daughter in law...it made me uncomfortable. I was also trying to find time to spend with my family (I got married in september). They have met my husband and liked him alot, he is great with my son and loves him very much. My son started calling him daddy and is very close to him, they have a great relationship, it is the only daddy he has known. My ex has never paid support and has seen him maybe 6 times in the last year. I started feeling like they were coaching my son not to call him daddy because he would come home confused as to what to call him, I feel like they were telling him it was bad. I asked the grandparents for space just to work on my family and have some weekends with my son and travel a bit, because when I did this my birthday was the week after and we wanted to go out of town and they flipped. Her response was "if you were dating someone for four years and they asked for space without a reasonable explination what would you do," I tried to explain that it wasnt permanent I just needed some time to make big decisions about my life and I wanted some uninterupted time with my son and she wouldnt listen. I said you can have him every other weekend starting on March 11th, I just want some space first (this was the last weekend in Jan). She never gave me the space and still kept blowing my phone up everyday so I tried to just ignore her for a while. On Monday I got served with papers requesting for grandparent visitation and to establish her sons paternity and get him visitation(which I understand). They claimed that I "unilaterally terminated visitation" and that it is in the childs best interest to get visitation. When I am a great mother and didnt actually terminate rights could they really win this? Could I be forced to give them visitation when they want it, I feel like its taking my rights away as his mom to control the situation. It just seems so unfair!