As a fiction author, nothing holds you back more than your own fear someone is going to read it and laugh at you. Or worse, judge you. Case in point:
Shortly after LIVING CANVAS came out, I ran a special. One of the ladies I work with asked me about my writing and I told her this was the perfect time for her to see for herself - pitching the special. She got herself a copy. A week later, she said she could hardly look at me without being embarrassed. Was I writing from personal experience? To give you reference, I followed the trend in romance novels and "opened the door" during intimate scenes. My coworker said every time she read the sexy parts, she pictured me. Okay, that's embarrassing for me, as well. I assured her the characters were in my imagination - NOT ME. Funny how she sees the interpersonal interaction between my characters as "based on real life" but the main character goes for a walk inside a painting. Does she imagine me doing that, as well? Granted, the story was inspired by a picture that hangs on my wall over my computer, but I've never disappeared inside the painting. Writing the genre standard scenes was merely that - writing to the genre. Even my mother told me I ought to beef up my writing with more sex and violence - until I did. Then it was too much, but you know. Moms.
I'd always dabbled with the supernatural. I loved scary movies and I'm a fan of Stephen King, so surprise! My books tended to lean ever so slightly into the paranormal. Emphasis on the ever so slightly, because what will people think? That I'm a whackadoodle? That I believe in the "woo-woo?" It took me some time before I gave myself permission to go full on ghost story/paranormal, maturing as a writer to get to the point beyond "what will people think?" I'm writing to entertain. People who pick up my books are looking for that particular brand of entertainment. So why is it, twenty some odd years later, that I'm standing on a precipice, writing my next book (genre: romantasy) and getting hung up on being too far out there?
As an author, it is important to know your genre - and genres are fluid. They change over time. What used to pass for a romance no longer does. Submissive women have become their own heroes, with men as decoration, unless you are specifically looking for submissive women. (Oversimplified, but you get the point.) Sex used to be taboo in a romance novel, then it became a requirement. Things have modified over time. Now you can get "sweet" or "spicy" (please don't get me started on "clean" or "dirty). Even spicy novels have toned it down a bit. Unless, of course, you're reading those books for the spice, in which case you might be looking for erotic romance. The point here is to understand the ever changing choices your audience is making.
The big publishing houses will dictate to their authors what to write (to a certain degree) to fit the demographics and the trends. Independent authors get to write what they want - and they demand a large share of the market. Genres that cross marketplaces. I can use Diana Gabaldon as an example here. She writes romance, and action/adventure, historical, time travel - all in one novel. The important part is to tell a story worth reading, and your audience will follow.