So really I am looking for help and advice on how to run a game I can both prepare for to develop interesting locations and characters (a strong point of mine) and run a more ‘open’ style of play where I am not railroading players into decisions.
Your issue does not really seem to be one of content, but one of preparation. You over-prepare a particular area and then drive your players through it whether they want to or not. This is actually pretty common in RPG games.
My suggestions would be as follows:
1) Pull back on the details. Give yourself something to "create" during the game so that you're not just delivering text and driving the players into your inflexible scenarios.
2) Save some time at the end of a session (and/or before the next) getting the players talking about what they want to do next. While you want your players to feel like they are in an open world, you can't prepare for 360 degrees at all times. You need to know what they are likely to do if you're going to leverage some of your prep time for a smoother gaming session. Trust me, being prepared half as much for two directions (or a third for 3) is better than being over-prepared for 1 direction your players really don't want to go.
3) Get your players to tell, and create, some of the story. For the telling, if you've got a lot of text to deliver, see if you can't hand some of it off to a character that might know some of it. Let the player sell it. For the creation, make sure what the players are doing changes their environment in ways that they can appreciate. And, most importantly...
4) Respect the spaces between the words. When people read a story their minds are constantly filling in the blanks. They are painting a picture that is unique to their view of the world their character's are inhabiting. For example, you may have a very different concept of what a "beautiful barmaid" might look like. If you spend 10 minutes fleshing her out you might actually be painting a picture of a woman that just isn't so beautiful to them. So try to focus on only the unique details that are needed to set the scenes and trust that you players are creative enough to flesh a lot of it out themselves.
5) And, finally, you do have to accept that not all players are the same. Just because this group (or a portion thereof) can't stay focused while you're in the middle of what you think is important expository doesn't mean that there aren't players out there that would really eat that up. If you're pounding square pegs into rounds holes... it may be an exercise in futility. Compatibility matters.