There are two basic styles of writers: Plotters and Seat of the Pantsers. A plotter spends time world building, character building, and developing the overall plot. This type of author can make great detective stories.
Seat of the Pantsers (like me) can sit down with just the first sentence and start writing. This is how George Martin writes. This tends to lead to very rich characters, although the stories can meander a bit more.
It sounds like you're wondering which style you should pursue, and there's no right answer for everyone. You have to decide what works for you. To me, working out extensive plot lines doesn't work. Oh, I can do it, but when I actually start writing, I find that the writing isn't as rich. I rush from main point to main point, but the reason my fans like my writing is because of the time I spend with the characters -- that's far more important to my style than moving the plot in a particular direction. My very best books have been when I didn't actually know where the story was going to go, so I just kept writing, and the story unfolded for me much the same way it eventually unfolds to my future readers.
If you like world building, if you like making up a cohesive system of magic or future science, or if you want to write plot-driven stories, then absolutely you should spend time doing those.
But only you can decide.
I will tell you what my fans tell me, and what I tell other writers.
Write what you love. If you love it, other people will, too.