When players first get into racing in Path of Exile, stumbles are to be expected as racing isn't exactly something most players of action role-playing games learn right from the start. It's more of something they learn later on in order to maximize their gameplay, and you can see how far that can be taken when you see the really good racers do their thing. It seems like they're on some sort of experience boost at first because they can level their characters so fast.
So you're sitting right there mulling over that gigantic gap between you and those high-tier racers who seem to work magic in the game. Do know that they started from the same position as you are in right now and had to learn everything that would later make them enviable to you from the ground up, from how they tackle every situation imaginable, what Path of Exile items and builds are best for racing, and so on. It's merely part of the process, and you can go through it and get there as well. However, the destination isn't exactly the point of racing.
Human beings are goal-oriented creatures, and having no goals usually results in stagnation. Perhaps that's too heavy for something that talks about playing a computer game in a certain way, but it's simply part of the whole exercise to begin with. As with everything else in life, the learning process in Path of Exile racing goes through these same stumbles as well. Upon realizing this, you must then ask yourself why you went into racing in the first place.
For those who don't question it because they're too busy having fun in race leagues, no matter how bad they are at it at the moment, good on them. As for those who may be finding it more than they can manage, there are a few things you can do to ease yourself in more. For one thing, don't worry too much about your performance, PoE items and PoE builds and just focus on playing in the race league. Similar to marathon runners, the running is the primary concern and finishing the race is paramount.
You then just try to learn more about racing in Path of Exile. There are some Twitch streamers and YouTube channels dedicated to Path of Exile racing, and you can just watch what they do to get an idea on what you can do. You can also delve into the forums and learn more about improving and enjoying race leagues. Most other people are casual players like you, so it's not like you'll find yourself in a place where you can't relate with anyone at all.
Racing is simply taking the idea of speed running in video games to the next level, putting it forward as a legitimate way to play Path of Exile. This is a move that Grinding Gear Games should be commended for as it's a developer of an action role-playing game acknowledging speed leveling as a big part of the genre to begin with.
As a casual player, the best thing you can do to show your appreciation for this show of thoughtfulness is to enjoy the race leagues as best as you can.