While the Illithid Powers can make your character stronger, they are completely optional, and the build works just fine without them. It is really up to you whether or not you’re happy to allow your Swashbuckler character to become more Illithid.