Twitter was a major component of the right-wing (and Russian) manipulation of Trump's first election. It is an active vector for huge amounts of hateful speech, organizing to abuse people and misinformation. In recent years, Twitter has done things to reduce its negative influence. Now it suppresses (very few, imho) posts that are hateful (n-word, nazi, anti-semitic). It labels factual assertions that are false as false. And, importantly, people that abuse its terms of service can be punished. The most famous case being Trump's lifelong ban.
Musk has made clear he opposes these things. He says he wants Twitter to be conduit for free speech and has at various times made clear that he thinks Twitters content moderation and terms of service should be eliminated. Most people who are not him and Trump think granting the ability for evil people to send messages to hundreds of millions of people at once results in terrible changes in our society.
Outside of his attitudes about Twitter, Musk is a garbage person. For example, you may recall the youth group caught in a cave in Thailand. He called one of the people who helped a pedophile for no reason. He denigrates people in public and private.
He has shown himself to be on the side of haters of various sorts, tweeting things about trans-people, allowing racist behaviors in his companies, comparing the Canadian prime minister to Hitler, speaking about Liz Warren in nasty, sexist terms to his hundreds of millions of Twitter followers.
He denied the value of COVID vaccines and reopened his factories while it was still very dangerous, has done stock market manipulation, cheated in cryptocurrency markets, allows his companies to use suppliers with child labor, he fired his longtime (dozen years) secretary for asking for a raise and produced an endless list of dumb, cruel, nasty tweets, comments and anecdotes.
To be clear, he has also made huge contributions. His professional accomplishments are amazing and important. Many people, including me, admire him as much as I hate him. I have long understood that people who change the world are not like you and me. To do such things requires personality and attitudes that are not good on a personal level (Henry Ford and Thomas Edison were both horrific assholes).
Because we live in an era where he can literally talk to a hundred million people at a time, we get to see his pernicious influence more clearly. Personally, I can mostly live with him being an asshole if he keeps doing the very important things he does but, when he applies his megalomania to the nervous system of our already shaky society, I am pissed. There are a lot of people who feel that way.