By Ben Morse, CNN
Updated 0833 GMT (1633 HKT) July 2, 2021
But Sifford did not give up.
With the backing of California attorney general Stanley Mosk and with fellow trailblazer Jackie Robinson by his side, Sifford became the first Black player to play on the PGA Tour in 1959.
And if you visit the Tiger Woods exhibition at the National Museum of African-American History and Culture in Washington, D.C., you'll see a photo of Sifford.
In breaking through the 'Caucasian only' clause in golf, Sifford helped open the door for other Black golfers, including the most famous Black golfer of all time, Woods.
And it's something Woods recognized, saying in 2015 following Sifford's death that he himself might not have been a professional golfer if it hadn't been for Sifford.
"He's like the Grandpa I never had," Woods said after a practice round in advance of the 2015 Farmers Insurance Open at Torrey Pines, the day after Sifford's death. "It's been a long night, and I'm sure it will be a long few days. He fought, and what he did, the courage for him to stick with it and be out here and play.
"I probably wouldn't be here (without Sifford). My dad would never have picked up the game. Who knows if the clause would still exist or not? But he broke it down."
While Sifford was the first Black player to make it through in golf, he had someone close to him to lean on.
Robinson, who broke the color barrier in Major League Baseball as a player in 1947, was a friend of Sifford's, and from his own experiences, passed on some advice to the golfer before he started his journey to make it onto the Tour.
"Jackie told him that he's going to have to face up to a lot of things, not respond to a lot of things because once he did that, it would be harder for him and harder for the people coming up behind him," Charles Sifford, Charlie's son, remembers.
"So he kept a stiff upper lip, bit his tongue and just dealt with what was presented towards him because he knew if he messed up that it would be even harder for the next guy coming along."
Having to move
Born in Charlotte, North Carolina in 1922, Sifford got into golf through the only avenue available to a young, Black kid -- caddying.
But he wanted to play the game -- at the age of 13, he could shoot a par round of 18 holes -- and not carry someone else's bag.
However, growing up in the age of segregation, opportunities for him to gain experience on courses weren't easy to come by.
He did eventually begin playing professionally in 1948, but -- because of the so-called "Caucasian only" clause which blocked Black players from playing with their White counterparts -- Sifford had to settle for playing in Black-only competitions.
By the time Sifford was in his 30s, segregation laws were slowly being abandoned, but golf proved less quick in moving with the times.
"In 1959, you still had the 'Caucasian only' clause, and it was easy to see how it could persist because golf was played on these private clubs, and they were able to continue to enforce the segregation rules," Nancy Churnin -- author of "Charlie Takes His Shot: How Charlie Sifford Broke the Color Barrier in Golf" -- told CNN Sport.
"So if you can't step foot in these private clubs, how are you going to play?"
Sifford's journey to play on the PGA Tour was not a spur-of-the-moment decision. It was something he worked to achieve for years.
His first attempt to break onto the Professional Golfers' Association of America (PGA) Tour in 1952 was met with vitriol and racial pressure.
At the Phoenix Open, Sifford and his all-Black foursome -- which included the heavyweight boxer Joe Louis -- found excrement in the cup of the first hole and had to wait nearly an hour for it to be replaced.
Unable to display his ability with all the best players, Sifford took his talents elsewhere -- to great success.
He won the United Golf Association's National Negro Open six times, securing consecutive victories from 1952 through 1956.
However, his dream was to showcase his abilities on the biggest stage in golf with the best in the business, and that meant making some sacrifices, as his son, Charles, remembers.
"When I was about 10 years old, I realized that we lived in Philadelphia, and my father really couldn't play in a lot of tournaments," he told CNN Sport. "There wasn't a lot of exposure to golf on the East Coast so we moved to the West Coast when I was 10. And that's when he told me that in order to succeed or have a chance to succeed, we had to move west."
Baseball star Robinson was a figure of inspiration and an example of what Sifford hoped to achieve in golf. But Sifford also realized he was going to need some legal help.
After his move out to the West Coast of the US, Sifford became friends with California attorney general Stanley Mosk.
Mosk was Jewish and had also experienced discrimination firsthand. He played golf at the Hillcrest Country Club in Los Angeles, which allowed members from the Jewish community when other clubs didn't allow them access.
Award-winning actor Billy Crystal, in his eulogy for Muhammad Ali at the great boxer's funeral in 2016, recalled an incident which highlighted golf's closed-door policy.
Ali invited his good friend Crystal to go for a run on a golf course, not realizing that the club didn't allow Jewish members.
"(Ali) was incensed: 'I'm a Black Muslim and they let me run there. Little brother, I'm never gonna run there again,'" Crystal remembers Ali saying.
Sifford's skill impressed Mosk immediately. And the fact that someone with such ability was unable to perform on the biggest stage angered him.
So Mosk set about helping Sifford in his quest to play on the PGA Tour.
As the attorney general of California, Mosk was able to bring about some political sway to Sifford's battle. Later, Mosk served as Associate Justice of the California Supreme Court for 37 years -- the longest tenure in that court's history.
After years of letters and conversations, Sifford finally earned a PGA Tour player card in 1960 at the age of 39, becoming the first Black player to play on the Tour.
A year later, following significant pressure, the PGA Tour dropped its "Caucasian only" membership clause.
However, Sifford was consistently subjected to racist abuse from White golfers and spectators.
His son Charles also remembers hearing stories about death threats sent to his father during those years.
"Several times when he was playing down in the South (region of the US), he had a couple of death threats," Charles explained. "People would call him at his hotel room and told him that if he showed up at the golf course, they were going to kill him.
"He said: 'Well, you're just going to have to do that because I'm showing up at the golf course.' So he was just that determined not to let anybody stand in his way and doing what he wanted to do. And he had that drive in him. The more you tried to stop him, the more he was going to try to succeed."
Change in the wind
Although he was late into his 30s when he made it onto the PGA Tour, Sifford was still able to show he could compete against the best golfers -- despite the hostility he encountered, both on and off the golf course.
Churnin recalls reading about hotels that wouldn't rent him rooms or clubs that still wouldn't let him eat with other professionals or use the locker room because of his skin color.
However, the 1967 Greater Hartford Open -- now the Travelers Championship -- in Connecticut proved a watershed. "That was the first time the crowd was on his side," according to Churnin.
And it appeared to make a difference, as Sifford claimed his maiden PGA Tour victory at the event, becoming the first Black player to earn a PGA Tour win.
Although he wasn't aware his dad had won because there wasn't wall-to-wall TV golf coverage as there is today, Charles remembers a palpable change in Sifford after the momentous victory.
"I saw it in the newspaper, and I was really excited for him because that was a lifelong dream to be able to win on the PGA Tour. And it took a lot of pressure off of him. He appeared to be more relaxed knowing that he did it once, and there was always a possibility he could do it again."
Sifford would go on to win the 1969 Los Angeles Open (now The Genesis Invitational) as well as the 1975 Senior PGA Championship and become an original member of PGA Tour Champions, where he won the Suntree Classic.
In 2004, he became the first Black golfer to be enshrined in the World Golf Hall of Fame.
Paving the way
President Barack Obama also awarded Sifford the 2014 Presidential Medal of Freedom "for altering the course of the sport and the country he loved."
Although Charles admits that Sifford was "highly disappointed" there hasn't been a huge influx of Black golfers following him, being the first Black player on the PGA Tour was something he was extremely proud of.
Churnin says it wasn't for lack of effort or commitment from Sifford that the number of Blacks following in his golfing footsteps wasn't massive.
"We all have different tools at our disposal," she explained. "Some of us use words; some of us use music; some of us will run for office, some of us will be legal scholars.
"We all come into this world and our job as we come into this world is to try to make the world a better place -- a better, a more equal, a more just, a kind, a more loving, a more inclusive place. This is a man who used the tool of the golf club to fight for justice. He knew he was not going to see all the fruits of that fight in his own lifetime.
"But he used his golf club for fairness, equality, to make the world a better place for others. And he got to see the promised land from where he was, because now that he knocked down this door, he had made it a place where others behind him could go and realize their dreams on the golf course."