Second President of the United States John Adams lived to age 90, longer than all but two U.S. Presidents.

In 1983, the so-called “Reagan Reforms” made some big changes to Social Security including eliminating the survivors benefits for college students, boosting the payroll tax, and raising the retirement age from 65 to 67.  As a result, anyone born after 1960 must now wait until age 67 to receive their full Social Security retirement benefits.

But now some conservative thought-leaders and wealthy CEOs are again championing lifting the retirement age, this time to 70. Their argument probably sounds familiar to anyone who remembers the 1983 reforms: as people live longer, the retirement age should adjust upward. It sounds reasonable to people who have white collar jobs working in air-conditioned offices – but for millions of working Americans, the reality is much different.

One of the most common arguments in favor of raising the retirement age is that average life expectancy has shot up since the inception of Social Security, from age 60 in 1930 to nearly 79 today. Don’t be misled. The change in overall life expectancy mostly reflects lower infant mortality, not longer lifespans for adults.

In 1939, infant mortality rates were extremely high, but once age 65 the average American could expect to live another 13.4 years, or to age 78. Today, better health care and fewer infant deaths means overall life expectancy has gone up. But life expectancy after age 65 – a more accurate way to predict how long people are really living in retirement – hasn’t changed nearly as much.

As of 2008, the average American who makes it to age 65 could expect to live 19.6 years. That’s just 6 years longer than in 1939, and less than 2 years longer than in 1979 – and even that number overgeneralizes, because it ignores other factors that affect life expectancy, including gender, race, and income. A Social Security Administration study found income inequality plays a big role in life expectancy. For workers in the top half of the earnings distribution, average life expectancy is 86.5, but for those in the bottom half it’s just 81 — a gap of more than 5 years that continues to grow.

Race is another important factor for life expectancy at age 65. The most recent data show black men reaching age 65 have an average life expectancy of just 81, three and-a-half years less than the average for the total U.S. population. Total life expectancy for African Americans is 74.5, while it is 78.8 for white Americans.

American workers that are living longer are, on average, better educated, more affluent, and white. Further raising the retirement age will undoubtedly have a profoundly negative impact on millions of Americans, primarily those with less education, lower earnings, and racial minorities.