Born in Decatur, Indiana on March 9, 1906, Smith grew up in Paulding, Ohio, where his father Harvey ran the Paulding Telephone Company and mother Golda taught school. He studied at Ohio University and the University of Notre Dame, but dropped out to become a welder on an automobile production line in South Bend, Indiana. He joined the Art Students League of New York in 1927. There, he discovered the works of Picasso, Mondrian, Kandinsky, and the Russian Constructivists, and became friends with Arshile Gorky, Willem de Kooning, Jan Matulka, and Jackson Pollock.

Profoundly influenced by the welded sculptures of Julio González and of Picasso, Smith started devoting himself entirely to metal sculptures, constructing compositions from steel and "found" scrap material. In the summer of 1929, Smith, along with his then wife Dorothy Dehner, bought a house in Bolton Landing, in upstate New York and won the Logan Medal of the arts.
In 1940, Smith moved permanently to Bolton Landing and created the Terminal Iron Works studio. In the long term, this allowed him to enlarge the size of many of his welded sculptures, moving to installations that increased in size as time passed by. In the short term, the Second World War disrupted Smith's supply of metal and reduced the demand for abstract art, leading Smith to draw and paint more than he had done previously. Smith painted prolifically throughout most of his career. He created landscapes, cubist abstractions and in the 1960s a series of sprayed pictographs that resemble visual studies for his Cubi sculptures.

However, with the end of conflict came a flood of new works, on a larger scale than ever before. In 1950, the Solomon R. Guggenheim Foundation awarded Smith a fellowship, removing the financial constraints and allowing him to spend more time sculpting. Furthermore, it allowed Smith to continue to create larger works and longer and more articulated series of works. The first of these were the Agricola (1951-1957) and Tanktotem (1952-1960) series.

In 1957, the Museum of Modern Art, New York City, presented a Smith retrospective, complete with work dating back to 1932. In 1961, MoMA organized a major traveling exhibition of his work.

In 1962, the government of Italy invited Smith to create two works for a festival, and gave him free access to an abandoned welding studio in the small town of Voltri, in Liguria. There, finding massive stockpiles of material, Smith decided to switch his plans from stainless steel to steel. The result was his Voltri series: 27 sculptures created in just 30 days. Still not satisfied with these, he shipped many tonnes of steel from Voltri back to the Bolton Landing, so that he could continue to work with the same material. Throughout late 1962 and early 1963, Smith produced a similarly-themed series, which he called Voltron, which was more varied and more prominently incorporated the trademark verticality of Tanktotem.

He began his Cubi series of monumental, geometric steel sculptures in 1961 (although he began in earnest only in 1963). They are considered some of the most important works of 20th century American sculpture. In recognition of his influence on abstract expressionism, Smith was appointed to the National Council on the Arts by President Lyndon B. Johnson in 1965. However, at the peak of his influence and still working on Cubi, he died in a car crash near Bennington, Vermont in May 1965.