The Collection and its history are inseparably linked with the development of the Academy as well as the arts and art politics of the Habsburg monarchy, with approaches ranging from the state's dominant position in the 18th century and the beginnings of the middle classes' responsibility in the 19th century to the state's withdrawal in the 21st century.

The 18th-century holdings of the Collection reflect the various stages of the Academy's development from Peter Strudel's private initiative of 1688, which was aimed at establishing an Italian studio academy, to the resumption of teaching activities in 1725, which had been interrupted for eleven years after Strudel's death and were now taken up again by Kammermaler Jacob van Schuppen and his successor Martin van Meytens, Empress Maria Theresia's Swedish court painter.

The inventory taken after van Schuppan's death in 1751 and Anton Weinkopf's descriptions from 1783 and 1790 constitute the earliest evidence of what the Collection comprised. The first real inventory, however, dates from 1792, when Chancellor Count Wenzel Kaunitz von Rietberg, one of the most important patrons of art and collectors of the era, reorganized the Academy and united the various special schools. While the Collection comprised the holdings of Schmutzer's Engravers Academy - admission pieces, nudes, competition works, studies, and models by students, important graduates, and professors of the different schools - in its beginnings, it was deliberately extended by way of commissions and acquisitions under Kaunitz. "Carelessness" called for a further inventory, which was agreed upon in 1775. After parts of it had been finished in 1790, it was finally completed in 1798 after the protectorate had been conferred upon Count Johann Philipp Cobenzl. At that time, the holdings of the various schools totaled 1,242 drawings, 670 prints, and 88 models, which clearly illustrates the institution's function as a teaching tool. The next inventory, this time an only summarizing list, dates from 1810; it was taken after Count Klemens Lothar Wenzel Metternich-Winneburg-Ochsenhausen had been elected new protector of the Academy.

Comprising a total of 6,500 drawings and 12,000 prints, seven important donations within the last twenty years before the Revolution of 1848 marked the most important change of the institution, transforming it from a teaching instrument into the second largest graphic collection of the Habsburg Monarchy after the Albertina. After 1850, regular acquisitions were made by the responsible ministries which included significant groups of works such as drawings of German and Austrian Romanticism. After having been moved to the new building by Theophil Hansen on Schillerplatz - its ceremonial opening took place in 1877 - the holdings of the institution were presented separately in a Picture Gallery ("Gemäldegalerie"), a Sculpture Gallery ("Glyptothek"), and a Graphic Collection ("Kupferstichkabinett").

During the 20th century, the Collection grew mainly through donations from former students and professors and specific purchases aimed at completing its holdings. Since 1996, an annual donation from the Society of Friends of the Fine Arts made it possible to build up a collection of young contemporary art focussing on works by graduates of the Academy of Fine Arts Vienna.