While some ivory scraps can be scavenged from the tusks of beasts, most ivory is mined. In an age long lost, behemoths trampled the land beneath feet which could crush the metropoli of this modern era. Their cylcopean corpses, buried now in vast elephantine graveyards beneath the surface of the earth, have left behind vast deposits of ivory.
The mines themselves are fantastical, but so are the crafts which this mined ivory allows. In the real world, one cannot find slabs of ivory as tall as a man (or taller). One cannot pave royal throne rooms with it. One cannot carve life-size statues from it. Nor can one marvel at the Ivory Palaces of the Seven Island Caliphates.
Ah, the oliphant … rich, massive pieces of ivory large enough to make double door lintels from.
I take exception to the real world argument. In times past, mammoth tusks as large as you suggest were found as a part of frozen carcasses in the northern shores of Siberia … but ivory diminishes over time, particularly that which is of lesser quality. But who knows what pieces the ancient and Medieval Chinese emperor had in his throne room?