Within the popular memory of the partition of India, the division of Bengal continues to evoke themes of political rupture, social tragedy and nostalgia. The refugees, or more broadly speaking, Hindu migrants from East Bengal, are often the central agents of such narratives. This article explores how the scholarship on East Bengali refugees portrays them either as hapless and passive victims of the regime of rehabilitation, or eulogises them as heroic protagonists who successfully battle overwhelming adversity to wrest resettlement from a reluctant state. This split image of the Bengali refugee, as victim/victor, obscures the complex nature of refugee agency. Through a case-study of the foundation and development of Bijoygarh colony, an illegal settlement of refugee-squatters on the outskirts of Calcutta, this article will argue that refugee agency in post-partition West Bengal was inevitably moulded by social status and cultural capital. However, the collective memory of the establishment of squatters’ colonies systematically ignores the role of caste and class affiliations in fracturing the refugee experience. Instead, it retells the refugees’ quest for rehabilitation along the mythic trope of heroic and masculine struggle. This article reads refugee reminiscences against the grain to illuminate their erasures and silences, delineating the mythic structure common to popular and academic refugee histories and exploring its significance in constructing a specific cultural identity for Bengali refugees.