Skip to main content

Research Repository

Advanced Search

Probabilistic Learning of Treatment Trees in Cancer

Yao, Tsung-Hung; Wu, Zhenke; Bharath, Karthik; Li, Jinju; Baladandayuthapani, Veerabhadran

Probabilistic Learning of Treatment Trees in Cancer Thumbnail


Tsung-Hung Yao

Zhenke Wu

Jinju Li

Veerabhadran Baladandayuthapani


Accurate identification of synergistic treatment combinations and their underlying biological mechanisms is critical across many disease domains, especially cancer. In translational oncology research, preclinical systems, such as patient-derived xenografts (PDX), have emerged as a unique study design evaluating multiple treatments administered to samples from the same human tumor implanted into genetically identical mice. In this paper we propose a novel Bayesian probabilistic tree-based framework for PDX data to investigate the hierarchical relationships between treatments by inferring treatment cluster trees, referred to as treatment trees (Rx-tree). The framework motivates a new metric of mechanistic similarity between two or more treatments, accounting for inherent uncertainty in tree estimation; treatments with a high estimated similarity have potentially high mechanistic synergy. Building upon Dirichlet diffusion trees, we derive a closed-form marginal likelihood, encoding the tree structure, which facilitates computationally efficient posterior inference via a new two-stage algorithm. Simulation studies demonstrate superior performance of the proposed method in recovering the tree structure and treatment similarities. Our analyses of a recently collated PDX dataset produce treatment similarity estimates that show a high degree of concordance with known biological mechanisms across treatments in five different cancers. More importantly, we uncover new and potentially effective combination therapies that confer synergistic regulation of specific downstream biological pathways for future clinical investigations. Our accompanying code, data, and shiny application for visualization of results are available at:


Yao, T., Wu, Z., Bharath, K., Li, J., & Baladandayuthapani, V. (2023). Probabilistic Learning of Treatment Trees in Cancer. Annals of Applied Statistics, 17(3), 1884-1908.

Journal Article Type Article
Acceptance Date Sep 24, 2022
Publication Date 2023-09
Deposit Date Oct 21, 2022
Publicly Available Date Sep 30, 2023
Journal Annals of Applied Statistics
Print ISSN 1932-6157
Electronic ISSN 1941-7330
Peer Reviewed Peer Reviewed
Volume 17
Issue 3
Pages 1884-1908
Keywords Approximate Bayesian Computation, Dirichlet Diffusion Trees, Patient Derived Xenograft, Precision Medicine, Tree-Based Clustering
Public URL
Publisher URL


You might also like

Downloadable Citations