Diffusion MRI (dMRI) tractography is an advanced imaging technique that enables in vivo reconstruction of the brain's white matter fiber pathways. It has become an important tool for computational brain analysis tasks, including population classification across different cohorts, for instance, sex classification and separating healthy subjects from disease populations. Whole-brain tractography data comprises millions of individual streamlines, which are typically parcellated into compact representations such as a connectivity matrix or streamline clusters for analysis. However, this coarse-graining discards the rich information inherent in individual streamlines. In this work, we introduce TractoMFormer, a parcellation-free framework that operates directly on information encoded at the individual streamline level and provides inherent interpretability through transformer-based attention. TractoMFormer offers two main innovations. First, we present DTractoEmbedding, a lightweight deep graph model that generates a 2D image representation that preserves the spatial organization of 3D streamlines while integrating microstructural attributes such as fractional anisotropy (FA), mean diffusivity (MD), together with tractography-derived fiber density. Second, we develop a deep classification model based on a multi-scale vision transformer (MViT), which aggregates connectivity information from coarse to fine anatomical scales, and an explainability scheme to identify the specific streamline pathways most discriminative for group classification. We demonstrate the utility of TractoMFormer through comprehensive evaluations on three classification tasks: (1) sex classification; (2) schizophrenia vs. health; and (3) Parkinson's disease vs. health. In comparative evaluations, TractoMFormer achieves the highest classification accuracy in each task, significantly outperforming the compared methods. Furthermore, the explainability module reveals task-specific discriminative white matter regions, offering valuable insights for identifying potential biomarkers at the group level.