Alzheimer’s disease (AD) is the most common form of neurodegenerative disease currently. It is widely accepted that AD is characterized by the self-assembly of amyloid beta (Aβ) peptides. The human glutaminyl cyclase (hQC) enzyme is characterized by association with Aβ peptide generation. The development of hQC inhibitors could prevent the self-aggregation of Aβ peptides, resulting in impeding AD. Utilizing structural knowledge of the hQC substrates and known hQC inhibitors, new heterocyclic and peptidomimetic derivatives were synthesized and were able to inhibit the hQC enzyme. The inhibiting abilities of these compounds were evaluated using a fluorometric assay. The binding mechanism at the atomic level was estimated using molecular docking, free energy perturbation, and quantum chemical calculation methods. The predicted log(BBB) and human intestinal absorption values indicated that these compounds are able to permeate the blood–brain barrier and be well-absorbed through the gastrointestinal tract. Overall, 5,6-dimethoxy-N-(3-(5-methyl-1H-imidazol-1-yl)propyl)-1H-benzo[d]imidazol-2-amine (1_2) was indicated as a potential drug for AD treatment.