Whatever was going on in the period Malcolm II/Duncan/Macbeth/Lulach, it seems that *all* the old rules had been thrown into the trash bin.
"Tanistry", as reconstructed by the scholars, involved selection of an heir-designate from all the adult male representatives of the *male* line.
Duncan wouldn't qualify. Macbeth wouldn't qualify. Thorfinn wouldn't qualify (assuming he did have a valid claim through *his* mother). Lulach wouldn't qualify.
Malcolm II is often credited with being the first to introduce the concept of hereditary monarchy in Scotland. He probably wasn't. His father, Kenneth II, is credited with/blamed for being the first to try to change the rules, and it allegedly got him assassinated. (Wikipedia, C2: According to John of Fordun (14th century), Kenneth II of Scotland (reigned 971-995) attempted to change the succession rules, allowing "the nearest survivor in blood to the deceased king to succeed", thus securing the throne for his own descendants. He reportedly did so to specifically exclude Constantine (III) and Kenneth (III), called Gryme in this source. The two men then jointly conspired against him, convincing Finnguala, daughter of Cuncar, Mormaer of Angus, to kill the king. She reportedly did so to achieve personal revenge, as Kenneth II had killed her own son. Entries in the Chronicles of the Picts and Scots, collected by William Forbes Skene, provide the account of Finnguala killing Kenneth II in revenge, but not her affiliation to Constantine or his cousins. These entries date to the 12th and 13th centuries.[14][15] The Annals of Ulster simply record "Cinaed son of Mael Coluim [Kenneth, son of Malcolm], king of Scotland, was deceitfully killed", with no indication of who killed him.[16][17])
Malcolm, out of dire necessity, went a lot farther than his father ever dreamed of. He threw out the *whole concept* of agnatic (male-line) inheritance, reverting to the old Pictish matrilineal tradition. He had to; he had no sons and only three(?) daughters.
But that left the succession open to *all* female-line descendants, including Duncan. Macbeth, and Thorfinn (plus others who may have seen fit not to press the point).
The conservative Scots weren't quite ready for absolute primogeniture (nor was the *British* monarchy until very, very recently), and once Duncan proved himself dangerously incompetent, it became winner-take-all. Macbeth was competent - very, *very* competent - and soon acquired the backing to take Duncan on and beat him. (Thorfinn seems to have been playing "Let's you and him fight", and watching developments, possibly hoping for a chance to take over a weakened kingdom.)
Seventeen years later, Macbeth had no (adult) sons and the situation was growing critical - so he used Malcolm's rules (which had made him eligible in the first place) to declare his wife's adult son Lulach his heir.
Unfortunately, Lulach wasn't up to it....