The majority of the China-ruled city's 7.3 million people have no say in deciding their leader, who is chosen from among several candidates by a 1200-person "election committee" stacked with pro-Beijing and pro-establishment loyalists.
Ms Lam, who will become Hong Kong's first female chief executive when she takes office on 1 July, won 777 votes compared with 365 for her closest rival, former financial secretary John Tsang, who polls show is more popular.
There were several invalid protest ballots including one that carried an obscenity.
"Hong Kong, our home, is suffering from quite a serious divisiveness," Ms Lam said in a victory speech.
"My priority will be to heal the divide and to ease the frustration, and to unite our society to move forward."
Ms Lam also pledged to follow through on election promises including introducing a "two-tier" profits tax, reducing tax to spur research and development, tackling the high cost of housing by increasing land supply and boosting education spending.
She also promised to defend the rule of law and freedom of expression as integral to underpinning prosperity.
"Hong Kong needs new thinking," she said.
Scuffles between protesters and police
Some scuffles broke out outside the voting centre between protesters and police, who used metal barricades to keep the demonstrations well away.
The activists denounced Beijing's "interference" amid widespread reports of lobbying of voters to back Ms Lam, rather than Mr Tsang.
Some protesters chanted "I want universal suffrage" and unfurled yellow umbrellas, a symbol of the civil disobedience "umbrella movement", when the result was announced.
"Lies, coercion, whitewash," read one banner. A big yellow banner calling for full democracy was hung from the Lion Rock peak overlooking the city.
"The central government has intervened again and again," said Carmen Tong, a 20-year-old student. "It's very unjust."
Hundreds of Lam's supporters waved Chinese flags and cheered inside and outside the venue after her win.
Since Hong Kong returned to Chinese rule in 1997, Beijing has gradually increased control over it even though it promised wide-ranging freedoms and autonomy not allowed on the mainland under the formula of "one country, two systems", along with an undated promise of universal suffrage.
Many, including opposition democrats, fear Ms Lam will stick to the tough policies of staunchly pro-Beijing incumbent Leung Chun-ying, who ordered the firing of teargas on pro-democracy protesters in 2014 and who was not seen to be defending Hong Kong's autonomy and core values.
"She doesn't have a strong foundation, nor will she have a honeymoon after she's elected," said political scientist Ivan Choy.
"But whether she will further divide society we still have to wait and see what she does, whether she will continue the approach of Leung."
All of Hong Kong's three other post-handover leaders have struggled to balance the demands of China's stability-obsessed Communist Party leaders, with the wish of many residents to preserve the global financial hub's liberal values and rule of law that have long underpinned its economic success.
In 2014, parts of the city were paralyzed when tens of thousands of protesters blocked major roads for nearly three months to demand Beijing allow full democracy; demands that were ignored amid some violent clashes.
China's Hong Kong and Macau Affairs Office congratulated Ms Lam, saying she should not disappoint the people and should seek to "comprehensively unite all sectors of society", strengthen development, and "work hard to forge a new situation", the official Xinhua news agency said.
Some city residents see China's creeping interference in business, media, politics, academia and the judiciary as tarnishing the city's international business allure.
The detention in 2015 of five Hong Kong booksellers who sold material critical of Beijing also dismayed many residents.
The upheavals over the city's autonomy and democratic reforms have roiled a new generation and weighed on the city's economy, ranked 33rd globally by the World Bank in 2015. Hong Kong's richest man, Li Ka-shing, warned this week the city couldn't afford another five years of strife.
Hong Kong had been presented with a reform package, offering the possibility of a direct vote for this leadership race, though only of candidates essentially pre-screened by Beijing. The blueprint was vetoed in 2015 by pro-democracy lawmakers as "fake" Chinese-style democracy.
Political and social divisions have led to some legislative and policy-making paralysis and the stalling of major projects, including a cultural hub and high-speed rail link to China.
While Hong Kong's proximity to China has been a boon, bringing investment and spending, businesses have also faced growing competition from mainland firms in sectors like services and property.
Housing prices, now among the world's highest, are widely seen to have been pushed up by a wave of buying from rich Chinese, intensifying anti-mainland China sentiment.