The government has been besieged by protests since it took office in July with support from Beijing, and a strong vote for democratic parties will be seen as a rejection of the mainland's growing influence in the former British colony.
Tens of thousands of student-led demonstrators surrounded government headquarters for a second consecutive night on Saturday, calling for the withdrawal of the unpopular plan to introduce Chinese patriotism classes in schools.
The rallies, which waxed and waned for 10 days straight and included hunger strikes and a Tiananmen Square-style democracy statue, became a rallying cry for democratic parties.
Critics of the policy said it amounted to Chinese Communist Party brainwashing, citing state-funded course materials praising the benefits of one-party rule.
In an election-eve policy reversal, the city's leader Leung Chun-ying dropped the 2016 deadline for the classes to be introduced and said they would no longer be mandatory.
"The schools are given the authority to decide when and how they would like to introduce the moral and national education," he told a news conference late Saturday, blaming the previous government for the policy.
The protests ended on Sunday but analysts said anger at the government's handling of the education row would not dissipate so quickly and could still boost turnout for the pro-democracy camp.
Polls closed at 10:30 pm with results not expected until Monday.
Turnout was about 53 percent, almost eight percent higher than the previous legislative elections held in 2008, and about 2.6 percent lower than the record turnout at the 2004 polls.
A high turnout rate could be positive for pro-democratic parties, which are thought to be better at marshalling their voters.
The new legislature could pave the way for universal suffrage as promised by Beijing in 2017 for the job of chief executive, and by 2020 for the parliament.
Forty of the 70 seats -- expanded from 60 in the outgoing assembly -- will be directly elected, the first time that more than half of the seats in the Asian financial centre have been decided by popular vote.
The remainder are chosen by relatively small "functional constituencies" of electors grouped along economic and professional lines, including wealthy business leaders with strong financial ties to the mainland.
Secretary for Constitutional and Mainland Affairs Raymond Tam said the expanded number of seats in the assembly "greatly enhanced" democracy, but the democracy camp and many independent analysts disagree.
"The impact will be very, very limited. The opposition will still remain in the minority, it still has no chance in securing a majority," City University of Hong Kong analyst Joseph Cheng told AFP.
Besides the protests over education policy, tensions have been brewing over corruption, the yawning gap between rich and poor, soaring property prices and the strains of coping with an influx of millions of mainland tourists.
Surveys show dissatisfaction with mainland rule is rising, especially among the young, while satisfaction with the Communist Party's performance in governing China is at its lowest point since the 1997 handover from Britain.
Pro-democracy campaigners are hoping to win the minimum 24 seats they need to retain a veto over constitutional amendments required for the introduction of universal suffrage.
They fear Beijing will try to force through a sanitised version of universal suffrage that gives the central authorities power to screen candidates.
Beijing-backed newspaper Wen Wei Po described the pro-democracy camp as people who "throw bananas", an apparent reference to the protests and the noisy antics of some radical lawmakers.
Hong Kong reverted to Chinese rule under the principle of "One Country, Two Systems", which guarantees a degree of democracy and freedoms not tolerated on the mainland.